Holistic Molecular Representation Learning via Multi-view Fragmentation
Authors: Seojin Kim
Jaehyun Nam
Junsu Kim
Hankook Lee
Sungsoo Ahn
Jinwoo Shin
Venue: Accepted by TMLR ()

Abstract: Learning chemically meaningful representations from unlabeled molecules plays a vital role in AI-based drug design and discovery. In response to this, several self-supervised learning methods have been developed, focusing either on global (e.g., graph-level) or local (e.g., motif-level) information of molecular graphs. However, it is still unclear which approach is more effective for learning better molecular representations. In this paper, we propose a novel holistic self-supervised molecular representation learning framework that effectively learns both global and local molecular information. Our key idea is to utilize fragmentation, which decomposes a molecule into a set of chemically meaningful fragments (e.g., functional groups), to associate a global graph structure to a set of local substructures, thereby preserving chemical properties and learn both information via contrastive learning between them. Additionally, we also consider the 3D geometry of molecules as another view for contrastive learning. We demonstrate that our framework outperforms prior molecular representation learning methods across various molecular property prediction tasks.

Jachyun Nam Published in Transactions on Machine Learning Research (05/2024)
Holistic Molecular Representation Learning via Multi-view Fragmentation
Seojin Kim*
oskkjs kc.kr
Korea Advanced Institute of Science & Technology (KAIST)
jachyun. wechpun.nammkaastt.cccm
Korea Advanced Institute of Science 8 Technology (KAIST)
Junsu Kim
junsuukmm@kaist..c...k
Korea Advanced Institute of Science 8 Technology (KAIST)
Hankook Lee
bankook.ie.@shku.co edu
Sungkyunkwan University (SKKU)
Sungsoo Ahn Pohang University of Science and Technology (POSTECH)
Lingsoo.ahn@posteeee
Jinwoo Shin
jinwossBaaist. oc.kr
Korea Advanced Institute of Science & Technology (KAIST)
* The authors contributed equally.
Reviewed on OpenReview: // net/ forum? id=ufDh555J1M
Abstract
Learning chemically meaningful representations from unlabeled molecules plays a vital role in Al-based drug design and discovery. In response this, several self-supervised learning methods have been developed, focusing either global (c.g., graph-level) local (c.g., motif-level) information of molecular graphs. However, it is still unclear which approach is more effective for learning better molecular representations. In this paper, we propose a novel holistic self-supervised molecular representation learning framework that effectively learns both global and local molccular information. Our key idea is to utilize fragmentation, which decomposes a molecule into set of chemically meaningful fragments (e.g., functional groups), to associate global graph structure set of local substructures, thereby preserv- ing chemical propertics and learn both information via contrastive learning between them. Additionally, also consider the 3D geometry of molecules another view for contrastive learning. We demonstrate that our framework outperforms prior molecular representation learning methods across various molecular property prediction tasks.
1 Introduction
Obtaining discriminative representations of molecules is long-standing problem in chemistry (Mrggan 1965). Such task is critical for many applications, such as drug discovery (Capecchi et al. 2020 and mate- rial design (Gmbar-Bombarrlll el al.] 2018), since it. is a fundamental building block for various downstream tasks, e.g., molecular property prediction (Duvenaud et al.) 2015) and molecular generation (Mahmood let al. 2021). Over the past decades, rescarchers have focused on handcrafting the molccular fingerprint representation which encodes the presence of chemically informative substructures, e.g., functional groups, in molecule (Rogers & Hahn 2010 Seo et al. 2020

---

Published in Transactions on Machine Learning Research (05/2024)
Projection
Positive
fragments
encoder
Attention pooling
Projection
fragments
2D GNN encader
(a) Fragment-basee single-view contrastive learning
M1 Graph-wise pooling pooling pairs
lis Boaue Bon Baga
P
2D Projection
Torsional angle decoder
Tan P
20 200, 200
3D GNN encoder (b) Fragment-based cross-view contrastive learning
3D Projection head
(c) Torsional angle prediction
between fragments
Figure 1: An overview of Holistic Molecular representation learning via multi-vie view fragmentation (Holi- Mol). (a) Fragment-based single-view contrastive learning: a set of fragments regarded as a positive view of the molecule. r; and -ー refer to representations of the molecular graph M, and its fragments, (b) Tragment-baseer cross-view contrastive learning: positive pairs are constructed using 2D and 3D views of a molecule and its fragments. (c) Torsional angle prediction between fragments: 3D contextual information is learned by predicting the torsional angle.
respectively.
Recently, graph neural networks (GNNs, Kipf & Welling 2017) have gained much attention as a framework to learn the molecular graph representation due to its remarkable performance in predicting chemical properties (Wu et al. 2018). However, they often suffer from overfitting when the number of labeled training samples is insufficient [Rong et al. 2020b). To resolve this, researchers have investigated superviid learning that generates supervisory signals without labels to utilize huge amount of unlabeled molecules (Rong et al. 2020a; Zhou et al. 2022).
The self-supervised learning approaches for 2D molecular graphs fall in two categories: (a) contrastive learning with graph augmentations, e.g., edge drop (Wang et al. 2021), and (b) masked substructure, e.g.; motifs prediction (Hu et al. 2020a Zhang et al. 2021]. Specifically, the former (a) learns molecular representations by maximizing the agreement of similar positive views while minimizing that of dissimilar negative views on the representation space. Although this approach can learn global graph structures, it often fails to capture chemically meaningful local information (c.g., functional groups) since the graph augmentations may not preserve such mubstructures In contrast, the latter approach (b) captures the local information well by predicting substructures directly from masked molecular graphs. Yet, it is unclear whether they learn the global information well.
In the field of chemistry, both global and local information molecular graphs are chemically important. For example, long chain-like global molccular structures are associated with high water-octanol partition coelficients (Bhal] 2007), while the molecules with the fluorobenzene substructure are highly likely to be active on blood- brain barrier permeability (i.e., 97% active ratio in the BBBP dataset (Wu et al. 2018)). This inspires us to explore the following research question: how to capture both global and local information for learning better molecular representations?
Contribution. Our key idea is to (i) associate a global structure with a set of substructures and (ii) learn their relationships via contrastive learning to capture both global and local information. Specifically, we utilize fragmentation that decomposes molecule into a set of chemically meaningful fragments, e.g., functional groups. In particular, the popular fragmentation strategy, Breaking of Retrosynthetically
2

---

Published in Transactions on Machine Learning Research (05/2024)
Interesting Chemical Substructures (BRICS, Degen et al. 2008 which preserves the functional groups Liu et al., 2017). We also utilize the 3D geometry as another global and substructure views of cantrastive learning and propose the 3D torsional angle prediction task between fragments. They enrich the molecular reprosentations since such 3D information related to various molecular propertics, c.g., polarizability (Anslyn & Dougherty 2006).
In an effort to jointly consider both global and local information acquired through fragmentation of the 2D and 3D multi-view unlabeled molecules, we introduce novel molecular representation learning framework, Holistic Molecular reprsentation learning via multi-view fragmentation (Holi-Mol). To be specific, Holi- Mol consists of the following components (see Figure
(a) Fragment-bascd singlc-view contrastive lcarning. To facilitate interaction between a molocule and its substructures, we consider set of fragments as a positive view of the molecule. We then apply contrastive learning between the whole molecular structure and the set of fragments for learning molecular representations via 2D and 3D molecule GNNs separately (i.e., single-view).
(b) Fragment-based cross-view contrastive learning. We consider molecule-level and fragment- level positive pairs between 2D topological and 3D geometric graphs. We then apply contrastive learning between the 2D and 3D views for learning 2D and 3D molecule GNNs jointly (i.e., cross- view). This cross-view objective enriches both global-level and local-level representations.
(c) Torsional angle prediction between fragments. To further exploit 3D geometric fragment- wise information, suggest solving the torsional angle prediction task between adjacent fragments. Note that the torsional angle, defined by the dihedral angle of four adjacent atoms, is related to several 3D contextual properties, e.g., the energy surface around the bond (Smith 2008
For evaluation, we follow the multi-view protraining sctup (Sääk ct al. 2022 Liu ct al. 2022b): 2D molecule GNN using an unlabeled molecular dataset containing both 2D and 3D information, and then (ti) fine-tune the GNN on downstream tasks without 3D information. This setup using only 2D GNN is practically important since 3D information is often unattainable in downstream tasks due to the high computation cost. We explore the usefulness of our pre-trained 3D GNN in Appendix G
: (i) pretrain
Through extensive experiments, we demonstrate the superiority of our Holi-Mol framework over existing pretraining methods. Specifically, our GNN pretrained by Holi-Mol on the GEOM (Axelrod Gomez- Bombarelli, 2022) dataset consistently outperforms the state-of-the-art the-art method, GraphMVP {Liu et al., 2022b), when transferred to both MoleculeNet classification (Wu et al., 2018) and QM9 regression (Ramakr- ishnan al. 2014] benchmarks (sce Table ] and 2
respectively). For example, we improve the average ROC-AUC score by 74.1 75.5 over the prior art MoleculeNet. We further demonstrate the potential of Holi-Mol for other applications:
httpupprriiiee/lllllssuperrer learning (see Table 3 and molecule
retricval (sce Table 4h.
2 Preliminaries
Multi-view molecular representation learning. The problem of interest in this paper is molecular representation. learning (MRL) which aims to (i) learn a generalizable neural network for molecules f : + Rd where M is the molecular space and IRd is the embedding space, and (ii) utilize the network f for various chemical downstream tasks (e.g., toxicity prediction). Usually, (i) utilizes only unlabeled molecules (Sun)
et al. 2019 You et al. 2020 2021 Liu et al. 2022b) due to the significant experimental cost to obtain label information of molccules compared the cost of the collection of unlabeled molccules.
In general, a molecule M M can be represented by an altributed graph M = (V, E, E,A,B, R) where V is set of nodes associated with atom features A R Vxxaae (e.g., atomic numbers), E CCVVVV is set of edges associated with bond features B E R [E| x x (c.g., bond types), and R E R V is an array of 3D atom
A malecule can represented by (a) 2D topological graph (V, F) nodes V and edges E (b) 3D geometric graph
(V, R) of nades V and 3D coordinates
3

---

Published in Transactions on Machine Learning Research (05/2024)
positions. Conventionally. M2 (V, A, B) and Ms A, R) are referred to 2D and 3D molecular 2022b). We note that obtaining accurate 3D information R is very expensive due to iterative quantum computations and thus many real-world applications suffer from the lack of such 3D information (Liu et al. 2022b
graphs, respectively (Stärk et al. 2022
Liu et al.
To consider a wide range of downstream tasks, we focus on learning a graph neural network (GNN) for 2D molecular graphs fr M2 - IRd where M2p is the 2D molecular graph space. To be specific, we (i) pretrain a 2D molecule GNN fzo using unlabeled set of molecules D, S M containing both 2D and 3D information, and then (ii) fine-tune fro downstream tasks without 3D information, ie., each task has dataset D S M20 x y where Y/ is the label space. Therefore, it is important to inject not only 2D topological information, but also 3D geometric information into the 2D molecule GNN fzn during pretraining. We remark that this multi-vieu pretraining setup has been recently investigated (Stärk et al. 2022 2022b) to alleviate the scarcity of 3D topological information.
Liu et al.
2D molecule GNN Jzo Man - Id. For any 2D molecule Ma - EV, Ma, GNNs for 2D molecules (2D-GNNs in short) compute molecular representations by applying (a) iterative neighborhood aggregation (i.e., message passing) to acquire node-level representations based on the graph (V, E¹) and then (b) readout function (e.g., mean pooling) to create graph-level representations. Formally, node- and graph- level representations of I-layer 2D-GNN are as follows:
h == MP(L (bl="",Bu (urlhe E [4],
fa (M) :- fzol (M)) Readout((h," hev),
where MP(-) is a message passing layer, Readout(-) is readout function, h A, is the atom feature for node v, Buv is the bond feature for an edge (u, E E, and NVv) is the set of adjacent nodes of v. In this work, we mainly use the graph isomorphism network (GIN) architecture (Xu et al. 2019) following the standard MRL sctup (Hu et al. 2020a).
3D molecule GNN fso MMD + Rd, For any 3D molecule Ma = (3D-GNNs in short) compute molecular representations by applying (a) iterative geometric interactions through distances and angles between nodes (i.c., atoms) to acquire node-level represcntations basod on the 3D geometry R and then (b) a readout function to create graph-level representation Formally, node- and graph- level representations of L-layer 3D-GNN are as follows:
E MaD. GNNs for 3D molecules
h = IB(b ((-)) RD (h!-") RulwEv((O)),EE [L],
(M) = fso (Mso) Readout ((hdouut(hhh)) (0) Juev). (ev)).
where IB(-) is an interaction block, Readout(.) is a readout function, h = A. and R, is the atom feature and the 3D position for a node u, respectively. In this work, we mainly use the SchNet architecture (Schüul lt al. 2017 following the setup of Liu et al. (2022b).
Contrastive learning. Contrastive learning aims to learn discriminative representations by attracting positive views while repelling negative views on the representation space, e.g., see Chen et al. 2020). A common practice for generating positive views is to utilize comantic-preserviin transformations. Let (x, x+) be a positive pair generated by the transformations and xx) be a negative pair obtained from different instances in a mini-batch. If z, zt, and z are the representations of x, x,x and x respectively, then the contrastive learning objective La, can be written as follows YYou et al. 2020 Chen et al. 2020
exp(sim(z,z) z )/T)
(z, 2* {z-})
log Σn exp(sim(z, -//7)
(1)
where (2, = z =T is the cosine similarity and T is temperature-sealer. Here, the set {z-} may include the positive x depending on the choice of objectives, e.g., NT-Xeet (Chen el al. 2020).
3 Holistic molecular representation learning via multi-view fragmentation
In this section, we propose a holistic molecular representation learning via multi-view fragmentation, coined Holi-Mol, which fully exploits both global and local molecular information using 2D and 3D multi-view
4

---

Published in Transactions on Machine Learning Research (05/2024)
fragmentation in order to learn good representation of 2D-GNN fzo First, introduce our fragmentation scheme (Section 3.1p. Thereafter, we describe details of our fragment-based components: single-view con- trastive learning (Section 3.20, cross-view contrastive learning (Section 3.3). and torsional angle prediction between fragments (Section 3.4) Our framework is illustrated in Figure
3.1 Molecular fragmentation
Our framework relies on the molecular fragmentation which decomposes molecule into a of chemically meaningful fragments (i.c., substructures). In this paper, mainly the popular strategy, Breaking of Retrosynthetically Interesting Chemical Substructures (BRICS, Degen et al. 2008), which is designed to preserve most chemically informative substructures and has been widely adopted in the prior literatures (Zhang et al.) [022 Yang et al., 2022p. The efficacy of BRICS is further verified through an analysis in Appendix H Formally, for a molecule M = (V, A, B, R), a fragmentation scheme decomposes into a set fragments (M² where M (V), E3, A3, Bi, RR) is the j-th connected component of the graph induced by the fragmentation. We refer 2D and 3D fragment as Ma and M respectively.
3.2 Fragment- contrastive learning: Single-view objective
To incorporate both global and local information into molecular representation, propose contrastive learn- ing objective upon molecular fragmentation. To achieve this, we first utilize fragmentation (see Section 3.1 Then, Holi-Mol learns the relationship between molecule-Ievel and fragment-level information by considering a set of fragments as positive view of a molccule. To be specific, given a training batch {M. we consider (M, {M, )==1) as a positive pair (i.e., they share the same chemical semantics) where n, is the number of fragments of the molecule M,. To aggregate representations of JM we use the attention pooling (Li et al. 2016). Formally, the representation for the set {M² is:
exp(a r b) r/. 1=1 cxp(a r b)
mix r
where r. = S(M)) is the representation for each fragment, E Rd and b E R are learnable parameters. Similarly, compute the molecular representation f(,)) for the whole structure. Then, we separately optimize the 2D-GNN fzo and the 3D-GNN fao along with projection heads 920 (or 3D) Rd Rd by the following contrastive objective with the fragment-basse based positive pairs:
Loingle Σ (Z20. 2 {2)b,) li4s) + LaL (Zso. 2,b, {zzb, }zt)),
(2)
where z; = g(:)) and 2gix g(12ix).
3.3
Tragment-basee contrastive learning: Cross-view objective
To bring 3D geometric information into molecular representation of 2D-GNN effectively, we propose cross- view contrastive objective. Since these views provide different chemical information (egg, atom-bond con- nectivity (Estrada et al. 1998 and energy surface (Smith 2008) in the 2D and 3D views, respectively). such a cross-view objective could make the representation f(M) more informative. To further utilize global and local information of 3D molccules, we use both of molecule-Iovel and fragment-level cross-view contrastive learning objectives into our framework.
Molecule-level objective. Here, consider (Mz, Mao.:) as positive pair. Then, the molecule-level contrastive objective can be written as follows:
Lerost.a0l = 2n
Σ (La (Z20,s, 230.i (Zzp, ( 1 + Lea (2z0,s 220,is (z20 (im1))
(3)
1=1
5

---

Published in Transactions on Machine Learning Research (05/2024)
This objective is well-known to be effective in general multi-view representation learning (Radford et al, 2021 Tian et al. 2020), and also investigated in molecular representation learning (Stärk et al. 2022 Liu et al. 2022b). However, modeling the cross-view contrastive objective based solely on the similarity of molecule-level representations may lack capturing fragment-level information (i.c., chemical property at a finer level). Therefore, inspired by loken-wise contrastive learning in the vision-language domain (Yao el al. 2022 we suggest fragment-level cross-view contrastive learning what follows.
Fragment-level objective. We consider (M M.s.)) (M M as fragment-level negative pairs for k + i. To achieve this, we consider contert-aware fragment representations p/, to incorporate the neighboring context of the fragments To be specific, we define the j-th fragment representation P of a molecule Mj via fragment-wise pooling as follows:
fragment-level positive pair and consider
p
Σ h DEV
(4)
where (he are the last-layer node representations of the whole molecular structure M, obtained by GNN f, and V is a of nodes the j-th fragment. Here, p contains global context of molecule since bu is obtained by message passing or interaction block. We then compute latent fragment. representations by a projector g, c.g., z=0 = 920 (P).)). Using these representations, we define the average of fragment-wise similarities S1 in molecule M, and 8, between molecules M, and Mg as follows:
sim (Z) zm,.), and
$1,5
T4.
i,k
(or 3D)
max
sim (or a0), 2:00 (or 20)
n4
15m5ng
where n4 is the number of fragments of the molecule M,, Mr. respectively. Finally, we formulate our fragment-level cross-view contrastive objective as follows:
1
eLS/T
Caroes,frag =
log @dnn/T + Ekgre ve/e + tog
(5)
2n
esnatt + Enaps cuele
To sum up, our cross-view objective is as follows:
Cerose (Corose,uo1 Loroas,trag)
3.4 Torsional angle prediction between fragments
To further incorporate the 3D geometry into 2D-GNN fao we propose an additional pretext task, where fro learns to predict torsional angles around the fragmented bonds given fragments of 2D molecular graphs 1oc {M,.))== A torsional angle is defined by atom quartet (s, u, v,t) which is a sequence of four atoms, cach pair connected by covalent bond. As illustrated in Figure [c), the torsional angle is the angle between the planes defined by (s, v) and (u, v, t), and it encodes important 3D local properties, e.g., energy surface around the atoms (Smith) 2008). Thus, by predicting the torsional angle from any two arbitrary fragments of molecule M, where they are originally connected by the bond (u, v), the 2D-GNN frD is expected to learn the 3D contextual properties around the fragmented bond.
We now define the torsional angle prediction task for each fragmented bond: for 2D molecule Map,; and fragmented bond (u,v) Ezo,1, randomly select non-hydrogen atoms and t adjacent to and
"Contrary the fragment reprosentations proposed in Section 3.2 which does not incorporate neighboring context nearby the fragments, consider context-aware fragment representations. These representations obtained by fragment- wise pooling after message passing (see Eq.

---

Published in Transactions on Machine Learning Research (05/2024)
Test ROC-AUC score on the MoleculeNet downstream molecular property classification bench- marks. We employ GIN (Xu et al., 2019) the 2D-GNN architecture and pretrain with 50k molecules from the GEOM dataset (Axelrod & Comez-Bombant
Table 1:
2022), following Liu et al. (2022b). We report mean and standard deviation over 3 different seeds. We mark the best mean score and scores within one standard deviation of the best mean score to be bold. We denote the scores obtained from Liu et al. (2022b) with (*). Otherwise, we reproduce scores under the same sctup. Scores obtained through fine-tuning of the officially
provided checkpoints are denoted by (t)
Methods
BBBP
Tox21
ToxCast
Sider
Clintox
MUV
HIV
Bace
Avg.
65.4=24
74.9=0.
616+12
58.0+2.4
58.8+5.5
71.0+2.
75.3
72.6+49
67.2
Pretrained with 50k 2D molocular graphs of CEOM and fino-tuned on 2D molocular graphs of MoloculeNet
AttrMask*
(uu al.
2020a
70.2
74.2
62.5+0.4
60.4
68.649.6
73.9
74.3+13
77.2=14
70.2
GPT-GNN
[Hu et
20206
64.521.1
75.320.5
62.210.1
57.5142
57.813.
76.1123
75.1=0a2
77.6200
68.3
Informxx*
Sun al.
2019
69.2.0..
73.010.7
62.010.3
59.210:2
75.1
74.011.
74.5118
73.912.
70.1
ContextPred'
Hu
20200
71.210.
73.310.5
62.810.3
59.3114
73.7140
72.5122
75.8-1.1
78, 18.61114
70.9
GraphLoG*
Xu al.
2021
67.8=1.7
73.0+0.3
62.2+0.4
57.4+2.3
62.0+1.
73.1+1.7
73.4+0.6
78.8
68.5
Contextual"
Rong
2020aa
70.3=1.6
75.220.3
62.6+0.3
58.420.
59.9+8.2
72.3+0.9
75.9=0.a
79.2+0.3
69.2
G-Motif*
[Rong ct al.
2020
66.4=a.4
73.2+n.s
62.610.5
60.6#1.1
77.8+20
73.3+2.0
73.8
4:4.
70.1
GraphCL.
You et al.
2020
67.5=a.3
75.0=0.s
62.8+0.2
60.1+1.5
78.9+4.2
77.1=1.0
75.0=0.4
78.7
70.1
J
You et al.
2021
66.0=n.6
74.4+0.7
62.7+a.6
60.7+1.0
66.3+a.9
77.0=222
76.6=0.5
72.9#2.
70.6
MCSSL
Zhang ct al.
2021
67.3...9
74.510.2
63.610.4
58.410.2
75.41a.
73.91114
77.2.225
76.211.3
70.8
MoICLR
Wang ct ml.
2021
67.6.n.6
74.411.3
62.910.2
58.711.1
57.91a.0
70.812.
75.4
74.613.5
67.8
D-SLA
Kim el. al.
2122
69.6=2.4
73.7+0.7
63.3+0.2
59.2+2.
60.5+1.0
75.3+0.
75.8=0a
81.2=2.5
69.8
BERT
[Xaa el al.
2023
69.8
74.920.5
63.720.6
58.520.
80.5+2.4
72.5+24
75.9=1.2
78.4
71.8
Pretrained with 50k 2D and 3D molocular graphs of GEOM and fine-tuned
2D molocular graphus MolcculeNct
3D- (tärk al. 2022
67.9-1.2
75.320.3
64.6
59.6
89.7#
76.7=0.
73.4+1.2
79.920.
73.4
GraphMVP Lau el al. 2226
69.620.2
75.620.7
63.720.3
61.320.6
89.021.4
75.721.0
75.120.3
80.9213
73.9
GraphMVP-GT Liu et a GraphMVP-Ct
2022b
70.1=a.7
75.3+0.9
64.2#0.9
61.0e0.5
89. 10.1
77.7=1.
75.3=0.8
80.2+1.5
74.1
[Lu et al.
20226
69.6=1.4
74.6+0.1
64.1#0.2
63.0=o.1
88.7#2.
73.9
74.7=2.0
81.3g007
73.7
Holl-Mol (Ours)
71.4+04
75.220.7
65.1+0.
61.0e0.6
95.2=1.0
77.6211.
76.3-9.4
82.3=1.6
75.5
respectively, and compute the torsional angle y the quartet (s, u, v, t) on the molecule M,. T is a collcction of the tasks for all fragments, our loss function can be written as follows:
Lior = T (L.......L)EET
Σ
(,,
(6)
where Lxe is the cross-entropy loss, is the binned label for the angle, and U,,s,,,,, == Ptor Otor([aa,, (hap,asaae is the prediction from the concatenation of node representations of atoms (s, u, of the molecule M using a multi-layer perceptron (MLP) (tor(-).
3.5 Overall training objective
By aggregating the objectives proposed in Section 3.2
3.3 and 3.4 we propose our total training loss function.
In summary, we consider a set of fragments as a positive view of a molecule while maximizing the consistency of the outputs of 2D and 3D-GNNs both the molccule-level and at the fragment-lovel. Additionally, we incorporate 3D contextual information by predicting the torsional angles around the fragmented bonds. The overall loss function is follows:
LHoli-Mol Leingle + Laross Lror-
(7)
Note that in Eq. et al. 2020
and is the only hyperparameter we introduced. We set 0.1 following You
3 MCraphMVP Liu al. 2022b pretrains with explicit hydrogens, but fine-tunes without explicit hydrogens. We report fine-tuning results with explicit hydrogens from official checkpoints. Thus, our reported average value slightly higher than the original paper.

---

Published in Transactions on Machine Learning Research (05/2024)
Table 2: Test MAE score on the QM9 downstream quantum property regression benchmarks. For ours and all basclines, we employ GIN (Xu et al.
2019) as the 2D-GNN architecture and pretrain with entire 310k 2022). We mark the best score bold.
molecules from the GEOM dataset (Axelrod & Komez-Bombare
Meshods
ZPVE v
4
LUMO
HOMO
Ggep
R 4
Uo
Una 4
Hous
Gous J
43.7
0.059
0.400
0.144
80.5
89.1
171.0
3.27
62.9
61.8
57.0
48.1
Preteained
310k
mokecuaar graphs of GEOM fine- luned 2D molecular graphs QM9
3D-Iufomax Stark
2022
27.0
0.051
0.355
0.126
63.4
55.2
103.8
2.99
38.8
45.6
41.0
40.8
GraphMVP G Liu 20226
24.1
0.051
0.367
0.123
39.1
53.8
100.4
2.97
39.9
44.2
41.0
40.3
Holi-Mol (Ours)
24.0
0.049
0.353
0,121
57.1
51.8
97.1
2.90
39.2
42.9
40.3
40.0
Table 3: Test MAE score of semi-supervised learning on the QM9 downstream quantum property regression benchmarks. We employ GIN (Xu et al.) 2019) as the 2D-GNN architecture and pretrain with the 110k QM9 training dataset. Then we fine-tune across different label fraction of the QM9 training dataset. We mark the best score bold.
Methods
ZPVE
LUMO 4.
HOMO
Uo L
Label Fraction (%)
20
50
100
20
50
100
20
50
100
20
50
100)
111.0
87.1
43.7
236.0
140.6
80.5
233.6
128.1
89.4
165.3
62.8
62.9
Pretrained on 110k 2D and 3D molecular graphs of QM9 and fine-tuned on 2D molecular graphs of QM9
3D-IIffomax Stärk et al.
2022
87.2
42.8
24.4
215.0
98.4
57.9
181.0
102.4
57.7
148.2
75.0
42.1
GraphMVP Lau 20226
24.4
214.3
99.7
59.7
177.3
100.0
145.7
74.5
Holi-Mol (Ours)
22.2
202.2
54.6
91.0
138.7
71.8
4 Experiments
In this section, extensively compare Holi-Mol with the existing molecular graph representation learning methods. We evaluate Holi-Mol and baselines on various downstream molecular property prediction tasks after pretraining on (unlabeled) molecular dataset. For an extensive evaluation in various downstream setups, we consider (1) transfer-learning, i.e., pre-training and fine-tuning distribution are different (Table 1 anddd 2). and (2) semi-supervised learning, i.e., pre-training and fine-tuning distribution are the same (Table 3). We further discuss the evaluation setups in Appendix(O) [ Also, we perform an ablation study to investigate the effect of each component of Holi-Mol for discriminating molecules.
4.1 Experimental setup
Baselines. We consider recently proposed multi-view molecular representation learning methods which utilize both 2D and 3D molecular graphs in pretraining: 3D-Infomax (Stärk et al.] 2022), GraphMVP, and GraphMVP-(G,CC (Liu et al. (022b). Following GraphMVP, we also compare with molecular representation learning methods which pretrain solely with 2D molecular graphs, including the recently proposed MGSSL (Zhang et al., 202T), , MolCL. (Wang al. 2021 and D-SL.A (Kim et al. 2022 We provide more details on basclines in Appendix B
Datasets. For pretraining, we consider the GEOM (Axelrod & iombazlBo 2022 and the QM9 RRmakkishnan et al.) 2014) datasets, which consist of 2D and 3D paired molecular graphs We consider (a) transfer learning on the binary classification tasks from the MoleculeNet benchmark (Wu ct al. 2018). and (b) transfer learning and semi-supervised learning on the regression tasks using QM9 (Ramakrishnan el al., 2014]. Following Liu et al. (2022b), we use scaffold split (Chen et al. 2012) in MoleculeNet experiments which splits the molocules based on their substructures. For QM9 experiments, we follow the setup of [Lu let al. (2021) which splits the dataset into 110,000 molecules for training, 10,000 molecules for validation, and 10,831 molecules for test. Detailed explanation about datasets be found in Appendix D
8

---

Published in Transactions on Machine Learning Research (05/2024)
Table 4: Visualization of the molecules retrieved by MGSSL (Zhang et al. 2021). GraphMVP-G (Liu et al. 2022b), and Holi-Mol (Ours). We report top-2 closest molocules from the Tox21 datasct with respect to the query molecule in terms of cosine similarity in representation space. We utilize models pretrained with the GEOM dataset [Axelrod & Gomez Bombarelli
2022). We mark BRICS fragments dotted lines.
Query
MGSSL
(Zhang et al.
2021
GraphMVP-G
Liu et al. 2022h
Holi-Mol (Ours)
Architectures. We employ 5-layer graph isomorphism network (GIN) Xu ct al. 2019 as 2D-GNN f2D and 6-layer SchNet (Schütt et al. 2017) as 3D-GNN fan- The configuration is drawn from GraphMVP et al. 2022b for a fair comparison.
Liu
Hardwares. We use single NVIDIA GeForce RTX 3090 GPU with 36 CPU cores (InteI(R) Cor)(TM) 19-10980XE CPU @ 3.00GHz) for self-supervised pretraining, and a single NVIDIA GeForce RTX 2080 Ti GPU with 40 CPU cores (Rutel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz) for fine-tuning.
4.2 Main results
MoleculeNet classification task. As reported in Table 1 Holi-Mol achieves the best average test ROC-
AUC score when transferred to the MoleculeNet (Wu et al.
2018) downstream tasks after pretrained with 50k
dataset. For example, Holi-Mol outperforms the recently proposed method for 2D molecular graph, Mole- BERT (Xia et al. 2023), in all downst Lasks. Furthermore, Holi-Mol improves the best average ROC-AUC score baseline, GraphMVP-G (Liu et al. 2022b). by 74.1 + 75.5, achieving the state-of-the-art performance on 7 out of downstream tasks. We emphasize that the improvement of Holi-Mol is consistent over the downstream tasks. For example, GraphMVP-C {Liul et al. 2022b) achieves the best performance on Sider, while it fails to generalize on Tox21, resulting in even lower ROC-AUC score compared to the model without pretraining. On the other hand, Holi-Mol shows the best average performance with no such a failure case, i.e., Holi-Mol learns well-generalizable representations over several downstream tasks. We provide further analysis in Appendix and F
molecules from the GEOM (Axelrod & Gomez- Bombarelli
2022
QM9 regression task. Table 2 and show the overall results of transfer learning and semi-supervised learning on the QM9 (Ramakrishnan et al. 2014) regression benchmarks, respectively. For transfer learning (Table , we pretrain with 310k molecules from the GEOM (Axelrod & Bomez-Bombard Holi-Mol outperforms the baselines, achieving the best performances on 11 out of 12 downstream tasks. We emphasize that Holi-Mol outperforms the baselines when transferred to both MoleculeNet and QM9 downstream tasks. For semi-supervised learning (Table 3), Holi-Mol successfully improves the baselines over all tasks and label fractions. In particular, Holi-Mol shows superior performance even in the fully supervised learning scenario (i.c., 100% label fraction), e.g., 89.4 > 48.4 for HOMO. This implics that Holi-Mol indeed finds "good initialization" " of GNN and shows wide applicability. More experimental results on the QM9 dataset can be found Appendix and N
2022) dataset.
We provide the regression results on PCQM4Mv2 in Appendix K
Molecule retrieval. In Table we further perform molecule retrieval task for qualitative analysis. Using pretrained models by MGSSL Zhang et al. 2021 } and GraphMVP-G (Liu et al. 2022b), and Holi-Mol, we visualize the molccules in the Tox 21 datasct, which have similar represcntations with the query molccule based on the cosine similarity. We observe that the state-of-the-art method, GraphMVP-G, does not find molecules with similar fragments to the query molecules. While MGSSL leverages local information of molecules in its training scheme and partially finds local substructure (indicated by dottod lines in Table 4 of molecules, Holi-Mol effectively retrieves molecules with common fragments and similar global structures of the query molecules. We further provide t-SNE plot regarding the obtained representations in Appendix
9

---

Published in Transactions on Machine Learning Research (05/2024)
Table 6: Effectiveness of each objective as mea-
Table 5: Average ROC-AUC score with different positive
sured the average ROC-AUC score across 8 downstream tasks in MoleculeNet.
view construction strategy across 8 downstream tasks in MoleculeNet.
Croes view interaction level
Positive view construction
Fragmentation strategy
Avg.
Pretraining data
Molecule Fragment
Torsion
Avg.
Nodedrop, Subgraph
You
73.4
Single-view (2D)
72.4
Random bond deletion
73.5
71.7
set of fragments (Ours)
Random non-ring bond deletion
74.0
BRICS decomposition (Ours)
75.5
Multi-view (2D&3D)
75.1
75.5
4.3 Ablation study
Fragment-based positive view construction. In Table 5 we investigate how our positive view construc- tion strategy is effective. We first compare our strategy with the alternative: using an augmented molecular graph (i.c., random subgraph) for positive-view construction does not improve the performance (73.4 - 73.5), since important substruc- tures of molecules (e.g., aromatic ring) can be easily broken by random deletion of bonds, which could lead to significant change in chemical properties. Preventing such ring deformation increases overall performance by 73.5 + 74.0. BRICS decomposition further incorporates chemical prior to obtain semantic-preservee local information, boosting the performance by 74.0 - 75.5. The result implies that considering chemically informative substructures key component of our framework. We provide detailed results in Appendix H
positive view (You al., 2020 We observe that deleting random bonds
Effectiveness of multi-view pretraining. In Table D evaluate the impact of individual objectives our total loss Losli-Nol. We observe that molecule-level cross-view contrastive learning (Lerozs,mal Eq. (3) between 2D and 3D molecular views improves the overall performance by 72.4 - 74.7. Introducing fragment- level cross-view contrastive learning (Frroas,tragi Eq. (5 further boosts the performance by 74.7 + 75.1,
capturing fine-grained semantics of molecules. Torsional angle prediction (Crozi Eq. (6) further improves the performance by 75.1 + 75.5 by directly injecting the 3D local information into 2D-GNN. These results confirm that Holi-Mol effectively utilizes both 2D and 3D fragments for multi-view pretraining. Notably, ours with only single-view (2D) learning outperforms Mole-BERT (Xia et al., 2023), which is the prior state-of-the-ard pretraining method on 2D molecular dataset. Detailed results can be found in Appendix !
5 Related work
Multi-view molecular representation learning. Recent works have incorporated mulliple views of a molecule (e.g., 2D topology and 3D geometry) into molecular representation learning (MRL) frameworks
(Zhu et al. 2021a Stärk al. 2022 Liu et al. 2022b Fang et al. 2022 Zhu et al., 2022 Luo et al.) 2002).. In particular, training 2D-GNNs with multi-view MRL has gained much attention to alleviate the cost to
obtain 3D geometry of molecules (Stäk et al. 2022 Liu et al. 2022b). However, they focus on molecule-level
objectives, which could lack capturing the local semantics Yao et al. 2022). In this work, we develop a fragment-based multi-view MRL framework to incorporate local semantics.
Single-view molecular representation learning. One of the single-view (i.e., 2D topological or 3D goometric graph) molccular representation lcarning techniques is predictive pretext tasks. For example. those methods reconstruct the corrupted input as pre-defined pretext tasks (Hamilton
al. 2017 Rong et al. 2020al Hu et al.] 2020a Zhang et al. 2021 Zhou et al. 2022 Jiao et al. 2022 Zaidi et al. 2022
Another large portion of technique is contrastive learning. For example, You et al. 2021 ) utilize augmentatinn
schemes to produce a positive view of molecular graphs, and Fang et al. (2021): Sun et al.) (2021) 1 Wang
et al. 2022 mitigate the effect of semlntically similar molecules in the negative samples (Zhu el al. 2021b)
Recently, substructures of molocules has also been considered in molccular represontation learning. For example, You et al. (2020): Wang et al. (2021): Zhang et al. (2020) construct a positive view of molecule as its single substructure (i.e., subgraph) and Wang et al. 2022
Luong & Singh (2023) repels representations of
10

---

Published in Transactions on Machine Learning Research (05/2024)
fragments from intra- and inter- molecular substructures. Compared to these works, framework considers both global (molecule-level) and local features (substructure-leve) of molecules in a unified framework.
6 Conclusion
We present Holi-Mol, new multi-view molecular representation learning method to focus on both global (molecule-wise) and local (fragment-wise) information of molecules. With this insight, we propose a con- trastive objective to regard the set of fragments as a positive view of a given molecule. Moreover, we introduce the fragment-based cross-view contrastive objective for 2D and 3D views of molecule and the torsional angle prediction task exploit 3D local information. Extensive experiments show that Holi-Mol putperforms prior methods in molecular property prediction lasks, thanks to our pretraining strategy guided by both molecule itself and its fragments.
Broader Impact Statement
This work will facilitate research in molecular representation learning, which can speed up the processing of many important downstream tasks such as predicting sidc-effect of drugs. However, malicious use of well-learned molecular expressions poses a potential threat of creating hazardous substances, such as loxic chemical substances or biological weapons. On the other hand, molecular representation is also essential for creating defense mechanisms against harmful substances, so the careful use of our work, Holi-Mol, can lead more positive effects.
References
Eric V Anslyn and Dennis Dougherty. Modern physical organic chemistry. University science books, 2006.
Simon Axelrod and Rafael
Gcom, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 9(1):1-14, 2022.
Sanjivanjit K Bhal. Logp Logp-making sense of the value. Advanced Chemistry Development: Toronto, ON, Canada, pp. 1 1-4, 2007.
Alice Capecchi, Daniel Probst, and Jean-Louis Reymond. One molecular fingerprint to rule them all: drugs, biomolecules, and the metabolome. Journal of cheminformatics, 12(1):1-15, 2020.
Lei Chen, Wei-Ming Zeng, Yu-Dong Cai, Kai-Yan Feng, and Kuo-Chen Chou. Predicting anatomical thera- peutic chemical (atc) classification of drugs by integrating chemical-chemical interactions and similarities. PloS one, 7(4):c35254, 2012.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In International conference machine learning, pp. 1597 1607. PMLR, 2020.
Jörg Degen, Christof Wegscheid- Gerlac,,, Andrea Zaliani, and Matthias Rarey. On the art of compiling and using' drug-like like' chemical fragment spaces. ChemMedChem: Chemistry Enabling Drug Discovery, 3(10): 1503-1507, 2008.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirze,, Alán Aspuru- Guzík, and Ryan P Adams. Convolutional networks graphs for learning molecular fingerprints. Advances in neural information processing systems, 28, 2015.
Ernesto Estrada, Luis Torres, Lissette Rodriguez, and Ivan Gutman. An atom-bond connectivity index: modelling the enthalpy of formation of alkanes. 1998.
Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong Hc, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geonetry-enhanced molecular representation learning for property prediction. Nature Machine Intelligence, 4(2):127-132) 2022.
11

---

Methods and Published in Transactions on Machine Learning Research (05/2024)
Yin Fang, Haihong Yang, Xiang Zhuang, Xin Shao, Xiaohui Fan, and Huajun Chen. Knowledge-aware contrastive molecular graph learning. arXiv preprinl ar 1rx:22103.1188 13047, 2021.
Rafael Gómez-Boo Bombarlll,,,,,,i,, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamin Sánchez-Lengelie Dennis Sheberla, Jorge eguilera-parrrguiree Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268 276, 2018.
William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: applications. arXiv preprint arvi 1xX::1709.0000 709.05584, 2017.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representa- tions, 2020a.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre- training of graph neural networks. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1857 1867, 2020b.
Rui Jiao, Jiaqi Han, Wenbing Huang, Yu Rong, and Yang Liu. Energy-motivated equivariant pretraining for 3d molecular graphs. arXiv preprint ar xrX2::2077.8882, 2022.
Dongki Kim, Jinheon Back, and Sung Ju Hwang. Graph self-supervised learning with accurate discrepancy learning. arXie preprint arXiv:2202. 022999,,0000999 2022.
Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arv preprint wxXiv:111.6998/ 2014.
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. In International Conference on Learning Represenlations, 2016.
Shengchao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with se (3)-invariant denoising distance matching. arXiv preprint Xiv:22200. 13602, 2022a.
Shengehao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. In International Conference on Learning Representa- tions, 2022b.
Tairan Liu, Misagh Naderi, Chris Alvin, Supratik Mukhopadhyay, and Michal Brylinski. Break down in order to build up: decomposing small molecules for fragment-based drug design with e molfrag. Journal of chemical information and modeling, 57(4):627 631, 2017.
Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.050:1 2021.
Shengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie- Yan Liu, Liwei Wang, and Di He. One trans- former can understand both 2d 3d molecular data. arXiv preprint wxXi:22200001666 2022.
Kha-Dinh Luong and Ambuj Singh. Fragment- pretraining and finetuning on molecular graphs. anXiv preprint 2rX1::2310.00 03274, 2023.
Omar Mahmood, Elman Mansimov, Richard Bonneau, and Kyunghyun Cho. Masked graph modeling for molecule generation. Nature communications, 12(1):1-12, 2021.
Harry L Morgan. The generation of a unique machine description for chemical structures-a technique devel- oped at chemical abstracts service. Journal of chemical documentation, 5(2):107-113, 1965.
12

---

Published in Transactions on Machine Learning Research (05/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
David Rogers and Mathew Hahn. Extendedd-commectvvtty fingerprints. Journal of chemical information and modeling, 50(5):742-775 2010.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang, Self- supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33:12559-122 12577, 2020a.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In International Conference on Learning Representations, 2020b.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnot: A contimuous- convolutional neural network for modoling quantum interactions. Advances in nearul information processing systems, 30, 2017.
Myungwon Sco, Hyun Kil Shin, Yoochan Myung, Sungbo Hwang, and Kyoung Tai No. Development of natural compound molecular fingerprint (ne-mfp) with the dictionary of natural products (dnp) for natural product-based drug development. Journal of Cheminformatics, 12(1):1 110:11,7, 2020.
Janice G Smith. Organic chemistry. McGraw-Hill, 2008
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günne- mann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. In International Conference Machine Learning, pp. 20479 20502. PMLR, 2022.
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908. 1FXN::1988.01000, 2019.
Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou. Mocl: data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 3585 3594, 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings Part XI 16, pp. 776-794. Springer, 2020.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molclr: Molecular contrastive
learning of representations via graph neural networks. arXie preprint arXiv:2102.10056, 2021.
Yuyang Wang, Rishikesh Magar, Chen Liang, and Amir Barati Farimani. Improving molecular contrastive learning via faulty negative mitigation and decomposed fragment contrast. Journal of Chemical Informa- tion and Modeling, 2022.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenct: benchmark for molecular machine lcarning. Chemical science, 9(2):513-538) 2018.
Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. 2023.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Represenlations, 2019.
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level represen- tation learning with local and global structure. In International Conference on Machine Learning, pp. 11548-11558, PMLR, 2021.
Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for dusti--disttibution molecular representations. In Advances in Neural Information Processing Systems, 2022.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image pre-training. In International Conference on Learning Representations, 2022.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Newral Information Processing Systems, 33:5812 5823, 2020.
Yuming You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning, pp. 12121 12132. PMLR, 2021.
Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez- Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-traine Pre-training via denoising for molec- ular property prediction. arXiu preprint arXio:2206.0011 2022.
Shichang Zhang, Ziniu Hu, Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations. Xiv preprint arXiv:2012.1255 2020.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu,, and Chee-Kong Lee. Motif-based graph self-supervised learning for molecular property prediction. Advances in Neural Information Processing Systems, 34, 2021.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. 2022.
Jinhua Zhu, Yingee Xia, Tao Qin, Wengang Zhou, Houqiang Li, and Tie- Yan Liu. Dual-view molecule pre-training, orXiv preprint arXio:2106. 10234, 2021a.
Jinhua Zhu, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Ya Liu. Unified 2d and 3d pre-training of molecular representations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2626 2636, 2022.
Yanqiao Zhu, Yichen Xu, Hejie Cui, Carl Yang, Qiang Liu, and Shu Wu. Structuro-aware hard negative mining for heterogencous graph contrastive lcarning. arXiv preprint arXiv:2108 13886, 2021b.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Supplementary Material
Appendix: Holistic Molecular Representation Leraning via Multi-view Fragmentation
A Experimental details
Self-supervised pretraining details. We follow the training setup considered in GraphMVP (Liu et al, 2022b): Spocifically, we usc a batch size of 256 and no weight decay. We use (Nodedrop, Attrmask, identity/ / randomly, i.e. probability for each fragment and the original 2D molecular graphs, and Gaussian noise I) each coordinate of 3D molecular graphs. When Nodedrop or Attrmask used, we drop/ hoop/makk/o the portion of 0.1 vertices from the total vertices. For self-supervised pretraining, we train for 100 epochs using Adam optimizer (Kingma & Ba. 2014) with learning rate of 0.001 and no dropout. For transfer learning to the QM9 (Ramakrishnan et al. 2014 dataset, we train with 310k entire unlabeled molecules from GFOM for 50 epochs. For semi-suueervised leraning for the QM9 dataset, we train with 110k training molecules (without labels) from QM9 for 50 epochs. Our code is based on open-source codes of GraphMVF
For Holi-Mol trained only with single view objective and other reproduced 2D basolines, we exclude explicit hydrogens in molecular graph, following the common frameworks of (You et al.) 2020 2021) for 2D moloc- ular graphs. For Holi-Mol, 3D-InfoMax, GraphMVP, GraphMVP-C, and GraphMVP-G we include explicit hydrogens into molecular graph, following (Liu et al. 2022b) that utilizes the 3D coordinates of hydrogen atoms provided in GEOM dataset (Axelrod & Boonaz-Bombarelii 2022). For torsional angle prediction task, we use 2-layer MLP for Foor and we construct the quartet of atoms (s, u, v,() () for the fragmented bond (u, v) so that 8, are non-hydrogen atoms, and the bimning of y splits O to 2x into 18 uniform bins.
Evaluation on MolcculeNet downstream tasks. Following Liu al. (202bb), we use scaffold split (Chen et al.) 2012) in MoleculeNet experiments which splits the molecules based on their substructures. We use the split ratio train:validationn:tst performance. For the consistency of the input graphs in pretraining and fine-tuning, hydrogen atoms of molecules in fine-tuning dataset for single-view pretrained Holi-Mol and other reproduced 2D baselines and we include implicit hydrogen atoms of molecules in fine-tuning dataset for Holi-Mol, 3D- InfoMax, GraphMVP, GraphMVP-C, and GraphMVP-G. Experimental detail follows GraphMVP {Liu et al,, 2022b); we fine-tune a pretrained 2D-GNN with an initialized linear layer for 100 epochs with Adam optimizer and a learning rate of 0.001, and dropout probability of 0.5. Our results are calculated by the test ROC- AUC score of the epoch with the best validation ROC-AUC score. Besides the ROC-AUC score of individual downstream tasks, we also report the average ROC-AUC score across downstream datasets.
80:10:10 for each downstream task dataset to evaluate the exclude implicit
Evaluation on QM9 downstream tasks. Following (Liu et al., 2021 we split the molecules in the QM9 (Ramakrishnan et al. 2014) dataset into 110,000 molecules for training, 10,000 molecules for validation, and 10,831 molecules for test. Our result is calculated by the test MAE score of the epoch with the best validation MAF score. We fine-tune a pretrained 2D-GNN with an initialized 2-layer multi layer perceptron for 1,000 epochs with Adam optimizer and StepLR scheduler with decay ratio of 0.5, and initial learning rate of 5c-4.
15

---

Published in Transactions on Machine Learning Research (05/2024)
B
Baselines details
We compare our method with an exlensive list of baseline methods the literature of graph representation learning:
No pretraining trains a model from scratch for downstream task.
AttrMask
(Hu et al.
2020a learns representation by recovering the vertex features after masking
them.
GPT-GNN (Hu et al.
2020b uses the graph generation task as pretext task.
maximizes mutual information between global representations (i.e., graph representations] and local representations (i.e. path representation).
Infomax (unn et al.
2019
ContentPred Hu al. node edge.
2020a learns representation by predicting surroundigg subgraph of specific
discriminates graph and subgraph pairs from their opposing pairs to preserve local similarity between various graphs, which leads to the embedding alignment of correlated graphs.
GraphLoG Xu et al. 2021 )
G-Conteatual (Rong et al. target nodes (or edges) and predicting these contextual properties from node embeddings.
2020a learns representations by randomly masking local subgraphs of
G-Motif Rong et al. chemical prior.
2020a predicts the occurrence of the semantic motifs extracted by using
GraphCL You et al. agnostic augmentation schemes, which do not use any molecule-specific knowledge.
2020 is generic graph contrastive learning method based on their graph-
2021 proposes min-max optimization processes to learn optimal data augmen- tation strategies dynamically from pre-fixed candidate set of augmentations.
JOAO You et al.
MGSSL [Zhngg al. tree.
2021 introduces a generative self-supervised objective to reconstruct motif-
MolCLR (Wang et al. constructing positive views of molecule by proposed molecule augmentation schemes.
2021 performs contrastive learning with NT-Xent (Chen et al. 2020
D-SLA (Kim al. 2022) extracts graph representations by learning the exact discrepancy between the original graph and the augmented graphs.
Mole-BERT (Xia al. 2023) utilizes VQ- AQEV (Van Den Oord et al. 2017 encode atoms meaningful discrete values and then perform masked atoms modeling and triplet masked contrastive learning.
3D-InfoMax Stääk et al. 2022) proposes to consider 2D topological molecule graph and 3D geometric molecule graph from the same molecule positive view of each other.
GraphMVP, GraphMVP-G, and GraphMVP-C (Liu et al., 2022bb regard 2D and 3D molecular graphs positive pair, and propose feature reconstruction of each view generative task.
16

---

Published in Transactions on Machine Learning Research (05/2024)
C
Graph neural networks
Graph Isomorphism Network (GIN). We provide detailed description of architecture of graph iso- morphism network (GIN) (Xu et al. 2019 which we mainly consider as the feature extractor f-o(-) in this
paper. Particularly, GIN learns representation h. by:
h = MLP(() (h (1((-1) + Σ
(h + ))
(8)
uEN(e)
where e(f-1) the embedding corresponding to the attribute of edge {u, 0] E
SchNet. We consider SchNet (Schütt al. 2017). which is a strong 3D graph neural network under
fair comparison (Liu et. al. 2022b as our (= in this paper. Particularly, SchNet learns representation he MLP(() by:
h
MLP Σ (0(h(-1), Fe, raorr))) WEV
(9)
where & the continuous-filte convolution layer and rv the 3D position of the vertex v.
17

---

Published in Transactions on Machine Learning Research (05/2024)
D Downstream dataset details
We perform transfer-learning on 8 benchmark binary classification datascts from MoleculeNet (Wu et al.
2018 More information on downstream tasks is described in Table 7
BBBP contains data whether the compound is permeable to the blood-brain barrier.
Tox21 measures the toxicity of a compound and was used in the 2014 Tox21 Data Challonge.
TorCast includes multiple toxicity annotations of compounds collected after performing high- throughput screening tests.
Sider refers to side effect resources, i.c., data on the marketed drugs and their side effects.
Clintor is a dataset of comparison results between drugs approved through the FDA and drugs removed because toxicity during clinical trials.
MUV is validation dataset of virtual screening technology. Specifically, it is subsampled in the PubChem BioAssay using refined nearest neighborhood analysis.
HIV consists of data about capability prevent HIV replication.
Bace is collected dataset of compounds that could prevent (BACE-1).
Table MoleculeNet downstream classification dataset statistics
Dataset
BBBP
Tox21
ToxCast
Sider
Clintox
MUV
HIV
Bace
Number of molecules
2,039
7,831
8,575
1,427
1,478
93,087
41,127 1
1,513
Number of tasks
1
12
617
27
2
17
1
Avg. Node
24.06
18.57
18.78
33.64
26.15
24.23
25.51
34.08
Avg. Degree
51.90
38.58
38.52
70.71
55.76
52.55
54.93
73.71
We also perform transfer-learnig on 12 benchmark regression tasks from QM9 More information on downstream tasks is described in Table g
(Ramakrishnan et al.
2014
Table 8: QM9 downstream regression tasks
Task
Summary
Unit meV D
ZPVE
Zero point vibrational energy
Dipole moment
a C LUMO HOMO
Isotropic polarizability
Qg
Heat, capacity at 298.15K
cal/mol
Lowest unoccupied molecular orbital energy Highest occupied molecular orbital energy
meV
meV
Egap R
Gap between HOMO and LUMO
meV
Electronic spatial extent Internal energy at 0K
ag²
Uu
meV
U298
Internal energy at 0K
meV
H298
Enthalpy at 0K
meV
G598
Gibbs energy at 0K
meV
18

---

Published in Transactions on Machine Learning Research (05/2024)
E Detailed analysis of results on MoleculeNet
Table 9: Statistics of training and test samples of MoleculeNet dataset under scaffold split (Chen et al.
2012). The number of atoms are calculated in consideration of hydrogen atoms.
Average of atoms
BBBP
Tox21
ToxCast
Sider
Clintox
MUV
IIV
Bace
Train
43.6
33.0
49.6
42.6
63.5
Test
52.1
49.8
52.3
102.0
43.0
41.8
45.5
67.9
Table 10: Relative ROC-AUC improvement to GraphMVP-G (the best performing baseline). We compare the results on the full test set (in Table D and the subset of the test set which contains the molecules with less atoms than the average number of atoms in training molocules.
Improvement of Holl-Mol from GraphMVP-G
BBBP
Bace
MUV
HIV
Full fest molecules
1.3
1.0
-0.1
1.0
Test molecules with less atoms than Avg. atoms training set
2.3
1.1
2.9
1.5
In Table 1 the improvement of Holi-Mol is most dramatic the Clintox dataset. We observe that Clintox is unique in that it has higher average number of atoms in the training set compared to the test set (see Table 9 Furthermore, we observe that the performance gap between Holi-Mol and GraphMVP-G (the strongest baseline) tends to widen as the number atoms in test molecules decreases. To illustrate this point, Table 10 compares the performance gap when evaluated on the full test set and on the molecules in the test set with fewer atoms than the average in the training set (we consider the downstream tasks with Avg. # of atoms in the training set molecules Avg. # of atoms in the test set molecules] < 10, due to the stability of evaluation). Holi-Mol's enhanced performance smaller molecules can be explained by its capacity to learn fine-grained molecular features through fragmentation.
F Complexity analysis
Table 11: Analysis of the training time of each method. We compare the most competitive baselines in Table D The training time is caleulated based the same sctup of Table I with single NVIDIA GeForce RTX 3090 GPU.
MGSSL
Mole-BERT
3D-Infomex
GrephMVP-G
Holi-Mol (Ours)
Training time (hours)
17.5
0.7
0.9
4.5
1.9
Test ROC-AUC score
70.8
71.8
73.4
74.1
75.5
In Table 11 we provide the rumning time analysis of each method and the corresponding test ROC-AUC score on the MoleculeNet benchmark. We observe that MGSSL (Zhang et al.) 2021) requires extensive training cost because of the rege-regressive predictive pretext task. GraphMVP-G (Liu et al. 2022b utilizes several 3D conformer structures, lcading to higher training complexity. Overall, our Holi-Mol significantly improves the prior best-performing baseline, GraphMVP-G [Liu al., 20022b, the test ROC-AUG score (74.1 + 75.5; higher better) with less training time (4.5 1.9; lower better).
19

---

Published in Transactions on Machine Learning Research (05/2024)
G Detailed results on semi-super learning
ZPVE
HOMO
No pret-an Th-nttedaa
prerran 3'HefflAncc
Ni protiain WDirthaae - EopphWiisGG [Duns]
ho pretrun 1D
CraptknAP-G
200
102-
200
([um)
200
IE
IAAE
IMKE:
100
100
50-
90-
20
60
(86
Comparison of test MAE score of semi-supervised learning with different fraction of labeled samples on QM9 downstream quantum property regression benchmarks. We pretrain GIN (Xu et al. 2019) as the 2D-GNN architecture with 110k molecules from QM9 pretraining dataset.
Figure 2:
Table 12: Comparison of test MAE score of semi-supervised learning on the QM9 downstream quantum property regression benchmarks. We pretrain GIN (Xu et al., 2019) as the 2D-GNN architecture with 110k QM9 training set and fine-tune on 10% subset of QM9 training set. We mark the best score bold.
Methods
ZPVE 4
C. 4
LUMO
HOMO
Egap
R² 4
Uo 4
U208
Haos
G288
173.1
0.339
2.67
0.882
415.5
340.7
680.8
20.6
278.0
301.3
299.9
274.1
Mole BERT Xia et al.
2023
163.6
0.330
2.65
0.877
402.2
329.0
671.2
20.2
266.6
287.1
285.3
250.9
Pretrained on 110k 2D and 3D molecular graphs of QM9 and fine-tuned on 10% 2D molecular graphs of QM9
Infoma Stärk al.
2022
168.7
0.325
2.59
0.878
395.3
332.7
672.7
20.4
257.5
284.1
283.9
240.4
GraphMVP-G Lai et al.
20222
162.6
0:324
2.58
0.872
388.3
325.8
602.7
19.9
255.4
281 .4
271.7
245.3
Holi-Mol (Ours)
151.5
0.322
2.51
650.5
19.8
252.9
279.4
269.1
243.6
Table 13: Comparison of test MAE score on QM9 downstream quantum property regression benchmarks when additional 3D geometric is available in fine-tuning. We utilize GIN (Xu et al. 2019) and SchNet Schütt et al. 2017 for 2D and 3D encoder, respectively. We mark the best score bold.
Methods
ZPVE +
LUMO 4
HOMO
Ug 4
1.67
26.8
32.9
14.5
GraphMVP-G Liu et al. 2022b
1.65
26.1
32.8
14.3
GooSSL-DDM Lit cb 2022a
1.63
25.8
32.0
14.1
3D-EMGP Jiao et al.
2002
1.63
25.9
31.9
14.0
Holi-Mol (Ours)
1.61
25.8
31.6
13.9
In this section, we provided detailed results for semi-supervisd learning on the QM9 (Ramakrishnan et al. 2014) dataset. Figure 2 shows the test MAE score across different label fractions after pretrained with the QM9 training dataset. We choose 4 downstream tasks which yields the highest performance gap after pretraining compared to non-pretraining (we exclude Egop |HOMO - LUMO| since we already include HOMO and LUMO). As visualized, Holi-Mol consistently outperforms the considered baselines. Table 12 shows the results for all 12 downstream tasks of QM9 when fine-tuned with 10% of training data. For all downstream tasks, Holi-Mol achieves the best performance.
Although our primary focus is to pretrain 2D-GNN (Stäkk et al. 2022 Liu et al. 2022b), we explore the possibility of Holi-Mol in the scenario when 3D information additionally available the line-tuning phase. We follow the same experimental setup as detailed in Table 2 (i.e., pretraining on QM9 and using SchNet the 3D GNN). We test 4 downstream tasks (ZPVE, LUMO, HOMO, and Uo) in the QM9 dataset following the setup in Table 3 and Figure 2 We additionally compare with recent 3D geometric pretraining basclines, 3D-EMGP (Jioo et al. 2022) and GeoSSL-DDM (Liu et al. (022a).. For Holi-Mol and GraphMVP-G, we consider fine-tuning 2D and 3D encoders jointly. Here, we note that one of the benefits of Holi-Mol compared to geometric pretraining basclines (with only 3D encoder) is that we can utilize both 2D and 3D encoders. In Table 13 we observe that Holi-Mol still shows reasonable performance when 3D information is additioanlly available in fine-tuning dataset.
20

---

Published in Transactions on Machine Learning Research (05/2024)
H Ablation on fragment-based positive view construction of Holi-Mol
Table 14: Comparison of positive view construction strategies for multi-view molecular contrastive learning framework. We report the test ROC-AUC score on the MoleculeNet downstream property classification benchmarks. We protrain GIN (Xu et al., 2019) as the 2D-GNN architecture with 50k molecules from the
GEOM dataset (Axelrod & Beanez-Bombarelli 2022 following Liu et al. (022b).. We report mean and
standard deviation over 3 different seeds. We bold the best average score.
Positive view construction
Fragromntatinn strategy
BRRP
Tow21
ToeClst
Sider
Clintox
MUV
HTV
Bare
Avg.
Nodeehop, Sabgraph
69.3
76.016.
63.7104
60.4g14
88.318
76.2ga
76.2
78.3164
73.4
Random bond deletion
69.3e1.
73.8
63.9e0s
39.9e122
91.42s
76.8
71.es.1
78.3e2.
73.5
set of fragments (Ours)
Random non-ring bond deletion
69.5ee..
73.7
64.0saa
60.5eas
98.2s11
77.3ea5
75.2saa
78.8
74.0
BRICS decomposiiinn (Ours)
71.4804
75.2sa7
65.1eas
61,0eao
95.2s10
77.6e10
76.3
82.3=1e
75.5
In Table 14 we provide a full result of Table in Section
4.3
We conduct an ablation study on regarding
the set of fragments as a positive view of molccule. Again, we emphasize that the result implics that considering chemically informative structures key component of Holi-Mol.
Component ablation of Holi-Mol
Table 15: Ablation of components for multi-view molecular contrastive learning framework. We report the test ROC-AUC score on the MoleculeNet downstream property classification benchmarks. We pretrain GIN (Xu et al. 2019) as the 2D-GNN architecture with 50k molecules from the GEOM dataset (Axelrod
& GomeezBoomareee Bombarelli 2022). following Liu et al. (2022b). We report mean and standard deviation over 3 different seeds. We mark the best mean score to be bold.
Multivvlew Internetion
Pretraining Molocule-evel Fragment- Jevel
Tomsnn level
TocCast
Arg.
Single-viow (21))
71.0a0
75.3a9
62.800
00.3a11
70.1as
74.3
75.9a1
80.72)
72.4
68.2200
75.6211
64.620
60.830a
94.92o
77.7g10
76.8308
70.520
74.7
Multivieww (2D 3D)
71.0aa
75.3a6
64.416a
61.1166
95.1g1
76-4414
76.2107
80.922
75.1
05.1esa
95.2a1a
76.3s04
82,3s16
In Table 15
we provide a full result of Table 0 in Section 4.3 We validate that cach components Holi-Mol
has an individual effect in improving the performance of multi-view pretraining.
Singlevee loss
Cross-ieeu loss
Torsion
20
60
100 20
60
80
220
20
60
100
opoch
spoch
opoch
Figure 3: Training curves of each component: (1) Single-view objective Loingler (2) Cross-view objective
Leross, and (3) Torsion objective Lror.
In Figure 3 we report the training curves of our objectives. Our training objectives carefully designed learn crucial but non-overlapping features of unlabeled molecules, e.g., sogmennt/ooeecullleee 2D/3D view. As a result, the individual losses consistently decrease during training.
21

---

Published in Transactions on Machine Learning Research (05/2024)
J
Case study
GraphMVP-G
Holi-Mol (Ours)
Malecules cyclohexyl
Molecules w/ cyclshexy.
Malecules benzyl
Molecules fluare benzy!
Figure 4: t-SNE plot of the representations obtained by GraphMVP-G (left) and Holi-Mol (right).
The global properties, i.e., labels, of a molecule are closely associated with its meaningful local substructures, ie., fragments. For example, the ratio of 1-labeled molecules in the BACE (0-1 labeled) dataset is 45%. However, the ratio drops to 29% for samples with cyclohexyl group, and the ratio goes up to 60% for samples with fluoro-benzyl group. These variations can be attributed to the hydrophobicity and the aromaticity or hydrophilicity of the fluoro-benzyl group. However, the current state-of-the-an meth- ods, e.g., GraphMVP-G, do not take into account the relationship between the local and global information of molccules. Thus, our method aims to capture such relationship in molecular representation learning. For empirical support, in Figure we provide t-SNE of the molecular representations containing the discussed semantic-determinnng groups (cyclohexyl and fluoro-benzyl) in the GEOM pre-training dataset. Holi-mol shows superior discriminability for such groups, which supports the overall improvements in downstream tasks in Table 2 and g This validates the efficacy of our novel approach, which captures the interaction between a molecule and its constituent fragments within a unified framework.
the cyclohexyl group
Results on PCQM4Mv2
Table Results on the PCQM4Mv2 dataset.
Methods
ID-Inmxxx
GraphMVP-G
Holi-Mol (Ours)
Validation MAE
0.1703
0.1692
0.1693
0.1688
In Table 16 we verify the effectiveness of our method in the PCQM4M<2 dataset. Specifically, we fine- tune the pre-trained models of each bascline with small fraction of the training data, i.c., 10%, to follow the conventional evaluation setup of molecular representation learning where the labeled data is fewer than unlabeled data. Our method achieves superior performance in terms of validation MAE.
22

---

Published in Transactions on Machine Learning Research (05/2024)
L
Comparison with fingerprints
Table 17: Comparison with fingerprint representation (Morgan
1965) on MoleculeNet.
Representatious
BEBP
Tox21
ToxCast
Sider
Clintox
MUV
HIV
Bacc
Avg.
Morgan fingerprints Morgan
1965
67.5215
70.420.
56.7=9.
57.720.5
75.422.1
66.4202
66.1102
76.4
67.1
Ours
71.4+0.4
75.2+0.7
65.1
61.0+o.6
95.2+1.0
77.0=10
76.3=0.4
82.3#16
75.5
In Table 17 we compare the representations obtained from our method with the commonly used Morgan fingerprints (Morgan 1965) of molecules and fragments, where the attention pooling used to aggregate the fingerprints of fragments. The results show that our learned r; and r consistently outperform the fingerprints, which verifies the effectiveness our learned representations.
M Ananlysis on fine-tuning
Cross-view
Plot of the view objective Ceroes and the MAE objecive in fine-tuning. Note that the objective function in fine-tuning is the MAE objective to the target label (not the cross-view objective).
Figure 5:
we provide an analysis on fine-tuning. We pre-train with a half of the QM9 dataset and then fine-tune exclusively on the 2D view of another half of the QM9 dataset (which has target labels). In the fine- tuning phase, the MAE improves while the contrastive loss between the 2D and 3D views increases, which is natural since do not impose the contrastive objective in fine-tuning (rather apply the supervised objective to learn the labels). In other words, our pre-training scheme learns "good initialization" of a GNN using both 2D and 3D views and then we additionally inject the target-specific 2D information of molecules using the learned GNN via fine-tuning.
In Figure 5
N Variance of QM9 experiment
Table 18: Standard deviation of the experiment in Table 2
The results are based on 3 seeds.
Methods
ZPVE
LUMO
HOMO J
Up 4
43.7
80.5
89.4
62.9
3D- Infomaxx Stärk 2022
27.0
63.4
55.2
38.8
G Liu (0222
24.1
59.1
53.8
39.9
Holi--Mol (Ours)
24.010.1
57.2
51.8103
39.0a13
23

---

Published in Transactions on Machine Learning Research (05/2024)
O Discussion on evaluation setup
Table 19:
Test. MAE on the QM9 downstream quantum property regression benchmarks. We mark the best score bold. The contents are the same Table Mechods
ZPVE
1 4
a
4
C. 4
LUMO 4 HOMO
Egop
Re
4
Uo
Una 4
Hoos
Grus J
13.7
0.059
0,400
0.144
80.5
89.1
171.0
3.27
62.9
61.8
57.0
48.1
Preleained on 310k 2D molecular graphs of GEOM fine- luned ou 2D molecular graphs QM9
3D-Iufomax Stark al. GraphiMPP-G
2022
27.0
0.051
0.355
0.126
63.4
55.2
103.8
2.99
38.8
45.6
41.0
40.8
2022b
0.051
59.1
53.8
100.4
2.97
41.0
40.3
Holi-Mol (Ours)
57.1
2.90
42.9
Table 20: Test ROC-AUC score on the MoleculeNet molecular property classification benchmarks. We report mean and standard deviation over 3 different. seeds. We mark the best mean score to be bold. Methods
BBBP
Tox21
ToxCast
Sider
Clintox
MUV
HIV
Bace
Avg.
65.4124
74.9.08
61.6112
58.0=2.4
58.8
71.0=2a
75.3108
72.614.9
67.2
Pretrained with 310k 2D and 3D molecular graphs of GEOM and fine-tuned on 2D molecular graphs of MoleculeNet
3D-InfoMax Stärk et al.
2022
68.8+0.2
75.0=1s
61.4104
61.7=02
89.7#1a
76.9=a.5
75.3 75.7+0.5
79.8an.
73.9
GraphMVP-GF (Lu 2022b
69.8+0.8
75.5=0.1
61.1+0.2
61.1=0.7
90.1
77.6+0.5
80.3
74.3
Holi-Mol (Ours)
71.210.
75.7103
64.910.4
61.71002
95.3
77.6114
76.510.4
82.410.
75.7
In Table 2 and 3 we choose different pre-training setups to consider various practical scenarios. We first
remark that the molecule space is very large. Thus, the fine-tuning distribution may often be quite different from the pre-training distribution. Therefore, divide the evaluation setup into two cases based the pre-training and fine-tuning distributions.
In Table 1 and 2
we assume the pre-training and fine-tuning distribuutioss are different. Specifically, we pre-train on the GEOM dataset and then fine-tune on the MoleculeNet (Table I and QM9 (Table 2), respectively.
In Table 3 we assume the pre-training and fine-tuning distributions are the same. Specifically, we use the QM9 dataset for both pre-training and fine-tuning.
We also that we use 50k molecules from the GEOM dataset in Table [ for a fair comparison with existing works reported in Liu al. (202bb). However, we find that there is no specific reason not to use the full GEOM dataset (310k molccules) for pro-training. Thus, in the evaluation QM9 (Table 2 the entire 310k molecules of the GEOM dataset for pre-training the models.
Nevertheless, it is possible to pre-train single model for all the downstream tasks. We further report the results on the downstream tasks using a single pre-trained model. Specifically, we use the pre-trained model based on the 310k molecules from the GEOM dataset (which the setup Table ? for each method. Table 19 and 20 show that the single pre-trained model also outperforms the baselines on the downstream tasks (MoleculeNet and QM9).
24


Reviews:
[{"review_id": "ganzFB2weP", "reviewer": "Reviewer_JUTC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- leveraging the fragmentation of Molecule is a reasonable approach to incorporate more prior information into the representation framework.\n- the cross-view contrastive learning is also interesting, where the model would hopefully learn the information across different channels.\n- Experimental results seem to be strong.\n\nWeakness:\n- Overall the representation learning framework and the multi-view idea is not new. The contribution on the fragmentation is also relatively straightforward and incremental, compared to the existing works.\n- Some of the claims are not well supported, and some more ablation studies might be needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a new molecule representation learning framework via semi-supervised learning. The core idea is to leverage the fragment decomposition of molecules to have a new representation parameterization. Regarding the training objective, it combines both the contrastive ones and the predictive one (using 3D information), where the contrastive one tries to provide the contrast between representations from both the 2D and 3D parameterizations. \n\nExperiments on benchmark datasets show that the proposed method is able to improve the performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The author claimed that the proposed approach is able to capture both the local and global information, however from the experiment it is not obvious to me. The authors only show the numerical comparison in terms of the final accuracy, but how the ability of obtaining local/global information relates to the accuracy is not clear. So it would be good to provide such kind of studies to show the benefit claimed by the paper.\n\nFurthermore, the fragmentation idea in Eq (2) and (3) seems to be yet another GNN that represent the graphs. It is known that GNNs are able to capture some high-order information, so it is not clear in terms of the motivation, why the explicit fragmentation is needed, or if the author can provide some theoretical justification to this (in terms of the representation power, e.g., try to prove that GNN is not able to represent the fragments or something like that). \n\nSeveral ablations might be needed as well:\n- try to see the performance if we just use the GNN (without the fragmentation) to do the single/cross-view and angle prediction\n- see if we just use the fragmented GNN and do the pretraining (without the vanilla GNN)\nthe purpose is to decouple the representation power of GNN v.s. the objective functions used in the paper.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lP6JxA2JT7", "reviewer": "Reviewer_iXqL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The manuscript is well-written and easy to follow.\n- The proposed method is simple and effective.\n\nWeaknesses:\n\n- The contribution of the proposed idea appears to be relatively incremental, merging concepts of 2D & 3D and fragment-based contrastive learning, both of which have been extensively explored in prior research. This raises questions about the significant advancement over existing methodologies.\n\n- The experimental design is somewhat confusing. The inclusion of three distinct training configurations (50k, 310k, and 110k datasets) in Tables 1, 2, and 3 lacks a clear justification. In MRL, the goal is usually to develop a single pre-trained model capable of addressing a variety of downstream tasks, rather than creating bespoke models for individual tasks. \n- It remains uncertain whether the proposed method can be applied to tasks requiring 3D inputs, as the paper currently only demonstrates its applicability to 2D input tasks. \n- Providing results on the PCQM4MV2 dataset, particularly on the test set, would offer a more comprehensive assessment of the model's effectiveness and applicability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a MRL model that decomposes a molecule into a set of fragments, to associate a global graph structure to a set of local substructures, and also consider the 3D geometry of molecules as another view for contrastive learning.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Referring to Weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0BLJHq7U2y", "reviewer": "Reviewer_wGgA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is novel in its approach to utilizing both holistic and segmented representations of molecules.\n\n2. It incorporates the use of both 2D and 3D perspectives for the self-training of molecules.\n\n3. It introduces the use of torsional angle prediction as a method to understand the relationships between molecular fragments.\n\n\nWeaknesses/Questions:\n\n1. **The Necessity of 2D GNN:** Is the application of a 2D GNN for molecule embedding mandatory? Could molecular fingerprints serve as an alternative to ￥r_i$  and $r_i^j$s?\n\n2. **Trend of Multimodal Large Models:** Given the current interest in multimodal large models, and considering multi-view as a subset of multimodal, why opt for contrastive learning across views instead of employing cross-attention mechanisms like those in CLIP?\n\n3. **Implementation of 3D GNN:** In employing SchNet for 3D positions, were the molecular fragments re-centered?\n\n4. **Loss Sensitivity:** With the model incorporating multiple types of losses, is there a risk of these losses competing against each other?\n\n5. **Fine-Tuning Forgetting Issue:** When the model is pre-trained on both 2D and 3D molecular graphs but fine-tuned solely on 2D graphs, would this enlarge the contrastive loss between the 2D and 3D representations?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces an innovative framework that utilizes a multi-view fragmentation approach for the pre-training of molecules, considering both holistic and fragmented representations. The concept of employing both 2D and 3D perspectives of molecules presents a novel approach. To understand the interactions between molecular fragments, the authors ingeniously devise a torsional angle prediction method, facilitating the learning of correlations among fragments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on the identified weaknesses and questions:\n\n1. **2D GNN Necessity:** Conduct an ablation study by substituting the 2D GNN embeddings for molecules and fragment sets with fingerprints to evaluate the critical role of 2D GNNs.\n\n2. **Loss Sensitivity:** Plot the training loss for all three types of losses to determine if any of the losses are, in fact, increasing during training.\n\n3. **Fine-Tuning Forgetting Issue:** Divide the GEOM dataset into two parts. Pre-train the model using both 2D and 3D views of the first part, then fine-tune it exclusively on the 2D view of the second part. Evaluate the contrastive loss between 2D and 3D views of the second part to investigate any increase in loss.\n\nMinor:\n\n4. **Standard Deviation Reporting:** It is advisable to include the standard deviation (STD) in Table 2, as the improvements depicted appear to be marginal. This additional data would provide a clearer understanding of the results' variability and significance.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewer JUTC is the only gatekeeper for this paper. After a thorough examination of the paper, the reviews, and the authors' response, I conclude that the remaining concerns raised by Reviewer JUTC are unfounded. The authors have indeed provided compelling evidence that demonstrates the underperformance of a molecule-level loss approach, which corresponds to the use of a standard GNN. I reached out to Reviewer JUTC for confirmation but have not received a reply.

Given the positive attributes of this work, including the innovative simultaneous consideration of both 2D and 3D perspectives and both holistic and fragmented views, the application of torsional angle prediction, and the strong empirical results, I am inclined to recommend acceptance.
Decision: Accept (poster)