Pre-training Molecular Graph Representation with 3D Geometry
Authors: Shengchao Liu
Hanchen Wang
Weiyang Liu
Joan Lasenby
Hongyu Guo
Jian Tang
Venue: ICLR 2022 Poster (2022)
Keywords: Pre-training
SSL
Molecule
3D Geometry
2D representation

Abstract: Molecular graph representation learning is a fundamental problem in modern drug and material discovery. Molecular graphs are typically modeled by their 2D topological structures, but it has been recently discovered that 3D geometric information plays a more vital role in predicting molecular functionalities. However, the lack of 3D information in real-world scenarios has significantly impeded the learning of geometric graph representation. To cope with this challenge, we propose the Graph Multi-View Pre-training (GraphMVP) framework where self-supervised learning (SSL) is performed by leveraging the correspondence and consistency between 2D topological structures and 3D geometric views. GraphMVP effectively learns a 2D molecular graph encoder that is enhanced by richer and more discriminative 3D geometry. We further provide theoretical insights to justify the effectiveness of GraphMVP. Finally, comprehensive experiments show that GraphMVP can consistently outperform existing graph SSL methods. Code is available on GitHub: https://github.com/chao1224/GraphMVP.

Published as a conference paper at ICLR 2022
PRE-TRAINING MOLECULAR GRAPH REPRESENTATION WITH 3D GEOMETRY
Shengchao Liu Hanchen Wang', Weiyang Liu"M, Joan Lasenby', Hongyu Guoi, Jian Tang
Mila Uniivrsitéé de Montréal UUniversity of Cambridge "MPT for Intelligent Systems, Tübingen
National Research Council Canada
'EEC Montréal
CIFAR Al Chair
ABSTRACT
Molecular graph representation learning is fundamental problem in modern drug and material discovery. Molecular graphs are typically modeled by their 2D topo- logical structures, but has been recently discovered that 3D geometric information plays more vital role in predicting molecular functionalities. However, the lack 3D information in real-world scenarios has significantly impeded the learning geometric graph representation. To cope with this challenge, we propose the Graph Multi- Pre-training (GraphMVP) framework where self-supervised learning (SSL) is performed by leveraging the correspondence and consistency between 2D topological structures and 3D geometric views. GraphMVP effectively learns a 2D molecular graph encoder that is enhanced by richer and more discriminative 3D geometry. We further provide theoretical insights to justify the effectiveness of GraphMVP. Finally, comprehensive experiments show that GraphMVP can consistently outperform existing graph SSL methods. Code is available on GitHub.
INTRODUCTION
In recent years, drug discovery has drawn increasing interest in the machine learning community. Among many challenges therein, how to discriminatively represent a molecule with a vectorized embedding remains a fundamental yet open challenge. The underlying problem can be decomposed into two components: how design a common latent space for molecule graphs (i.e., designing a suitable encoder) and how to construct an objective function to supervise the training (i.e., defining learning target). Falling broadly into the second category, our paper studies self-supervised molecular representation learning by leveraging the consistency between 3D geometry and 2D topology.
Motivated by the prominent success of the trained graph neural networks for molecules yields promising performance downstream tasks and becomes increasingly popular [42, 54, 82, 90, 103, 104]. The key to pre-training lies in finding an effective proxy task (i.e., training objective) to leverage the power of large unlabeled datasets. Inspired by [54, 58, 79] that molecular properties [29, 54] can be better predicted by 3D geometry due its encoded energy knowledge, aim make use of the 3D geometry of molecules in pre-training. However, the stereochemical structures are often very expensive to obtain, making such 3D geometric information scarce downstream tasks. To address this problem, we propose the Graph Multi-View Pre-training (GraphMVP) framework, where 2D molecule encoder is pre-trained with the knowledge of 3D geometry and then fine-tuned on downstream tasks without 3D information. Our learning paradigm, during pre-training, injects the knowledge of 3D molecular geometry to a 2D molecular graph encoder such that the downstream tasks can benefit from the implicit 3D geometric prior even if there is no 3D information available.
caneuinnnnnnnuuuiingggg pipeline [17], unsupervisedly pre-
We attain the aforementioned goal by leveraging two pretext tasks on the and 3D molecular graphs: one contrastive and one generative SSL. Contrastive SSL creates the supervised signal at an inter-molecule level: the 2D and 3D graph pairs are positive they are from the same molecule, and negative otherwise; Then contrastive SSL [93] will align the positive pairs and contrast the negative pairs simultaneously. Generative SSL [38, 49, 91], on the other hand, obtains the supervised signal in an intra-molecule way: it learns 2D/3D representation that reconstruct its 3D/2D counterpart view for each molecule itself. To cope with the challenge of measuring the quality of reconstruction molecule 2D and 3D space, we further propose novel surrogate objective function called variation

---

Published as a conference paper at ICLR 2022
representation reconstruction (VRR) for the generative SSL task, which can effectively measure such quality in the continuous representation space. The knowledge acquired by these two SSL tasks is complementary, so GraphMVP framework integrates them to form more discriminative 2D molecular graph representation. Consistent and significant performance improvements empirically validate the effectiveness of GraphMVP.
We give additional insights to justify the effectiveness of GraphMVP. First, GraphMMPP is self- supervised learning approach based on maximizing mutual information (MI) between 2D and 3D views, enabling the learnt representation to capture high-level factors [6, 7, 86] in molecule data. Second, we find that 3D molecular geometry is form of privileged information [88. 89]. It has been proven that using privileged information in training can accelerate the speed of learning. We note that privileged information only used in training, while is not available in testing. This perfectly matches our intuition of pre-training molecular representation with 3D geometry.
Our contributions include (1) To our best knowledge, we are the first to incorporate the 3D geometric information into graph SSL; (2) We propose one contrastive and one generative SSL tasks for pre- training. Then we elaborate their difference and empirically validate that combining both can lead to a better representation; (3) We provide theoretical insights and case studies to justify why adding 3D geometry is beneficial; (4) We achieve the SOTA performance among all the SSL methods.
Related work. We briefly review the most related works here and include more detailed summa- rization in Appendix A. Self-supervised learning (SSL) methods have attracted massive attention to graph applications [57, 59, 97, 99]. In general, there are roughly two categories of graph SSL: con- trastive and generative, where they differ on the design of the supervised signals. Contrastive graph SSL [42, 82, 90, 103, 104] constructs the supervised signals at the inter-graph level and learns the representation by contrasting with other graphs, while generative graph SSL [34, 42, 43, 54] focuses on reconstructing the original graph at the intra-graph level. One of the most sigmificant differences that separate our work from existing methods is that all previous methods merely focus on 2D molecular topology. However, for scientific tasks such as molecular property prediction, 3D geometry should be incorporated as it provides complementary and comprehensive information [58, 79]. To fill this gap, we propose GraphM to leverage the 3D geometry in graph self-supervised pre-training.
2 PRELIMINARIES
We first outline the key concepts and notations used in this work. Self-supervieed learning (SSL) is based on the view design, where each view provides specific aspect and modality of the data. Each molecule has two natural views: the 2D graph incorporates the topological structure defined by the adjacency, while the 31) graph perspective, 3D geometric graphs focus on the energy while 2D graphs emphasize the topological information; thus they can be composed for learning more informative representation in GraphM Transformation is an atomic operation in SSL that can extract specific information from each view. Next, we will briefly introduce how to represent these two views.
better reflect the geometry and spatial relation. From chemical
2D Molecular Graph represents molecules as 2D graphs, with atoms as nodes and bonds as edges respectively. We denote each 2D graph as G2D (X, E). where X the atom attribute matrix and E is the bond attribute matrix. Notice that here E also includes the bond connectivity. Then we will apply a transformation function Tap on the topological graph. Given a 2D molecular graph gzD, its representation h22 can be obtained from 21) graph neural network (GNN) model:
hpp = GNN-2D (Gx)) = GNN-2D(TT.(X, E)).
(1)
3D Molecular Graph additionally includes spatial positions of the atoms, and they are needless to be static since atoms are in continual motion on a potential energy surface [4]. The 3D structures at the local minima on this surface are named conformer. As the molecular properties are conformers ensembled [36], GraphMVP provides novel perspective on adopting 3D conformers for learning better representation. Given a conformer gaD = (X, Rt,, its representation via a 3D GNN model is:
hap = GNN-3D(THD(Qw)) = GNN-3D(T:0DXX. R)),
(2)
A more rigorous way of defining conformer is in [65] conformer is isomer of a molecule that differs from another isomer by the rotation of single hond in the molecule.
2

---

Published as a conference paper at ICLR 2022
where R is the 3D-coordinate matrix and Tap is the 3D transformation. In what follows, for notation simplicity, we use a and y for the 2D and 3D graphs, i.e., a representations are denoted as h and hy.
_ 92D and _ 93D- Then the latent
GRAPHMVP: GRAPH MULTI- VIEW PRE-TRAINING
Our model, termed Graph Multi- View Pre-training (GraphMVP), conducts self-supervised learning (SSL) pre-training with 3D information. The 3D conformers encode rich information about the molecule energy and spatial structure. which are complementary to the 2D topology. Thus, applying SSL between the 2D and 3D views will provide a better 2D representation, which implicitly embeds the ensembles of energies and geometric information for molecules.
In the following, we first present overview of GraphMVP, and then introduce two pretext tasks specialized concerning conformation structures. Finally, we summarize a broader graph SSL. family that prevails the 2D molecular graph representation learning with 3D geometry.
3.1 OVERVIEW OF GRAPHMVP
Align
Contrastive See 3.2
Contrast
Generative See 3.3
x
h
Zx
h
Z-5
Project
Reparameterize
Figure 1: Overview the pre-training stage in GraphMVP. The black dashed circles denote subgraph masking, and we mask the same region in the 2D and 3D graphs. Multiple views of the molecules (herein: Halicin) are mapped to the representation space via 2D and 3D GNN models, where we conduct GraphMVP for SSL pre-training, using both contrastive and generative pretext tasks.
In general, GraphMVP exerts 2D topology and 3D geometry as two complementary views for cach molecule. By performing SSL between these views, it is expected to learn 2D representation enhanced with 3D conformation, which can better reflect certain molecular properties.
As generic SSL pre-training pipelines, GraphMVP has stages: pre-training then fine-tuning. In the pre-training stage, we conduct SSL via auxiliary tasks on data collections that provide both 2D and 3D molecular structures. During fine-tuning, the pre-trained 2D GNN models are subsequently fine-tuned on specific downstream tasks, where only 2D molecular graphs are available.
At the SSL pre-training stage, we design two pretext tasks: one contrastive and one generative. We conjecture and then empirically prove that these two tasks are focusing on different learning aspects, which summarized into the following learning, contrastive SSL utilizes inter-data knowledge and generative SSL utilizes intra-data knowledge. For contrastive SSL. one key step to obtain the negative view pairs for inter-data contrasting: while generative SSL. focuses on each data point itself, by reconstructing the key features at an intra-data level. (2) From the perspective distribution learning, contrastive SSI. and generative SSL are learning the data distribution from local and global manner, respectively. Contrastive SSL learns the distribution locally by contrasting the pairwise distance at an inter-data level. Thus, with sufficient number of data points, the local contrastive operation can iteratively recover the data distribution. Generative SSL, on the other hand, learns the global data density function directly.
points. (1) From the perspective representation
Therefore, contrastive and generative SSL are essentially conducting representation and distribution learning with different intuitions and disciplines, and we expect that combining both can lead to a better representation. We later carry ablation study (Section 4.4) verify this empirically. In

---

Published as a conference paper at ICLR 2022
addition, to make the pretext tasks more challenging, we take views for each molecule by randomly masking M nodes (and corresponding edges) as the transformation function, i.e., T2d = Tap = mask. This trick has been widely used in graph SSL [42, 103, 104] and has shown robust improvements.
3.2 CONTRASTIVE SELF- SELF-SUPERVISED LEARNING BETWEEN 2D AND 3D VIEWS
The main idea of contrastive self-supervised learning (SSL) [10, 69] is first to define positive and negative pairs of views from an inter-data level, and then to align the positive pairs and contrast the negative pairs simultaneously [93]. For each molecule, we first extract representations from 2D and 3D views, i.e., Pi and by. Then we create positive and negative pairs for contrastive learning: the 2D-3D pairs (z. y) for the same molecule are treated as positive, and negative otherwise. Finally, we align the positive pairs and contrast the negative ones. The pipeline is shown in Figure 1. In the following, we discuss two common objective functions contrastive graph SSL.
InfoNCE is first proposed in [69], and its effectiveness has been validated both empirically [10, 37]
and theoretically [3]. Its formulation is given follows:
exp(fa v))
exp(fu(1. æ²)
LINANNE: Bp(m.)))
log ex v)) + exp(fn(xi, + log explfuly. x)) + (3), )
where w², 3y are randomly sampled 2D and 3D views regarding to the anchored pair (æc, y). f (3x, y) and f (y, a) are scoring functions for the two corresponding views, with flexible formulations. Here we adopt fz y) fy( (har hy)) More details are in Appendix D.
Energy-Based Model with Noise Contrastive Estimation (EBM-NCE) is an alternative that has been widely used in the line of graph contrastive SSL [42, 82, 103, 104]. Its intention is essentially the same as InfoNCE, to align positive pairs and contrast negative pairs, while the main difference is the usage of binary cross-entropy and extra noise distribution for negative sampling:
LERMACE Ep(w) (pnttt log (1 (ff v))) Ep(nt log (f y))
(4)
Bo(e)) (p..(uim) log aff((y x))) Bptu, log logo (fs ())
where Pr. is the noise distribution and a is the sigmoid function. We also notice that the final formulation of EBM-NCE shares certain similarities with Jensen-Shannon estimation (JSE) [68]. However, the derivation process and underlying intuition are different: EBM-NCE models the conditional distributions in MI lower bound (Equation (9)) with EBM, while JSE is special case of variational estimation of f-divergence. Since this is not the main focus of GraphMVP, we expand the more comprehensive comparison in Appendix D. plus the potential benefits with EBM-NCE.
Few works [35] have witnessed the effect on the choice of objectives in graph contrastive SSL. In GraphMVP, we treat it as hyper-parameter and further ablation studies them, i.e., to solely use either InfoNCE (Lc = LINONNC) or EMB-NCE (Lc = LEEMANCE).
3.3 GENERATIVE SELF-SUPERVISED LEARNING BETWEEN 2D AND 3D VIEWS
Generative SSL another classic track for unsupervised pre-training [11, 48, 49, 51]. It aims at learning an effective representation by self-reconstructing each data point. Specifically to drug discovery, we have one 2D graph and certain number of 3D conformers for each molecule, and our goal is to learn a robust 2D/3D representation that can, to the most extent, recover its 3D/2D counterparts. By doing so, generative SSL can enforce 2D/3D GNN to encode the most crucial geometry/topolog information, which can improve the downstream performance.
There are many options for generative models, including variational auto-encoder (VAE) [49], generative adversarial networks (GAN) [30], flow- model [18], etc. In GraphMVP, we prefer VAE-like method for the following reasons: (1) The mapping between two molecular views is stochastic: multiple 3D conformers correspond to the same 2D topology: (2) An explicit 2D graph representation (i.e., feature encoder) is required for downstream tasks; (3) Decoders for structured data such graph are often highly nontrivial to design, which make them suboptimal choice.
Variational Molecule Reconstruction. Therefore we propose a light VAE-like generative SSL, equipped with crafty surrogate loss, which we describe in the following. We start with an example
4

---

Published as a conference paper at ICLR 2022
for illustration. When generating 3D conformers from their corresponding 2D topology. we want to model the conditional likelihood p z)). By introducing a reparameterized variable 2 = U + Oe, where Ue and ar two flexible functions on hgg é ~ NV, I) and is the element-wise production. we have the following lower bound:
logp(yl 2 (m(( [logp(y>==)) K [g (x) [p(=)))
The expression for log p(rl y) be similarly derived. Equation (5) includes a conditional log- likelihood and a KL.-divergence term, where the bottleneck is to calculate the first term for structured data. This term has also been recognized as the reconstruction term: it is essentially reconstruct the 3D conformers (y) from the sampled 2D molecular graph representation (zæ). However, performing the graph reconstruction on the data space is not trivial: since molecules (e.g., atoms and bonds) are discrete, modeling and measuring on the molecule space will bring extra obstacles.
Variational Representation Reconstruction (VRR). To cope with this challenge, we propose a novel surrogate loss by switching the reconstruction from data space to representation space. Instead of decoding the latent code As to data space, we can directly project it to the 3D representation space, denoted as Gar (Zn) Since the representation space is continuous, we may as well model the conditional log-likelihood with Gaussian distribution, resulting in L2 distance for reconstruction, (b) Here SG is the stop-gratd noopggradeett operation, assuming that fy is fixed
i.e., ae (zx) learnt representation function. SG has been widely adopted in the SSL literature to avoid model collapse [12, 31]. We call this surrogate loss variational representation reconstruction (VRR):
1 Eyte m) (xx) - (G(N))ll + ( ( [1vu(=x) (1
La LVrR=
2
(6)
(=](((=)<)))(( + K (L(g(z)))))ll(a _/2
3
We give simplified illustration for the generative SSL pipeline in Figure 1 and the complete derivations in Appendix E. As will be discussed in Section 5.1, VRR is actually maximizing MI, and MI is invariant to continuous bijective function [7]. Thus, this surrogate loss would be exact if the encoding function h satisfies this condition. However, we find that GNN, though does not meet the condition, can provide quite robust performance, which empirically justify the effectiveness of VRR.
3.4 MULTI-TASK OBJECTIVE FUNCTION
As discussed before, contrastive SSL and generative SSL essentially learn the representation from distinct viewpoints. reasonable conjecture is that combining both SSL methods can lead to overall better performance, thus arrive minimizing the following complete objective for GraphMVP:
LompaMvP = a1 Le + ag Lo,
(7)
where a1, ag are weighting coefficients. A later performed ablation study (Section 4.4) delivers two important messages: (1) Both individual contrastive and generative SSL on 3D conformers can consistently help improve the 2D representation learning; (2) Combining the two SSL strategies can yield further improvements. Thus, we draw the conclusion that GraphMVP (Equation (7)) is able to obtain an augmented 2D representation by fully utilizing the 3D information.
As discussed in Section 1, existing graph SSL methods only focus the 2D topology, which is in parallel to GraphMVP: 2D graph SSL focuses on exploiting the 2D structure topology, and GVP takes advantage of the 3D geometry information. Thus, we propose merge the 2D SSL into GraphMVP. Since there are two main categories in 2D graph SSL: generative and contrastive, we propose two variants GraphMVP-G and GraphMVP-C accordingly. Their objectives are as follows:
LompaMoR-G = LamspMMV + Leieerative 2D-551.,
Lomponvec: LauphMNP + 03 LCantative 20-55I. (8)
Later, the empirical results also help support the effectiveness of GraphMVP-G and GraphMVP-C, and thus, we can conclude that existing 2D SSL is complementary to GraphMVP.
EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL SETTINGS
Datasets. We treitrain models on the same dataset then fine-tune on the wide range of downstream tasks. We randomly select 50k qualified molecules from GEOM [4] with both 2D and 3D structures
5

---

Published as a conference paper at ICLR 2022
Table 1: Results for molecular property prediction tasks. For each downstream lask, we report the mean (and standard deviation) ROC-AUC of 3 seeds with scaffold splitting. For GraphMVP, we set M 0.15 and
5. The best and second best results are marked bold and bold, respectively.
Pre-training
BBBP
Tox21
ToxCast
Sider
Clin'ox
MUV
HIV
Bace
Avg
65.4(2.4)
74.9(0.8)
61.6(1.2) 58.0(2.4)
58.8(5.5) 71.0(2.5) 75.3(0.5) 72.6(4.9)
67.21
EdgePred
64.5(3.1)
74.5(0.4) 60.8(0.5) 56.7(0.1) 55.8(6.2) 73.3(1.6) (0.8) 64.6(4.7)
65.64
AttrMask
70.2(0.5)
74.2(0.8) 62.5(0.4) 60.4(0.6)
68.6(9.6) 73.9(1.3) 74.3(1.3)
77.2(1.4)
70.16
GPT-GNN
64.5(1.1)
75.3(0.5) 62.2(0.1) 57.5(4.2) $7.8(.11)
76.1(2.) 75.1(0.2) 77.6(0.5)
68.27
InfoGraph
69.2(0.8)
73.0(0.7) 62.0(0.3) 59.2(0.2) 75.1(5.0)
74.0(1.5) 74.5(1.8 73.9(2.5) 70.10
ContextPred
71.2(0.9)
73.3(0.5) 62.8(0.3) 59.3(1.4) 73.7(4.0) 72.5(2.2) 75.8(1.1) 78.6(1.4)
70.89
GraphLoG
67.8(1.7)
73.0(0.3) 62.2(0.4) 57,4(2.3) 62.001.8) 73.1(1.7) 73.4(0.))
78.8(0.7)
68.47
G G-Contextual
70.3(1.6)
75.2(0.3) 62.6(0.3) 58.4(0.6)
59.9(8.2) 72.3(0.) 75.9(0.9) 79.2(0.3)
69.21
G-Motif
66.4(3.4)
73.2(0.8) 62.6(0.5) 60.6(1.1) 77.8(2.0) 73.3(200) 73.8(1.4)
73.4(4.0)
70.14
GraphCI.
67.5(3.3)
75.0(0.3) 62.8(0.2) 60.1(1.3) 78.9(4.2) 77.1(1.0) 75.0(0.4)
68.7(7.8) 70.64
JOAO
66.0(0.6)
74.4(0.7)
62.7(0.6) 60.7(1.0)
66.3(399
77.0(2.2)
76.6(0.5)
72.9(2.0)
69.57
GraphMVP
68.5(0.2)
74.5(0.4) 62.7(0.1) 62.3(1.6) 79.0(2.5)
75.0(1.4
74.8(1.4)
76.((1.1)
71.69
GraphMVP-G 70.8(0.5)
(5.9(0.5) 63.1(0.2) 60.2(1.1) 79.1(2.8)
77.7(0.6)
76.0(0.1)
79.3(1.5)
72.76
GraphMVP-C 72.4(1.6) 74.4(0.2) 63.1(0.4) 63.9(1.2) 77.5(4.2) 75.0(1.0)
77.0(1.2) 81.2((0.9) 73.07
for the pre-training. As clarified in Section 3.1, conformer ensembles can better reflect the molecular property, thus we take C' conformers of each molecule. For downstream tasks, we first stick to the same setting of the main graph SSL work [42, 103, 104], exploring 8 binary molecular property prediction tasks, which are all in the low-data regime. Then explore 6 regression tasks from various low-data domains to be more comprehensive. We deseribe all the datasets in Appendix F.
2D GNN. We follow the research line of SSL on molecule graph [42, 103, 104], using the same Graph Isomorphism Network (GIN) [100] as the backbone model, with the same feature sets.
3D GNN. We choose SchNet [79] for geometrie modeling, since SchNet: (1) is found be strong geometric representation learning method under the fair benchmarking: (2) can be trained more efficiently, comparing to the other recent 3D models. More detailed explanations in Appendix B.2.
4.2 MAIN RESULTS ON MOLECULAR PROPERTY PREDICTION.
We carry out comprehensive comparisons with 10 SSL baselines and random initialization. For pre-training, we apply all SSL methods on the same dataset based on GEOM [4]. For fine-tuning, we follow the same setting [42, 103, 104] with 8 low-data molecular property prediction tasks.
Baselines. Due to the rapid growth of graph SSL [59, 97, 99]. we are only able to benchmark the most well-acknowledged baselines: EdgePred [34], InfoGraph [82], GPT-GNNI43J, AttrMask & Contex(Pred[42], Grap1Lo[[1011, G-[Contextual, [77] GraphCL[104), JOAO[103].
Our method. GraphMVP has two key factors: i) masking ratio (M) and ii) number of conformers for each molecule (C). We set M = 0.15 and C by default, and will explore their effects in the following ablation studies in Section 4.3. For EBM-NCE loss, adopt the empirical distribution for noise distribution. For Equation (8), we pick the empirically optimal generative and contrastive 21) SSL method: that is AttrMask for GraphMVP-G and ContextPred for GraphMVP-C.
The main results on molecular property prediction tasks are listed Table 1. We observe that the performance of GraphMVP is significantly better than the random initialized one, and the average performance outperforms the existing SSL methods by a large margin. In addition, GraphMVP-G and GraphMVP-C consistently improve the performance, supporting the claim: 3D geometry is complementary to the 2D topology. GraphMVP leverages the information between 3D geometry and 2D topology, and 2D SSL. plays the role as regularizer to extract more 2D topological information; they are extracting different perspectives of information and are indeed complementary to each other.
4.3 ABLATION STUDY: THE EFFECT OF MASKING RATIO AND NUMBER OF CONFORMERS
We analyze the effects of masking ratio M and the number of conformers C in GraphMVP. In Table 1, set the M 0.15 since it has been widely used in existing SSL methods [42, 103, 104], and

---

Published as a conference paper at ICLR 2022
Table 2: Ablation of masking ratio M, C' = 5.
Table 3: Ablation of # conformer C, M = 0.15.
M
GraphMVP-G GraphMVP
GruphMVP-C
GruphMVP
GraphMVP-G
GruphMVP-C
71.12
72.15
72.66
71.61
72.80
72.46
71.60
73.08 72.59
72.76
0.15
71.60
72.76
73.08
10
72.20
73.09 73.00
0.30
71.79
72.91
73.17
20
72.39
73.02
is set to 5, which we will explain below. We explore on the range of M E {0, 0.3) and Ce (1,5, 5, 10, 20), and report the average performance. The complete results are in Appendix G.2.
As seen in Table 2, the improvement is more obvious from M = 0 (raw graph) to M = 0.15 than from M = 0.15 to M = 0.3. This can be explained that subgraph masking with larger ratio will make the SSL tasks more challenging, especially comparing to the raw graph (M 0).
Table shows the effect for C. We observe that the performance is generally better when adding more conformers, but will reach a plateau above certain thresholds. This observation matches with previous findings [5]: adding more conformers to augment the representation learning is not as helpful as expected: while we conclude that adding more conformers can be beneficial with little improvement. One possible reason is, when generating the dataset, we are sampling top-C' conformers with highest possibility and lowest energy. In other words, top-5 conformers are sufficient cover the most conformers with equilibrium state (over 80%), and the effect of larger C' is thus modest.
To sum up, adding more conformers might be helpful, but the computation cost can grow linearly with the increase in dataset size. On the other hand, enlarging the masking ratio will not induce extra cost, yet the performance slightly better. Therefore, we would encourage tuning masking ratios prior to trying a larger number of conformers from the perspective of efficiency and effectiveness.
4.4 ABLATION STUDY: THE EFFECT OF OBJECTIVE FUNCTION
In Section 3, we introduce a new contrastive learning objective family called EBM-NCE, and we take either InfoNCE and EBM-NCE as the contrastive SSL. For the generative SSL. task, we propose a novel objective function called variational representation reconstruction (VRR) in Equation (6). As discussed in Section 3.3, stochasticity important for GraphMVP since it can capture the conformer distribution for each 2D molecular graph. To verify this, we add an ablation study on representation reconstruction (RR) by removing stochasticity VRR. Thus, here deploy comprehensive ablation study to explore the effect for each individual objective function (InfoNCE, EBM-NCE, VRR and RR), followed by the pairwise combinations between them.
Table 4: Ablation the objective function.
GraphMVP Loss
Contrastive
Generative
Avg
Random
67.21
InfoNCE only EBM-NCE only VRR only
68.85
70.15 15
69.29
RR only
68.89
InfoNCE + VRR EBM-NCE- VRR InfoNCE + RR
70.67
71.69
70.60
EBM-NCE + RR
70.94
The results in Table 4 give certain constructive insights as follows: (1) Each individual SSL objective function (middle block) can lead to better performance. This strengthens the claim that adding 3D information is helpful for 2D representation learning. (2) According to the combination of those SSL objective functions (bottom block), adding both contrastive and generative SSL consistently improve the performance. This verifies our claim that conducting SSL at both the inter-data and intra-data level is beneficial. (3) We can see VRR is consistently better than RR on all settings, which verifies that stochasticity is an important factor in modeling 3D conformers for molecules.
4.5 BROADER RANGE OF DOWNSTREAM TASKS
The 8 binary downstream tasks discussed so far have been widely applied the graph SSL research line molecules [42, 103, 104], but there Here we test 4 extra regression property prediction tasks and 2 drug-targe affinity tasks.
tasks where the 3D conformers can be helpful.
About the dataset statistics, more detailed information be found in Appendix F, and may as well briefly describe the affinity task here. Drug- target affinity (DTA) is a crucial task [70, 71, 96] in drug discovery, where it models both the molecular drugs and target proteins, with the goal predict
7

---

Published as a conference paper at ICLR 2022
Table 5: Results for four molecular property prediction tasks (regression) and two DTA tasks (regression). We report the mean RMSE of 3 seeds with scaffold splitting for molecular property downstream tasks, and mean MSE for seeds with random splitting on DTA tasks. For GraphMVP we set M = 0.15 and C = 5. The best performance for each task is marked in bold, We omit the std here since they are very small and indistinguishable. For complete results, please check Appendix G.4.
Molecular Property Prediction
TargeTarget Affinity
Pre-training
ESOL
Lipo
Malaria
CEP
Avg
Davis
KIBA
Avg
1.178
0.744
1.127
1.254
1.0756
0.286
0,206
0,2459
AM
1.112
0.730
1.119
1.256
1.0542
0.291
0.203
0.2476
CP
1.196
0.702
1.101
1.243
1.0606
0.279
0.198
0.2382
JOAO
1.120
0.708
1.145
1.293
1.0663
0.281
0.196
0.2387
GraphMVP
1.091
0.718
1.114
1.236
1.0397
0.280
0.178
0.2286
GraphMVP-G
1.064
0.691
1.106
1.228
1.0221
0.274
0.175
0.2248
GraphMVP-C
1.029
0.681
1.097
1.244
1.0128
0.276
0.168
0.2223
their affinity scores. One recent work [66] is modeling the molecular drugs with 2D GNN and target protein (as an amino-acid sequence) with convolution neural network (CNN). We adopt this setting by pre-training the 2D GNN using GraphMVP. As illustrated in Table 5, the consistent performance gain verifies the effectiveness of our proposed GraphMVP.
4.6
CASE STUDY
We investigate how GraphMVP helps when the task objectives are challenging with respect to the 2D topology but straightforward using 3D geometry (as shown in Figure 2). We therefore design two case studies to testify how GraphMVP transfers knowledge from 3D geometry into the 2D representation.
The first case study is 3D Diameter Prediction. For molecules, usually, the longer the 2D diameter is, the larger the 3D diameter (largest atomic pairwise 12 distance). However, this does not always hold, and we interested in using the 2D) graph predict the 3D diameter. The second case study is Long- Long-Range Donor-Acceptor Detection. Molecules possess a special geometric structure called donor-acceptor bond, and we want to use 2D molecular graph to detect this special structure. We validate that GraphMVP consistently brings improvements on these 2 case studies, and provide more detailed discussions and interpretations in Appendix G.6.
Diameter Donor-Acceptor
marked O
420 = 22
diam = 26 diam 10.9Å
dxp = 2.63
Figure 2: We select the molecules whose properties can be easily resolved via 3D but not 2D. The randomly initialised 2D GNN achieves accuracy of 38.9 + 0.8 and 77.9 + 1.1, respectively. The GraphMVP pre-trained ones obtain scores of 42.3 + 1.3 and 81.5 + 0.4. outperforming all the precedents in Section 4.2. We plot cases where random initialization fails but GraphMVP is correct.
THEORETICAL INSIGHTS
In this section, we provide the mathematical insights behind GraphMVP. We will first discuss both contrastive and generative SSL methods (Sections 3.2 and 3.3) are maximizing the mutual information (MI) and then how the 3D geometry, privileged information, can help 2D representation learning.
5.1 MAXIMIZING MUTUAL INFORMATION
Mutual information (MD) measures the non-linear dependence [7] between two random variables: the larger MI, the stronger dependence between the variables. Therefore for GraphMVP, we can interpret it maximizing MI between 2D and 3D views:
obtain more robust 2D/3D representation
8

---

Published as a conference paper at ICLR 2022
by sharing more information with its 3D/2D counterparts. This is also consistent with the sample complexity theory [3, 20, 26] where SSL as functional regularizer can reduce the uncertainty in representation learning. We first derive lower bound for MI (see derivations in Appendix C), and the corresponding objective function LMI is
1 2
T(X;Y) 2 L
[log
log p(rly))
Contrastive Supe-Supe Learning. InfoNCE was initialized proposed to maximize the MI directly [69]. Here in GraphMVP, EBM-NCE estimates the conditional likelihood in Equation (9) using EBM, and solves with NCE [32]. As a result, EBM-NCE can also seen as maximizing MI
between 2D and 3D views. The detailed derivations can be found in Appendix D.2.
Generative Sell-Supervised Learning. One alternative solution is to use a variational lower bound to approximate the conditional log-likelihood terms in Equation (9). Then we can follow the same pipeline in Section 3.3, ending up with the surrogate objective, i.e., VRR in Equation (6).
5.2 3D GROMETRY AS PRIVILEGED INFORMATION
We show the thcoretical insights from privileged information that motivate GraphMVP. We start by considering a supervised learning setting where (u, 4,) is a feature-label pair and u, is the privileged information [88, 89]. The privileged information is defined to be additional information about the input u, 4,) in order to support the prediction. For example, u, could be some CT images of a particular disease, I; could be the label of the disease and ul is the medical report from a doctor. VC theory [87, 88] characterizes the learning speed of an algorithm from the capacity of the algorithm and the amount of training data. Considering a binary classifier f from a function class F with finite VC-dimension VCD(F). With probability 1 - 8, the expected error is upper bounded by
R() < (J) o (VD((F) logo (
(10)
n
where (f) denotes the training error and is the number of training samples. When the training data is separable, then (f) will diminish to zero and 3 is equal to 1. When the training data is non-separable, 8 is 3.2 Therefore, the rate of convergence for the separable case is of order 1/n.
In contrast, the rate for the non-separable case is of order 1/ //. We note that such difference is huge, since the same order of bounds require up to 100 training samples versus 10,000 samples. Privileged information makes the training data separable such that the learning can be more efficient. Connecting the results to GraphMVP, we notice that the 3D geometric information of molecules can be viewed as a form of privileged information, since 3D information can effectively make molecules more separable for some properties [54, 58, 79]. Besides, privileged information is only used in training, and well matches our usage of 3D geometry for pre-training. In fact, using 3D structures as privileged information has been already shown quite useful in protein classification [89], which serves as a strong evidence to justify the effectiveness of 3D information in graph SSL pre- pre-training.
6 CONCLUSION AND FUTURE WORK
In this work, we provide a very general framework, coined GraphMVP. From the domain perspective, GraphMVP (1) is the first to incorporate 3D information for augmenting 2D graph representation
learning and (2) is able take advantages of 3D conformers by considering stochasticity in modeling. From the aspect of technical novelties, GraphMVP brings following insights when introducing 2 SSL tasks: (1) Following Equation (9), GraphMVP proposes EBM-NCE and VRR, where they are modeling the conditional distributions using EBM and variational distribution respectively. (2) EBM-NCE is similar JSE, while we start with a different direction for theoretical intuition, yet EBM opens another promising venue in this area. (3) VRR, as a generative SSL method, is able to alleviate the potential issues in molecule generation [25, 106]. (4) Ultimately, GraphMVP combines both contrastive SSL (InfoNCE EBM-NCE) and generative SSL (VRR) for objective function. Both empirical results (solid performance improvements on 14 downstream datasets) and thcoretical analysis can strongly support the above domain and technical contributions.
We want to emphasize that GraphMVP is model-agnostic and has the potential to be expanded to many other low-data applications. This motivates broad directions for future exploration, including but not limited to: (1) More powerful 2D and 3D molecule representation methods. (2) Different application domain other than small molecules, e.g., large molecules like proteins.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of the empirical results, we include our code base in the supplementary material, which contains: instructions for installing conda virtual environment, data preprocessing scripts, and training scripts. Our code is available on GitHub for reproducibility. Complete derivations of equations and clear explanations are given in Appendices to E.
ACKNOWLEDGEMENT
This project is supported by the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronics Co., Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bing Gift Fund and a NRC Collaborative R&D Project (AI4D-CORE-06) This project was also partially funded by IVADO Fundamental Research Project grant PRF-2019-358313979
REFERENCES
[1] Moayad Alnammi, Shengchao Liu, Spencer S Ericksen, Gene E Ananiev, Andrew F Voter, Song Guo, James Keck, Michael Hoffmann, Scott Wildman, and Anthony Gitter. Evaluating scalable supervised learning for synthesize-on-eemand chemical libraries. 2021. 18
[2] Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. arXiv preprint arXiv:2003.05033, 2020. 23,
[3] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation leaming. arXiv preprint 192.v:190..09229, 2019. 4,
[4] Simon Axelrod and Rafael Gomez-Bombarell Bombarelli. Geom: Fnergy-annotated molecular conformations for property prediction and molecular generation, arXiv preprint arXiv:2006 055531,,,,,,, 2020. 2, 5, 6, 19
[5] Simon Axelrod and Rafael Gomer-Bombarelli. Molecular machine learning with conformer ensembles.
arXiv preprint arXiv:20/2.08495 2020.
[6] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.0091 2019. 2
[7] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. International Conference on Machine Learning, pages 531-540. PMLR, 2018. 2, 8, 26
[8] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.1 /4165, 2020. 17
[9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.n0 09882, 2020. 17
[10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on Machine Learning, pages 1597-1607, 2020. 4, 17
[11] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Inter- pretable representation learning by information maximizing generative adversarial nets. Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 2180-2188, 2016.
[12] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 15750-15758, 2021. 17, 26
[13] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličkovié, Principal neighhour- hood aggregation for graph nets. arXiv preprint arXiv:2004.05718, 2020. 18,
[14] Mindy Davis, Jeremy P Hunt, Sanna Herrgard, Pietro Ciceri, Lisa M Wodicka. Gabriel Pallares, Michael Hocker, Daniel K Treiber, and Patrick P Zarrinkar. Comprehensive analysis of kinase inhibitor selectivity. Nature biotechnology, 29(1 1004611116,, 2011.
2011.
10

---

Published as a conference paper at ICLR 2022
[15] John S Delaney. Esol: estimating aqueous solubility directly from molecular structure. Journal of chemical information and computer sciences, 44(3):1000-1003 2004. 27
[16] Mehmet Demirel, Shengchao Liu, Siddhant Garg, and Yingyu Liang. An analysis attentive walk- aggregating graph neural networks. arXiv preprint arXiv:2110.02667, 2021. 18
[17] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Loe, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805. 2018. 1, 17
[18] Laurent Dinh, Jascha Sohl- biil-Diictttin, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 2016. 4
[19] Yilun Du Shuang Li, Joshua Tenenbaum. and Igor Mordatch. Improved contrastive divergence training energy based models. arXiv preprint arXiv:20/2.013/6 2020. 22, 24
[20] Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre- training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 201 208. JMLR Workshop and Conference Proceedings, 2010.
[21] Yin Fang. Qiang Zhang. Haihong Yang. Xiang Zhuang, Shumin Deng, Wen Zhang. Ming Qin. Zhuo Chen, Xiaohui Fan, and Huajun Chen. Molecular contrastive learning with chemical element knowledge graph. arXiv preprint arXiv:21 2tti::212..0000 00544, 2021. 19
[22] Elizabeth H Finn, Gianluca Pegoraro, Sigal Shachar, and Tom Misteli. Comparative analysis of 2d and 3d distance measurements to study spatial genome organization. Methods, 123:47-55, 2017. 30
[23] Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto-
translation equivariant attention networks. arXiv preprint arXiv:2006. 10503, 2020. 18, 19
[24] Francisco-Javicr Gamo, Laura M Sanz, Jaume Vidal, Cristina De Cozar, Emilio Alvarez, Jose-Luis Lavandera, Dana E Vanderwall, Darren VS Green, Vinod Kumar, Samiul Hasan, et al. Thousands of chemical starting points for antimalarial lead identification. Nature, 465(7296)3005, 2010. 28
[25] Wenhao Gao and Connor W Coley. The synthesizability of molecules proposed by generative models. Journal of chemical information and modeling, 60(12):5714- 2020. 9
[26] Siddhant Garg and Yingyu Liang. Functional regularization for representation learning: A unified thcoretical perspective. arXiv preprint arXiv:2008. 02447, 2020.
[27] Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 40(D1):D1100 D1107, 2012. 27
[28] Kaitlyn M Gayvert, Neel S Madhukar. and Olivier Elemento. A data-driven approach to predicting
successes and failures of clinical trials. Cell chemical biology. 23(10): 2016. 27
[29] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pages 1263-1272.
PMLR, 2017. 1, 18
[30] Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014. 4
[31] Jean-Basticn Grill, Florian Strub, Florent Altché, Corentin Tallec Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 2020. 5, 26
[32] Michael Gutmann and Aapo Hyvärinen. Noisc-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010. 9, 22
[33] Johannes Hachmann, Roberto Olivares-Amaya Sule Atahan Atahan-Evrenk, Carlos Amador-Bedolla, Roel S Sánchez-Carera, Aryeh Gold-Parked Leslie Vogt, Anna M Brockway. and Alán Aspuru- Guzik. The harvard clean energy project: large-scale computational screening and design of organic photovoltaics on the world community grid. The Journal of Physical Chemistry Letters, 217))2241-2251, 2011, 27
11

---

Published as a conference paper at ICLR 2022
[34] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, NeurIPS, 2017. 2, 6,
[35] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive nulti-view representation learning on graphs. In International Conference on Machine Learning, pages 4116 4126. PMLR, 2020. 4
[36] Paul CD Hawkins. Conformation generation: the state of the art. Journal of Chemical Information and Modeling, 57(8): 1747-1766,, 2017. 2, 19
[37] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9729-9738, 2020. 4, 17
[38] Irina Higgins, Loic Matthey, Arka Pal. Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations. 2017. 1, 26
[39] Maya Hirohara, Yutaka Saito, Yuki Koda, Kengo Sato, and Yasubumi Sakakibara. Convolutional neural network based on SMILES representation of compounds for detecting chemical motif. BMC bioinformatics, 19(19)88399) 2018. 19
[40] Devon Hjelm, Alex Fedorov, Samuel Mavoie-Marchildee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.066 166,00660,, 2018.2 24
[41] Qianjiang Hu, Xiao Wang. Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1074-1083, 2021. 24
[42] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovee. Strategies for treitrannng graph neural networks. In International Conference Learning Representa- tions, ICLR, 2020. 1,
17, 18
[43] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre- training of graph neural networks. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD, pages 1857-1867, 2020. 2, 6, 17
[44] Dejun Jiang, Zhenxing Wu, Chang-Yo Yu Hsieh, Guangyong Chen, Ben Liao, Zhe Wang. Chao Shen, Dongsheng Cao, Jian Wu, and Tingjun Hou. Could graph neural networks learn better molecular representation for drug discovery? a comparison study of descriptor-based and graph-based models. Journal of cheminformatics, 13(1):1-24 2021. 18
[45] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židck, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, pages 1 11, 2021. 18, 19
[46] Ramandeep Kaur, Fabio Possanza, Francesca Limosani, Stefan Bauroth, Robertino Zanoni, Timothy Clark, Giorgio Arrigoni, Pictro Tagliatesta, and Dirk M Guldi. Understanding and controlling and long-range eleetron/charge-trame transferrrtrrrrrrrre processes electron donor-acceptor conjugates. Journal of the American Chemical Society, (42(17):898-79) 2020. 30
[47] Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. 1362, 2020. 24
[48] Diederik Kingma and Prafulla Dhariwal. Glow: generative flow with invertible x convolutions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 10236-1000 10245, 2018. 4, 25
[49] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013. 1,4,25 25
[50] Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. The sider database of drugs and side effects. Nucleic acids research, D1075) 2015. 27
[51] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In European conference on computer vision, pages 577-593, 2016. 25
12

---

Published as a conference paper at ICLR 2022
[52] Shengchao Liu, Moayad Alnammi, Spencer S Ericksen, Andrew F Voter, Gene E Ananiev, James Keck, F Michael Hoftmann, Scott Wildman, and Anthony Gitter. Practical model selection for prospective virtual screening. Journal of chemical information and modeling, 59(1):282 293, 2018. 18
[53] Shengchao Liu, Andreea Deac. Zhaocheng Zhu, and Jian Tang. Structured multi-view representations for drug combinations. In NeurlPS 2020 MI. for Molecules Workshop, 2020. 19
[54] Shengchao Liu, Mehmet Furkan Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised representation for graphs, with applications molecules. arXiv preprint
2018. 1, 2,2, 9,
17, 18, 19
[55] Shengchao Liu, Yingyu Liang, and Anthony Gitter. Loss-balanced task weighting to reduce negative transfer in multi-task learning. In Proceedings of the AAAl conference on artificial intelligence, volume 33, pages 9977-9978, 2019. 18
[56] Shengchao Liu, Meng Qu, Zuobai Zhang, Huiyu Cai, and Jian Tang. Structured multi-task learning for molecular property prediction. arXiv preprint arXiv:2203. 04695, 2022. 19
[57] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021. 2, 17.21 21
[58] Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.05013 2021. 2, 9, 17, 18, 19
[59] Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S Yu. Graph self supervveee learning: A survey. arXiv preprint vrXiv:2103.00111 2021. 2, 6. 17, 24
[60] Yu Shen Liu, Qi Li, Guo-Qu Qin Zheng, Karthik Ramani, and William Benjamin. Using diffusion distances for flexible molecular shape comparison. BMC bioinformatics, 11(1):1 15, 2010. 30
[61] lnes Filipa Martins, Ana L Teixeira, Luis Pinheiro, and Andre O Falcao. A bayesian approach to in silico blood-brain barrier penetration modeling. Journal of chemical information and modeling, 52(6): 5266):686-1699 1697, 2012. 27
[62] RVN Melnik, A Uhlherr. Hodgkin, and F De Hoog. Distance geometry algorithums in molecular modelling of polymer and composite systems. Computers Mathematics with Applications, 45(1-3):515-
534, 2003. 30
[63] Jesse G Meyer, Shengchao Liu, Ian J Miller, Joshua Coon, and Anthony Gitter. Learning drug functions from chemical structures with convolutional neural networks and random forests. Journal of chemical information and modeling, 59 (1)):443384444 2019. 18
[64] Andriy Maih and Yee Whye Teh. A fast and simple algorithm for training neural probabilistic language models. arXiv preprini /206. 2012. 23
[65] Gerry Moss. Basic terminology stereochemistry (iupac recommendations 1996). Pure and applied chemistry,
1996. 2
[66] Thin Nguyen, Hang Le, and Svetha Venkatesh. Graphdta: prediction drug-target binding affinity using graph convolutional networks. BioRxiv, page 684662, 2019. 8
[67] Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom. Ole Winther, and Max Welling. Survae flows: Surjections to bridge the gap between vaes and flows. Advances in Neural Information Processing Systems, 33, 2020. 25
[68] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural
Information Processing Systems, pages 271 2016. 4, 24
[69] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv 037448 2018. 4.9. 9. 17,21
[70] Hakime Öztürk, Arzucan Özgür, and Elif Ozkirimli. Deepdta: deep drug target binding affinity prediction. Bioinformatics, 34(17):1821-i829, 2018.1 78
[71] Tapio Pahikkala, Antti Airola, Sami Pietilä, Sushil Shakyawar, Agnieszka Szwajda, Jing Tang. and Tero Aittokallio. Toward more realistic drug-target interaction predictions. Briefings in bioinformatics, 16()):25-337,200 2015.
13

---

Published as a conference paper at ICLR 2022
[72] Lagnajit Pattanaik, Octavian-Eugen Ganea, Iam Coley, Klavs F Jensen, William II Green, and Con- nor W Coley. Message passing networks for molecules with tetrahedral chirality. arXiv preprint arXiv:2012.00094, 2020. 31
[73] Ben Poole. Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds mutual information. In International Conference on Muchine Learning, pages 5171-5180. PMLR. 2019. 24
[74] Zhuoran Qiao, Anders S Christensen, Frederick Manhy, Matthew Welhorn, Anima Anandkumar, and Thomas Miller III. Unite: Unitary n-body tensor equivariant network with applications to quantum chemistry. arXiv preprint (ttp:://05..45555. 2021. 19
[75] Bharath Ramsundar, Steven Kearnes, Patrick Riley, Dale Webster, David Konerding, and Vijay Pande. Massively multitask networks for drug discovery. arXiv preprint arXiv:1502.02000 02072, 2015. 18
[76] Sebastian G Rohrer and Knut Baumann. Maximum unbiased validation (muv) data sets for virtual screening based on pubchem bioactivity data. Journal of chemical information and modeling, 1962):169- 184, 2009. 27
[77] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. In Advances in Neural Information Processing Systems, NeurIPS, 2020. 6, 17,28
[78] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. (n) equivariant graph neural networks. arXiv preprint arXiv:2102.9884,, 2021. 18, 19
[79] Kristof T Schütt, Pieter-Jan Kindermans, Huziel E Sauceda, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-fille convolutional neural network for modeling quantum interactions. arXiv preprint (iXiv:170.008566 2017. 1, 2, 6, 9, 17, 18, 19
[80] Yang Song and Diederik Kingma. How to train your energy-based based models. arXiv preprint arXiv:2101.03288 2021. 22, 23, 24
[81] Yang Song. Jascha Sohk-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020. 22, 24
[82] Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, ICLR, 2020. 1, 2, 4, 6, 17.24
Unsupervised and semi-
[83] Jing Tang, Agnieszka Szwajda, Sushil Shakyawar, Tao Xu, Petteri Hintsanen, Krister Wennerberg, and Tero Aittokallio. Making sense of large-scale kinase inhibitor bioactivity data sets: comparative and integrative analysis. Journal of Chemical Information and Modeling, 54(3):335-74,, 2014. 28
[84] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs, arXiv preprint arXiv:2102.068/ 2021. 26
[85] Tox21 Data Challenge. Tox21 data challenge 2014.
nih. http:/ippdddiii.ooooooo////////
2014. 27
[86] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.16625, 2019. 2
2019.
[87] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013. 9
[88] Vladimir Vapnik, Rauf Izmailov, et al. Learning using privileged information: similarity control and knowledge transfer. J. Mach. Learn. Res., 16(1):2023-2049, 2015. 2,
[89] Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information.
Neural networks, 22(5-6):544-557, 2009. 2, 9
[90] Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint arxiv: 1809, 10341, 2018. 1,2, 17
[91] Pascal Vincent, Hugo Larochelle. Yoshua Bengio, and Pietre-Antoine Manzagol. Extracting and compos- ing robust features with denoising autoencoders. In Proceedings the 25th international conference on Machine learning, pages 1096-1103, 2008
[92] Hanchen Wang, Qi Liu, Xiangyu Yue, Joan I Lasenby, and Mathew J. Kusner. Unsupervised point cloud pre-training via view-point occlusion, completion. In ICCV, 2021. 17
14

---

Published as a conference paper at ICLR 2022
[93] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. Interaatonnal Conference on Machine Learning. ICML, 2020. 1, 17, 21
[94] Yingheng Wang. Yaosen Min, Xin Chen, Wu. Multi-view graph contrastive representation learning for drug-drug interaction prediction. In Proceedings of the Web Conference 202/, pages 2921-2933, 2021. 19
[95] David Weininger. Smiles, a chemical language and information system. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-33) 1988. 19
[96] Ming Wen, Zhimin Zhang, Shaoyu Niu, Haozhi Sha, Ruihan Yang. Yonghuan Yun, and Hongmei Lu. Decp-lcarning based drug target interaction prediction. Journal of proteome research, 16(4):100114099 2017. 7
[97] Lirong Wu, Haitao Lin, Zhangyang Gao, Cheng Tan. Stan Li, al. Self-supervised on graphs: Contrastive, generative, or predictive. arXiv preprint arXiv: 105.07334, 2021. 2, 6, 17, 21, 24
[98] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9():513-5300 2018. 19, 27, 28
199] Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. arXiv preprint 10757, 2021. 6, 17, 21, 24
[100] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv: arXiv:1810.0000 1810.00826, 2018. 6, 18
[101] Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-super graph- representation learning with local and global structure. International Conference on Machine Learning. ICML, 2021. 6, 17, 28
[102] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perezz Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing Iearned molccular representations for property prediction. Journal of chemical information and modeling, 59(8):3370-3388 2019. 18, 19
[103] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning, ICMI,, 2021. 1,2, 4, 6, 7, 17, 18, 28
[104] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Advances in Neural Information Processing Systems, NeurIPS, 2020. 1, 2, 4, 6, 7, 17, 18
[105] Daniel Zaharevitz. Aids antiviral screen data, 2015. 28
[106] Alex Zhavoronkov, Yan A Ivanenkov, Alex Aliper, Mark S Veselov, Vladimir A Aladinskiy, Anastasiya Aladinskaya, Victor A Terentiev, Daniil A Polykovskiy, Maksim D Kuznetsov, Arip Asadulaev, et al. Deep learning enables rapid identification of potent ddrl kinase inhibitors. Nature biotechnology. 373) 2019. 9
[107] Jinhua Zhu, Yingee Xia, Tao Qin, Wengang Zhou, Hougiang Li, and Tie-Yan Yan Liu. Dual-view molecule pre-training, arXiv preprint a0Xiv:2106. 10234, 2021. 19
15

---

Published as a conference paper at ICLR 2022
Appendix
Table of Contents
A Self-Supervieed Learning Molecular Graph
17
A.1
Contrastive graph SSL
A.2 Generative graph SSL
A.3
Predictive graph SSL
B Molecular Graph Representation
18
B.1 2D Molecular Graph Neural Network
B.2 3D Molecular Graph Neural Network
B.3 Summary
19
C Maximize Mutual Information
20
C.I Formulation
20
C.2 A Lower Bound to MI
20
D Contrastive Supf-viee Learning
21
D.1 InfoNCE
21
D.2 EBM-NCE
22
D.3
EBM-NCE v.s. JSE and InfoNCE
24
E
Generative Self-Supervised Learning
25
E.1
Variational Molecule Reconstruction
25
E.2
Variational Representation Reconstruction
25
E.3
Variational Representation Reconstruction and ConCConrrattive SSL
26
F
Datasct Overview
F.1
Pre-Training Dataset Overview
F.2
Downstream Dataset Overview
27
G Experiments Details
28
G.1 Self-supervised Learning Baselines
28
G.2
Ablation Study: The Effect of Masking Ratio and Number of Conformers
29
G.3 Ablation Study: Effect of Each Loss Component
29
G.4 Broader Range of Downstream Tasks: Molecular Property Prediction Prediction
30
G.5 Broader Range of Downstream Tasks:
Affinity Prediction
30
G.6 Case Studies
30
16

---

Published as a conference paper at ICLR 2022
A SELF-SUPERVISED LEARNING ON MOLECULAR GRAPH
Supe-vupe learning (SSL) methods have attracted massive attention recently, trending from vision [9, 10, 12, 37, 92], language [8, 17, 69] to graph [42, 54, 82, 90, 103, 104]. In general, there are two categories of SSL: contrastive and generative, where they differ on the design of the supervised signals. Contrastive SSL realizes the supervised signals at the inter-data level, learning the representation by contrasting with other data points; while generative SSL focuses reconstructing the original data at the intra-data level. Both venues have been widely explored [57, 59, 97, 99].
A.1 CONTRASTIVE GRAPH SSL
Contrastive graph SSL first applies transformations to construct different views for each graph. Each view incorporates different granularities of information, like node-, subgraph-, and graph-level. It then solves two sub-tasks simultaneously: (1) aligning the representations views from the same data; (2) contrasting the representations of views from different data, leading to a uniformly distributed latent space [93]. The key difference among existing methods is thus the design view constructions. InfoGraph [82, 90] contrasted the node (local) and graph (global) views. ContextPred [42] and G-Contextual [77] contrasted between node and context views. GraphCL and IOAO [103, 104) made comprehensive comparisons among four graph-level transformations and further learned select the most effective combinations.
A.2 GENERATIVE GRAPH SSL
Generative graph SSL aims at reconstructing important structures for each graph. By so doing, consequently learns representation capable of encoding key ingredients of the data. EdgePred [34] and AttrMask [42] predicted the adjaceney matrix and masked tokens (nodes and edges) respectively. GPT-GNN [43] reconstructed the whole graph in an auto-regressive approach.
A.3 PREDICTIVE GRAPH SSL
There are certain SSL methods specific to the molecular graph. For example, one central task in drug discovery is to find the important substructure motif in molecules that can activate the target interactions. G-Motif [77] adopts domain knowledge to heuristically extract motifs for each molecule, and the SSL task is make prediction the existence of each motif. Different from contrastive and generative SSL, recent literature [97] takes this as predictive graph SSL, where the supervised signals self- self-generated labels.
SSL for Molecular Graphs. Recall that all previous methods in Table 6 merely focus on the 2D) topology. However, for science-centric tasks such as molecular property prediction, 3D geometry should be incorporated as it provides complementary and comprehensive information [58, 79]. To
Table 6: Comparison between GraphMVP and existing graph SSL methods.
Graph View
SSL Category
SSL Pre-training
2D Topology
3D Geometry
Generative
Contrastive
Predictive
EdgePred [34]
AttrMask [42]
GPT-GNN [43]
InfoGraph [82, 90]
ContexPred [42]
GraphLoG [101]
G G-Contextual [77]
GraphCL [104]
JOAO [103]
G-Motif [77]
GraphMVP (Ours)
17

---

Published as a conference paper at ICLR 2022
mitigate this gap, propose GVaphMVP to leverage the 3D geometry with unsupervised graph pre-training.
B MOLECULAR GRAPH REPRESENTATION
There two main methods for molecular graph representation learning. The first one is the molecular fingerprints. It is a hashed bit vector to describe the molecular graph. There has been re-discoveries fingerprint--based methods [1, 44, 52, 55, 63, 75], while its has main drawback: Random forest and XGBoost are very strong learning models on fingerprints, but they fail to take benefits of the pre-training strategy.
Graph neural network (GNN) has become another mainstream modeling methods for molecular graph representation. Existing methods can be generally split into two venues: 2D GNN and 3D GNN, depending on what levels of information is considered. 2D GNN focuses on the topological structures of the graph, like the adjacency among nodes, while 3D GNN is able model the "energy" of molecules by taking account the spatial positions of atoms.
First, we want to highlight that GraphMVP is model-agnostic, i.e. can be applied to any 2D and 3D GNN representation function, yet the specific 2D and 3D representations are not the main focus of this work. Second, we acknowledge there are a lot of advanced 3D [23, 45, 58, 78] and 2D [13, 16, 29, 54, 100, 102] representation methods. However, considering the graph SSL literature and graph representation liteature (illustrated below), we adopt GIN [100] and SchNet [79] in current GraphMVP.
B. 2D MOLECULAR GRAPH NEURAL NETWORK
The 2D representation is taking each molecule as a 2D graph, with atoms as nodes and bonds as edges, ie., 92D (X, E). X Raxd, is the atom attribute matrix, where is the number of atoms (nodes) and dn is the atom attribute dimension. E E Rmxd. is the bond attribute matrix, where m is the number of bonds (edges) and dm is the bond attribute dimension. Notice that here E also includes the connectivity. Then we will apply transformation function T2D on the topological graph. Given 2D graph g2D. its 2D molecular representation is:
(11) The core operation of 2D GNN is the message passing function [29], which updates the node representation based adjacency information. We have variants depending the design of message and aggregation functions, and we pick GIN [100] in this work.
12n = GNN-2D(Tmn(um)))
GNN-2D(TTnn> (X, E))-
There has been long research line 2D graph representation learning [13, 16, 29, 54, 100, 102). Among these, graph isomorphism network (GIN) model [100] has been widely used as the backbone model in recent graph self superviedd learning work [42, 103, 104]. Thus, as well adopt GIN as the base model for 2D representation.
GIN
Recall each molecule is represented feature matrices for atoms and bonds respectively. Then the message passing function is defined as:
molecular graph, i.e., 92D (X, E), where X and E are
(k+1)
MLP(1+1)
(k+1)
Zi
(k)
+
Σ JEN(()
(z) (A)
+ MLP((++)
hond (E.))))
(12)
the (1 1)-th MLP layers the atom- and bond-level respectively. Repeating this for K times, and we can encode K-hopp neighborhood information for each center atom in the molecular data, and take the last layer for each node/atom representation. The graph-level molecular representation is the mean of the node representation:
where 2g X and MLP((+1) (k+1)
stom and MLP(#+1) bond
2(x)
Σ (K)
(13)
N
B.2 3D MOLECULAR GRAPH NEURAL NETWORK
Recently, the 3D geometric representation learning has brought breakthrough progress in molecule modeling [23, 45, 58, 78, 79]. 3D molecular graph additionally includes spatial locations of the atoms,
18

---

Published as a conference paper at ICLR 2022
which needless to be static since, in real scenarios, atoms are in continual motion potential energy surface [4]. The 3D structures at the local minima on this surface are named molecular conformation conformer. As the molecular properties
function of the conformer ensembles [36], this reveals another limitation of existing mainstream methods: to predict properties from a single 2D or 31) graph cannot account for this fact [4], while proposed method can alleviate this issue certain extent.
For specific 3D molecular graph, additionally includes spatial positions of the atoms. We rep- resent each conformer as gan = (X, R), where R E IRax is the 3D-coordinate matrix, and the corresponding representation is:
hDD GNN- GDN-3D(() GNN-3
(14)
where It is the 3D-coordinate matrix and Tap is the 3D transformation. Note that further information such plane and torsion angles can be solved from the positions.
SchNet
SchNet [79] is composed of the following key steps:
embedding(z.)
2
(++1)
MLP
ES ((a)5-1) Tsy (i)
(15)
h; MLP(2 (K)).
where K is the number of hidden layers, and
f1,,, = Iÿ ex(r: rr)) xj exp(->ill|r; -ryll2 -m13)
(16)
is the continuous-filee convolution layer, enabling the modeling of continuous positions of atoms.
We adopt SchNet for the following reasons. (1) SchNet is very strong geometric representation method after fair benchmarking. (2) SchNet can be trained more efficiently, comparing to the other recent 3D models. To support these two points, we make comparison among the most recent 3D geometric models [23, 58, 78] on QM9 dataset. QM9 [98] is a molecule dataset approximating 12 thermodynamie properties calculated by density functional theory (DFT) algorithm. Notice: UNiTE [74] is the state-oftthe-e art 3D GNN, but it requires a commercial software for feature extraction, thus we exclude for now.
Table 7: Reproduced MAE on QM9, 100k for training, 17,748 for val, 13,083 for test. The last
column is the approximated running time.
alpha
gap
homo
lumo
I11
CV
g298
h298
u298
zpvc
time
SchNet
0.077
50
32
26
0.030
0.032
15
14
0.122
14
14
1.751
3h
3E(3)-Trams [23]
EGNN [78]
0.075
49
29
26
0.030
0.012
11
10
0.076
10
10
1.562
24h
SphereNet [58]
0.054
41
0.028
0.027
0.295
1.401
Table 7 shows that, under fair comparison (w.r.t. data splitting, seed, cuda version, etc), SchNet can reach pretty comparable performance, yet the efficiency of SchNet is much better. Combining these two points, adopt SchNet current version of GraphMVP.
B.3 SUMMARY
To up, in GraphMVP. the most important message want to deliver is how design well- motivated SSL algorithm to extract useful 3D geometry information to augment the 2D representation for downstream fine-tuning. GraphMVP is model-agnostic, and may well leave the more advanced 3D [23, 45, 58, 78] and 2D [13, 54, 102] GNN for future exploration.
In addition. molecular property prediction tasks have rich alternative representation methods, in- cluding SMILES [39, 95], and biological knowledge graph [56, 94]. There have been another SSL research line on them [21, 53, 107], yet they are beyond the scope of discussion in this paper.
19

---

Published as a conference paper at ICLR 2022
C MAXIMIZE MUTUAL INFORMATION
In what follows, we will use and Y to denote the data space for 27 graph and 37 graph respectively. Then the latent representations are denoted as hg and hy-
C.I FORMULATION
The standard formulation for mutual information (MI) is
p( y)
I(X;)) By(,)) log p(z))(())
(17)
Another well-explained MI inspired from wikipedia is given in Figure 3.
H(X, Y)
H(X
I(X; H(YI))
H(X)
H(Y)
Figure 3: Venn diagram of mutual information. Inspired by wikipedia.
Mutual information (MI) between random variables measures the corresponding non-linear depen- dence. As can be seen in the first equation in Equation (17), the larger the divergence between the joint (p(x, y) and the product of the marginals p(z))(()) the stronger the dependence between X and Y.
Thus, following this logic, maximizing MI between 2D and 3D views can force the 3D/2D representa- tion to capture higher-level factors, e.g., the occurrence of important substructure that is semantically vital for downstream tasks. Or equivalently, maximizing MI can decrease the uncertainty in 21) representation given 3D geometric information.
C.2 A LOWER BOUND TO MI
To solve MI, we first extract lower bound:
log p(æ, y)
I(X;Y)= Ep(ar.))
p(x))(y)
p(a, y)
Ep(m..))
log Vr(œ)p(y)
1
log (p)
(18)
2 Ep(a.)))
p(x)p(())
log (ogp(xl))) + E( - logp(v/())
2 p()
[((|X)+ H(XY)).
Thus, we transform the MI maximization problem into minimizing the following objective: +
CMi (
(19)
In the following sections, will describe two self sulf-super learning methods for solving MI. Notice that the methods are very general, and can be applied to various applications. Here we apply it mainly for making 3D geometry useful for 2D representation learning molecules.
20

---

Published as a conference paper at ICLR 2022
D CONTRASTIVE SELF-SUPERVISED LEARNING
The essence of contrastive self- self-supervised learning is to align positive view pairs and contrast negative view pairs, such that the obtained representation space is well distributed [93]. We display the pipeline in Figure 4. Along the research line in graph SSL [57, 59, 97, 99], InfoNCE and EBM-NCE are the two most-widely used, as discussed below.
Align
2D GNN
Contrast
3D GNN
Figure 4: Contrastive SSL in GraphMVP. The black dashed circles represent subgraph masking.
D.1 INFONCE
InfoNCE [69] is first proposed to approximate MI Equation (17):
exp(f= y)) LIMONCE - log exp(fa v)) E, exp(fa(x') y) + log exp(fu x)) + Σ, exp fu x))
exp(fu(y.x)) =))
(20)
where x², y' are randomly sampled 2D and 3D views regarding to the anchored pair (x) y). fa (x. y). (y. x) are scoring functions for the two corresponding views, whose formulation can be quite flexible. Here we use fa, y) (y x) = exp((h=: hy)).
Derivation of InfoNCE
T(X; Y) -log( K)) = Eppea [log p(æ, y)
K p(z))(U)) p(x', ,y))
Σ
log K p(x=)(())) )r(rr)
log (1 + (K - 1) p( (p(x)) plxx" y²)
2 Σ
y
p(w² -v)
(K- 1)
(21)
Σ
log
plac [p(""
p(ai .y")
y
p(a Ip(y")
p(a" .y p( PP
(K
1) # p 1p(y")
p(x .y
log
11
p( v²)
cxp( y²))
log
(x y') + 5a
where we set fa y²) = log
pia" pla >
Notice that in D. we are using data x E X as the anchor points. If we use the y E Y as the anchor points and follow the similar steps, we can obtain
exp(fy (,))))))))
I(X; Y) log(K)
>
Σ
log exp f x²) + E exp(fy x())
(22)
Thus, by add both together, have the objective function as Equation (20).
21

---

Published as a conference paper at ICLR 2022
D.2 EBM-NCE
We here provide an alternative approach to maximizing MI using energy-based model (EBM). To our best knowledge, we the first to give the rigorous proof of using EBM maximize the MI.
D.2.1 ENERGY-BASED MODEL (EBM)
Energy-based model (EBM) is powerful tool for modeling the data distribution. The classic formulation is:
exp(- 1 ((a))
p(x)
(23)
A
where the bottleneck is the intractable partition function = exp(-E((r))da, Recently, there have been quite lot progress along this direction [19, 32, 80, 81]. Noise Contrastive Estimation (NCE) [32] is one of the powerful tools here, as we will introduce later.
D.2.2 EBM FOR MI
Recall that our objective function is Equation (19): LMi = H(X])) the conditional likelihood with energy-based model (EBM). This gives us
+
Then we model
1 CEBM Bp(,,)) w)) log ex,(f
exp(fy(U.xx)
log
(24)
Azly
Ayla
where f=((;; (xx y) (xxly) yy and (y x) = Ely (+) are the negative energy functions, and Axjy and Ayje j are the corresponding partition functions.
Under the EBM framework, if we solve Equation (24) with Noise Contrastive Estimation (NCE) [32]. final EBM-NCE objective
E pun((w) Ep, (mll)) log o1 (ffe( y))) P lo |)>l|
LEBM-NCE
(25)
1 E (x) Ep.. (u a)) flog (1 - (f x m (0m
2
Next we will give the detailed derivations.
D.2.3 DERIVATION CONDITIONAL EBM WITH NCE
WLOG, let's consider the (x)l)) first, and by EBM is follows:
exp(-E(xxl)) ()) po(aly) J exp( Exp(-EE(s E( (
exp(S=(x,g) exp(Sa(x, y))
(26)
Anly
Then solve this using NCE. NCE handles the intractability issue by transforming it binary classification task. We take the partition function Anly as a parameter, and introduce noise distribution Pn- Based on this, we introduce a mixture model:
if the conditional x: y is from Pn (xly), and if æly is from Piata (x (y). So the joint distribution is:
Pa. (x] yy) p(= Pdnaa (x) (y) p(z 0)p, (æly)
The posterior of p(z 0|2, 3) is
p(z = (xly))
1/ Pr. (x
daa (z 0/2r y) = 01p. + p(z Pdaata (2]1g) 2 (P)(())))) + Pdatal (aaua(F)) by)
where v p(zm0 p(z=1)
Similarly, have the joint distribution under EBM framework as:
(a) = p(2 = ()) + p(2 1xpo(x))))
And the corresponding posterior is:
p(z = (0)p((
V Pa. (x) (x)y)
(=, = 00xx v) p(= = (x + p(z (xpo(
V Pm (x) + po
22

---

Published as a conference paper at ICLR 2022
We indirectly match po (y) to Pauaa (aly)) by fitting Pr, y) to Pro (2 y) by minimizing their KL-divergence:
min DKi. PKI.(Pn,dauu (Pn, (2 y) Il ar, y))
(p..d [log Pr, (2)22 y)]
.f Σ Pridana (ly) Po (z|ar, y)
/ { p(= = O)P (x 0) Pn. 0|zr y)
(27)
+ p(z = 1) (z0/2 = 1, 1,)) log pn, = 1 1/,, y) Jole
log (z (z= = 0 0/,,80 3)) + 1 (aln)))
log Pn, = 1/2, y)
= v. pa. (n/l)
1. Pr, (r 3)
(o
= P. Ep. (mlv)
log
+ (1l())))))
log V. Pm y) pe
V. (m y) pa (x y
This optimal distribution an estimation to the actual distribution (or data distribution), i.e. (=)l)) ~ Paly) We can follow the similar steps for Po(y(æ) = Pasa Paau(ylz)). Thus follow- ing Equation (27), the objective function to maximize
(x)l) v. Epunnn (p)())) log V. Pm Pa (mly)) + po (x(ly)
(P log po(xly) (n + po xz y
(28)
The we will adopt three strategies to approximate Equation (28):
Self-normalization. When the EBM is very expressive, i.e., using deep neural network for modeling, we can assume it is able to approximate the normalized density directly [64, 80]. In other words, we can set the partition function = This a self-normalized EBM-NCE, with normalizing constant close to 1, i.., p(x) = exp(- E(x)) exp(f(x)) in Equation (23).
2. Exponential tilting term. Exponential tilting term [2] is another useful trick. It models the distribution as pa (x) ((x) exx())xxp(E where ((x) is the reference distribution. If we use the same reference distribution the noise distribution, the tilted probability is
(x = (x) exp( - ( (x)) in Equation (23).
3. Sampling. For many cases, we only need to sample negative points for each data, i.e.,
V
Following these three disciplines, the objective function optimize pa(x)) (a(x))) becomes
log Pro (x)g))
Do (z y
E pa (m/w)
( log
Pr (2) y) + po po y
Pn (x by) + po (a
= -
log
B Bapa
[og
Po (x (y)
+ po (xly)
1 po po (
(29)
log exp( - fan (z. y))
1 exp(-falae v)) +1
-E -Ep.(ml))
+ E (mlle)
log
exp(- y))
=E -Ep. (xlv)
log (1 - (f (x, y))) + E (plm))
(1)
Thus, the final FBM-NCE contrastive SSL objective
LEBM-NCE =
2 (p))
Ep., (m/a)) log (1 a (f (z, b))) + ( of vi)
(30)
E Em)
(m)
E Ep.(timm log (1 o (f x))) - B log (fg
2
23

---

Published as a conference paper at ICLR 2022
D.3 EBM-NCE V.S. JSE AND INFONCE
We acknowledge that there are many other contrastive objectives [73] that can be used to maximize MI. However, in the research line of graph SSL, summarized in several recent survey papers [59, 97, 99], the two most used ones are InfoNCE and Jensen-Shannon Estimator (JSE) [40, 683.
We conclude that JSE is very similar to EBM-NCE, while the underlying perspectives are totally different, as explained below.
1. Derivation and Intuition. Derivation process and underlying intuition are different. JSE [68] starts from f-divergence, then with variational estimation and Fenchel duality function f. Our proposed EBM-NCE is more straightforward: it models the conditional distribution in the MI lower bound Equation (19) with EBM, and solves it using NCE.
2. Flexibilityy Modeling the conditional distribution with EBM provides a broader family of algorithms. NCE is just one solution to it, and recent progress on score matching [80. 81] and contrastive divergence [19], though no longer contrastive SSI., adds on more promising directions. Thus, EBM can provide a potential unified framework for structuring our understanding of self-supervised learning.
3. Noise distribution. Starting from [40], all the following works on graph SSL [59, 82, 97, 99] have been adopting the empirical distribution for noise distribution. However, this is not the case EBM-NCE. Classic EBM-NCE uses fixed distribution, while more recent work [2] extends with adaptively learnable noise distribution. With this discipline, more advanced sampling strategies (w.r.t. the noise distribution) can be proposed, egg., adversarial negative sampling in [41].
In the above, conclude three key differences between EBM-NCE and JSE, plus the solid and straightforward derivations on EBM-NCE. We believe this can provide a insightful perspective of SSL to the community.
According to the empirical results Section 4.4, we observe that EBM-NCE better than InfoNCE.
This can be explained using the claim from [47], where the main technical contribution is to construct many positives and many negatives per anchor point. The binary cross-entropy in EBM-NCE is able realize this some extent: make all the positive pairs positive and all the negative pairs negative, where the softmax-bast based cross cross-entropy fails to capture this, as in InfoNCE.
To conclude, we are introduce using EBM in modeling MI, which opens many potential venues. As for contrastive SSL, EBM-NCE provides a better perspective than JSE, and is better than InfoNCE on graph-level self-super learning.
24

---

Published as a conference paper at ICLR 2022
E GENERATIVE
LEARNING
Generative SSL is another classic track for unsupervised pre-training [48, 49, 51]. though the main focus is on distribution learning. In GraphMVP, we start with VAE for the following reasons:
1. One of the biggest attributes of our problem is that the mapping between two views stochastic: multiple 3D conformers can correspond to the same 2D topology. Thus, we expect stochastic model [67] like VAE, instead of the deterministic ones.
2. For pre-training and fine-tuning, we need to learn explicit and powerful representation function that can be used for downstream tasks.
3. The decoder for structured data like graph are often complicated, e.g., the auto-regressive generation. This makes them suboptimal.
To cope with these challenges, in GraphMVP, we start with VAE-like generation model, and later propose light-weighted and smarl surrogate loss as objective function. Notice that for notation simplicity, for this section, we use hy and hg to delegate the 2D and 3D GNN respectively.
E.1 VARIATIONAL MOLECULE RECONSTRUCTION
As shown in Equation (19), our main motivation to model the conditional likelihood:
1 B (log (logp(æly) + log p(y(()
LMI
2
By introducing a reparameterized variable 2M Pr + an O where jmm and or are two flexible functions on /t.. e ~ (N, 7) and @ the element-wise production, we have a lower bound on conditional likelihood:
logp(y<x) 2 Ba(ssla)) logp(uizz)) -=((q(=/=))l(((e))
(31)
Similarly, we have
log pepp( 2 Eg(=ylv) log p(x) (x) - (L(g(z,1)))l(zz)))
(32)
where 2y Hy + By 3 e. Here My and By are flexible functions on hy, and N(0, For implementation, we take multi-layer perceptrons (MLPs) for Pæss Myy Tw. Ty.
Both the above objectives are composed of a conditional log-likelihood and KL-divergence. The conditional log-likelihood has also been recognized as the reconstruction term: essentially to reconstruct the 3D conformers (y) from the sampled 2D molecular graph representation (zz).
However, performing the graph reconstruction on the data space not easy: since molecules are discrete, modeling and measuring are not trivial.
E.2 VARIATIONAL REPRESENTATION RECONSTRUCTION
To cope with data reconstruction issue, we propose a novel generative loss termed variation represen- tation reconstruction (VRR). The pipeline is in Figure 5.
Reparameterize
Project
Represent
Figure 5: VRR SSL in GraphMVP. The black dashed circles represent subgraph masking.
25

---

Published as a conference paper at ICLR 2022
Our proposed solution is very straightforward. Recall that MI is invariant to continuous bijective function [7]. So suppose we have a representation function hy satisfying this condition, and this guide surrogate loss by transferring the reconstruction from data space to the continuous representation space:
(ogpl) = -By(,,
(x)(==,_=)]]l])) ( (gn((z)) - by (v)l +
where 9N the decoder and C' is a constant, and this introduces using the mean meean-suureed error (MSE) for reconstruction on the representation space.
Then for the reconstruction, current formula has two steps: the latent code 2 is first mapped to molecule space, and ii) it is mapped to the representation space. We can approximate these two mappings with one projection step, by directly projecting the latent code 2Ø to the 3D representation space, i.e., 9u (Zx) = hy (2x)). This gives us a variation representation reconstruction (VRR) SSL objective as below:
(logg
[l) (zx)) hy (v) +
3-VAE We consider introducing 8 variable [38] to control the disentanglement of the latent representation. To be more specific, we would have
log p(y x) By(zal=) (oyp(yy) zx)] 3 LL(g(za/<)]ll(z
(33)
Stop-gradient For the optimization variational representation reconstruction, related work have found that adding the stop- vop-gradient operator (SG) as a regularizer can make the training more stable without collapse both empirically [12, 31] and theoretically [84]. Here, may as well utilize this SG operation the objective function:
(x (z) p(y| = -= [11x===())) (Ghh, +
(34)
Objective function for VRR objective function for VRR is:
Thus, combining both two regularizers mentioned above, the final
1
LVRR
q *=
(x))==)) [1la( SG(h,))) |1 1 [llg, [|n) SG(h ||3
2
(35)
+ 3 [[[g(Zz|<))lp(> + KL(g y) ,)(p((z)))
Note that MI is invariant to continuous bijective function [7], thus this surrogate loss would be exact the encoding function hy and ih satisfy this condition. However, we find GNN (both GIN and SchNet) can, though do not meet the condition, provide quite robust performance empirically, which justify the effectiveness of VRR.
E.3 VARIATIONAL REPRESENTATION RECONSTRUCTION AND NON-CONTRASTIVE SSL
By introducing VRR, we provide another perspective understand the generative SSL, including the non-contrastive SSL [12, 31].
provide unified structure on the intra-data generative SSL:
Reconstruction the data space, like Equations (5), (31) and (32).
Reconstruction the representation space, i.e., VRR in Equation (35).
If we remove the stochasticity, then it is simply the representation reconstruction (RR), as we tested in the ablation study Section 4.4.
we remove the stochasticity and assume two views are sharing the same represen- tation function, like CNN for multi-view learning images, then it is reduced the BYOL [31] and SimSiam [12]. In other words, these pecently-proposed non-contrastive SSL methods indeed special cases of VRR.
26

---

Published as a conference paper at ICLR 2022
F DATASET OVERVIEW
F.1 PRE- TRAINING DATASET OVERVIEW
In this section, we provide the basic statisties of the pre-training dataset (GEOM).
In Figure 6, we plot the histogram (logarithm scale on the y-axis) and cumulative distribution on the number of conformers of each molecule. As shown by the histogram and curves, there certain number of molecules having over 1000 possible 3d conformer structures, while over 80% of molecules have less than 100 conformers.
cumulstive percentage
10a
101
0.8
104
0.8
1000
2000
3000
4000
7000
2000
3000
40.00
5000
6000
number conformers per mel
number conformers per mol
Figure 6: Statistics on the conformers of each molecule
In Figure 6, plot the histogram of the summation of top (descending sorted by weights) [0,5, conformer weights. The physical meaning of the weight is the portion of each conformer occurred in nature. We observe that the top 5 10 conformers all the natural observations. Such long-tailed distribution is also alignment with our findings in the ablation studies. We find that utilizing top five conformers in the GraphMVP has reached an idealised spot between effectiveness and efficiency.
sufficient they have dominated nearly
7500
60000
5000
20000
sum weights conformers
sum weights conformers
150000
counte
100000
50000
0.6 sum of weights for the first 10 conformers
0.2
0.8
1.0
0.2
0.6
0.8
sum of weights for the first conformers
Figure 7: Sum of occurrence weights for the top major conformers
F.2 DOWNSTREAM DATASET OVERVIEW
In this section, we review the four main categories datasets used for downstream tasks.
Molecular Property: Pharmacology The Blood-Brain Barrier Penetration (BBBP) [61] dataset measures whether molecule will penetrate the central nervous system. All three datasets, Tox21 [85], ToxCast [98], and ClinTox [28] related to the toxicity of molecular compounds. The Side Effect Resource (SIDER) [50] dataset stores the adverse drug reactions on a marketed drug database.
Physical Chemistry Dataset proposed in [15] measures aqueous solubility of the molecular compounds. Lipophilicity (Lipo) dataset is subset of ChEMBL. [27] measuring the molecule octanol/water distribution coefficient. CEP dataset a subset of the Havard Clean Energy Project (CEP) [33], which estimates the organic photovoltaic efficiency.
Molecular Property:
Molecular Property: Biophysics Maximum Unbiased Validation (MUV) [76] is another sub- database from PCBA, and is obtained by applying refined nearest neighbor analysis. HIV is from
27

---

Published as a conference paper at ICLR 2022
the Drug Therapeutics Program (DTP) AIDS Antiviral Screen [105], and it aims predicting inhibit HIV replication. BACE measures the binding results for set inhibitors of f-secretase (BACE-) 1), and is gathered in MoleculeNet [98]. Malaria [24] measures the drug efficacy against the parasite that causes malaria.
TrrgeTarget Affinity Davis [14] measures the binding affinities between kinase inhibitors and kinases, scored by the K2 value (kinase dissociation constant). KIBA [83] contains binding affinities for kinase inhibitors from different sources, including Kü Ka and ICso. KIBA scores [70] are constructured to optimize the consistency among these values.
Table Summary for the molecule chemical datasets.
Dataset
Task
# Tasks
Molecules
Proteins
# Molecule-Pronee pairs
BBBP
Classification
1 2,039
Tox21
Classification
7,831
ToxCast
Classification
8,576
Sider
Classification
27
1,427
CoinTox
Classification
2
1,478
MUV
Classification
17
93,087
HIV
Classification
41,127
Bace
Classification
1,513 1
Delaney
Regression
1,128
Lipo
Regression
1
4,200
Malaria
Regression
1
9,999
CEP
Regression
29,978 1
Davis KIBA
Regression
68
379
30,056 118,254
Regression
2,068
229
G EXPERIMENTS DETAILS
G.1 SELF-SUPERVISED LEARNING BASELINES
For the SSL baselines in main results (Table 1), generally we can match with the original paper, even though most of them using larger pre- pre-training datasets, like ZINC-2m. Yet, would like to add some specifications.
G- G- (Cntextual, Motif [77] proposes new GNN model for backbone model, and does pre-training larger dataset. Both settings are different from us.
JOAO [103] has two versions in the original paper. In this paper, we run both versions and
report the optimal one.
Almost all the graph SSL basclines reporting the test performance with optimal validation error, while GraphloGG [101] reports 73.2 in the paper with the last-epoch performance. This can be ovtr-optdd in terms of overfitting, and here rerun with the same downstream evaluation strategy as a fair comparison.
28

---

Published as a conference paper at ICLR 2022
G.2
ABLATION STUDY: THE EFFECT MASKING RATIO AND NUMBER OF CONFORMERS
Table 9: Full results for ablation of masking ratio M (C = 0.15), MVP is short for GraphMVP.
M
BBBP
Tox21
ToxCast
Sider
ClinTox
MUV
HIV
Bace
Avg
65.4(244
74.9(0.) 61.6(1.2) 58.0(2.4) 58.8(5.5) 71.0(2.5 75.3(0.5) 72.6(4.9)
67.21
0
(1.0)
75.3 (0.5) 62.8 (0.2) 61.9
74,6 (1.4) (1.3)
(1.0)
(200
71.12
MVP
0.15
68.5 (0.2) (4.5
62.7 (0.1)
(1.6
(25)
74,8 (1.4)
(1.1
71.69
0.3
68.6 (0.3) 74.9
62.8 (0.4) 60.0
74.8 (7.8)
74.7 (0.8)
75.5
(1.7)
71.79
0
72.4 (1.3) 74.7 (0.6)
62.4 (02) 60.3 (0.7) (6.7
76.4 (1.7
(1.0
72.15
MVP-G
0.15
70.8 (0.5)
63.1 (0.1 (0.9
79.1
(2.1
(0.)
(6.0
79.3
72.76
0.3
69.5 (0.5) 74.6 (0.6)
62.7 (0.) 60.8 (1.2) 80.7
77.8 (25)
76.2 (0.5)
(1.0
72.91
0
(0.9)
75.4 (0.)
63.6 (0.)) 61.8
75.8 (0.6)
76.1 (0.9)
(.4)
72.66
MVP-C
0.15
72.4.1..)
74.4 (0.2)
63.1 (0.4 (1.2
77.5
75.0
77.0 (1.2) (09
73.07
0.3
70.7 (0.8) 74.6 (0.3) 63.8 (0.7) 60.4 (0.6) 83.5
74.2 (1.6) 76.0 (L.) 82.2 (2.2) 73.17
Table 10: Full results for ablation of # conformers C (M 0.5), MVP is short for GraphMVP.
C
BBBP
Tox21
ToxCast
Sider
ClinTox
MUV
HIV
Bace
Avg
65.4(2.4)
74.9(0.8)
61.6(1.2) 58.0(2.4) 58.8(5.5)
71.0(2.5) 75.3(0.5)
72.6(4.9)
67.21
69.2 (1.0) 74.7 (0.4) 62.5 (0.2) 63.0 (0.4) 73.9
76.2 (0.4)
75.3
(0.5)
71.61
MVP
5 68.5 (0.2) 74.5
(0.4
62.7
(0.1
(2.3
79.0 (2.5
75.0 (1.)
74.8
(1.4)
(1.1)
71.69
10 68.3 (0.5) 74.2
632 (0.5
80.6 (1.0
75.4 (2.4)
75.5
(2.3) 72.20
20 68.7 (0.5) 74.9
(0.)
(0.)
75.8
(1.5)
77.4
82.3
72.39
70.9 (0.4) (.7) (05) 61.2 (0.6) 814 (2.1)
(6.4
8072 72.80
MVP-G
70.8 (0.5) (..) 63.
60.2
79.1
(0.6)
(6.0
(1.3 72.76
10 70.2 (0.9) 74.9 (.4) 63.4
(0.)
(1.0
80.6 (04)
(2.0)
77.0
(03)
(1.4 72.59
20 69.5 (0.4) (.4)
63.3
60.8 (03)
81.2 (0.5)
77.3 (2.7)
76.9
(0.3)
80.1 (0.5)
73.00
69.7 (0.9) (4.
64.1
61.0
78.3 (2.7) (1.5)
74.7 (0.)
81.3 (0.7) 72.46
MVP-C
72.4 (1.6) 74.4
(02)
63
(04
63.9
77.5
(4.2
75.0
(1.0
812 (0.9) 73.07
10 69.5 (1.5) 74.5 (05) 6.9
60.9
81
(1.)
76.8 (1.5)
76.0 (0.8)
(1.0 73.09
20 72.1 (0.4) 73.4 (0.7) 63.9 63.0
78.8 (2.4) 74.1
(1.0) 74.8
(0.9)
84.1 (0.6) 73.02
G.3 ABLATION STUDY: EFFECT OF EACH Loss COMPONENT
Table 11: Molecular graph property prediction, we set C=5 and 1= for GraphMPP methods.
BBBP
Tox21
ToxCast
Sider
ClinTox
MUV
HIV
Bace
Avg
# Molecules
2,039
7,831
8,575
1,427
1,478
93,087
41,127
1,513
# Tasks
617
27
2
65.4(2.4) 74.9(0.8) 61.6(1.2)
58.0(2.4)
58.8(5.5)
71.0(255 75.3(0.5) 72.6(4.9) 67.21
InfoNCE only
68.9(1.2) 74.2(0.3) 62.8(0.2) 59.7(0.7) 57.8(11.5)
73.6(1.8) 76. (0.6) 77.6(0.3) 68.85
EBM-NCE only
68.0(0.3) 74.3(0.4) 62.6(0.3) 61.3(0.4) 66.0(6.0)
73.1(1.6) 76.4(1.) 79.6(1.7) 70.15
VAE only
67.6(1.8) 73.2(0.5) 61.9(0.4) 60.5(0.2) 59.7(1.6) 78.6(0.7) 77.40.6) 75.4(2.) 69.29
AE only
70.5(0.4) 75.0(0.4) 62.4(004) 61.0(1.) 53.8(1.0)
74.1(2.9) 76.3(0.5) 77.9(0.)) 68.89
InfoNCE VAE
69.6(1.1) 75.4(0.6) 63.2(0.3) 59.9(0.4) 69.3(14.)) 76.5(1.3) 76.3(.22 75.2(2.7) 70.67
EBM-NCE + VAE 68.5(0.2) 74.5(0.4) 62.7(0.1) 62.3(1.6) 79.0(..5) 75.0(1.4) 74.8(1.4) 76.8(1.1) 71.69
InfoNCE AE 63.1(3.1) 75.4(0.7) 62.5(0.5) 59.2(0.6) 77.2(18) 72.4(1.) 75.8(0.6) 77.1(0.) 70.60
EBM-NCE + AE
69.4(1.0) 75.2(0.)) 62.4(0.4) 61.5(099) 71.1(6.0) 73.3(.33 75.2(0.6) 79.3(1.1) 70.94
29

---

Published as a conference paper at ICLR 2022
G.4 BROADER RANGE OF DOWNSTREAM TASKS: MOLECULAR PROPERTY PREDICTION PREDICTION
Table 12: Results for four molecular property prediction tasks (regression). For each downstream task, we report the mean (and standard variance) RMSE of 3 seeds with scaffold splitting. For GraphMVP, we set M = 0.15 and 5. The best performance for each task marked in bold.
ESOL
Lipo
Malaria
CEP
Avg
1.178 (0.044)
0.744 (0.007)
1.127 (0.003)
1.254 (0.030)
1.07559
AM
1.112 (0.048)
0.730 (0.004)
1.119 (0.014)
1,256 (0.000)
L05419
CP
1.196 (0.037)
0.702 (0.020)
1,101 (0.015)
1,243 (0.025)
1.00599
JOAO
1.120 (0.019)
0.708 (0.007)
1.145 (0.010)
1.293 (0.003)
1.06631
GraphMVP
1.091 (0.021)
0.718 (0.016)
1.114 (0.013)
1,236 (0.023)
1.03968
GraphMVP-G
1.064 (0.045)
0.691 (0.013)
1.106 (0.013)
1.228 (0.001)
1.02214
GraphMVP-C
1.029 (0.033)
0.681 (0.010)
1.097 (0.017)
1.244 (0.009)
1.01283
G.5 BROADER RANGE OF DOWNSTREAM TASKS: DRUG-TARGER AFFINITY PREDICTION
Table 13: Results for two drug-target affinity prediction tasks (regression). For each downstream task, we report the mean (and standard variance) MSE of seeds with random splitting. For GraphMVP, we set M = 0.15 and C' = 5. The best performance for each task is marked in bold.
Davis
KIBA
Avg
0.286 (0.006)
0,206 (0.004)
0.24585
0,291 (0.007)
0,203 (0.003)
0,24730
CP
0.279 (0.002)
0.198 (0.004)
0.23823
JOAO
0.281 (0.004)
0.196 (0.005)
0.23871
GraphMVP
0,280 (0.005)
0,178 (0.005)
0.22860
GraphMVP-C
0.274 (0.002)
0.175 (0.001)
0.22476
GraphMVP-C
0.276 (0.004)
0.168 (0.001)
0.22231
G.6 CASE STUDIES
Shape Analysis (3D Diameter Prediction). Diameter is an important measure in molecule [60, 62], and genome [22] modelling. Usually, the longer the 2D diameter (longest adjacency path) is, the larger the 3D diameter (largest atomic pairwise 12 distance). However, this is not always true. Therefore, we are particularly interested in using the 2D graph to predict the 3D diameter when the 2D and 3D molecular landscapes are with large differences (as in Figure 2 and Figure 8). We formulate it as n-cllass recognition problem, where n is the number class after removing the consecutive intervals. We provide numerical results in Table 14 and more visualisation examples in Figure 9.
dzp dap
d2p
dxp
10.9
Figure 8: Molecules selection, we select the molecules that lies in the black dash box.
Long-Range Donor-Acceptor Detection. Donor-Acceptor structures such as hydrogen bonds have key impacts on the molecular geometrical structures (collinear and coplanarity), and physical proper- ties (melting point, water affinity, viscosity etc.). Usually, atom pairs such as "O...." that closed donor-acceptor
in the Euclidean space are considered as the donor acceptor structures [46]. On this basis, we are particularly interested in using the 2D graph recognize (i.e., binary classification) donor-acceptor
30

---

Published as a conference paper at ICLR 2022
Table 14: Accuracy on Recognizing Molecular Spatial Diameters
Random
AttrMask
ContextPred
GPT-GNN GraphCL
JOAOv2
MVP
MVP-G MVP-C
38.9 (0.8) 37.6 (0.6)
41.2 (0.7)
39.2 (1.1) 38.7 (2.0) 41.3 (1.2) 42.3 (1.9) 41.9 (0.7) 42.3 (1.3)
structures which have larger ranges in the 2D adjacency (as shown Figure 2). Similarly, we select the molecules whose donor-acceptor are close in 3D Euclidean distance but far in the 2D adjacency. We provide numerical results in Table 15. Both tables show that MVP the MVP :)
Table 15: Accuracy on Recognizing Long-Range Donor- Aeco--occepoor Structures
Random
AttrMask
ContextPred
GPT-GNN GraphCL JOAOv2
MVP
MVP-G
MVP-C
77.9 (1.1) 78.6 (0.3) 80.0 (0.5) 77.5 (0.9) 79.9 (0.7) 79.2 (1.0) 80.0 (0.4) 81.5 (0.4) 80.7 (0.2)
Chirality. We have also explored other tasks such as predicting the molecular chirality, it is challenging setting only 2D molecular graphs are provided [72]. We found that GraphMVPbring negligible improvements due to the model capacity of SchNet. We save this in the ongoing work.
31

---

Published as a conference paper at ICLR 2022
clatto
196-14-1558
581-8-800
579-8-884
558-11-996
533-16-1472
516-9-223
go
80243-13.
479-10-551
47716-1653
471-9-941
470-12-1069
454-14-1377
377-12-1302
374-10-367
350-14-1056
349 349-14-1384
342-13-1041
324-12-1388
309-13-1342
300-13-1030
205-9-8.48
204-19-1453
286 286-10-958 95-954
279-10-1341
249-13-1361
Eueros
144 144-1-11557
143-10-328
139-10 1016
137 19-148
130-17-144 14-1433
114 17 17.00000
107-22-1338
106-14-1057
64-13-1015
63-14-10222
56-15-1390
54-12-1301
Cr c0
Figure 9: Molecule examples where GraphMVP successfully recognizes the 3D diameters while random initialisation fails, legends are format of "molecule -d" diameter". "3d diameter".
32


Reviews:
[{"review_id": "uH-f01ZOQf2", "reviewer": "Reviewer_K6sw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper proposed GraphMVP to pretrain molecular representations by using 3D information with SSL tasks. Overall, the motivation is strong, and the method is technically sound. My remaining concern is that the experiments could be more convincing by using large datasets and considering tasks that are more dependent on 3D information.\n\n#####Pros#####\n\n1. The intuition of using expensive 3D geometry information to pretrain molecular representation is convincing since (1) 3D geometry is essentially vital for molecular property and (2) it is usually unavailable for downstream tasks. Therefore, the motivation of this work is clear and strong.\n\n\n2. To achieve the pretraining with considering 3D information, this paper develops one generative and one contrastive SSL task. Although generative and contrastive SSL are widely studied in graph representation learning, extending them into considering both 2D and 3D inputs is non-trivial and well-tackled by this work. One example is that for the generative SSL task this work proposes to do the reconstruction in the latent space instead of data space, where considering reconstruction is much more complicated. This insight is interesting and novel.\n\n\n3. The ablation studies are extensive and well-presented. The writing and organization of this paper are great.\n\n#####Cons#####\n\n1. The main remaining concern for this paper is that the experiments could be more convincing. The current results are mostly on small datasets (See Table 8 in supplementary) and it is not clear that if the included tasks are highly related to 3D information. I suggest the authors to consider larger datasets and more convincing tasks. The recent OGB KDD Cup task (https://ogb.stanford.edu/kddcup2021/) could be a good candidate. The task is to predict the quantum chemical property, which is highly dependent on the 3D geometry.\n\n\n#####Questions#####\n\n1. In Section 3.1, it is mentioned that the contrastive SSL learns the distribution locally while the generative SSL learns the global distribution. This description is confusing to me. More explanations of this point should be considered.\n", "summary_of_the_paper": "This paper proposes to leverage 3D information to pretrain graph neural networks for learning molecular representations. To achieve this, contrastive and generative SSL strategies are developed accordingly. The empirical results on several datasets show that the improvements of this method over previous 2D graph pretraining methods are consistent and obvious.", "summary_of_the_review": "The current version of this paper is good and above the acceptance threshold. More convincing experiments could improve the quality a lot. Hence, I recommend a weak accept to the current version and will consider improving the score if the experiments could be improved significantly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "na9LxWDoEEZ", "reviewer": "Reviewer_zY9H", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "For molecular structure, it is natural and necessary to consider 3D geometry information together with graph structure. This would be a great direction to explore. The proposed method makes sense.\n\n My concern is the experimental part where only molecular property prediction has been discussed. It would be necessary to illustrate the proposed graph and 3D geometry method can indeed improve reconstruction and generation for molecular data by comparing with other state-of-the-art graph-based methods. ", "summary_of_the_paper": "The manuscript proposes a method for molecular graph representation learning by combing graph topological structure with 3D geometric information. ", "summary_of_the_review": "I like the proposed idea. However, some experiments on reconstitution and generation would better illustrative the improvements using the proposed method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hkI-_pdUUuF", "reviewer": "Reviewer_A6B1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths are :\n1. The idea of using 3D information to augment the 2D graph representation learning is interesting and novel. Indeed, 3D geometric information is useful and provides a complementary view of the data, as evidenced in existing works such as protein classification. \n2. Fusing contrastive and generative losses can empirically further improve the performances by providing both inter-data (local) and intra-data (global) viewpoints. \n3. Extensive experiments were conducted (e.g., two tasks, a number of recent baselines, and 14 datasets are involved). Via the experiments, we can see that the proposed method shows promising performances across various datasets. Some useful insights are given in the ablation study.\n4. The paper is well-written and easy to follow.\n\nYet, there are also some weaknesses:\n1. My major concern is the fairness of the comparison. I am afraid that the major improvement is brought by simply incorporating 3D information as additional features, which raises a question:  If other baseline methods have 3D information as additional features, will they perform better (or even better than GraphMVP)? If so, the contribution might be weakened.\n2. It seems that x and y notations are flipped. In preliminary sections, x denotes 2D and y denotes 3D; In the method section, x denotes 3D and y denotes 2D. This is a little bit confusing to readers.\n3. The improvements on some datasets seem to be marginal. More results analyses are required.\n\nMinor comments:\n1. Typo: and are different (page 4).\n2. The appendix can be condensed.\n\n", "summary_of_the_paper": "This work aims to leverage additional 3D geometric information for molecular graph representation learning and proposes a multi-view pre-training framework, GraphMVP. Specifically, in GraphMVP, both 2D and 3D information are used, and a combined loss function (i.e., the combination of contrastive self-supervised learning loss and generative self-supervised learning loss ) is adopted to enhance the quality of representations. To verify the model's effectiveness, they conducted experiments and comparisons on a number of datasets. Ablation studies are also carried out to inspect the inner workings of GraphMVP.", "summary_of_the_review": "This paper proposes integrating 3D geometric information to improve molecule graph representation learning. The idea is interesting and the experiments are comprehensive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FxNrsEful4p", "reviewer": "Reviewer_1JZt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow. The focused problem of taking into account 3D information in GNN pretraining is also very interesting and important. The idea sounds interesting, but at the same time, a big question on how to handle \"conformer ensemble\" seems not well addressed. 3D information is not unique for a single molecule, and this would be the main difficulty why past work didn't take into account it well. For now, it would be a bit unconvincing that this method worked as intended.\n\nSo a big natural question is how this method could handle multiple 3D poses of a molecule. Just all of the possible 3D conformers fed into learning by data augmentation? As the paper also mentioned, \"the molecular properties are a function of the conformer ensembles\", and 3D information is not unique for a given molecule. But it is unclear how to address this ensemble problem. In theory, a molecule has infinitely many 3D shapes in most cases, because any single carbon-carbon bond can be rotated to any degree. We can continuously rotate every C-C bond to produce massive sets of 3D geometries? How to handle them? Or pick up locally energy-stable ones in some ways...? This point seems to remain unclear. Also, the datasets in experiments mainly come from tasks predicting the bioactivities of molecules, and molecules would have a huge degree of such degrees, as well as actual 3D shape in action, might be determined by other interacting molecules (such as target proteins) and also statistical-mechanically fluctuated continuously, and so it would be unconvincing that learning only a few of stable 3D conformations is sufficient.\n\nFor your interest: Bio QSAR tasks would require some handling of the combinatorial multiplicity of 3D conformations of a molecule when we try to include 3D information (typically by docking simulations or something?). I guess that's why (2D) GNNs are sufficient usually. We can ignore complicated problems by continuous/statistical-mechannical 3D confomations and their dynamics and abstract a molecule as a graph (dots and lines) to see statistical trends. This method might be good for a quantum chemical task like QM9 where we are trying to predict the energy (or properties) of a molecule with a specified 3D conformation (an energetically stable structure).\n\n", "summary_of_the_paper": "This paper presents a pre-training method of GNNs for molecular graphs by self-supervised learning (SSL) on not only 2D topologies of graphs but also 3D geometries of molecules.  In particular, the paper develops two SSL pretext tasks learning inter-molecule and intra-molecule associations in 2D and 3D. The \"inter-molecule\" task is a graph classification on whether a 2D, 3D graph pairs come from the same molecule or not, while the \"intra-molecule\" task is a generative task to generate 2D from 3D as well as 3D from 2D information. Empirical evaluations over several datasets demonstrate that jointly learning these two SSL tasks brings nice improvement in prediction accuracies.", "summary_of_the_review": "The paper is well written and easy to follow. It provides interesting and effective pretraining methods for GNNs. However, it is still not clear how to handle the combinatorial multiplicity of 3D molecular information in a principled way even though we can observe empirical improvement in prediction accuracies by this pretraining in a limited set of datasets.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of how to use 3D molecular geometry information during training to improve performance during prediction time when 3D information is not available. This is a highly interesting problem as obtaining 3D molecular geometry information requires expensive calculations and such information is usually not available in practice during prediction, while there are some training data with both 2D and 3D information. The work proposes to use self-supervised, predictive and generative approaches to make use of such information. The reviewers overall expressed mixed recommendations. One of the reviewers who scored 5 did not provide further feedback after author response even being prompted multiple times. The other reviewer who scored 5 actively participated in discussion and it seems most of the concerns have been addressed. Given the importance of this problem, and this work seems to be among the first to address this problem, I lean toward accept.
Decision: Accept (Poster)