PRE-TRAINING MOLECULAR GRAPH REPRESENTATION WITH 3D GEOMETRY
Year: 2022

Abstract:
Molecular graph representation learning is a fundamental problem in modern drug and material discovery. Molecular graphs are typically modeled by their 2D topological structures, but it has been recently discovered that 3D geometric information plays a more vital role in predicting molecular functionalities. However, the lack of 3D information in real-world scenarios has significantly impeded the learning of geometric graph representation. To cope with this challenge, we propose the Graph Multi-View Pre-training (GraphMVP) framework where self-supervised learning (SSL) is performed by leveraging the correspondence and consistency between 2D topological structures and 3D geometric views. GraphMVP effectively learns a 2D molecular graph encoder that is enhanced by richer and more discriminative 3D geometry. We further provide theoretical insights to justify the effectiveness of GraphMVP. Finally, comprehensive experiments show that GraphMVP can consistently outperform existing graph SSL methods. Code is available on GitHub.

heading: 1 INTRODUCTION
text: In recent years, drug discovery has drawn increasing interest in the machine learning community. Among many challenges therein, how to discriminatively represent a molecule with a vectorized embedding remains a fundamental yet open challenge. The underlying problem can be decomposed into two components: how to design a common latent space for molecule graphs (i.e., designing a suitable encoder) and how to construct an objective function to supervise the training (i.e., defining a learning target). Falling broadly into the second category, our paper studies self-supervised molecular representation learning by leveraging the consistency between 3D geometry and 2D topology.
Motivated by the prominent success of the pretraining-finetuning pipeline [16], unsupervisedly pretrained graph neural networks for molecules yields promising performance on downstream tasks and becomes increasingly popular [39, 48, 72, 80, 91, 92]. The key to pre-training lies in finding an effective proxy task (i.e., training objective) to leverage the power of large unlabeled datasets. Inspired by [50, 69] that molecular properties [27, 48] can be better predicted by 3D geometry due to its encoded energy knowledge, we aim to make use of the 3D geometry of molecules in pre-training. However, the stereochemical structures are often very expensive to obtain, making such 3D geometric information scarce in downstream tasks. To address this problem, we propose the GraphMulti-View Pre-training (GraphMVP) framework, where a 2D molecule encoder is pre-trained with the knowledge of 3D geometry and then fine-tuned on downstream tasks without 3D information. Our learning paradigm, during pre-training, injects the knowledge of 3D molecular geometry to a 2D molecular graph encoder such that the downstream tasks can benefit from the implicit 3D geometry even if there is no 3D information available.
We attain the aforementioned goal by leveraging two pretext tasks on the 3D and 2D molecular graphs: one contrastive and one generative SSL. Contrastive SSL creates the supervised signal at an inter-molecule level: the 3D and 2D graph pairs are positive if they are from the same molecule, and negative otherwise; Then contrastive SSL [83] will align the positive pairs and contrast the negative pairs simultaneously. Generative SSL [36, 45, 81], on the other hand, obtains the supervised signal in an intra-molecule way: it learns a 2D/3D representation that can reconstruct its 3D/2D counterpart view for each molecule itself. To cope with the challenge of measuring the quality of reconstruction on molecule 3D and 2D space, we further propose a novel surrogate objective function called variation
representation reconstruction (VRR) for the generative SSL task, which can effectively compute such quality in the continuous representation space. The knowledge acquired by these two SSL tasks is complementary, so our GraphMVP framework integrates them to form more discriminative 2D molecular graph representation. Consistent performance improvements empirically validate the effectiveness of GraphMVP.
We give additional insights to justify the effectiveness of GraphMVP. First, GraphMVP is a selfsupervised learning approach based on maximizing mutual information (MI) between 3D and 2D views, enabling the learnt representation to capture high-level factors [5, 6, 76] in molecule data. Second, we find that 3D molecular geometry is a form of privileged information [78, 79]. It has been proven that using privileged information in training can accelerate the speed of learning. We are aware that privileged information is only used in training, while it is not available in testing. This perfectly matches our intuition of pre-training molecular representation with 3D geometry.
Our major contributions include (1) To our best knowledge, we are the first to incorporate the 3D geometric information into graph SSL; (2) We propose one contrastive and one generative SSL tasks for pre-training. Then we elaborate their differences and empirically validate that combining both can lead to a better representation; (3) We provide theoretical insights and case studies justify why adding 3D geometry is beneficial; (4) We achieve the SOTA performance among all the SSL baselines.
Related work. We briefly review the most related works here and include a more detailed summarization in Appendix A. Self-supervised learning (SSL) methods have attracted massive attention to graph applications [49, 51, 85, 87]. In general, there are roughly two categories of graph SSL: contrastive and generative, where they differ on the design of the supervised signals. Contrastive graph SSL [39, 72, 80, 91, 92] constructs the supervised signals at the inter-graph level and learns the representation by contrasting with other graphs, while generative graph SSL [32, 39, 40, 48] focuses on reconstructing the original graph at the intra-graph level. One of the most significant differences that separate our work from existing methods is that all previous methods merely focus on 2D molecular topology. However, for scientific tasks such as molecular property prediction, 3D geometry should be incorporated as it provides complementary and comprehensive information [50, 69]. To fill this gap, we propose GraphMVP to leverage the 3D geometry in graph self-supervised pre-training.

heading: 2 PRELIMINARIES
text: We first outline the key concepts and notations used in this work. Self-supervised learning (SSL) is based on the view design, where each view provides a specific aspect and modality of the data. Each molecule has two natural views: the 2D graph incorporates the topological structure defined by the adjacency, while the 3D graph can better reflect the geometry and spatial relation. From a chemical perspective, 3D geometric graphs focus on the energy while 2D graphs emphasize the topological information; thus they can be composed for learning more informative representation in GraphMVP. Transformation is an atomic operation in SSL that can extract specific information from each view. Next, we will briefly introduce how to represent these two views of molecular graphs.
2D Molecular Graph represents molecules as 2D graphs, with atoms as nodes and bonds as edges. We denote it as g2D = (X,E), where X is the atom attribute matrix and E is the bond attribute matrix. Notice that hereE also includes the bond connectivity. Then we will apply one transformation function T2D on the topological graph. Given a 2D molecular graph g2D, its representation h2D can be obtained from a 2D graph neural network (GNN) model:
h2D = GNN-2D(T2D(g2D)) = GNN-2D(T2D(X,E)). (1)
3D Molecular Graph additionally includes spatial positions of the atoms, which needless to be static since atoms are in continual motion on a potential energy surface [3]. 1 The 3D structures at the local minima on this surface are named conformer. As the molecular properties are conformers ensembled [34], GraphMVP provides a novel perspective on adopting 3D conformers for learning better representation. Given a conformer g3D = (X,R), its representation via a 3D GNN model is:
h3D = GNN-3D(T3D(g3D)) = GNN-3D(T3D(X,R)), (2)
1A more rigorous way of defining conformer is in [56]: a conformer is an isomer of a molecule that differs from another isomer by the rotation of a single bond in the molecule.
where R is the 3D-coordinate matrix and T3D is the 3D transformation. In what follows, for notation simplicity, we use x and y for the 2D and 3D graphs, i.e., x , g2D and y , g3D. Then the latent representations are denoted as hx and hy .

heading: 3 GRAPHMVP: GRAPH MULTI-VIEW PRE-TRAINING
text: Our model, termed as Graph Multi-View Pre-training (GraphMVP), conducts self-supervised learning (SSL) pre-training with 3D information. The 3D conformers encode rich information about the molecule energy and spatial structure, which are complementary to the 2D topology. Thus, applying SSL between the 3D and 2D views will provide a better 2D representation, which implicitly embeds the ensembles of energies and geometric information for molecules.
In the following, we first present an overview of proposed GraphMVP, and then introduce two pretext tasks specialized concerning 3D conformation structures. Finally, we summarize a broader graph SSL family that prevails the 2D molecular graph representation learning with 3D geometry.
3.1 OVERVIEW OF GRAPHMVP
3
As aforementioned, GraphMVP exerts 2D topology and 3D geometry as two complementary views for each molecule. By proceeding SSL between these views, it is expected to learn a 2D representation enhanced with 3D conformation, which can better reflect certain molecular properties.
As generic SSL pre-training pipelines, GraphMVP has two stages: pre-training then fine-tuning. In the pre-training, we conduct SSL via auxiliary tasks on data collections providing both 3D and 2D molecular structures. During fine-tuning, the pre-trained 2D GNN models are subsequently fine tuned on specific downstream tasks, where usually only 2D structures are available.
At the SSL pre-training stage, we design two pretext tasks: one contrastive and one generative. We conjecture then empirically prove that these two tasks are focusing on different learning aspects, which are concluded into following two points. (1) From the perspective of representation learning, contrastive SSL is learning from inter-data and generative SSL is learning by intra-data. For contrastive SSL, one key step is to obtain the negative view pairs from inter-data for contrasting; while generative SSL focuses on each data point itself, by reconstructing the key features at the intra-data level. (2) From the perspective of distribution learning, contrastive SSL and generative SSL are learning the data distribution from local and global manner, respectively. Contrastive SSL learns the distribution locally by contrasting the pairwise distance at the inter-data level. Thus, with sufficient number of data, the local contrastive operation can iteratively recover the data distribution. Generative SSL, on the other hand, learns the global data density function directly.
Therefore, contrastive and generative SSL are essentially conducting representation and distribution learning with different intuitions and disciplines, and we expect that combining these two can lead to better representation. We later carry out an ablation study (Section 4.4) to verify this empirically. In
addition, to make the pretext tasks more challenging, we take views for each molecule by randomly masking M nodes (and corresponding edges) as the transformation function, i.e., T2D = T3D = mask. This smart trick has been widely used in graph SSL [39, 91, 92] and has shown robust improvements.

heading: 3.2 CONTRASTIVE SELF-SUPERVISED LEARNING BETWEEN 3D AND 2D VIEWS
text: The main idea of contrastive self-supervised learning (SSL) [9, 60] is first to define positive and negative pairs of views from the inter-data level, and then to align the positive pairs and contrast the negative pairs simultaneously [83]. For each molecule, we first extract representations from 2D and 3D views, i.e., hx and hy. Then we create positive and negative pairs for contrastive learning: the 2D-3D pairs (x,y) for the same molecule are treated as positive, and negative otherwise. Finally, we align the positive pairs and contrast the negative ones. The pipeline is shown in Figure 1. In the following, we discuss two common objective functions on contrastive graph SSL.
InfoNCE is first proposed in [60], and its effectiveness has been validated both empirically [9, 35] and theoretically [2]. Its formulation is given as follows:
LInfoNCE = − 1
2 Ep(x,y)
[ log
exp(fx(x,y)) exp(fx(x,y)) + ∑ j exp(fx(xj ,y) ) + log
exp(fy(y,x)) exp(fy(y,x)) + ∑ j exp(fy(yj ,x))
] , (3)
where x(j),y(j) are randomly sampled 2D and 3D views regarding to the anchored pair (x,y). fx(x,y) and fy(y,x) are scoring functions for the two corresponding views, whose formulation can be flexible. Here we use fx(x,y) = fy(y,x) = 〈hx, hy〉. More details are in Appendix D. Energy-Based Model with Noise Contrastive Estimation (EBM-NCE) is an alternative that has been widely used in the research line of graph contrastive SSL [39, 72, 91, 92]. Its intention is essentially the same as InfoNCE, to align positive pairs and contrast negative pairs, while the main difference is the usage of binary cross-entropy and extra noise distribution for negative sampling:
LEBM-NCE = − 1
2 Ep(y)
[ Epn(x|y) log ( 1− σ(fx(x,y)) ) + Ep(x|y) log σ(fx(x,y)) ] − 1
2 Ep(x)
[ Epn(y|x) log ( 1− σ(fy(y,x)) ) + Ep(y,x) log σ(fy(y,x)) ] ,
(4)
where pn is the noise distribution and σ is the sigmoid function. We also notice that the final formulation of EBM-NCE shares certain similarities with Jensen-Shannon estimation (JSE) [59]. However, the derivation process and underlying intuition are different: EBM-NCE models the conditional distribution in MI lower bound (Equation (9)) with EBM, while JSE is a special case of variational estimation of f-divergence. Since this is not the main focus of GraphMVP, we expand the a more comprehensive comparison in Appendix D, plus some potential benefits with EBM-NCE.
Few works [33] have witnessed the effect on the choice of objectives in graph contrastive SSL, therefore in GraphMVP, we treat it as a hyper-parameter and further run ablations on them, i.e., to solely use either InfoNCE (LC = LInfoNCE) or EMB-NCE (LC = LEBM-NCE).

heading: 3.3 GENERATIVE SELF-SUPERVISED LEARNING BETWEEN 3D AND 2D VIEWS
text: Generative SSL is another classic track for unsupervised pre-training [10, 44, 45, 47]. It aims at learning an effective representation by reconstructing each data point itself. Specifically to drug discovery, we have one 2D graph and a certain number of 3D conformers for each molecule, and our goal is to learn a robust 2D/3D representation that can, to the most extent, recover its 3D/2D counterparts. By doing so, generative SSL can enforce 2D/3D GNN to encode the most inherent geometry/topology information, which can logically improve the downstream performance.
There are many options for generative models, including variational auto-encoder (VAE) [45], generative adversarial networks (GAN) [28], flow-based model [17], etc. In GraphMVP, we prefer VAE-like method for the following reasons: (1) Mapping between two molecular views is stochastic: multiple 3D conformers correspond to the same 2D topology; (2) An explicit 2D graph representation (i.e., feature encoder) is required for downstream tasks; (3) Decoders for structured data such as graph are often complicated and hard to design, which make them a suboptimal choice.
Variational Molecule Reconstruction. Therefore we propose a light VAE-like generative SSL, equipped with a crafty surrogate loss, which we describe follows. Take one direction for illustration,
when generating 3D conformers from their corresponding 2D topology, we want to model the conditional likelihood p(y|x). By introducing a reparameterized variable zx = µx + σx , where µx and σx are two flexible functions on hx, ∼ N (0, I) and is the element-wise production, we have the following lower bound:
log p(y|x) ≥ Eq(zx|x) [ log p(y|zx) ] −KL(q(zx|x)||p(zx)). (5)
The expression is similar for log p(x|y). The above objective is composed of a conditional loglikelihood and a KL-divergence, where the bottleneck is to calculate the first term for structured data. This term has also been recognized as the reconstruction term: it is essentially to reconstruct the 3D conformers (y) from the sampled 2D molecular graph representation (zx). However, performing the graph reconstruction on the data space is not easy: since molecules (e.g., atoms and bonds) are discrete, modeling and measuring on the molecule space will bring extra obstacles.
Variational Representation Reconstruction (VRR). To cope with this challenge, we propose a novel surrogate loss by transferring the reconstruction from data space to representation space. Instead of decoding the latent code zx to data space, we can directly project it to the 3D representation space, denoted as qx(zx). Since the representation space is continuous, we may as well model the conditional log-likelihood with Gaussian distribution, resulting in L2 distance for reconstruction, i.e., ‖qx(zx)− SG(hy(y))‖2. Here SG is short for stop-gradient, assuming that hy is a fixed learnt representation function, which has been widely adopted in the SSL literature [11, 29]. We term this surrogate loss as VRR, and we take it for the generative SSL loss:
LG = LVRR = 1
2
[ Eq(zx|x) [ ‖qx(zx)− SG(hy)‖2 ] + Eq(zy |y) [ ‖qy(zy)− SG(hx)‖22 ]] + β 2 · [ KL(q(zx|x)||p(zx)) +KL(q(zy|y)||p(zy)) ] .
(6)
We provide the generative SSL pipeline in Figure 1 and the complete derivations in Appendix E. As will be discussed in Section 5.1, VRR is actually maximizing MI, and MI is invariant to continuous bijective function [6]. Thus, this surrogate loss would be exact if the encoding function h satisfies this condition. However, we find that GNN, though does not meet the condition, can provide quite robust performance, which empirically justify the effectiveness of VRR.

heading: 3.4 MULTI-TASK OBJECTIVE FUNCTION
text: As discussed before, contrastive SSL and generative SSL essentially learn the representation from distinct viewpoints. A reasonable conjecture is that combining both SSL methods can lead to overall better performance, thus we arrive at minimizing the following complete objective for GraphMVP:
LGraphMVP = α1 · LC + α2 · LG, (7) where α1, α2 are weighting coefficients. A later performed ablation study (Section 4.4) delivers two important messages: (1) Both individual contrastive and generative SSL on 3D conformers can consistently help improve the 2D representation learning; (2) Combining the two SSL strategies can yield further improvements. Thus, we draw the conclusion that GraphMVP (Equation (7)) is able to obtain an augmented 2D representation by fully utilizing the 3D information.
As discussed in Section 1, existing graph SSLs only focus on the 2D topology, which is in parallel to GraphMVP: 2D graph SSL focuses on exploiting the 2D structure topology, and GraphMVP takes advantage of the 3D geometry information. Thus, we propose to merge the 2D SSL into GraphMVP. Since there are two main categories in 2D graph SSL: generative and contrastive, we propose two variants GraphMVP-G and GraphMVP-C accordingly. Their objectives are as follows: LGraphMVP-G = LGraphMVP + α3 · LGenerative 2D-SSL, LGraphMVP-C = LGraphMVP + α3 · LContrastive 2D-SSL. (8)
Later, the empirical results also help support the effectiveness of GraphMVP-G and GraphMVP-C, and thus, we can conclude that existing 2D SSL is complementary to GraphMVP.

heading: 4 EXPERIMENTS AND RESULTS

heading: 4.1 EXPERIMENTAL SETTINGS
text: Datasets. We pre-train models on the same dataset then fine-tune on the wide range of downstream tasks. We randomly select 50k qualified molecules from GEOM [3] with both 2D and 3D structures
for the pre-training. Clarified in Section 3.1, conformer ensembles can better reflect the molecular property, thus we take C conformers of each molecule. For downstream tasks, we first stick to the same setting of the main graph SSL work [39, 91, 92], exploring 8 binary molecular property prediction tasks, which are all in the low-data regime. Then we explore 6 regression tasks from various low-data domains to be more comprehensive. We describe all the datasets in Appendix F.
2D GNN. We follow the research line of SSL on molecule graph [39, 91, 92], using the same Graph Isomorphism Network (GIN) [88] as the backbone model, with the same feature sets.
3D GNN. We choose SchNet [69] for geometric modeling, since SchNet: (1) is found to be a strong geometric representation learning method under the fair benchmarking; (2) can be trained more efficiently, comparing to the other recent 3D models. We provide details in Appendix B.2.

heading: 4.2 MAIN RESULTS ON MOLECULAR PROPERTY PREDICTION.
text: We carry out comprehensive comparisons with 10 SSL baselines and random initialization. For pre-training, we apply all SSL methods on the same dataset based on GEOM [3]. For fine-tuning, we follow the same setting [39, 91, 92] with 8 low-data molecular property prediction tasks.
Baselines. Due to the rapid growth of graph SSL [49, 51, 85, 87], we are only able to benchmark the most well-acknowledged, peer-reviewed baselines: EdgePred [32], InfoGraph [72], GPT-GNN [40], AttrMask & ContextPred[39], GraphLoG[89], G-{Contextual, Motif}[67], GraphCL[92], JOAO[91].
Our method. GraphMVP has two key factors: i) masking ratio (M ) and ii) number of conformers for each molecule (C). We set M = 0.15 and C = 5 by default, and will explore their effects in the following ablation studies in Section 4.3. For EBM-NCE loss, we adopt the empirical distribution for noise distribution. For Equation (8), we pick the empirically optimal generative and contrastive 2D SSL method: that is AttrMask for GraphMVP-G and ContextPred for GraphMVP-C.
The main results on 8 molecular property prediction tasks are listed in Table 1. We observe that the performance of GraphMVP is significantly better than the random initialized one, and the average performance outperforms the existing SSL methods by a large margin. In addition, GraphMVP-G and GraphMVP-C consistently improve the performance, supporting the claim: 3D geometry is complementary to the 2D topology. GraphMVP leverages the information between 3D conformers and 2D topology, and 2D SSL plays the role as regularizer to extract more 2D topological information; they are extracting different perspectives of information and are indeed complementary to each other.

heading: 4.3 ABLATION STUDY: THE EFFECT OF MASKING RATIO AND NUMBER OF CONFORMERS
text: We analyze the effects of masking ratioM and the number of conformers C in GraphMVP. In Table 1, we set the M as 0.15 since it has been widely used in existing SSL methods [39, 91, 92], and
Table 2: Ablation of masking ratio M , C ≡ 5.
M GraphMVP GraphMVP-G GraphMVP-C
0 71.12 72.15 72.66 0.15 71.60 72.76 73.08 0.30 71.79 72.91 73.17
Table 3: Ablation of # conformer C, M ≡ 0.15.
C GraphMVP GraphMVP-G GraphMVP-C
1 71.61 72.80 72.46 5 71.60 72.76 73.08
10 72.20 72.59 73.09 20 72.39 73.00 73.02
C is set to 5, which we will explain below. We explore on the range of M ∈ {0, 0.15, 0.3} and C ∈ {1, 5, 10, 20}, and report the average performance. The complete results are in Appendix G.2. As seen in Table 2, the improvement is more obvious from M = 0 (raw graph) to M = 0.15 than from M = 0.15 to M = 0.3. This can be explained that subgraph masking with larger ratio will make the SSL tasks more challenging, especially comparing to the raw graph (M = 0).
Table 3 shows the effect for C. We observe that the performance is generally better when adding more conformers, but will reach a plateau above certain thresholds. This observation matches with previous findings [4]: adding more conformers to augment the representation learning is not as helpful as expected; while we conclude that adding more conformers can be beneficial with little improvement. One possible reason is, when generating the dataset, we are sampling top-C conformers with highest possibility and lowest energy. In other words, top-5 conformers are sufficient to cover the most conformers with equilibrium state (over 80%), and the effect of larger C is thus modest.
To sum up, adding more conformers might be helpful, but the computation cost can grow linearly with the increase in dataset size. On the other hand, enlarging the masking ratio will not induce extra cost, yet the performance is slightly better. Therefore, we would encourage tuning masking ratios prior to trying a larger number of conformers from the perspective of efficiency and effectiveness.

heading: 4.4 ABLATION STUDY: THE EFFECT OF OBJECTIVE FUNCTION
text: In Section 3, we introduce a new contrastive learning objective family called EBM-NCE, and we take either InfoNCE and EBM-NCE as a contrastive loss. For the generative SSL task, we propose a novel objective function called variational representation reconstruction (VRR) in Equation (6). As discussed in Section 3.3, stochasticity is important for GraphMVP since it can capture the conformer distribution for each 2D molecular graph. To verify this, we add an ablation study on representation reconstruction (RR) by removing stochasticity in VRR. Thus, here we deploy an ablation study to explore the
effect for each individual objective function (InfoNCE, EBM-NCE, VRR and RR), followed by the pairwise combinations between them.
The results in Table 4 give certain constructive insights as follows: (1) Each individual SSL objective function (middle block) can lead to better performance. This strengthens the claim that adding 3D information is helpful for 2D representation learning. (2) According to the combination of those SSL objective functions (bottom block), adding both contrastive and generative SSL can consistently improve the performance. This verifies our claim that conducting SSL at both the inter-data and intra-data level is beneficial. (3) We can see VRR is consistently better than RR on all settings, which verify that stochasticity is an important factor in modeling 3D conformers for molecules.

heading: 4.5 BROADER RANGE OF DOWNSTREAM TASKS
text: The 8 binary downstream tasks discussed so far have been widely applied in the graph SSL research line on molecules [39, 91, 92], but there are more tasks where the 3D conformers can be helpful. Here we test 4 extra regression property prediction tasks and 2 drug-target affinity tasks.
About the dataset statistics, more detailed information can be found in Appendix F, and we may as well briefly describe the task here. Drug-target affinity (DTA) is a crucial task [61, 62, 84] in drug discovery, where it models both the molecular drugs and protein targets, with the goal to predict
their affinity scores. One recent work [57] is modeling the molecular drugs with 2D GNN and target protein (as an amino-acid sequence) with convolution neural network (CNN). We adopt this setting by pre-training the 2D GNN using GraphMVP. As illustrated in Table 5, the consistent performance gain verifies the effectiveness of our proposed GraphMVP.

heading: 4.6 CASE STUDY
text: We investigate how GraphMVP helps when the task objectives are challenging with respect to the 2D topology but straightforward using 3D geometry (as shown in Figure 2). We therefore design two case studies to testify how GraphMVP transfers knowledge from 3D geometry into the 2D representation.
The first case study is 3D Diameter Prediction. For molecules, usually, the longer the 2D diameter is, the larger the 3D diameter (largest atomic pairwise l2 distance). However, this is not always true, and we are interested in using the 2D graph to predict the 3D diameter. The second case study is Long-Range Donor-Acceptor Detection. Molecules possess a special geometric structure called donor-acceptor bond, and we want to use 2D molecular graph to detect this special structure. We validate that GraphMVP consistently brings improvements on these 2 case studies, and provide more detailed discussions and interpretations in Appendix G.6.

heading: 5 THEORETICAL INSIGHTS
text: In this section, we provide the mathematical insights behind GraphMVP. We will first discuss both contrastive and generative SSL methods(Sections 3.2 and 3.3) are maximizing the mutual information (MI) and then how the 3D geometry, as privileged information, can help 2D representation learning.

heading: 5.1 MAXIMIZING MUTUAL INFORMATION
text: Mutual information (MI) measures the non-linear dependence [6] between two random variables: the larger MI, the stronger dependence between the variables. Therefore for GraphMVP, we can interpret it as maximizing MI between 3D and 2D views: to obtain a more robust 2D/3D representation
8
by sharing more information with its 3D/2D counterparts. This is also consistent with the sample complexity theory [2, 19, 24] where SSL as functional regularizer can reduce the uncertainty in representation learning. We first derive a lower bound for MI (see derivation in Appendix C), and the corresponding objective function LMI is
I(X;Y ) ≥ LMI = 1
2 Ep(x,y)
[ log p(y|x) + log p(x|y) ] . (9)
Contrastive Self-Supervised Learning. InfoNCE was initialized proposed to maximize the MI directly [60]. Here in GraphMVP, EBM-NCE estimates the conditional likelihood in Equation (9) using EBM, and solves it with NCE. As a result, EBM-NCE can also be seen as maximizing MI between 3D and 2D views. The detailed derivations can be found in Appendix D.2.
Generative Self-Supervised Learning. One alternative solution is to use a variational lower bound to approximate the conditional log-likelihood terms in Equation (9). Then we can follow the same pipeline in Section 3.3, ending up with the surrogate objective, i.e., VRR in Equation (6).

heading: 5.2 3D GEOMETRY AS PRIVILEGED INFORMATION
text: We show the theoretical insights from privileged information that motivate GraphMVP. We start by considering a supervised learning setting where (ui, li) is a feature-label pair and u∗i is the privileged information [78, 79]. The privileged information is defined to be additional information about the input (ui, li) in order to support the prediction. For example, ui could be some CT images of a particular disease, li could be the label of the disease and u∗i is the medical report from a doctor. VC theory [77, 78] characterizes the learning speed of an algorithm from the capacity of the algorithm and the amount of training data. Considering a binary classifier f from a function class F with finite VC-dimension VCD(F). With probability 1− δ, the expected error is upper bounded by
R(f) ≤ Rn(f) +O ((VCD(F)− log δ
n )β) (10) where Rn(f) denotes the training error and n is the number of training samples. When the training data is separable, then Rn(f) will diminish to zero and β is equal to 1. When the training data is non-separable, β is 12 . Therefore, the rate of convergence for the separable case is of order 1/n. In contrast, the rate for the non-separable case is of order 1/ √ n. We note that such a difference is huge, since the same order of bounds require up to 100 training samples versus 10,000 samples. Privileged information makes the training data separable such that the learning can be more efficient. Connecting the results to GraphMVP, we notice that the 3D geometric information of molecules can be viewed as a form of privileged information, since 3D information can effectively make molecules more separable for some properties [50, 69]. Besides, privileged information is only used in training, and it well matches our usage of 3D geometry for pre-training. In fact, using 3D structures as privileged information has been already shown quite useful in protein classification [79], which serves as a strong evidence to justify the effectiveness of 3D information in pre-training.

heading: 6 CONCLUSION AND FUTURE WORK
text: In this work, we provide a very general framework, coined GraphMVP. From the domain perspective, GraphMVP (1) is the first to incorporate 3D information for augmenting 2D graph representation learning and (2) is able to take advantages of 3D conformers by considering stochasticity in modeling. From the perspective of technical novelties, GraphMVP brings following insights when introducing 2 SSL tasks: (1) Following Equation (9), GraphMVP proposes EBM-NCE and VRR, where they are modeling the conditional distribution using EBM and variational distribution respectively. (2) EBM-NCE is similar to JSE, while we start with a different direction for derivation and mathematical intuition, yet EBM opens another promising venues in this area. (3) VRR, as a generative SSL method, is able to alleviate the potential issues in molecule generation [23, 94]. (4) Ultimately, GraphMVP combines both contrastive SSL (InfoNCE or EBM-NCE) and generative SSL (VRR) for objective function. Both empirical results (solid performance improvements on 14 downstream datasets) and theoretical analysis can strongly support the above domain and technical contributions.
We want to emphasize that GraphMVP is model-agnostic and has the potential to be expanded to many other low-data applications. This motivates broad directions for future exploration, including but not limited to: (1) More powerful 3D and 2D molecule representation methods. (2) Different application domain other than small molecule, e.g., large molecules like proteins.

heading: REPRODUCIBILITY STATEMENT
text: To ensure the reproducibility of the empirical results, we include our code base in the supplementary material, which contains: instructions for installing conda virtual environment, data preprocessing scripts, and training scripts. Our code is available on GitHub for reproducibility. Complete derivations of equations and clear explanations are given in Appendices C to E.

heading: Table of Contents

heading: A Self-Supervised Learning on Molecular Graph 16
text: A.1 Contrastive graph SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.2 Generative graph SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.3 Other graph SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

heading: B Molecular Graph Representation 17
text: B.1 2D Molecular Graph Neural Network . . . . . . . . . . . . . . . . . . . . . . 17 B.2 3D Molecular Graph Neural Network . . . . . . . . . . . . . . . . . . . . . . 17 B.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

heading: C Maximize Mutual Information 19
text: C.1 Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 C.2 A Lower Bound to MI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

heading: D Contrastive Self-Supervised Learning 20
text: D.1 InfoNCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 D.2 EBM-NCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 D.3 EBM-NCE v.s. JSE and InfoNCE . . . . . . . . . . . . . . . . . . . . . . . . 23

heading: E Generative Self-Supervised Learning 24
text: E.1 Variational Molecule Reconstruction . . . . . . . . . . . . . . . . . . . . . . . 24 E.2 Variational Representation Reconstruction . . . . . . . . . . . . . . . . . . . . 24 E.3 Variational Representation Reconstruction and Non-Contrastive SSL . . . . . . 25

heading: F Dataset Overview 26
text: F.1 Pre-Training Dataset Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 26 F.2 Downstream Dataset Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 26

heading: G Experiments Details 27
text: G.1 Self-supervised Learning Baselines . . . . . . . . . . . . . . . . . . . . . . . 27 G.2 Ablation Study: The Effect of Masking Ratio and Number of Conformers . . . 28 G.3 Ablation Study: Effect of Each Loss Component . . . . . . . . . . . . . . . . 28 G.4 Broader Range of Downstream Tasks: Molecular Property Prediction Prediction 29 G.5 Broader Range of Downstream Tasks: Drug-Target Affinity Prediction . . . . . 29 G.6 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

heading: A SELF-SUPERVISED LEARNING ON MOLECULAR GRAPH
text: Self-supervised learning (SSL) methods have attracted massive attention recently, trending from vision [8, 9, 11, 35, 82], language [7, 16, 60] to graph [39, 48, 72, 80, 91, 92]. In general, there are two categories of SSL: contrastive and generative, where they differ on the design of the supervised signals. Contrastive SSL realizes the supervised signals at the inter-data level, learning the representation by contrasting with other data points; while generative SSL focuses on reconstructing the original data at the intra-data level. Both venues have been explored [49, 51, 85, 87] on the graph applications.

heading: A.1 CONTRASTIVE GRAPH SSL
text: Contrastive graph SSL first applies transformations to construct different views for each graph. Each view incorporates different granularities of information, like node-, subgraph-, and graph-level. It then solves two sub-tasks simultaneously: (1) aligning the representations of views from the same data; (2) contrasting the representations of views from different data, leading to a uniformly distributed latent space [83]. The key difference among existing methods is thus the design of view constructions. InfoGraph [72, 80] contrasted the node (local) and graph (global) views. As an extension, GraphLoG [89] learned the context (subgraph or motif) view using clustering and contrasted it with both node and graph views. ContextPred [39] contrasted between node and context views. GraphCL and JOAO [91, 92] made comprehensive comparisons among four graph-level transformations and further learned to select the most effective combinations.

heading: A.2 GENERATIVE GRAPH SSL
text: Generative graph SSL aims at reconstructing important structures for each graph. By so doing, it consequently learns a representation capable of encoding key ingredients of the data. EdgePred [32] and AttrMask [39] predicted the adjacency matrix and masked tokens (nodes and edges) respectively. GPT-GNN [40] reconstructed the whole graph in an auto-regressive approach.

heading: A.3 OTHER GRAPH SSL
text: There are certain SSL methods specific to the molecular graph. For example, one central task in drug discovery is to find the important motif or substructure in molecules that can activate the target interactions. Grover [67] adopts domain knowledge to heuristically extract motifs for each molecule, and the SSL task is to make prediction on the existence of each motif. Different from contrastive and generative SSL, recent literature [85] takes this as predictive graph SSL, where the supervised signals are self-generate informative labels.
Recall that all previous methods in Table 6 merely focus on the 2D topology. However, for sciencecentric tasks such as molecular property prediction, 3D geometry should be incorporated as it provides complementary and comprehensive information [50, 69]. To mitigate this gap, we propose GraphMVP to leverage the 3D geometry with unsupervised graph pre-training.

heading: B MOLECULAR GRAPH REPRESENTATION
text: Graph neural network (GNN) has become the mainstream modeling methods for molecular graph representation. Existing methods can generally be split into two venues: 2D GNN and 3D GNN, depending on what levels of information is being considered. 2D GNN focuses on the topological structures of the graph, like the adjacency among nodes, while 3D GNN is able to model the “energy” of molecules by taking account the spatial positions of atoms.
First, we want to highlight that GraphMVP is model-agnostic, i.e., it can be applied to any 2D and 3D GNN representation function, yet the specific 3D and 2D representations are not the main focus of this work. Second, we acknowledge there are a lot of advanced 3D [21, 41, 50, 68] and 2D [12, 15, 27, 48, 88, 90] representation methods. However, considering the graph SSL literature and graph representation liteature (illustrated below), we adopt GIN [88] and SchNet [69] in current GraphMVP.
B.1 2D MOLECULAR GRAPH NEURAL NETWORK
The 2D representation is taking each molecule as a 2D graph, with atoms as nodes and bonds as edges, i.e., g2D = (X,E). X ∈ Rn×dn is the atom attribute matrix, where n is the number of atoms (nodes) and dn is the atom attribute dimension. E ∈ Rm×de is the bond attribute matrix, where m is the number of bonds (edges) and dm is the bond attribute dimension. Notice that here E also includes the connectivity. Then we will apply one transformation function T2D on the topological graph. Given a 2D graph g2D, its 2D molecular representation is:
h2D = GNN-2D(T2D(g2D)) = GNN-2D(T2D(X,E)). (11)
The core operation of 2D GNN is the message passing function [27], which updates the node representation based on adjacency information. We have variants depending on the design of message and aggregation functions, and we pick GIN [88] in this work.
GIN There has been a long research line on 2D graph representation learning [12, 15, 27, 48, 88, 90]. Among these, graph isomorphism network (GIN) model [88] has been widely used as the backbone model in recent graph self-supervised learning work [39, 91, 92]. Thus, we as well adopt GIN as the base model for 2D representation.
Recall each molecule is represented as a molecular graph, i.e., g2D = (X,E), where X and E are feature matrices for atoms and bonds respectively. Then the message passing function is defined as:
z (k+1) i = MLP (k+1) atom ( z (k) i + ∑ j∈N (i) ( z (k) j + MLP (k+1) bond (Eij) )) , (12)
where z0 = X and MLP (k+1) atom and MLP (k+1) bond are the (l + 1)-th MLP layers on the atom- and bond-level respectively. Repeating this for K times, and we can encode K-hop neighborhood information for each center atom in the molecular data, and we take the last layer for each node/atom representation. The graph-level molecular representation is the mean of the node representation:
z(x) = 1
N ∑ i z (K) i (13)
B.2 3D MOLECULAR GRAPH NEURAL NETWORK
Recently, the 3D geometric representation learning has brought breakthrough progress in molecule modeling [21, 41, 50, 68, 69]. 3D Molecular Graph additionally includes spatial locations of the atoms, which needless to be static since, in real scenarios, atoms are in continual motion on a potential energy surface [3]. The 3D structures at the local minima on this surface are named molecular conformation or conformer. As the molecular properties are a function of the conformer ensembles [34], this reveals another limitation of existing mainstream methods: to predict properties from a single 2D or 3D graph cannot account for this fact [3], while our proposed method can alleviate this issue to a certain extent.
For specific 3D molecular graph, it additionally includes spatial positions of the atoms. We represent each conformer as g3D = (X,R), where R ∈ Rn×3 is the 3D-coordinate matrix, and the corresponding representation is:
h3D = GNN-3D(T3D(g3D)) = GNN-3D(T3D(X,R)), (14)
where R is the 3D-coordinate matrix and T3D is the 3D transformation. Note that further information such as plane and torsion angles can be solved from the positions.
SchNet SchNet [69] is composed of the following key steps:
z (0) i = embedding(xi)
z (t+1) i = MLP ( n∑ j=1 f(x (t−1) j , ri, rj) ) hi = MLP(z (K) i ),
(15)
where K is the number of hidden layers, and
f(xj , ri, rj) = xj · ek(ri − rj) = xj · exp(−γ‖‖ri − rj‖2 − µ‖22) (16)
is the continuous-filter convolution layer, enabling the modeling of continuous positions of atoms.
We adopt SchNet for the following reasons. (1) SchNet is a very strong geometric representation method after fair benchmarking. (2) SchNet can be trained more efficiently, comparing to the other recent 3D models. To support these two points, we make a comparison among the most recent 3D geometric models [21, 50, 68] on QM9 dataset. QM9 [86] is a molecule dataset approximating 12 thermodynamic properties calculated by density functional theory (DFT) algorithm. Notice: UNiTE [65] is the state-of-the-art 3D GNN, but it requires a commercial software for feature extraction, thus we exclude it for now.
Table 7 shows that, under a fair comparison (w.r.t. data splitting, seed, cuda version, etc), SchNet can reach pretty comparable performance, yet the efficiency of SchNet is much better. Combining these two points, we adopt SchNet in current version of GraphMVP.

heading: B.3 SUMMARY
text: To sum up, in GraphMVP, the most important message we want to deliver is how to design a wellmotivated SSL algorithm to extract useful 3D geometry information to augment the 2D representation for downstream fine-tuning. GraphMVP is model-agnostic, and we may as well leave the more advanced 3D [21, 41, 50, 68] and 2D [12, 48, 90] GNN for future exploration.

heading: C MAXIMIZE MUTUAL INFORMATION
text: In what follows, we will useX and Y to denote the data space for 2D graph and 3D graph respectively. Then the latent representations are denoted as hx and hy .

heading: C.1 FORMULATION
text: The standard formulation for mutual information (MI) is I(X;Y ) = Ep(x,y) [ log p(x,y)
p(x)p(y)
] . (17)
Another well-explained MI inspired from wikipedia is given in Figure 3.
4
Mutual information (MI) between random variables measures the corresponding non-linear dependence. As can be seen in the first equation in Equation (17), the larger the divergence between the joint (p(x,y) and the product of the marginals p(x)p(y), the stronger the dependence between X and Y .
Thus, following this logic, maximizing MI between 3D and 2D views can force the 3D/2D representation to capture higher-level factors, e.g., the occurrence of important substructure that is semantically vital for downstream tasks. Or equivalently, maximizing MI can decrease the uncertainty in 2D representation given 3D geometric information.

heading: C.2 A LOWER BOUND TO MI
text: To solve MI, we first extract a lower bound: I(X;Y ) = Ep(x,y) [ log p(x,y)
p(x)p(y) ] ≥ Ep(x,y) [ log
p(x,y)√ p(x)p(y) ] = 1
2 Ep(x,y)
[ log (p(x,y))2
p(x)p(y) ] = 1
2 Ep(x,y)
[ log p(x|y) ] + 1
2 Ep(x,y)
[ log p(y|x) ] = −1
2 [H(Y |X) +H(X|Y )].
(18)
Thus, maximizing MI is equivalent to minimizing the following objective function:
LMI = 1
2 [H(Y |X) +H(X|Y )] (19)
In the following sections, we will describe two self-supervised learning methods for solving MI. Notice that the methods are very general, and can be applied to various applications. Here we apply it mainly for making 3D geometry useful for 2D representation learning on molecules.

heading: D CONTRASTIVE SELF-SUPERVISED LEARNING
text: The essence of contrastive self-supervised learning is to align positive view pairs and contrast negative view pairs, such that the obtained representation space is well distributed [83]. We display the pipeline in Figure 4. Along the research line in graph SSL [49, 51, 85, 87], InfoNCE and EBM-NCE are the two most-widely used, as discussed below.
1
D.1 INFONCE
InfoNCE [60] is first proposed to approximate MI Equation (17):
LInfoNCE = − 1
2 E
[ log
exp(fx(x,y)) exp(fx(x,y)) + ∑ j exp(fx(x j ,y) ) + log
exp(fy(y,x)) exp(fy(y,x)) + ∑ j exp fy(y j ,x)
] ,
(20) where xj ,yj are randomly sampled 3D and 2D views regarding to the anchored pair (x,y). fx(x,y), fy(y,x) are scoring functions for the two corresponding views, whose formulation can be quite flexible. Here we use fx(x,y) = fy(y,x) = exp(〈hx, hy〉).

heading: Derivation of InfoNCE
text: I(X;Y )− log(K) = Ep(x,y) [ log 1
K
p(x,y)
p(x)p(y) ] = ∑ xi,yi [ log 1 K p(xi,yi) p(xi)p(yi)
] ≥ −
∑ xi,yi [ log ( 1 + (K − 1)p(x i)p(yi) p(xi,yi) )]
= − ∑ xi,yi [ log p(xi,yi) p(xi)p(yi) + (K − 1) p(xi,yi) p(xi)p(yi) ] ≈ − ∑ xi,yi [ log p(xi,yi) p(xi)p(yi) + (K − 1)Exj 6=xi p(xj ,yi) p(xj)p(yi) p(xi,yi) p(xi)p(yi) ] // 1©
= ∑ xi,yi [ log
exp(fx(x i,yi)) exp(fx(xi,yi)) + ∑K j=1 fx(x j ,yi)
] ,
(21)
where we set fx(xi,yi) = log p(xi,yi) p(xi)p(yi) .
Notice that in 1©, we are using data x ∈ X as the anchor points. If we use the y ∈ Y as the anchor points and follow the similar steps, we can obtain
I(X;Y )− log(K) ≥ ∑ yi,xi [ log
exp(fy(y i,xi)) exp fy(yi,xi) + ∑K j=1 exp(fy(y j ,xi))
] . (22)
Thus, by add both together, we can have the objective function as Equation (20).

heading: D.2 EBM-NCE
text: We here provide an alternative approach to maximizing MI using energy-based model (EBM). To our best knowledge, we are the first to give the rigorous proof of using EBM to maximize the MI.

heading: D.2.1 ENERGY-BASED MODEL (EBM)
text: Energy-based model (EBM) is a powerful tool for modeling the data distribution. The classic formulation is:
p(x) = exp(−E(x))
A , (23) where the bottleneck is the intractable partition function A = ∫ x exp(−E(x))dx. Recently, there have been quite a lot progress along this direction [18, 30, 70, 71]. Noise Contrastive Estimation (NCE) [30] is one of the powerful tools here, as we will introduce later.

heading: D.2.2 EBM FOR MI
text: Recall that our objective function is Equation (19): LMI = 12 [H(Y |X) +H(X|Y )]. Then we model the conditional likelihood with energy-based model (EBM). This gives us
LEBM = − 1
2 Ep(x,y)
[ log exp(fx(x,y))
Ax|y + log
exp(fy(y,x))
Ay|x
] , (24)
where fx(x,y) = −E(x|y) and fy(y,x) = −E(y|x) are the negative energy functions, and Ax|y and Ay|x are the corresponding partition functions.
Under the EBM framework, if we solve Equation (24) with Noise Contrastive Estimation (NCE) [30], the final EBM-NCE objective is
LEBM-NCE =− 1
2 Epdata(y)
[ Epn(x|y)[log ( 1− σ(fx(x,y)) ) ] + Epdata(x|y)[log σ(fx(x,y))] ] − 1
2 Epdata(x)
[ Epn(y|x)[log ( 1− σ(fy(y,x)) ) ] + Epdata(y|x)[log σ(fy(y,x))] ] . (25)
Next we will give the detailed derivations.

heading: D.2.3 DERIVATION OF CONDITIONAL EBM WITH NCE
text: WLOG, let’s consider the pθ(x|y) first, and by EBM it is as follows:
pθ(x|y) = exp(−E(x|y))∫ exp(−E(x̃|y))dx̃ = exp(fx(x,y))∫ exp(fx(x̃|y))dx̃ = exp(fx(x,y)) Ax|y . (26)
Then we solve this using NCE. NCE handles the intractability issue by transforming it as a binary classification task. We take the partition function Ax|y as a parameter, and introduce a noise distribution pn. Based on this, we introduce a mixture model: z = 0 if the conditional x|y is from pn(x|y), and z = 1 if x|y is from pdata(x|y). So the joint distribution is:
pn,data(x|y) = p(z = 1)pdata(x|y) + p(z = 0)pn(x|y)
The posterior of p(z = 0|x,y) is
pn,data(z = 0|x,y) = p(z = 0)pn(x|y)
p(z = 0)pn(x|y) + p(z = 1)pdata(x|y) = ν · pn(x|y) ν · pn(x|y) + pdata(x|y) ,
where ν = p(z=0)p(z=1) .
Similarly, we can have the joint distribution under EBM framework as:
pn,θ(x) = p(z = 0)pn(x|y) + p(z = 1)pθ(x|y) And the corresponding posterior is:
pn,θ(z = 0|x,y) = p(z = 0)pn(x|y)
p(z = 0)pn(x|y) + p(z = 1)pθ(x|y) = ν · pn(x|y) ν · pn(x|y) + pθ(x|y)
We indirectly match pθ(x|y) to pdata(x|y) by fitting pn,θ(z|x,y) to pn,data(z|x,y) by minimizing their KL-divergence:
min θ DKL(pn,data(z|x,y)||pn,θ(z|x,y))
= Epn,data(x,z|y)[log pn,θ(z|x,y)]
= ∫ ∑ z pn,data(x, z|y) · log pn,θ(z|x,y)dx
= ∫ { p(z = 0)pn,data(x|y, z = 0) log pn,θ(z = 0|x,y)
+ p(z = 1)pn,data(x|z = 1,y) log pn,θ(z = 1|x,y) } dx
= ν · Epn(x|y) [ log pn,θ(z = 0|x,y) ] + Epdata(x|y) [ log pn,θ(z = 1|x,y) ] = ν · Epn(x|y) [ log
ν · pn(x|y) ν · pn(x|y) + pθ(x|y)
] + Epdata(x|y) [ log
pθ(x|y) ν · pn(x|y) + pθ(x|y)
] .
(27)
This optimal distribution is an estimation to the actual distribution (or data distribution), i.e., pθ(x|y) ≈ pdata(x|y). We can follow the similar steps for pθ(y|x) ≈ pdata(y|x). Thus following Equation (27), the objective function is to maximize ν · Epdata(y)Epn(x|y) [ log
ν · pn(x|y) ν · pn(x|y) + pθ(x|y)
] + Epdata(y)Epdata(x|y) [ log
pθ(x|y) ν · pn(x|y) + pθ(x|y)
] .
(28)
The we will adopt three strategies to approximate Equation (28):
1. Self-normalization. When the EBM is very expressive, i.e., using deep neural network for modeling, we can assume it is able to approximate the normalized density directly [55, 70]. In other words, we can set the partition function A = 1. This is a self-normalized EBM-NCE, with normalizing constant close to 1, i.e., p(x) = exp(−E(x)) = exp(f(x)) in Equation (23).
2. Exponential tilting term. Exponential tilting term [1] is another useful trick. It models the distribution as p̃θ(x) = q(x) exp(−Eθ(x)), where q(x) is the reference distribution. If we use the same reference distribution as the noise distribution, the tilted probability is p̃θ(x) = pn(x) exp(−Eθ(x)) in Equation (23).
3. Sampling. For many cases, we only need to sample 1 negative points for each data, i.e., ν = 1.
Following these three disciplines, the objective function to optimize pθ(x|y) becomes
Epn(x|y) [ log
pn(x|y) pn(x|y) + p̃θ(x|y)
] + Epdata(x|y) [ log
p̃θ(x|y) pn(x|y) + p̃θ(x|y) ] =Epn(x|y) [ log 1
1 + pθ(x|y)
] + Epdata(x|y) [ log
pθ(x|y) 1 + pθ(x|y) ] =Epn(x|y) [ log
exp(−fx(x,y)) exp(−fx(x,y)) + 1
] + Epdata(x|y) [ log
1
exp(−fx(x,y)) + 1 ] =Epn(x|y) [ log ( 1− σ(fx(x,y)) )] + Epdata(x|y) [ log σ(fx(x,y)) ] .
(29)
Thus, the final EBM-NCE contrastive SSL objective is
LEBM-NCE = − 1
2 Epdata(y)
[ Epn(x|y) log ( 1− σ(fx(x,y)) ) + Epdata(x|y) log σ(fx(x,y)) ] − 1
2 Epdata(x)
[ Epn(y|x) log ( 1− σ(fy(y,x)) ) + Epdata(y,x) log σ(fy(y,x)) ] . (30)

heading: D.3 EBM-NCE V.S. JSE AND INFONCE
text: We acknowledge that there are many other contrastive objectives [64] that can be used to maximize MI. However, in the research line of graph SSL, as summarized in several recent survey papers [51, 85, 87], the two most used ones are InfoNCE and Jensen-Shannon Estimator (JSE) [37, 59].
We conclude that JSE is very similar to EBM-NCE, while the underlying perspectives are totally different, as explained below.
1. Derivation and Intuition. Derivation process and underlying intuition are different. JSE [59] starts from f-divergence, then with variational estimation and Fenchel duality on function f . Our proposed EBM-NCE is more straightforward: it models the conditional distribution in the MI lower bound Equation (19) with EBM, and solves it using NCE.
2. Flexibility. Modeling the conditional distribution with EBM provides a broader family of algorithms. NCE is just one solution to it, and recent progress on score matching [70, 71] and contrastive divergence [18], though no longer contrastive SSL, adds on more promising directions. Thus, EBM can provide a potential unified framework for structuring our understanding of self-supervised learning.
3. Noise distribution. Starting from [37], all the following works on graph SSL [51, 72, 85, 87] have been adopting the empirical distribution for noise distribution. However, this is not the case in EBM-NCE. Classic EBM-NCE uses fixed distribution, while more recent work [1] extends it with adaptively learnable noise distribution. With this discipline, more advanced sampling strategies (w.r.t. the noise distribution) can be proposed, e.g., adversarial negative sampling in [38].
In the above we conclude three key differences between EBM-NCE and JSE, plus the solid and straightforward derivations on EBM-NCE, we would like to share this general contrastive SSL framework to the community.
According to the empirical results Section 4.4, we observe that EBM-NCE is better than InfoNCE. This can be explained using the claim from [43], where the main technical contribution is to construct many positives and many negatives per anchor point. The binary cross-entropy in EBM-NCE is able to realize this to some extent: make all the positive pairs positive and all the negative pairs negative, where the softmax-based cross-entropy fails to capture this, as in InfoNCE.
To conclude, we are introduce using EBM in modeling MI, which opens many potential venues. As for contrastive SSL, EBM-NCE provides a better perspective than JSE, and is better than InfoNCE on graph-level self-supervised learning.

heading: E GENERATIVE SELF-SUPERVISED LEARNING
text: Generative SSL is another classic track for unsupervised pre-training [44, 45, 47], though the main focus is on distribution learning. In GraphMVP, we start with VAE for the following reasons:
1. One of the biggest attributes of our problem is that the mapping between two views are stochastic: multiple 3D conformers can correspond to the same 2D topology. Thus, we expect a stochastic model [58] like VAE, instead of the deterministic ones.
2. For pre-training and fine-tuning, we need to learn an explicit and powerful representation function that can be used for downstream tasks.
3. The decoder for structured data like graph are often complicated, e.g.., the auto-regressive generation. This makes them suboptimal.
To cope with these challenges, in GraphMVP, we start with VAE-like generation model, and later propose a light-weighted and smart surrogate loss as objective function. Notice that for notation simplicity, for this section, we use hy and hx to delegate the 3D and 2D GNN respectively.
E.1 VARIATIONAL MOLECULE RECONSTRUCTION
As shown in Equation (19), our main motivation is to model the conditional likelihood:
LMI = − 1
2 Ep(x,y)[log p(x|y) + log p(y|x)]
By introducing a reparameterized variable zx = µx + σx , where µx and σx are two flexible functions on hx, ∼ N (0, I) and is the element-wise production, we have a lower bound on the conditional likelihood:
log p(y|x) ≥ Eq(zx|x) [ log p(y|zx) ] −KL(q(zx|x)||p(zx)). (31)
Similarly, we have log p(x|y) ≥ Eq(zy|y) [ log p(x|zy) ] −KL(q(zy|y)||p(zy)), (32)
where zy = µy + σy . Here µy and σy are flexible functions on hy, and ∼ N (0, I). For implementation, we take multi-layer perceptrons (MLPs) for µx, µy, σx, σy .
Both the above objectives are composed of a conditional log-likelihood and a KL-divergence. The conditional log-likelihood has also been recognized as the reconstruction term: it is essentially to reconstruct the 3D conformers (y) from the sampled 2D molecular graph representation (zx). However, performing the graph reconstruction on the data space is not easy: since molecules are discrete, modeling and measuring are not trivial.
E.2 VARIATIONAL REPRESENTATION RECONSTRUCTION
To cope with data reconstruction issue, we propose a novel generative loss termed variation representation reconstruction (VRR). The pipeline is in Figure 5.
24
Our proposed solution is very straightforward. Recall that MI is invariant to continuous bijective function [6]. So suppose we have a representation function hy satisfying this condition, and this can guide us a surrogate loss by transferring the reconstruction from data space to the continuous representation space:
Eq(zx|x)[log p(y|zx)] = −Eq(zx|x)[‖hy(gx(zx))− hy(y)‖ 2 2] + C,
where gx is the decoder and C is a constant, and this introduces to using the mean-squared error (MSE) for reconstruction on the representation space.
Then for the reconstruction, current formula has two steps: i) the latent code zx is first mapped to molecule space, and ii) it is mapped to the representation space. We can approximate these two mappings with one projection step, by directly projecting the latent code zx to the 3D representation space, i.e., qx(zx) ≈ hy(gx(zx)). This gives us a variation representation reconstruction (VRR) SSL objective as below:
Eq(zx|x)[log p(y|zx)] = −Eq(zx|x)[‖qx(zx)− hy(y)‖ 2 2] + C.
β-VAE We consider introducing a β variable [36] to control the disentanglement of the latent representation. To be more specific, we would have
log p(y|x) ≥ Eq(zx|x) [ log p(y|zx) ] − β ·KL(q(zx|x)||p(zx)). (33)
Stop-gradient For the optimization on variational representation reconstruction, related work have found that adding the stop-gradient operator (SG) as a regularizer can make the training more stable without collapse both empirically [11, 29] and theoretically [74]. Here, we may as well utilize this SG operation in the objective function:
Eq(zx|x)[log p(y|zx)] = −Eq(zx|x)[‖qx(zx)− SG(hy(y))‖ 2 2] + C. (34)
Objective function for VRR Thus, combining both two regularizers mentioned above, the final objective function for VRR is:
LVRR = 1
2
[ Eq(zx|x) [ ‖qx(zx)− SG(hy)‖2 ] + Eq(zy|y) [ ‖qy(zy)− SG(hx)‖22 ]] + β 2 · [ KL(q(zx|x)||p(zx)) +KL(q(zy|y)||p(zy)) ] .
(35)
Note that MI is invariant to continuous bijective function [6], thus this surrogate loss would be exact if the encoding function hy and hx satisfy this condition. However, we find GNN (both GIN and SchNet) can, though do not meet the condition, provide quite robust performance empirically, which justify the effectiveness of VRR.
E.3 VARIATIONAL REPRESENTATION RECONSTRUCTION AND NON-CONTRASTIVE SSL
By introducing VRR, we provide another perspective to understand the generative SSL, including the recently-proposed non-contrastive SSL [11, 29].
We provide a unified structure on the intra-data generative SSL:
• Reconstruction to the data space, like Equations (5), (31) and (32). • Reconstruction to the representation space, i.e., VRR in Equation (35).
– If we remove the stochasticity, then it is simply the representation reconstruction (RR), as we tested in the ablation study Section 4.4.
– If we remove the stochasticity and assume two views are sharing the same representation function, like CNN for multi-view learning on images, then it is reduced to the BYOL [29] and SimSiam [11]. In other words, these recently-proposed non-contrastive SSL methods are indeed special cases of VRR.

heading: F DATASET OVERVIEW

heading: F.1 PRE-TRAINING DATASET OVERVIEW
text: In this section, we provide some basic statistics of the pre-training dataset (GEOM).
In Figure 6, we plot the histogram (logarithm scale on the y-axis) and cumulative distribution on the number of conformers of each molecule. As shown by the histogram and curves, there are certain number of molecules having over 1000 possible 3d conformer structures, while over 80% of molecules have less than 100 conformers.
In Figure 6, we plot the histogram of the summation of top (descending sorted by weights) {1,5,10,20} conformer weights. The physical meaning of the weight is the portion of each conformer occurred in nature. We observe that the top 5 or 10 conformers are sufficient as they have dominated nearly all the natural observations. Such long-tailed distribution is also in alignment with our findings in the ablation studies. We find that utilizing top five conformers in the GraphMVP has reached an idealised spot between effectiveness and efficiency.

heading: F.2 DOWNSTREAM DATASET OVERVIEW
text: In this section, we review the four main categories of datasets used for downstream tasks.
Molecular Property: Pharmacology The Blood-Brain Barrier Penetration (BBBP) [53] dataset measures whether a molecule will penetrate the central nervous system. All three datasets, Tox21 [75], ToxCast [86], and ClinTox [26] are related to the toxicity of molecular compounds. The Side Effect Resource (SIDER) [46] dataset stores the adverse drug reactions on a marketed drug database.
Molecular Property: Physical Chemistry Dataset proposed in [14] measures aqueous solubility of the molecular compounds. Lipophilicity (Lipo) dataset is a subset of ChEMBL [25] measuring the molecule octanol/water distribution coefficient. CEP dataset is a subset of the Havard Clean Energy Project (CEP) [31], which estimates the organic photovoltaic efficiency.
Molecular Property: Biophysics Maximum Unbiased Validation (MUV) [66] is another subdatabase from PCBA, and is obtained by applying a refined nearest neighbor analysis. HIV is from
the Drug Therapeutics Program (DTP) AIDS Antiviral Screen [93], and it aims at predicting inhibit HIV replication. BACE measures the binding results for a set of inhibitors of β-secretase 1 (BACE-1), and is gathered in MoleculeNet [86]. Malaria [22] measures the drug efficacy against the parasite that causes malaria.
Drug-Target Affinity Davis [13] measures the binding affinities between kinase inhibitors and kinases, scored by the Kd value (kinase dissociation constant). KIBA [73] contains binding affinities for kinase inhibitors from different sources, including Ki, Kd and IC50. KIBA scores [61] are constructured to optimize the consistency among these values.

heading: G EXPERIMENTS DETAILS

heading: G.1 SELF-SUPERVISED LEARNING BASELINES
text: For the SSL baselines in main results (Table 1), generally we can match with the original paper, even though most of them are using larger pre-training datasets, like ZINC-2m. Yet, we would like to add some specifications.
• GraphLoG [89] reports 73.2 in the paper while the downstream tasks are not picking the test performance with optimal validation. If we follow the same evaluation strategy, then the downstream performance (pre-trained on zinc-2m) is about 71.
• G-{Contextual, Motif}[67] proposes a new GNN model for backbone model, and does pre-training on a larger dataset. Both settings are different from us.
• JOAO [91] has two versions in the original paper, and we reported the optimal one.

heading: G.2 ABLATION STUDY: THE EFFECT OF MASKING RATIO AND NUMBER OF CONFORMERS
text: MVP 0 69.4 (1.0) 75.3 (0.5) 62.8 (0.2) 61.9 (0.5) 74.4 (1.3) 74.6 (1.4) 74.6 (1.0) 76.0 (2.0) 71.120.15 68.5 (0.2) 74.5 (0.4) 62.7 (0.1) 62.3 (1.6) 79.0 (2.5) 75.0 (1.4) 74.8 (1.4) 76.8 (1.1) 71.69 0.3 68.6 (0.3) 74.9 (0.6) 62.8 (0.4) 60.0 (0.6) 74.8 (7.8) 74.7 (0.8) 75.5 (1.1) 82.9 (1.7) 71.79
MVP-G 0 72.4 (1.3) 74.7 (0.6) 62.4 (0.2) 60.3 (0.7) 76.2 (5.7) 76.6 (1.7) 76.4 (1.7) 78.0 (1.1) 72.150.15 70.8 (0.5) 75.9 (0.5) 63.1 (0.2) 60.2 (1.1) 79.1 (2.8) 77.7 (0.6) 76.0 (0.1) 79.3 (1.5) 72.76 0.3 69.5 (0.5) 74.6 (0.6) 62.7 (0.3) 60.8 (1.2) 80.7 (2.0) 77.8 (2.5) 76.2 (0.5) 81.0 (1.0) 72.91
MVP-C 0 71.5 (0.9) 75.4 (0.3) 63.6 (0.5) 61.8 (0.6) 77.3 (1.2) 75.8 (0.6) 76.1 (0.9) 79.8 (0.4) 72.660.15 72.4 (1.6) 74.4 (0.2) 63.1 (0.4) 63.9 (1.2) 77.5 (4.2) 75.0 (1.0) 77.0 (1.2) 81.2 (0.9) 73.07 0.3 70.7 (0.8) 74.6 (0.3) 63.8 (0.7) 60.4 (0.6) 83.5 (3.2) 74.2 (1.6) 76.0 (1.0) 82.2 (2.2) 73.17
MVP 1 69.2 (1.0) 74.7 (0.4) 62.5 (0.2) 63.0 (0.4) 73.9 (7.2) 76.2 (0.4) 75.3 (1.1) 78.0 (0.5) 71.615 68.5 (0.2) 74.5 (0.4) 62.7 (0.1) 62.3 (1.6) 79.0 (2.5) 75.0 (1.4) 74.8 (1.4) 76.8 (1.1) 71.69 10 68.3 (0.5) 74.2 (0.6) 63.2 (0.5) 61.4 (1.0) 80.6 (0.8) 75.4 (2.4) 75.5 (0.6) 79.1 (2.3) 72.20 20 68.7 (0.5) 74.9 (0.3) 62.7 (0.3) 60.8 (0.7) 75.8 (0.5) 76.3 (1.5) 77.4 (0.3) 82.3 (0.8) 72.39
MVP-G 1 70.9 (0.4) 75.3 (0.7) 62.8 (0.5) 61.2 (0.6) 81.4 (3.7) 74.2 (2.1) 76.4 (0.6) 80.2 (0.7) 72.805 70.8 (0.5) 75.9 (0.5) 63.1 (0.2) 60.2 (1.1) 79.1 (2.8) 77.7 (0.6) 76.0 (0.1) 79.3 (1.5) 72.76 10 70.2 (0.9) 74.9 (0.4) 63.4 (0.4) 60.8 (1.0) 80.6 (0.4) 76.4 (2.0) 77.0 (0.3) 77.4 (1.3) 72.59 20 69.5 (0.4) 74.9 (0.4) 63.3 (0.1) 60.8 (0.3) 81.2 (0.5) 77.3 (2.7) 76.9 (0.3) 80.1 (0.5) 73.00
MVP-C 1 69.7 (0.9) 74.9 (0.5) 64.1 (0.5) 61.0 (1.4) 78.3 (2.7) 75.7 (1.5) 74.7 (0.8) 81.3 (0.7) 72.465 72.4 (1.6) 74.4 (0.2) 63.1 (0.4) 63.9 (1.2) 77.5 (4.2) 75.0 (1.0) 77.0 (1.2) 81.2 (0.9) 73.07 10 69.5 (1.5) 74.5 (0.5) 63.9 (0.9) 60.9 (0.4) 81.1 (1.8) 76.8 (1.5) 76.0 (0.8) 82.0 (1.0) 73.09 20 72.1 (0.4) 73.4 (0.7) 63.9 (0.3) 63.0 (0.7) 78.8 (2.4) 74.1 (1.0) 74.8 (0.9) 84.1 (0.6) 73.02

heading: G.3 ABLATION STUDY: EFFECT OF EACH LOSS COMPONENT

heading: G.4 BROADER RANGE OF DOWNSTREAM TASKS: MOLECULAR PROPERTY PREDICTION PREDICTION

heading: G.5 BROADER RANGE OF DOWNSTREAM TASKS: DRUG-TARGET AFFINITY PREDICTION

heading: AM 0.291 (0.007) 0.203 (0.003) 0.24730

heading: G.6 CASE STUDIES
text: Shape Analysis (3D Diameter Prediction). Diameter is an important measure in molecule [52, 54], and genome [20] modelling. Usually, the longer the 2D diameter (longest adjacency path) is, the larger the 3D diameter (largest atomic pairwise l2 distance). However, this is not always true. Therefore, we are particularly interested in using the 2D graph to predict the 3D diameter when the 2D and 3D molecular landscapes are with large differences (as in Figure 2 and Figure 8). We formulate it as a n-class recognition problem, where n is the number of class after removing the consecutive intervals. We provide numerical results in Table 14 and more visualisation examples in Figure 9.
Long-Range Donor-Acceptor Detection. Donor-Acceptor structures such as hydrogen bonds have key impacts on the molecular geometrical structures (collinear and coplanarity), and physical properties (melting point, water affinity, viscosity etc.). Usually, atom pairs such as “O...H” that are closed in the Euclidean space are considered as the donor-acceptor structures [42]. On this basis, we are particularly interested in using the 2D graph to recognize (i.e., binary classification) donor-acceptor
29

heading: 38.9 (0.8) 37.6 (0.6) 41.2 (0.7) 39.2 (1.1) 38.7 (2.0) 41.3 (1.2) 42.3 (1.9) 41.9 (0.7) 42.3 (1.3)
text: structures which have larger ranges in the 2D adjacency (as shown in Figure 2). Similarly, we select the molecules whose donor-acceptor are close in 3D Euclidean distance but far in the 2D adjacency. We provide numerical results in Table 15. Both tables show that MVP is the MVP :)

heading: 77.9 (1.1) 78.6 (0.3) 80.0 (0.5) 77.5 (0.9) 79.9 (0.7) 79.2 (1.0) 80.0 (0.4) 81.5 (0.4) 80.7 (0.2)
text: Chirality. We have also explored other tasks such as predicting the molecular chirality, it is a challenging setting if only 2D molecular graphs are provided [63]. We found that GraphMVP brings negligible improvements due to the model capacity of SchNet. We save this in the ongoing work.

Review summaries:
This paper presents a pre-training method of GNNs for molecular graphs by self-supervised learning (SSL) on not only 2D topologies of graphs but also 3D geometries of molecules.  In particular, the paper develops two SSL pretext tasks learning inter-molecule and intra-molecule associations in 2D and 3D. The "inter-molecule" task is a graph classification on whether a 2D, 3D graph pairs come from the same molecule or not, while the "intra-molecule" task is a generative task to generate 2D from 3D as well as 3D from 2D information. Empirical evaluations over several datasets demonstrate that jointly learning these two SSL tasks brings nice improvement in prediction accuracies.

This work aims to leverage additional 3D geometric information for molecular graph representation learning and proposes a multi-view pre-training framework, GraphMVP. Specifically, in GraphMVP, both 2D and 3D information are used, and a combined loss function (i.e., the combination of contrastive self-supervised learning loss and generative self-supervised learning loss ) is adopted to enhance the quality of representations. To verify the model's effectiveness, they conducted experiments and comparisons on a number of datasets. Ablation studies are also carried out to inspect the inner workings of GraphMVP.

This paper proposes to leverage 3D information to pretrain graph neural networks for learning molecular representations. To achieve this, contrastive and generative SSL strategies are developed accordingly. The empirical results on several datasets show that the improvements of this method over previous 2D graph pretraining methods are consistent and obvious.