Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching
Authors: Shengchao Liu
Hongyu Guo
Jian Tang
Venue: ICLR 2023 poster (2023)
Keywords: molecule
pretraining
representation
geometry
denoising score matching

Abstract: Molecular representation pretraining is critical in various applications for drug and material discovery due to the limited number of labeled molecules, and most existing work focuses on pretraining on 2D molecular graphs. However, the power of pretraining on 3D geometric structures has been less explored. This is owing to the difficulty of finding a sufficient proxy task that can empower the pretraining to effectively extract essential features from the geometric structures. Motivated by the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space forms a smooth potential energy surface, we propose GeoSSL, a 3D coordinate denoising pretraining framework to model such an energy landscape. Further by leveraging an SE(3)-invariant score matching method, we propose GeoSSL-DDM in which the coordinate denoising proxy task is effectively boiled down to denoising the pairwise atomic distances in a molecule. Our comprehensive experiments confirm the effectiveness and robustness of our proposed method.

Published as a conference paper at ICLR 2023
MOLECULAR GEOMETRY PRETRAINING WITH SE(3) -IVAAIANA DENOISING DISTANCE MATCHING
Shengehao Liu
Hongyu Guo
Jian Tang
Mila Québec AI Institute
National Research Council Canada University of Ottawa hongyu. guo uottawa.ca
Mila Québec Al Institute
Université de Montréal liusheng @ mila. quebec
HEC Montréal
CIFAR Al Chair
jian.tang @ hec.ca
ABSTRACT
Molecular representation pretraining is critical in various applications for drug and material discovery due to the limited number of labeled molecules, and most ex- isting work focuses pretraining 2D molecular graphs. However, the power of pretraining on 3D geometric structures has been less explored. This is owing to the difficulty of finding sufficient proxy task that can empower the pretraining ef- fectively extract essential features from the geometric structures. Motivated by the dynamic nature of 3D molecules. where the continuous motion of a molecule in the 3D Euclidean space forms a smooth potential energy surface, we propose GeoSSL, a 3D coordinate denoising pretraining framework to model such an energy land- scape. Further by leveraging an SE(3)-invariant score matching method, we propose GeoSSL-DDM which the coordinate denoising proxy task effectively boiled down to denoising the pairwise atomic distances in molecule. Our comprehensive experiments confirm the effectiveness and robustness of our proposed method.
INTRODUCTION
Learning effective molecular representations is critical in variety of tasks in drug and material dis- covery, such as molecular property prediction [14, 20, 21, 74], de novo molecular design and optimiza- tion [7,36,3 37, 33, 77], and retrosynthesis and reaction planning [4, 22, 52, 64). Recent work based on graph neural networks (GNNs) [20] has shown superior performance thanks the simplicity and effectiveness of GNNs in modeling graph-structure data. However, the problem remains challenging due to the limited number of labeled molecules as it is in general expensive and time-consuming to label molecules, which usually requires expensive physics simulations or wet-lab experiments.
As result, recently, there has been growing interest in developing pretraining or self-supervised learning methods for learning molecular representations by leveraging the huge amount of unlabeled molecule data [28, 35, 63, 75]. These methods have shown superior performance on many tasks, especially when the number of labeled molecules is insufficient. However, one limitation of these approaches is that they represent molecules as topological graphs, and molecular representations are learned through pretraining 2D topological structures (i.e., based on the covalent bonds). But intrinsi- cally, for molecules, a more natural representation is based on their 3D geometric structures, which largely determine the corresponding physical and chemical properties. Indeed, recent works [20, 38] have empirically verified the importance of applying 3D geometric information for molecular property prediction tasks. Therefore, a more promising direction is to pretrain molecular representations based on their 3D geometric structures, which is the main focus of this paper.
The main challenge for molecule geometric pretraining arises from discovering an effective proxy task to empower the pretraining to extract essential features from the 3D geometric structures. Our proxy task motivated by the following observations. Studies [48] have shown that molecules are not static but in continuous motion in the 3D Euclidean space, forming potential energy surface (PES). As shown in Figure 1, it is desirable to study the molecule in the local minima of the PES, called conformer. However, such stable state conformer often comes with different noises for the following reasons. First, the statistical and systematic errors in conformation estimation unavoidable [11]. Second, it has been well-acknowledged that conformer have vibrations

---

Published as a conference paper at ICLR 2023
around the local minima in PES. Such characteristics of the molecular geometry motivate to attempt to denoise the molecular coordinates around the local minima, to mimic the computation errors and conformation vibration within the corresponding local region. The denoising goal is to learn molecular representations that are robust to such noises and effectively capture the energy surface around the local minima.
To achieve the aforementioned goal, we first introduce a general geometric self-supervised learning framework called GeoSSL. Based on this, we fur- ther propose an SE(3)-invariant de- noising distance matching pretraining algorithm, GeoSSL-DDDM. In nut- shell, to capture the smooth energy surface around the local minima, we aim to maximize the mutual informa- tion (MI) between given stable ge- ometry and its perturbed version (i.e., gi and 92 in Figure 1). In practice, is difficult to directly maximize the mutual information between two ran- dom variables. Thus, we propose to maximize an equivalent lower bound of the above mutual information, which amounts to a pretraining framework on denoising geometric structure, coined GeoSSL. Moreover, directly denoising such noisy coordinates remains challenging because one may need to effectively constrain the pairwise atomic distances while changing the atomic coordinates. To cope with this obstacle, we further leverage SE(3)-invariant score matching method, GeoSSL- DDM, to successfully transform the coordinate denoising desire to the denoising of pairwise atomic distances, which then can be effectively computed. In other words, our pretraining proxy task, namely mutual information maximization, effectively boils down to achicving an intuitive learning objec- tive: denoising molecule's 's pairwise atomic distances. Using 22 downstream geometrie molecular prediction tasks, we empirically verify that our method outperforms nine pretraining baselines.
Potential Energy Surface
8 Original Geometry
8: Perturbed Geometry
Perturbation Local Minims
82
Figure 1: Illustration on coordinate geometry of molecules. The molecule is in continuous motion, forming potential energy surface (PES), where each 3D coordinate (x-axis) corresponds to an energy value (y-axis). The provided molecules, í.e., conformers, are in the local minima (g1). It often comes with noises around the minima (e.g., statistical and systematic errors vibrations), which be captured using the perturbed geometry (ga).
Our main contributions summarized follows. (1) We propose novel geometric self supervied learning framework, GeoSSL. To the best of our knowledge, it is the first pretraining framework focusing on the pure 3D molecular data !. (2) To overcome the challenge of attaining the coordinate denoising objective in GeoSSL, we propose GeoSSL-DDMM, an
score matching strat- egy to successfully transform such objective into the denoising of pairwise atomic distances. (3) We empirically demonstrate the effectiveness and robustness of GeoSSL-DDM on 22 downstream tasks.
2 RELATED WORK
2.1 EQUIVARIANT GEOMETRIC MOLECULE REPRESENTATION LEARNING
Geometric representation learning. Recently, 3D geometric representation learning has been widely explored in the machine learning community, including but not limited 3D point clouds [8, 44, 55, 67]. N-body particle [45, 47], and 3D molecular conformation [6, 31, 32, 41, 50, 51, 56], amongst many others. The learned representation should satisfy the physical constraints, e.g., it should be equivariant to the rotation and transition in the 3D Fuclidean space. Such constraints can be described using group symmetry introduced below.
SE(3)-invariant energy. Constrained by the physical nature of 3D geometric data, a key principle we need follow is to learn an -Eu3)-equivariant representation function. The SE(3) is the special Eu- clidean group consisting of rigid transformations in the 3D Cartesian space, where the transformations include all the combinations of translations and rotations. Namely, the learned representation should be equivariant to translations and rotations for molecule geometries. We also note that the represen- tation function needlessly satisfies the reflection equivariance for certain tasks like molecular chiral- ity [1]. For more rigorous discussion, please check [17, 19, 65). In this work, we will design an SE(3)- invariant energy (score) function in addition to the SE(3) equivarraa representation backbone model.
During the rebuttal of our submission, one of the reviewers pointed us to this parallel work [76], which is also under review. We provide a detailed comparison with this work in Appendix G.
2

---

Published as a conference paper at ICLR 2023
2.2 SULERVSSER LEARNING FOR MOLECULE REPRESENTATION LEARNING
In general, there are two categories of self-supervised learning (SSL) [40, 42, 71, 72]: contrastive and generative, and the main difference is if the supervised signals are constructed in inter-data or intra-data manner. Contrastive SSL extracts two views from the data and determines the supervised signals by detecting whether the sampled view pairs are from the same data. Generative SSL learns structural information by reconstructing partial information from the data itself.
2D molecular graph (topology) sell-supervised learning. One of the mainstream research lines for molecule pretraining is on the 2D molecular graph. It treats the molecules as 2D graphs, where atoms and bonds are nodes and edges, respectively. It then carries out a pretraining task by either detecting if the two augmentations (e.g. neighborhood extraction, node dropping, edge dropping, etc) correspond to the same molecular graph [28, 63, 75] or if the representation can successfully reconstruct certain substructures of the molecular graphs [28, 35].
3D molecular graph (geometry) self-supervised learning. Self-supervised learning for 3D molec- ular graphs is still underexplored. The only related works are [16, 38], which leverage both 2D topology and 3D conformation to improve the molecule representation learning. For example, GEMMRL-GEM [16] designs novel model using 2D and 3D molecular graphs. Regarding SSL, it utilizes the geometry information by conducting distance and angle prediction as the generative pretraining tasks. GraphMVP [38] introduces extra 2D topology and employs detection and reconstruction tasks simultaneously between 2D and 3D graphs, yet it focuses on 2D downstream tasks due the small scale of the pretraining dataset. To the best of our knowledge, our work is the first to explicitly do SSL on pure 3D geometry along the molecule representation learning research line. We note that there a parallel work [76], which is also under review; Appendix E provides a detailed comparison, highlighting the fact that the parallel work is special case of GeoSSL-DDM.
3 PRELIMINARIES
Molecular geometry graph. Molecules can be naturally featured in geometric formulation, i.e. all the atoms are spatially located in 3D Euclidean space. Note that the covalent bonds are added heuristically by expert rules, so they are only applicable in 2D topology graphs. Besides, atoms are not static but in a continual motion along a potential energy surface [2]. The 3D structures at the local minima on this surface are named conformer, as shown in Figure 1. Conformers at such an equilibrium state possess nice properties, and we would like to model them during pretraining.
Geometric neural network. We denote each conformer as g (X, R). Here X E Rnxd is the atom attribute matrix and R E Rx3 is the atom 3D-coordinate matrix, where n is the number of atoms and d is the feature dimension. The representations for the i-th node and whole molecule are:
h, GNN)3D(T(g)); GNN-3D(T(X... R)); ht (EA.
(ho.
(1)
where T is the transformation function like atom masking, and READOUT is the readout function. In this work, we take the mean over all the node representations as the readout function.
Energy-based model and denoising score matching. Energy-based model (EBM) is flexible and powerful tool for modeling data distribution. It has the form of Gibbs distribution as (a) = exp(-Eiz))/A4 where (x the model distribution and A denotes the normalization constant. The computation of such probability is intractable due to the high cardinality of the data space. Recently, great progress has been made in solving this intractable function, including contrastive divergence [12], noise contrastive estimation [25], and score matching (SM) [30. 60, 61]. For example, SM solves this by first introducing the concept score, the gradient of the log- likelihooh with respect to the data, and then matching the model score with the data score using Fisher divergence. This approach has been further improved by combining SM with denoising auto-encoding. forming the promising denoising score matching (DSM) strategy [69]. In this work, we will explore the potential of leveraging DSM for molecule geometry representation learning. We aim to utilize pairwise distance information, one of the most fundamental factors in the geometric molecule data.
Problem setup. Our goal here is to apply a self-supervised pretraining algorithm on a large molecular geometric dataset and adapt the pretrained representation for fine-tuning geometric downstream tasks. For both the pretraining and downstream tasks, only the 3D geometric information is available, and our solution is agnostic in terms of the backbone geometric neural network.
3

---

Published as a conference paper at ICLR 2023
4 METHOD
This section first introduces the GeoSSL framework and then proposes the GeoSSL-DDM algorithm. We start with exploring the coordinate perturbation for molecular data in Section 4.1. Then introduce a coordinate-aware mutual information (MI) maximization formula and turn it into a coordinate denoising framework in Section 4.2. Nevertheless, the coordinate denoising is non-trivial since it requires geometric data reconstruction, and we adopt the score matching for estimation, as proposed in Section 4.3. The ultimate training objective is discussed in Section 4.4.
4.1 COORDINATE PERTURBATION FOR GEOMETRIC DATA
The mainstream self-supervised learning community designs the pretraining task by defining multiple views from the data, and these views share common information some degree. Thus. by designing generative or contrastive tasks to maximize the mutual information (MI) between these views, the pre- trained representation can encode certain key information. This will make the representation more ro- bust and more generalizable to downstream tasks. In our work, we propose GeoSSL-DDM, an SE(3)- invariant self-supervised learning (SSL) method for molecule geometric representation learning.
The 3D geometric information or the atomic coordinates are critical to molecular properties. We carry an additional ablation study verify this in Appendix B. Then based on this acknowledgment, we introduce geometry perturbation, which adds small noises to the atom coordinates. For notation, following Section 3, we define the original geometry graph and an augmented geometry graph as two views, denoted as 9I (X1, R1) and 92 (X2, R2), respectively. The augmented geometry graph can be seen as a coordinate perturbation to the original graph with the same atom types, i.e., X2 X1 and R2 = R + where is drawn from normal distribution.
4.2 COORDINATE DENOISING WITH MI MAXIMIZATION FRAMEWORK: GEOSSL.
The views defined above share certain common information. By maximizing the mutual informa- tion (MI) between them, we expect that the learned representation can better capture the geometric information and is robust to noises and thus can generalize well downstream tasks. To maximize the MI, we turn to maximize the following lower bound on the two geometry views, leading to the geometric self supervieee learning framework, GeoSSL:
1
Liiessi
Bptan .92) logp())lo>) + logr(ozl11)
(2)
In Equation (2), we transform the MI maximization problem into maximizing the summation of two conditional log-likelihoods. In addition, these two conditional log-likelihoods are in the mirroring di- rection, and such symmetry can reveal certain nice properties, e.g., highlights the equal importance and uncertainty of the two views and can lead to more robust representation of the geometry.
To solve Equation (2), we adopt the energy-based model (EBM) for estimation. EBM has been acknowledged flexible framework for its powerful usage in modeling distribution over highly- structured data, like molecules [27, 34]. To adapt it for GeoSSL, the objective can be turned into:
log Dogr(R) (Ri))))) 1
LGoosSI E P2))1992)
Fr)) oz) log p( logg(Ralg)))
exp(s(Hi, 92)) Pizz000) [Tog exp( g:))
log
ARi
ARylon
where the f(-) are the negative of energy functions, and AR. and A the intractable partition functions. The first equation in Equation (3) is because the two views share the same atom types. This equation can be treated as denoising the atom coordinates of one view from the geometry of the other view. In the following, we will explore how to use the score matching for solving the above EBM estimation problem, and further transform the coordinate aware GeoSSL the denoising distance matching as the final objective.
4.3 FROM COORDINATE DENOISING TO DISTANCE DENOISING: GEOSSL-DDM
Before going into details, first, would like briefly discuss denoising matching (DSM). DSM has three main advantages that inspire us to apply it for solving the coordinate-awei aware GeoSSL. (1) The DSM solution has nice formulation, such that the final objective function can be simplified
4

---

Published as a conference paper at ICLR 2023
g1
d
Step 1: Distance Extraction
Step Distance Perturbation
82
Step 3: Distance Denoising
Figure 2: Pipeline for GeoSSL-DDM. The 9i and ga around the same local minima, yet with coordinate noises perturbation. Originally we want to conduct coordinate denoising between these two views. Then as proposed in GeoSSL-DDM, we transform to an equivalent problem, ie.. distance denoising. This figure shows the three key steps: extract the distances from the two geometric views, perform distance perturbation, and denoise the perturbed distances. Notice that the covalent bonds in the 3D data are added for illustration only.
with an intuitive explanation: GeoSSL-DDM can be seen as solving the denoising pairwise distance at multiple noise levels. (2) The score defined in geometric data can be viewed as a coordinate- based pseudo-force. Such pseudo-force can play an important role in the corresponding geometric representation learning. (3) In terms of the MI maximization, existing methods like InfoNCE [68], EBM-NCE, and Representation Reconstruction [38] map the data the representation space for either inter-data contrastive learning or intra-data reconstruction. This operation can avoid the decoding design issue for highly-structured data [13], yet the trade-off is losing the data-inherent information by a certain degree. In other words, the data-level reconstruction task (e.g.. DSM) is expected lead more robust representation. Thus, considering the above points, we adopt DSM to our framework and propose GeoSSL-DDM, We expect that it can learn an expressive geometric representation function by solving the coordinate- aware GeoSSL. Additionally, the two terms in Equation (3) are in the mirroring direction. Thus in what follows, we adopt a proxy task that calculate the two directions separately, and take for illustration, egg. log exp(f(Rn.ga) AR
4.3.1 DENOISING DISTANCE MATCHING
Score. The score defined as the gradient of the log-likelihool w.r.t. the data, i.e., the atom coordinates in our case. Because the normalization function is constant regarding the data, it will disappear during the score calculation. To adapt into our setting, the score is obtained as gradient of the negative energy function w.r.t. the atom coordinates, as:
s(R, 92) VRi l(g (92) J
(4)
If we assume that the learned optimal energy function, i.e., f-)), possesses certain physical or chemical information, then the score Equation (4) can be viewed special form of the pseudo-force. This may require more domain-specift, knowledge, which we leave for future exploration.
Score decomposition: from coordinates distances. Through propaaaaoonnnonnn [54], the score on atom coordinates can be further decomposed into the scores attached to pairwise distances:
g2) 0f(R1.g2)
Ad1.3 = -Ed,
dr.ss -sdd,, 92)s (ri, 1..--11.) ri.);
0d1,s
Ori.:
$$1
where F1,a the i-th coordinate in 91, d1,ú denotes the pairwise distance between the i-th and j-th af(R1.92). Such decomposition has nice intuition from the pseudo- the summation of
nodes in 91. and s(d1,g2) force perspective: the pseudo-force on each atom can be further decomposed pseudo-forces attached to the pairwise distances between this atom and all its neighbors. Note that here the pairwise atoms connected in the 3D Euclidean space, not by the covalent bonds.
0di.'s
Denoising distance matching (DDM). Then we adopt the denoising score matching (DSM) [69] to task. To be more concrete, take the Gaussian kernel the perturbed noise distribution each

---

Published as a conference paper at ICLR 2023
pairwise distance, i.e., 90 (d11g2)) = B pan 192) (d1 [1]], where Ø is the deviation in Gaussian perturbation. One main advantage of using the Gaussian kernel is that the following gradient of conditional log-likelihood has a closed-form formulation: Vd, log 9- lo (d] (1, 92) = (dj di)/o?, and the objective function of DSM is train score network to match it. This trick first introduced in [69]. and has been widely utilized in deep generative modeling tasks [58, 59].
To adapt to our setting, this is essentially saying that we want to train a score network, ie., so (d) l92). to match the distance perturbation, pairwise distances from the pseudo-force aspect. By taking the Fisher divergence as the discrepancy metric and the trick mentioned above, the estimation objective can be simplified
can say aims at matching the pseudo-foree with the
Dr (go (di (x))llpedd (=))) = (11 lh (d) (2-92) [[ss 92) di di
1
+C
(6)
2
σ²
For more detailed derivations, please refer to Appendix C. this section, we turn the coordinate-awan GeoSSL framework into a distance perturbation matching problem, which is equivalent to denoising distance matching, i.e., GeoSSL-DDM. The corresponding pipeline is illustrated in Figure
4.3.2 SE(3)-INVARIANT SCORE NETWORK MODELING
The objective function in Equation (6) is essentially doing the distance denoising. Since the distance is type-0 feature [65], we simply design SE(3)-invarias score network sg(+). ). For modeling ht-). we take an SE(3)-eqiivarian 3D geometric graph neural network as the geometric representation backbone model. Following the notations in Section 3 and 92 modeling, we have h(gz), 3D-GNN(T(g2))),
h(g2) h(g2): + h(g2));
for the atom-level and atom pairwise- representation. Then we define the score network as: so sa g2);; = MLP(MMP(dd..)) @ h(g:)<))
(8)
where @ is the concatenation and MI.P is the multi-layer perception. GeoSSL-DDM is agnostic to the backbone geometric representation function, and its main module the score network Equation (8). Thus, GeoSSL-DDM is an SE(3)-invarian) [19] pretraining algorithm. Meanwhile, the type-0 distance can be modeled in more expressive SE(3)-qquivariant manner, and we leave that for future work.
4.4 ULTTIMATE OBJECTIVE
With the above score network modeling, we can formulate the ultimate objective function. We adopt the following four training tricks from [38, 58, 59] to stabilize the score matching training process. (1) We carry out the distance denoising at L-level of noises. (2) We add a weighting coefficient X(a)= = の for each noise level, where acts the annealing factor. (3) We scale the score network by a factor of 1/o. (4) We sample the same atoms from the two geometry views with a masking ratio
Ultimately, the objective function for GeoSSL-DDM,
follows:
80 (o g2)
di di
?7
LGeuSSL-DDD DDM
Bpun (1112) (d oz)
2L
o²
2
(9)
80 (d 91)
d2 dg 2
+
0 Epan (d2 01 (dds)da.n)))
2L
of
The algorithm in Algorithm
Algorithm 1 GeoSSL-DDM pretraining
Comparison with score match-
Input: 3D geometry dataset and levels Gaussian noise.
ing in generative modeling. We
2: Output: pre-trained 3D representation function k(-).
note that score matching has
3:
for cach 3D geometry graph 91 do
been widely used for genera-
4: Obtain g2 by adding Gaussian noises to atom coordinates g1. for each noise level E [1, [1, L} do
5:
tive modeling tasks. One of the main drawbacks in the generative setting is the long mixing time for MCMC sampling. However,
6:
Add noise to pairwise distance with d, = d1 + at de da +aj- Get the score so(d; .2), so (d2. (d2.9)) with Equutionn (8) accordingly. end for
7:
8:
9: Update 3D GNN representation function h(-) using Equation (9).
our work aims at representation 10:
end for
learning, so such a sampling issue will not affect our task. We further note that there also exists a series of works exploring the score matching for conformation generation [54]. However, their scores or srcudo-forces are attached to the 2D topology (the covalent bonds), while our work is for the pure geometric data and is attached the pairwise distances defined in the 3D Fuclidean space.

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTS
In this section, we compare our method with nine 3D geometric pretraining baselines, including one randomly initialized, one supervised, and seven self-supervised approaches. For the downstream tasks, we adopt 22 tasks covering quantum mechanics prediction, force prediction, and binding affinity prediction. We provide all the experiment details and ablation studies in Appendix D.
5.1 BACKBONE MODELS
Our proposed GeoSSL-DDM is model-agnostic, and here we evaluate our method using one of the tate-offthhe-rt geometric graph neural networks, PaiNN [51]. We carry out the exact same experiments on another backbone model, SchNet [49], and present the results in Appendix D.
PaiNN [51] is follow-up work of SchNet [49]. It addresses the limitation of rotational equivariance in SchNet by embracing rotational invariance, attaining a more expressive 3D geometric model.
Other backbone models. First, we want to highlight that what we propose is a general solution and agnostic to the backbone 3D geometric models. And addition to the PaiNN model, we want to acknowledge that, recently, there have been several works along this research line, including but not limited to [6, 17, 17, 32, 41, 47, 56]. Yet, they may require large computation resources and may be infeasible (e.g., out of GPU memory) in our setting. The decision is made by considering the model performance, computation efficiency, and memory cost. For more benchmark results and detailed comparisons of the 3D geometric models, please check Appendix A.
5.2 BASELINES AND PRETRAINING DATASET
Pretraining dataset. The PubChemQC database is large-scale database with around 4M molecules with 3D geometries, and it calculates both the ground-state and excited-state 3D geometries using DFT (density functional theory). Due to the high computational cost, only several thousand molecules can be processed every day, and this dataset takes years of effort in total. Following this, Molecule3D [73] takes the ground-state geometries from PubChemQC and transforms the data formats into deep learning-friee friendly way. It also parses essential quantum properties for each molecule, including energies of the highest occupied molecular orbital (HOMO) and the lowest occupied molecular orbital (LUMO), the energy gap between HOMO-LUMO, and the total energy. For our molecular geometry pretraining, we take subset of 1M molecules with 3D geometries from Molecule3D.
Self-supervised learning pretraining baselines. We first consider the four
SSL methods: (1) Type Prediction is to predict the atom type of masked atoms; (2) Distance Prediction aims to predict the pairwise distances among atoms; (3) Angle Prediction is to predict the angle among triplet atoms, i.e., the bond angle prediction; (4) 3D InfoGraph adopts the contrastive learning paradigm by taking the node-grae gode-graph pair from the same molecule geometry as positive and negative otherwise. Next, following the coordinate-awan GeoSSL framework introduced in Equation (2), include two contrastive and one generative SSL. baselines. (5) GeoSSI--IfoNCE [68] and (6) GeoSSI-EBBM-NCH [38] are the two widely-used contrastive learning loss functions, where the goal align the positive views and contrast the negative views simultaneously. Finally, (7) GeoSSL-RR (RR for Representation Reconstruction) [38] is a generative SSL that is a proxy to maximize the MI. RR more general form of non non-contras SSL methods like BOYL [23] and SimSiam [9], and the goal is to reconstruct each view from its counterpart in the representation space. Following this, our proposed GeoSSL-DDM, can be classified as generative SSL for distance denoising.
Supervised pretraining baseline. We also compare our method with a supervised pretraining baseline. aforementioned, the large-scale pretraining dataset uses the DFT to calculate the energy and extracts the most stable conformers with the lowest energies, which reveal the most fundamental properties of molecules in the 3D Euclidean space. Thus, such energies can be naturally adopted as supervised signals, and we take this supervised pretraining baseline.
5.3 DOWNSTREAM TASKS ON QUANTUM MECHANICS AND FORCE PREDICTION
QM9 [46] is a dataset of 134K molecules consisting of 9 heavy atoms. It includes 12 tasks that related to the quantum properties. For example, UO and U298 are the internal energies at OK at OK and 298.1 15K respectively, and U298 and G298 are the other two energies that can be transferred from
7

---

Published as a conference paper at ICLR 2023
Table Downstream results on quantum mechanies prediction tasks from QM9, We take 110K for training.
10K for validation, and 1 11K for test. The evaluation is mean absolute error, and the best results are in bold.
Pretraining
Alpha
Gup
HOMOO]
LUMO 1
Mu L
Cv
G298 4
11298.
R2
U298 4
UD
Zpve
21.11
0.025
7.67
0.132
7.77
7.89
1.322
Supervised
0.049
45.33
26.61
21.77
0.016
0.026
8.97
8.59
0.170
8.35
8.19
1.346
Type Prediction
0.050
47.28
30.56
23.18
0.016
0,024
9.32
9.10
0.163
8.94
8.60
1.157
Distance Prediction
0.063
47.62
29.18 18
22.40
0.019
0.045
12.02
12.31
0.636
11.76
12.22
1.840
Angle Prediction
0.027
10.13
0.143
9.70
1.643
3D InfoGraph
0.053
44.79
27.09
21.66
0.016
0.027
9.22
8.78
0.143
8.94
9.11
1.465
GeoSS1 -RR
0.048
44.85
25.42
20.82
0.015
0.025
8.56
8.20
0.133
7.89
7.62
1.329
0.052
45.65
26.70
21.87
0.016
0.027
9.17
9.62
0.130
8.77
8.63
1.519
GeoSS1. EBM NCE
0.049
44.18
21.46
0.026
0.126
7.96
1.447
GooSSL-DDM (ours)
0,046
40.22
23,48
19.42
0,015
0,024
7.65
7.09
0.122
6.99
6.92
1.307
Table 2:
Downstream results on 8 force prediction tasks from MD17. We take 1K for training, IK for validation, and the number of molecules for test are varied among different tasks, ranging from 48K to 991K. The evaluation mean absolute error, and the best results are bold.
Pretraining
Aspirin 4
Benzene
Fihanol + Malonuldehydk...
Naphthalene +
Salicylic L
Toluene
Uracil
0.556
0.052
0.213
0.338
0.138
0.288
0.155
0.194
Supervised
0.478
0.145
0
0.434
0.460
0.527
0
0,404
Type Prediction
I 1,656
0.349
0.414
0.886
1.684
1.807
0.660
1,020
Distance Prediction
1.434
0.090
0.378
1.017
1.569
0.350
0.415
Angle Prediction
0.839
0.105
0.337
0.517
0.772
0.931
0.274
0.676
3D InfoGraph
0.844
0.114
0.344
0.741
1,062
0.945
0.373
0.812
GeoSSL-RRC
0.502
0.052
0.219
0.334
0.312
0.192
JefSNIIIn InfoNCE
0,881
0.066
0.275
0.550
0.356
0.607
0.186
0.559
GeoSSL-EBM-NCE
0.598
0.073
0.237
0.518
0,246
0.416
0,178
0.475
GeoSSL-DDM (ours)
0.453
0.051
0.166
0,288
0.129
0,266
0,122
0,183
H298 respectively. The other 8 tasks are quantum mechanics related to the DFT process. MD17 [10] is a dataset on molecular dynamics simulation. It includes eight tasks, corresponding to eight organic molecules, and each task includes the molecule positions along the potential energy surface (PES), as shown in Figure 1. The goal is to predict the energy conggyccnn interatomic forces for each atom in each molecule position. We follow the literature [31, 41, 50, 51] of using 1K for training and IK for validation, while the test set (from 48K to 991 K) is much larger.
The results on QM9 and MD17 are displayed Tables and respectively. From Tables and we can observe that most the pretraining baselines tested perform on par with or even worse than the randomly-inittalizzd baseline. The top performing baseline is the representation reconstruction method (RR), which optimizes the coordinate-aware MI; it outperforms the other baselines on 5 out of 12 tasks in QM9 and out of tasks in MD17. This implies the potential of applying generative SSL for maximizing this coordinate- aware MI. Promisingly, achieves consistently improved performance on all 12 tasks in QM9 and 8 tasks in MD17. All these observations empirically verify the effectiveness of the distance denoising in GeoSSL-DDM, which models the most determinant factor in molecule geometric data.
proposed GeoSSL-DDM.
5.4 DOWNSTREAM TASKS ON BINDING AFFINITY PREDICTION
Atom3D [66] is a recently published dataset. It gathers several tasks for 3D molecules, including binding affinity. The binding affinity prediction is to measure the strength of binding interaction between a small molecule to the target protein. Here we will model both the small molecule and protein with their 3D atom coordinates provided. We follow Atom3D in data preprocessing and data splitting. For more detailed discussions and statistics, please check Appendix D.
During the binding process, there is a cavity in protein that can potentially possess suitable properties for binding small molecule (ligand), and it is termed a pocking [62]. Because of the large volume of the protein, we follow [66] by only taking the binding pocket, where there are no more than 600 atoms for each molecule and protein pair. To more concrete, we consider two binding affinity tasks. (1) The first task is ligand binding affinity (LBA). It is gathered from [70] and the task is to predict the binding affinity strength between a small molecule and a protein pocket. (2) The second task is ligand efficacy prediction (LEP). We have molecule bounded to pockets, and the goal is to detect if the same molecule has a higher binding affinity with one pocket compared to the other one.
Results in Table 3 illustrate that, for the LBA task, two pretraining baseline methods fail generalize to LBA (the loss gets too large), and all the other pretraining baselines cannot beat the randomly initialized baseline. For the LEP task, the supervised and two contrastive learning pretraining
8

---

Published as a conference paper at ICLR 2023
Table 3:
Downstream results on binding affinity tasks. We select three evaluation metrics for LBA: root mean squared error (RMSD), the Pearson correlation (Htp) and the Spearman correlation (Rs). LEP is a binary classification task, and we use the area under the curve for receiver operating characteristies (ROC) and precision-recal (PR) for evaluation. We run cross validation with 5 seeds, and the best results are in bold.
LBA
LEP
Pretraining
RMSD
Rr f
Re 1
ROC 1
PR f
1.463
0.06
0.572 + 0.02
0.568
+ 0.02
0.675 + 0.04
0.549 + 0.05
Supervised
1.551 + 0.08
0.539
0.03
0.533
- 0.03
0.696
0,554 + 0.03
Charge Prediction
2.316
0.80
0.387
0.11
0.400
0.11
0.30
0.557 1: 0.07
Distance Prediction
1.542
0.08
0.545
0.540 1
0.07
0.479
Angle Prediction
0.504 0.04
3D InfoGraph
0.540
GeoSSL RR
1,515 + 0.07
0.545
0.03
0.539 0.03
0.654
0,518
1.564
0.05
0.508
0.03
0.497 = 0.05
0.693
0,571
SeoSSL-EEB-NCCE
1.499
0.06
0.547
0.03
0.534 1 0.03
0.691 1 0.05
0.603
0.07
GeoSSL-DDM (ours)
1,451 +: 0.03
0.577 1 0.02 0.572 1 0.01
0.776 1 0.03 0.694 1 0.06
baselines stand out for both ROC and PR metrics. Meaningfully, for both tasks, GeoSSL-DDM is able to achieve promising improvement, revealing that modeling the local region around conformer with distance denoising can also benefit binding affinity downstream tasks.
5.5 DISCUSSION: CONNECTION WITH MULTI-TASK PRETRAINING
In the above experiments, we test multiple self-supervised and supervised pretraining tasks separately, Yet, all these pretraining methods are not contradicted but could be complementary instead. Existing work has successfully shown the effect of combining them in various ways. For example, [28] shows that jointly doing supervised and supervipe pretraining can augment the pretrained representation. [38, 57] prove that contrastive and generative SSL pretraining methods can be learned simultaneously as a multi-task pretraining. In addition, terms of the molecule-specific pretraining, [38] empirically verifies that 2D topology and 3D geometry views share certain information, and maximizing their mutual information together with 2D topology SSL for pretraining is beneficial.
With these insights, we would like to claim that all of these points are worth exploring in the future, especially in the line of pretraining for molecular geometry. Because pretraining datasets often come with multiple quantum properties and the 2D molecular topology can be obtained heuristically. Yet as the first step to explore self-supervised learning using only the 3D geometric data (i.e., without covalent bonds), study here would like leave multi-task pretraining for future exploration.
6 CONCLUSIONS AND FUTURE DIRECTIONS
We proposed novel coordinate denoising method, coined GeoSSL-DDM, for molecular geometry pretraining. GeoSSL-DDM leverages an SE(3)-invariant score matching strategy, under the GeoSSL framework, to successfully decompose its coordinate denoising objective into the denoising of pairwise atomic distances in molecule, which then can be effectively computed and directly target the determinant factors in molecular geometric data. We empirically verified the effectiveness and robustness of our method, showing its superior performance to nine state-of-the-arr pretraining baselines on 22 benchmarking geometric molecular property prediction and binding affinity tasks.
Our work opens up venues for multiple promising directions. First, from the machine learning perspec- tive, we propose general pipeline on using EBM for MI maximization on geometric data pretraining. Yet, there are more explorations on the success of EBM, like GelowNet [3], and it would be interesting to explore how to combine with molecular geometric data along this systematic path. In addition, GeoSSL-DDM does not utilize the 2D structure (i.e., covalent bonds for molecules), and it would be desirable to consider how to utilize the distance denoising together with the 2D topology information.
In terms of applications, our proposed GeoSSL-DDM is general framework, and it can be naturally applied to other geometric data, such as point clouds and protein pretraining. In addition, our current goal is to perform denoising in the local region, yet it would be interesting to explore larger regions. From this aspect, the denoising can be viewed as recovering the molecular dynamics trajectory, and would explore how generalizable this pretrained representation is to downstream tasks.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
We authors acknowledge that have read and committed to adhering to the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of the empirical results, we provide the implementation details (hyper- parameters, dataset statistics, etc.) in Section 5 and appendix D, and publically share source code through this GitHub link. Besides, the complete derivations equations and clear explanations are shown in Section 4 and appendix C.
ACKNOWLEDGEMENT
We would like to thank Anima Anandkumar, Chaowei Xiao, Weili Nie, Zhuoran Qiao, Chengpeng Wang, and Pierre-André Noël for their insightful discussions. This project is supported by the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronics Co., Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bing Gift Fund and two NRC Collaborative R&D Projects (AI4D-CORE-06 AI4D-CORE-08) This project was also partially funded by IVADO Fundamental Research Project grant PRF-2019-35831
REFERENCES
[I] Kenneth Atz, Francesca Grisoni, and Gisbert Schneider. Geometric deep learning on molecular representations. Nature Machine Intelligence, pp. 1-10, 2021. 2
[2] Simon Axelrod and Rafael Gomez-Bombarelli, Geom: Energy- annoottaddt molecular conforma- tions for property prediction and molecular generation. arXiv preprint arXiv:2006.05531, 2020.
3
[3] Yoshua Bengio, Tristan Deleu, Edward J Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio.
Gflownet foundations. arXiv preprint arXiv:211 1.09266, 2021. 9, 27
[4] Hangrui Bi. Hengyi Wang. Chence Shi, Connor W. Coley, Jian Tang, and Hongyu Guo. Non- autoregressive electron redistribution modeling for reaction prediction. In ICML, 2021.
[5] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training, In Proceedings of the eleventh annual conference on Computational learning theory, pp. 92-100, 1998. 20
[6] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Geometric and physical quantities improve e(3) equivariant message passing, arXiv preprint 1rXiv22110002900 2021. 2, 7, 15
[7] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: bench- marking models for de novo molecular design. Journal of chemical information and modeling. 99(3):1096-1100 2019.
[8] Jintai Chen, Biwen Lei, Qingyu Song, Haochao Ying, Danny Z Chen, and Jian Wu. A hierarchical graph network for 3d object detection point clouds. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 312-39221111 2020. 2
[9] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758,
2021. 7
[10] Stefan Chmiela, Alexandre Tkatchenko, Huziel E Sauceda, Igor Poltavsky, Kristof T Schüit, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, 36)30155 2017. 8, 21
[11] John D Chodera and Frank Noé. Markov state models of biomolecular conformational dynamics. Current opinion in structural biology, 25:135-144, 2014. 17
10

---

Published as a conference paper at ICLR 2023
[12] Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316, 2020. 3
[13] Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengchao Liu. Molgensurvey: A systematic survey machine learning models for molecule design. arXiv preprint arXiv:2203 14500, 2022. 5
[14] David Duvenaud, Dougal Maclaurin, Jorge -paralrur,,,,,gurrree, Rafael Komez-Boomaardd Timothy Hirzel, Alán Aspuru-Guzik and Ryan P Adams. Convolutional networks graphs for learning molecular fingerprints. arXiv preprint arXiv:1509.0929) 2015.
[15] Thomas Engel and Johann Gasteiger. Applied chemoinformatics achievements and future opportunities. John Wiley & Sons, 2018. 16
[16] Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Chemrl-gem: Geometry enhanced molecular representation learning for property prediction. arXiv preprint arXiv:2106.066 0006006633,. 2021. 3
[17] Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling. Se(3) Se(3)-transsee transffrrmers: 3d roto-translation equivariant attention networks. arXiv preprint arXiv:2006. 10503, 2020. 2, 7, 15
[18] Siddhant Garg and Yingyu Liang, Functional regularization for representation learning: A unified theoretical perspective. Advances Neural Information Processing Systems, 33:17187- 17199, 2020. 20
[19] Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks. arXiv preprint arXiv:2207.09453 2022. 2,6
[20] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp.
1263-1272. PMLR, 2017. 1
[21] Jonathan Godwin, Michael Schaarschmidt. Alexander Gaunt, Alvaro Gonzzlez. Yulia Rubanova, Petar Veličković, James Kirkpatrick, and Peter Battaglia. Simple GNN regularisation for 3d molecular property prediction and beyond. In International Conference on Learning Representations, 2022. 1, 17
[22] Sai Krishna Gottipati, Boris Sattarov, Sufeng Niu, Yashaswi Pathak, Haoran Wei, Shengchao Liu, Simon Blackburn, Karam Thomas, Connor Coley, Jian Tang, et al. Learning to navigate the synthetically accessible chemical space using reinforcement learning. In International Conference on Machine Learning, pp. 3668-3679. PMLR, 2020.
[23] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271 -22222444 2020. 7
[24] Yuzhi Guo, Jiaxiang Wu, Hehuan Ma, and Junzhou Huang. Self-supervised pre-training for protein embeddings using tertiary structures. 2022. 15
(25) Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international con- ference on artificial intelligence and statistics, 297-304. JMLR Workshop and Conference Proceedings, 2010. 3, 17, 18,
[26] Thomas A Halgren. Merck molecular force field. i. basis, form, scope, parametcrization, and performance of mmff94, Journal of computational chemistry, 17(5-6):40--51, 1996. 16
[27] Ryuichiro Hataya, Hideki Nakayama, and Kazuki Yoshizoe. Graph energy- based model for molecular graph generation. In Energy Based Models Workshop-ICLA 2021, 2021. 4
[28] Weihua Hu. Bowen Liu, Joseph Gomes, Marinka Zimik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks, In International Conference on Learning Representations, ICLR, 2020. 1,3,9

---

Published as a conference paper at ICLR 2023
[29] Ziniu Hu, Yuxiao Dong, Kuansan Wang. Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. InACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD, pp. 1857-1867, 2020. 3
[30] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[31] Johannes Klicpera, Shankari Giri, Johannes T Margraf, and Stephan Günnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. arXiv preprint arXiv:2011.14115, 2020. 2, 8, 15
[32] Johannes Klicpera, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. In Conference on Neural Information Processing Systems
(NeurIPS), 2021. 2, 7, 15
[33] Greg Landrum al. RDKit: A software suite for cheminformatics, conpiiaoommmticscccmoutttttaaala chemistry, and predictive modeling, 2013. 16
[34] Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. Graphebm: Molecular graph genera- tion with energy-based models. arXiv preprint arXiv:2102.0000 000.06666, 2021.
[35] Shengchao Liu, Mehmet Furkan Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised representation for graphs, with applications to molecules. arXiv preprint arXiv:1806.09226. 2018. 1,3
[36] Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Anima Anandkumar. Multi-modal molecule structure-text model for text-based retrieval and editing. arXiv preprint arXiv:2212.1078)) 2022.
[37] Shengchao Liu, Chengpeng Wang. Weili Nie, Hanchen Wang. Jiarui Lu, Bolei Zhou, and Jian Tang. GraphCG: Unsupervised discovery of steerable factors in graphs. In NeurIPS 2022 Workshop: New Frontiers in Graph Learning, 2022. 1
[38] Shengchao Liu, Hanchen Wang. Weiyang Liu, Joan Lasenby. Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. In International Conference on Learning Representations, 2022. 3, 5, 6, 7, 9,
[39] Shengehao Liu, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Anthony Gitter, Chaowei Xiao, Jian Tang, Hongyu Guo, and Anima Anandkumar. text-guided protein design framework. arXiv preprint 2tt.:23322.4611,, 2023. 1
[40] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering. 2021. 3
[41] Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.05013, 2021. 2, 7, 8, 15, 21
[42] Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip Yu. Graph sule-super learning: A survey. arXiv preprint arXiv: /03.0000 1, 2021.
[43] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 1600. 03983, 2016 15,
[44] Francesca Pistilli, Giulia Fracastoro, Diego Valsesia, and Enrico Magli. Learning graph- convolutional representations for point cloud denoising, In European conference on computer vision, pp. 103-118. Springer, 2020. 2
[45] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017. 2
[46] Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014. 7, 21
12

---

Published as a conference paper at ICLR 2023
[47] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling, E(n) equivariant graph neural networks. arXiv preprint arXiv:2102.098444 2021. 2. 15
[48] H. Bernhard Schlegel. Exploring potential energy surfaces for chemical reactions: an overview of some practical methods. Journal of computational chemistry, 24(12):15141127, 2003.
[49] Kristof T Schütt, Pieter-Jan Kindermans, Huziel E Sauceda, Stefan Chmicla, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-fltee convolutional neural network for modeling quantum interactions. arXiv preprint arXiv: 08566, 2017. 7, 15
[50] Kristof T Schütt,, Huziel Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241722, 2018. 2, 8, 15, 21
[51] Kristof T Schüitt, Oliver T Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. arXiv preprint arXiv:2102.033 2021.
2, 7, 8, 15, 21
[52] Chence Shi, Minkai Xu, Hongyu Guo, Ming Zhang, and Jian Tang. A graph to graphs framework for retrosynthesis prediction. In International Conference on Machine Learning, pp. 8818-8827.
PMLR, 2020. 1
[53] Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a flow-based autoregressive model for molecular graph generation. arXiv preprini arXiv:2001003882, 2020. 1
[54] Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In International Conference on Machine Learning, pp. 9558-9568. PMLR, 2021. 5, 6, 19
[55] Weijing Shi and Raj Rajkumar. Point-gnn: Graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1711-1719, 2020. 2
[56] Muhammed Shuaibi, Adeesh Kolluru, Abhishek Das, Aditya Grover, Anuroop Sriram, Zachary Ulissi, and C Lawrence Zitnick. Rotation invariant graph neural networks using spin convolu- tions. arXiv preprint arXiv:2106.09575 2021. 2, 7, 15
[57] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv: 106.:2200600142,, 2021. 9
[58] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019. 6, 19, 20, 27
[59] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 13:12448-118 2020. 6, 19, 20
[60] Yang Song and Diederik P Kingma. How to train your energy -aaedd models. arXiv preprint arXiv:2101.03288. 2021. 3, 17
[61] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 133466, 2020. 3, 17, 20, 24
[62] Antonia Stank, Daria B Kokh, Jonathan C Fuller, and Rebecca C Wade. Protein binding pocket dynamics. Accounts of chemical research, 49(5):809-81) 815, 2016. 8, 21
[63] Fan- Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, ICLR, 2020. 3,
[64] Ruoxi Sun, Hanjun Dai, Li, Steven Kearnes, and Bo Dai. Energy- based view of retrosynthesis. arXiv preprint arXiv:2007. 2020.
13

---

Published as a conference paper at ICLR 2023
[65] Nathaniel Thomas, Tess Smidt, Steven Kearne, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and eqquiaaiioneuuiaaiiant neural networks for 3d point clouds. arXiv preprint arXiv: 082.9,82999.. 2018. 2,
[66] Raphael JL Townshend, Martin Vögele, Patricia Suriana, Alexander Derry Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Brandon Anderson, Stephan Eismann, Risi Kondor, al. Atom3d: Tasks on molecules in three dimensions, arXiv preprint arXiv:2012.04035, 2020.
8, 22
[67] Mikaela Angelina Uy Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF international conference computer vision,
pp. 1588-1597, 2019.
[68] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pp. arXiv-1807, 2018. 5,
[69] Pascal Vincent. connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674 2011. 3, 5, 6, 19, 27
[70] Renxiao Wang. Xueliang Fang, Yipin Lu, Chao- The Yang, and Shaomeng Wang. The pdbbind database: methodologies and updates. Journal of medicinal chemistry, 48(12)4411144119 2005. 8, 22
[71] Lirong Wu, Haitao Lin, Zhangyang Gao, Cheng Tan, Stan Li, al. Self-supervised on graphs: Contrastive, generative, or predictive. arXiv preprint arXiv:2105.07342 2021. 3
[72] Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: unified review. arXiv preprint arXiv:2 10757, 2021.
[73] Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3d: A benchmark for predicting 3d geometries from molecular graphs. arXiv preprint 1r000:2110.01717, 2021. 7
[74] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molec- ular representations for property prediction. Journal of chemical information and modeling, 59 (8):3370-3388, 2019.
[75] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Advances in Neural Information Processing Systems, NeurlPS, 2020. 1,3
[76] Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. arXiv preprint XXXv:2200600033 2022. 2, 3, 27
[77] Chengxi Zang and Fei Wang. Moflow: an invertible flow model for generating molecular graphs. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 617-626, 2020. 1
14

---

Published as a conference paper at ICLR 2023
A BENCHMARKS AND RELATED WORK
A.1 GEOMETRIC NEURAL NETWORKS
Recently, geometric neural networks have been actively proposed, including SchNet [50], TFN [17],
DimeNet++ [31], SE(3)- Trans [17]. EGNN [47], SEGNN [6], SphereNet [41], SpinConv [56].
PaiNN [51], and GemNet [32]. We reproduce most of them on the QM9 dataset as shown in Table
Among this, we would like to highlight two models: SchNet and PaiNN.
SchNet [49] composed the following key steps:
(0) cmbedding(a.),,
MLP far (..r)))). h, MLP(2
2
(10)
where K is the number of hidden layers, and
f(zy, (i. ry) = zÿ ex(r, r5) xj exp(->lll1r - ryll2
- - -m113)
(11)
the continuous- filter convolution layer, enabling the modeling of continuous coordinates of atoms.
PaiNN [51] is an improved work of SchNet [49]. It addresses the limitation of rotational equivariance SchNet by embracing rotational invariance, attaining more expressive -qquivaaava neural
network model.
A.2 BENCHMARK ON QM9
Current work is using different optimization strategies and different data split (in terms the splitting size). Originally there 133,885 molecules in QM9, where 3,054 are filtered out, leading 130,831 molecules. During the benchmark, we find that:
The performance QM9 is very robust either using (1) 110K for training, 10K for val, 10,831 for test or using (2) 100K for training, 13,083 for val and 17,748 for test.
The optimization, especially the learning rate scheduler is very critical. During the bench- marking, we find that using cosine annealing learning rate schedule [43] is generally the most robust.
For more detailed discussion on QM9, please refer to Appendix D. We show the benchmark results on QM9 in Table 4.
Table 4: Benchmark results 12 quantum mechanics prediction tasks from QM9. We take 110K for training, 10K for validation, and 11K K for test. The evaluation is mean absolute error (MAE).
Alpha
Gap 1
HOMO
LUMO J.
Mu 4
Cv
G298
H298 4.
R2
U298 4
UO
Zpve
SchNet
0.070
50.38
31.81
25.76
0.029
0.031
14.60
14.24
0.131
13.99
14.12
1.686
SE(3) Trans
0.136
58.27
35.95
35.41
0.052
0.068
68.50
70.22
1.828
70.14
72.28
5.302
EGNN
0.067
24.44
11.02
11.07
10.83
10.70
1.578
DimeNat++
0.046
38.14
21.23
17.57
0.029
0.022
7.98
7.19
0.306
6.86
6.93
1.204
SphereNet
0.050
39.54
18.66
0.026
0.025
8.65
7.43
0.262
8.28
1.390
SEGNN
0.057
41.08
22,46
21.46
0.025
0.028
13.07
13.94
0.472
14.64
13.89
1.662
PaiNN
0.048
44.50
26.00
21.11
0.016
0.025
8.31
7.67
0.132
7.77
7.89
1.322
A.3 RELATED WORK
We acknowledge that there is a parallel work called Protein Tertiary SSL (PTSSL) [24] working on the geometric self-supervised learning. Yet, there are some fundamental differences between theirs and ours, as listed below: (1) Key notion on pseudo-force. PTSSL directly applies the denoised score matching method into protein tertiary structures, yet our focus is on how the notion of pseudo-force can come into the play, which possess better generalization ability. (2) Task setting. PTSSL works protein and utilize both the 2D and 3D information, and our work is purely working on the 3D geometric information. (3) Technical novelty. PTSSL designs the DSM objective for SSL, and what propose is systematic tool: using energy based model and score matching to solve the geometric SSL problem opens a new venue this field. (4) Objective. PTSSL directly designs objective function, which is denoising from one view the other. Ours starts from the
15

---

Published as a conference paper at ICLR 2023
lower bound of MI, which is symmetric in terms of the denoising directions. We believe that such symmetry are treating the two views equally, and can better reveal the mutual concept, making the pre-trained representation more robust the position augmentations. (5) Empirical baseline. PTSSL lacks the comparisons with other pre-training methods, while we compare with 7 SOTA pre-training methods, especially those driven by maximizing the MI with the same augmentations. Without such comparisons, it is hard to tell the effectiveness of the pseudo-force matching for geometric data. (6) Score network. Last but not least, the score network designed in PTSSL does not satisfy the SE(3) equivariant property.
B AN EXAMPLE ON THE IMPORTANCE OF ATOM COORDINATES
First, it has been widely acknowledged [15] that the atom positions or molecule shapes are important factors to the quantum properties. Here we carry out an evidence example empirically verify this. The goal here is to make predictions on 12 quantum properties in QM9.
The molecule geometric data includes two main components as input features: the atom types and atom coordinates. Other key information can be inferred accordingly, including the pairwise distances and torsion angles. We consider corruption each of the components to empirically test their importance accordingly.
Atom type corruption. There are total 118 types atom types, and the standard embedding option is to apply the one-hot encoding. In the corruption case, we replace all the atom types with a hold-out index, Le., index 119.
Atom coordinate corruption. Originally QM9 includes atom coordinates that are in the stable state, and now we replace them with the coordinates generated with MMFF [26] from RDKit [33].
Table 5: An evidence example on molecular data. The goal is to predict 12 quantum properties (regression tasks) of 3D molecules (with 3D coordinates on each atom). The evaluation metric is MAE.
Model
Mode
Alpha
Gap
HOMOL
LUMO
Mu
Cv
G298
H798
R2
U/98
LO..]
Zpve
Geometry
14.55
0.122
14.44
SehNet
Type Corruption
0.2331
Courdinate Comuption
110.59
58.92
18.619
60.71
59.32
Stable Geometry
0.048
44.50
26.00
21.11
0.016
0.025
8.31
7.67
0.132
7.77
7,89
1.323
PaiNN
Type Compptiin 0,057
45.61
25.22
22.16
8.066
0.025
11 18
11.60
0.181
11.15
10.89
1.339
Coordinate Compption
0.223
108.31
73.43
72.35
0.391
0.095
48.40
51.82
16.828
51.43
48.95
4.395
We take SchNet and PaiNN as the backbone 3D GNN models, and the results are in Table 5. We can observe that (1) Both corruption examples lead to performance decrease. (2) The atom coordinate corruption may lead to more severe performance decrease than the atom type corruption. To put this into another way is that, when corrupt the atom types with the same hold-out type, it is equivalently to removing the atom type information. Thus, this can be viewed as using the equilibrium atom coordinates alone, and the property prediction is comparatively robust. This observation can also be supported from the domain perspective. According to the valence bond theory, the atom type information can be implicitly and roughly inferred from the atom coordinates.
Therefore, by combining all the above observations and analysis, one can draw the conclusion that, for molecule geometry data, the atom coordinates reveal more fundamental information for representation learning.
16

---

Published as a conference paper at ICLR 2023
C MUTUAL INFORMATION MAXIMIZATION WITH ENERGY-BASED MODEL
In this section, we will give a detailed discussion on mutual information (MI) maximization with the energy-based model (EBM).
First, we can get a lower bound of MI. Assuming that there exist (possibly negative) constants a and b such that a S (X) and S HY),, i.e., the lower bounds to the (differential) entropies, then have:
1
I( = (H + H(Y) H(X - (((+))
-
2
1
2 2
(a + b - H(XIT))
(12)
(a +66 L...
where the loss CMi defined as:
CMI
E2 pp(...) logp(xly) log p(ylz))
(13)
Empirically, we use energy- based models to model the distributions. The existence of a and b can be understood as the requirements that the two distributions (Px: Py) are not collapsed. Notice that to keep consistent with the notations in Section 3, will be using 9I and g2 the two variables. Then the goal is equivalent to optimizing the following equation:
CoOssL
Epis .aa) log p(gn (ga) + logal
(14)
Thus, we transform the MI maximization problem into maximizing the summation of two conditional log-like bogllikelhhoods. Such an objective function opens a wider venue for estimating MI, e.g., using the EBM to estimate Equation (14).
Adaptation to Geometric Data The 3D geometric information the atomic coordinates critical to molecular properties. Then based on this, we propose geometry perturbation, which adds small noises to the atom coordinates. This geometry perturbation possesses certain motivations from both domain and machine learning perspectives. (1) From the practical experiment perspective, the statistical and systematic errors [11] on conformation estimation are unavoidable. Coordinate perturbation is a natural way to enable learning representations robust to such noises. (2) From the domain aspect, molecules are not static but continuous motion the 3D Euclidean space, and we can obtain a potential energy surface accordingly. We are interested in modeling the conformer, i.., the 3D coordinates with the lowest energy. However, even the conformer at the lowest energy point can have vibrations, and coordinate perturbation can better capture such movement yet with the same order of magnitude on energies. (3) As will be illustrated later, our proposed method can be simplified as denoising atomic distance matching. (4) Leveraging coordinate perturbation for model regularization has also been empirically verified its effectiveness for supervised molecule geometric representation learning [21]. Such characteristics of molecular geometry motivate to apply coordinate perturbation. If we take each of the two views as adding noise to the coordinates from the other view, then the objective in Equation (14) essentially states that we want to conduct coordinate denoising, as shown in Figure 3. Yet, this is not trivial task due to the complicated geometric space (e.g., 3D coordinates) reconstruction.
C.1 AN EBM FRAMEWORK FOR MI ESTIMATION
The lower bound in Equation (14) is composed of two conditional log-likelihood terms, and then we model the conditional likelihood with EBM. This gives us:
LOeosSL EDM = (ptor log exp(Sn (g1. + log (22.
(15)
Agalo2
Apzlo:
where fq. (g1, (2) = E(g (92) and fog (92.91) g1) E(gg (g)) are the negative energy functions, and Agi ig. and Agalg: are the corresponding partition functions. The energy functions can be flexibly defined, thus the bottleneck here is the intractable partition function due to the high cardinality. To solve this, existing methods include noise-conine contrastttt estimation (NCE) [25] and score matching (SM) [60, 61]. and we will describe how apply them for MI maximization.
17

---

Published as a conference paper at ICLR 2023
81
82
Goal: Coordinate Denoising
Figure 3: Pipeline for denoising coordinate matching.
C.2 EBM-NCE FOR MI ESTIMATION
Under the EBM framework, if we solve Equation (15) with Noise-Contrastive Estimation (NCE) [25]. the final objective is termed EBM-NCE, as:
CoeUSS-EEMANCE
2 E Peam Epnios loog (1 - o(fn (g), (2))) + Bun e2) llog (g) ga))g
(x) (enloolo flog (1 ofo. (02. 9 1))] Ban 0n fog aff. 9 [1))]]
(16)
All the detailed derivations can be found in [25]. Specifically, EBM-NCE is equivalent to the Jensen- Shannon estimation for MI, while the mathematical intuitions and derivation processes are different. Besides, it also belongs to the contrastive SSL venue. That is, it aims at aligning the positive pairs and contrasting the negative pairs.
C.3 EBM-SM FOR MI ESTIMATION: GEOSSL-DDM
g1
d
Step I: Distance Extraction
Step 2: Distance Perturbation
82
Step 3: Distance Denoising
Figure 4: Pipeline for GeoSSL-DDM. The 9i and g2 are around the same local minima, yet with coordinate noises perturbation. Originally we want to do coordinate denoising between these two views. Then as proposed in GeoSSL-DDM, we transform it to equivalent problem, i.e., distance denoising. This figure shows the three key steps: extract the distances from the geometric views, perform distance perturbation, and denoise the perturbed distances.
In this subsection, we will focus on geometric data like molecular geometry. Recall that we have two views: 9i and g2, and the goal is maximize the lower bound of the mutual information
18

---

Published as a conference paper at ICLR 2023
in Equation (14). Because the two views share the same atomic features, it can be reduced to:
Liiosst
pog [ex) Ep(or.
log (ggalsl9n) lsn)
(( Ri)) )1 (1, Re))
p .22)
logel R2) (X),,
2
(17)
log 92)
2 12))
12)
(91)
Bllan .22) log exp( (R1, (2)))
(x2(((R2991))
pi)
log
ARi
A
where the f()) are the negative of energy functions, and AR, and ARalgn are the intractable partition functions. The first equation in Equation (17) results from that the two views share the same atom types. This equation can be treated as denoising the atom coordinates of one view from the geometry of the other view. In the following, we will explore how to use the score matching for solving EBM, and further transform the coordinate-awan mutual information maximization to the denoising distance matching (GeoSSI-DDDM) as the final objective.
Score Definition The two terms in Equation (3) are in the mirroring direction. Thus in what follows, we may as well adopt proxy task that these two directions can calculated separately, and take one direction for illustration, e.g., log (Rp(f(Rs..o)) An,102 The score is defined as the gradient of the log-likelihood w.r.t. the data, i.e., the atom coordinates in our case. Because the normalization function is a constant w.r.t. the data, will disappear during the score calculation. To adapt into our setting, the score is obtained the gradient of the negative energy function w.r.t. the atom coordinates, as:
s(H.. g2) = Vn, Pog (g2) = Vn, (Hi,
(18)
we assume that the learned optimal energy function, i.e., ((), possesses certain physical or chemical information, then the score in Equation (18) can viewed special form of the pseudo-force. This may require more domain-specific knowledge, and we leave this for future exploration.
Score Decomposition: From Coordinates To Distances Through back-propagation [54], the score atom coordinates can be further decomposed into the scores attached to pairwise distances:
s(R1, 92),
Ōri,1
Σ
Rf 92)
Ad1,0
Od1,15
Pri,4
(E
(19)
Σ
1
(f
(Fi. r1.s)
di.si
Ødy.5
(EX(()
Σ JEN(()
1
s(h, g2)g - (ri,-11.s))
d1,s
where s(d,, 92)i
adi,ss Such decomposition has a nice underlying intuition from the pseudo-
force perspective: the pseudo-force on each atom can be further decomposed as the summation of pseudo-forces on the pairwise distances starting from this atom. Note that here the pairwise atoms are connected in the 3D Euclidean space, not by the covalent- covalent-bonding. bonding.
Denoising Distance Matching (DDM) Then we adopt the denoising score matching (DSM) [69] to our task. To be more concrete, we take the Gaussian kernel as the perturbed noise distribution each pairwise distance, i.e., yo (d1]92) = Epan, (d, 1g2) g (d, [d1)), )], where is the deviation in Gaussian perturbation. One main advantage of using the Gaussian kernel is that the following gradient of conditional log- likelioodo has a closed-fome formulation: V Va, log 9a (d |d,, 92) = (di d/2/ and the goal of DSM is to train score network match it. This trick was first introduced in [69], and has been widely utilized in the score matching applications [58, 59].
To adapt this into setting, this is essentially saying that want train "distance network", i.e. so (d, (g2), match the distance perturbation, or we can say it aims matching the pseudo-force
19

---

Published as a conference paper at ICLR 2023
with the pairwise distances from another aspect. By taking the Fisher divergence the discrepancy metric and the trick mentioned above, the estimation se sa(d1.g2) simplified to the following:
V d, log q(d) d1 92) can be
Dr (d, (dllgg)llpp( g2)) = (um (92)) B (dd.ddd..))) [Ilsodd, ([s((di.2)) g2)
1
di di
[1
C.
(20)
o²
Final objective. We adopt the following four model training tricks from [38, 58, 59] stabilize the score matching training process. (1) We carry out the distance denoising at L-level of noises. (2) We add weighting coefficient X(or) の for each noise level, where 3 the annealing factor. (3)
We scale the score network by a factor of 1/o. (4) We sample the exactly same atoms from the two geometry views with masking ratio Finally, by considering the two directions and all the above tricks, the objective function becomes the follows:
86 (di 92)
di di
CGeaSSI-DD -DDM
oj pum ( a (d,
Ø1
of
2
(21)
1 2T
sp d2, 91)
d2 d2 22
+
of Bysu (ds 01) B (dd2ldaa)))
01
of
C.4 DISCUSSIONS
Using the energy-based model (EBM) to solve MI maximization open novel venue, espe- cially for high-structured data like molecular geometry. To solve EBM, existing methods include noise-contrastive estimation (NCE) [25], score matching (SM) [61], etc. To put this under the MI maximization setting, EBM-NCE is essentially contrastive learning method, where the goal is to align the positive pairs and contrast the negative pairs simultaneously. While EBM-SM or GeoSSL- DDM, is a generative self-supervised learning (SSL) on distance denoising, and it is especially appealing in the field for geometric data representation learning.
Further interpretation of pseudo-force. Score matching can be smoothly adopted to 3D geometric setting. Because scores are defined as gradients of the energy function with respect the atom positions, it can be thought of a form of pseudo-forces, forces. Following this, GeoSSL-DDM, can be viewed as a pseudo-force matching, which more natural to the molecular structures. However, further understanding of this requires more domain knowledge in understanding or designing of the energy function. This beyond the score this paper, and we would like to leave for future exploration.
Multi-view pretraining: complementary information with 2D topological graph. Recently, there have been certain works [38] proving that 3D geometric information useful for 2D topology. Here want to conjecture that the reverse direction is also meaningful: 2D topology can be also useful for 3D representation. This may not seem reasonable from the domain perspective, since 2D topology can be heuristically obtained from the 3D geometry, i.e., all the 2D information is redundant 3D geometry. However, from the machine learning theory perspective [5, 18). this still helpful in reducing the sample complexity. From a higher level perspective, we want explicitly point out that such gap between machine learning and scientific domain has been widely existed, and it would be an interesting direction for further exploration.
20

---

Published as a conference paper at ICLR 2023
D EXPERIMENTS
In this section, we would like to discuss the experiment details of our work. The main structure is as follows:
In Appendix D.I, we introduce the computation resources.
In Appendices D.2 D.4, we introduce the downstream datasets.
Notice that because the performance of QM9 and MD17 is quite stable after fixing the seed (e.g., : 42), we we will not run cross-validation. This also follows the main literature [41, 50, 51].
Yet, for LBA LEP, these two datasets are quite small and are very sensitive to data splitting, so we pick up 5 seeds (12, 22, 32, 42, and 52) and run cross-validation them.
In Appendix D.5, list the key hyperparameters for all the pretraining baselines and GeoSSL-DDM.
In Appendix D.6, show the empirical results using SchNet the backbone model.
D.1 COMPUTATIONAL RESOURCES
We have around 20 V100 GPU cards for computation internal cluster. Each job can be finished within 3-24 hours (each job takes one single GPU card).
D.2 DATASET: QM9
QM9 [46] a dataset of 134K molecules consisting heavy atoms. It includes 12 tasks that are related to the quantum properties. For example, UO and U298 are the internal energies at OK at OK and 298. 15K respectively, and U298 and G298 are the other two energies that can be transferred from H298 respectively. The other 8 tasks are quantum mechanics related to the DFT process. We follow [50] in preprocessing the dataset (including unit transformation for each task).
Current work is using different data split (in terms of the splitting size). Originally there 133,885 molecules in QM9, where 3,054 are filtered out, leading to 130,831 molecules. During the benchmark, find that the performance on QM9 is very robust either using (1) 110K for training, 10K for val, 10,831 for test or using (2) 100K for training, 13,083 for val and 17,748 for test. In this paper, we are using option (1).
D.3 DATASET: MD17
MD17 [10] is dataset molecular dynamics simulation. It includes eight tasks, corresponding to eight organic molecules, and each task includes the molecule positions along the potential energy surface (PES), as shown in Figure 1. The goal is predict the for each atom at each molecule position. We list some basic statistics in Table 6. We follow [41, 51] in preprocessing the dataset (including unit transformation for each task).
interatomic forces
Table 6: Some basic statistics on MD17.
Pretraining
Aspirin
Benzene
Ethanol 4
Malonaldehyde
Naphthalene 4
Salicylic Toluene
Uracil
Train
IK
1K
IK
1K
IK
1K
1K
1K
Validation
1K
IK
1K
1K
Test
209.762
47,863
553,092
991.237
324,250
318.231
440.790
131,770
D.4 DATASET: LBA & LEP
Atom3D [66] is a newly published dataset. It gathers several core tasks for 3D molecules, including binding affinity. The binding affinity prediction is to measure the strength of binding interaction between a small molecule to the target protein. Here we will model both the small molecule and large molecule (protein) with their 3D atom coordinates provided.
During the binding process, a cavity in a protein can potentially possess suitable properties for binding small molecule (ligand), and it is termed pocking [62]. Because of the large volume of
21

---

Published as a conference paper at ICLR 2023
Table 7: Some basic statistics on LBA & LEP. For LBA, we use split- we--succnc--idenity--000 we split prolein-ligand complexes such that no protein in the test dataset has more than 30% sequence identity with any protein the training dataset. For LEP, we split the complex pairs protein target.
Pretraining
LBA
LEP
Train
3,507
304
Validation
466
110
Test
490
104
Split
Seiitby-identit--33
pplit-by-targer
protein, follow [66] by only taking the binding pocket, where there
than 600 atoms
for each molecule and protein pair. To be more concrete, we consider two binding affinity tasks. (1) The first task is ligand binding affinity (LBA). is gathered from [70] and the task predict the binding affinity strength between small molecule and a protein pocket. (2) The second task is ligand efficacy prediction (LEP). We have a molecule bounded to pockets, and the goal detect the
same molecule has higher binding affinity with one pocket compared to the other one. We list some basic statisties in Ta
D.5 HYPERPARAMETER SPECIFICATION
We list all the detailed hyperparameters in this subsection. For all the methods, optimization strategy, ie., with learning rate as 5e-4 and cosine annealing learning rate sched- ule [43]. The other hyperparameters for each pretraining method are listed in Table For the other hyperparameters, we are using the default hyperparameters, as attached in the codes.
the same
Table 8: Hyperparameter specifications.
Pretraining
Hyperparameter
Value
Supervised
task
(total energy]
Type Prediction
masking ratio
(0.15, 0.3)
Distance Prediction
prediction rate
(1)
Angle Prediction
prediction rate
(1e-3, le-4)
perturbed noise μ
(0)
RR
perturbed noise
(0.3)
masking ratio r
(0, 0.3)
perturhed noise y
(0)
InfoNCE
perturbed noise a
(0.3, 1]
masking ratio r
(0, 0.3)
perturbed noise (0)
EBM-NCE
perturbed noise a
(0.3, 1)
masking ratio r
(0, 0.3)
perturbed noise μ
(0)
perturbed noise
(0.3)
masking ratio r
(0. 0.3)
GeoSSL-DDM
L
[30, 50)
01
(0.01)
aL
10]
annealing factor B (0.05, 0.2, 2, 10)
22

---

Published as a conference paper at ICLR 2023
D.6 SCHNET AS BACKBONE MODEL
We want to highlight that some backbone models (e.g., DimeNet++ and SphereNet) may perform better or on par with the PaiNN, as shown in Table 4. Yet they will be out of GPU memory. Thus, considering all (including the model performance, computation efficiency, and memory cost) together, we adopt PaiNN as the backbone model in the main paper.
In this section, сarry experiments using SchNet as the backbone model. We follow the same process as in Section 5, i.e., we compare our method with one randomly-initialzzed and seven pretraining baselines. The results on QM9, MD17, LBA and LEP are in Tables 9 to accordingly. From these three tables, we can observe that in general, GeoSSL-DDM can reach the most optimal results, yielding 21 best performance in 22 downstream tasks, and can reach comparative performance on the remaining task (within top 2 model). This can largely support the effectiveness of our proposed method, GeoSSL-DDM. In addition, we also want mention that a lot of pretraining tasks show the negative transfer issue. Comparing to the results in Section 5, we conjecture that this is related to the task (both pretraining and downstream tasks) and the backbone model. Yet, this is beyond the scope of our work, and we would like to leave this as a future direction.
Table 9: Downstream results on 12 quantum mechanics prediction tasks from QM9, We take I 110K for training, 10K for validation, and 1 IK for test. The evaluation is mcan absolute error, and the best results are in bold.
Pretraining
Alpha
Gap
HOMO]
LUMO
Mu L
Cv
G298
H298
R2
U298
U0
Zpve
0.070
50.59
32.53
26.33
0.029
0.032
14.68
14.85
0.122
14.70
14.44
1.698
Supervised
0.070
51.34
32.62
27.61
0.030
0.032
14.08
14.09
0.141
14.13
13.25
1.727
Type Prediction
0.084
56.07
34.55
30.65
0.040
0,034
18.79
19.39
0.201
19.29
18.86
2,001
Distunce Prediction
0.122
1.676
Angle Prediction
0.084
57.01
37.51
30.92
0.037
0.034
15.81
15.89
0.149
16.41
15.76
1,850
3D InfoGraphh
0.076
51.33
33.92
28.55
0.030
0.032
15.97
16.28
0.117
16.17
15.96
1.666
GeuSSL-RR RR
0.073
52.57
34.44
28.41
0.033
0.038
15.74
16.11
0.194
15,58
14.76
1.804
27.03
0.033
15.53
0.125
15.79
1.675
GeoSSI. EBM NCTE
0.073
52.86
33.74
28.07
0.031
0.032
14.02
13.65
0.121
13.70
13.45
1.677
GooSSL-DDM (ours)
0.066
48.59
30,83
25,27
0.028
0.031
13.06
12.33
0.117
12.48
12.06
1.631
Downstream results force prediction tasks from MD17. We take IK for training, IK for validation, and the number of molecules for test are varied among different tasks, ranging from 48K to 991K. The evaluation is mean absolute error, and the best results are bold.
Table 10:
Pretraining
Aspirin
Benzene
Ethanol J.
Malomaldehyde
Naphthalene J. Salicylic
Toluene
Uracil
1.196
0.404
0.542
0.879
0.534
0.786
0,562
0.730
Supervised
1 1863
0.413
0.512
| 1,254
0.846
1,005
0.529
0.899
Type Prediction
1.293
0.787
0.547
0.879
1,030
1.076
0.614
0.738
Distance Prediction
1.414
0.453
0.845
1.371
0,591
0,819
0.588
0.993
Angle Prediction
3,030
0.450
0.485
0.845
1.112
1,214
0.791
1.016
3D InfoGraph
I 1,545
0.448
0.640
1,080
0.827
1.096
0.735
0,760
GeoSSL-RR
1.878
0.450
0.690
2.255
0.960
1.382
0.784
1.188
GeoSSL InfoNCE
1 1,266
0.396
0.512
1.007
0.778
1,060
0.667
0.933
GeoSSI-EEM-NCCE
1.271
0.400
0.570
0.972
0.605
0.862
0.576
0.790
DDD (ours)
1.176
0.368
0..334
0.779
0.460
0.700)
0.561
0.679
Table 11:
Downstream results hinding affinity tasks. We select three evaluation metries for LBA: the root mean squared error (RMSD), the Pearson correlation (R₂) and the Spearman correlation (Rs). LEP a hinary classification task, and we use the area under the curve for receiver operating characteristies (ROC) and precision-recall (PR) for evaluation. We run cross- validation with 5 seeds, and the best results are in bold.
LBA
LEP
Pretraining
RMSD
Rp
Re t
ROC 1
PR
1.489 + 0.02
0.522
+ 0.01
0.501
+ 0.01
0.436 + 0.03
0.369 + 0.02
Supervised
1.477 + 0.04
0.528
+ 0.02
0.503
0.03
0.462
- 0.05
0.392 + 0.03
Type Prediction
1.483 + 0.04
0.498
0.03
0.03
0.481
0.03
0,570
0.04
0.509
Distance Prediction
1.461
+ 0.06
0.04
0.512
0.04
0.502-0..0
0.415
Angle Prediction
1.499
0.01
0
0.462
+ 0.02
0.532
0.06
1.49+0000
3D InfoGraph
1.467 - 0.06
0.526+0.00
0.500 + 0.03
0.515
GeoSSL-RR
0.439 + 0.04
0.365
+ 0.02
GenSSI. SaSSSTL-Iffiin CIInfoNCE
1.528 + 0.05
0.483 + 0.02
0.464
0.02
0.588
0.06
0.523 + 0.05
ScoSSL-EBM-NCC
1.499 + 0.03
0.509 0.02
0.498
0.02
0.493
- 0.07
0.429 + 0.06
GeoSSL-DDM (ours)
1.432 + 0.02
0.550 + 0.02
0.529 + 0.02
0.633 - 0.03
0.541 + 0.03
23

---

Published as a conference paper at ICLR 2023
E ABLATION STUDIES
E THE EFFECT OF ANNEALING FACTOR IN GEOSSL-DDM
Among all the hyperparameters (see Table for GeoSSL-DDM, we find that the annealing factor is one of the most sensitive ones. Annealing factor | is applied on the weighting coefficient \(o) aa. In this section, we carry out an ablation study to verify this by pretraining GeoSSL-DDM with annealing factors five different scales.
Table 12: Ablation study on the effect of annealing factor on 12 quantum mechanies prediction tasks from QM9. We take 110K for training, 10K for validation, and 1K for test. The backbone model is PaiNN, and the evaluation is mean absolute error.
3
Alpha +
Gap +
HOMOL
LUMO 4
Mu J
Cv +
G298
H298 +
12
U298 4
U0 +
Zpve
0.05
0.047
40.10
23.71
19.40
0.016
0.025
7.72
7.15
0.131
7.30
7.07
1.312
0.2
0.046
40.22
23.48
19.42
0.015
0.024
7.65
7.09
0.122
6.99
6.92
1.307
0.049
40.88
23.96
19.89
0.015
0.029
8.60
7.95
0.136
7.81
7.62
1.357
0.056
45.01
26.36
20.68
0.016
0.030
9.97
9.56
0.136
9.81
9.46
1.597
44.41
26.87
21.13
0.015
0.027
0.133
9.42
9.47
1.592
As can observed Table 12, the models are more stable with smaller annealing values (e.g., 0.2 and 0.05). With large annealing values, the model performance can degrade drastically.
E.2 THE EFFECT ON THE NUMBER OF NOISE LAYERS IN GEOSSL-DDM
Another important hyperparameter listed in Table 8 is the number of noise layers, L. Here we conduct an ablation study on and the results are shown in Table 13.
Table 13: Ablation study on effect noise layer on 12 quantum mechanics prediction tasks from QM9. We take 110K for training, 10K for validation, and 11K for test. The backbone model is PaiNN, and the evaluation is the mean absolute error.
Alpha
Gap
HOMO]
LUMO
Mu
Cv
G298
H298
r2
U298
Zpve
(random init)
0.048
44,50
26.00
21.11
0.016
0.025
8,31
7.67
0.132
7.77
7,89
1.322
0.052
42.75
25.12
20.46
0.015
0.027
9.40
9.08
0.121
8.73
8.80
1.585
30
0.048
40.08
23.95
19.71
0.016
0.025
8.16
7.48
0.137
7.42
7.17
1.311
50
40.22
19.42
7.09
0.122
1.307
In Table 13, we can observe that in general, GeoSSL-DDM can attain better performance with more denoising layers. This is in fact consistent with that in vision applications [61]. Promisingly, even with smaller T. (e.g., I 1), GeoSSL-DDDM can still achieve modest improvement some extent.
24

---

Published as a conference paper at ICLR 2023
F STRONG MODEL ROBUSTNESS WITH RANDOM SEEDS
To further illustrate that our proposed GeoSSL-DDM is robust and insensitive to certain random seeds, we further provide the downstream results with more random seeds. We list the key details as follows:
Dataset. We conduct downstream experiments with random seeds on two datasets: QM9 and MD17.
Backbone models. We run two backbone models: PaiNN in Appendix F.1 and SchNet in Appendix F.2.
Seeds. Up till now, for both the main tables (Tables 1, 2, and 10) and ablation studies (in Appendix E), we use a fixed seed 42. In this section, we provide results with two additional seeds 22 and 32.
Basclines. We here compare against the most optimal baselines: random initialization (without any pretraining). distance prediction, representation reconstruction (RR), and EBM-NCE.
Reported results. We report both the mean and standard deviation with seeds 22, 32, and 42 for all the experiments.
F.1 PAINN
Here we take the PaiNN as the backbone model. The results on QM9 and MD17 are reported in Tables and 15 respectively. Such empirical results match with the main result in Tables and 2, and they do verify that our proposed GeoSSL-DDM is indeed learning more robust representation.
Table 14: Downstream results 12 quantum mechanics prediction tasks from QM9. We take 110K for training.
1OK for validation, 11K for test. The evaluation metric is mean absolute error, the best results are in bold. We report both the mean and standard deviation seeds 22, 32, and 42.
Presivsg
Me
[3]
2,00 ma)
4431
076
835
31
D0W
3017
DM
8
5.1 8 811
M367 300
5:41 6.11
6.2 0.00
13811 1.005
mm9
mm
MM
801 837
0181
831
Ceebol Jelvers
3.005
0007
1786
11.02 17.54
5.009
44.1
(L31
8835
(M
0100
M131
530
1:31 0:011
9.27 0:38
GeeSS DEA (ean)
8.85 (99)
00.34 0.39
31.2
1451
0815
RM
2.55 836
5.77 635
6138 840
5,55 635
6-961 4.11
1.392:1.
Downstream results on force prediction tasks from MD17. We take IK for training, IK for validation, and the number of molecules for test are varied among different tasks, ranging from 48K to 991K. The evaluation is mean absolute error, and the best results are in bold. We report both the mean and standard deviation for sceds 22, 32, and 42.
Table 15:
Pretraining
Aspiria
Benzene
Ethunol
Molomsldehyde
Naphchalene
Salicylic
0.01
0.00
0.000
0.203
0.156
Supervised
0.507
0.02
0.180
0.08
0.312
0.03
0.450
0.01
0.299
0.11
0.169
0.07
0.238
0.03
0:435
Distance Prediction
1,701
0.20
0.1:16
0.04
0.368
0.07
0.757
1: 0.23
0.731
0.07
1/193
0.35
0.340
0.01
0.766
0.29
GooSSL-RR
0.527
0.053
0.00
0.223
0.03
0.342
0.02
0.136
0.01
0.296
0.149
0.190
0.00
CroSSL-InfeNCT
0.999
0.11
0.108
0.03
0.263
0.02
0.469
1: 0.06
0.415
0.16
0.516
0.08
0.189
0.01
0.506
0.724
0.10
0.105
1 0.02
0.267
0.02
0.479
1: 0.03
0.362
0.13
0.517
0.13
0.241
0.07
0.468
0.02
GeoSSL-DDDe DDM (ours)
0.439
0.01
0.051
1: 0.00
0.170 1: 0.00
0.290 4: 0.01
0.133 0.01
0.267
0.00
0.122
0.00
0.192 0.01
25

---

Published as a conference paper at ICLR 2023
F.2 SCHNET
Here we take the SchNet as the backbone model. The results on QM9 and MD17 are reported in Tables 16 and 17 respectively. Such empirical results match with the main result in Tables 9 and 10, and they do verify that our proposed GeoSSL-DDM is indeed learning a more robust representation.
Table 16: Downstream results on 12 quantum mechanics prediction tasks from QM9. We take 110K for training 10K for validation, and 11K for test. The evaluation is absolute emoor and the best results in bold. We report both the mean and standard deviation for seeds 22, 32, and 42.
Patraring
Appl
ROMO.
1130
Mr.1
0V
5006
R2
1748
E00
Susasted
Prodetie
R000 = 000)
5147 2.54
2130
41422 2.11
0585
0232
1438
1150 51R
8142 5010
2.034
1341 2.13
1-15 = BU3
80778
000
6:45
0334
1.30-
Gedd01 Invvace
31
0.09
19
15:34
10772
$3.64
007
3144
0.44
0031
1.00
1347
13.36
0134
1:000
1544
4.33
1.61
GodSL-OUM Swar)
0J966 0.90
46.5 0..33
25.53
833
0826
2000
0831
12.00 8.19
11.35 6M0
12:5::0.44
12.02:008 0.6
1437 6:91
Downstream results on force prediction tasks from MD17. We take 1K for training, 1K for validation, and the number of molecules for test are varied among different tasks, ranging from 48K to 991K. The evaluation metric is mean absolute error, and the best results are in bold. We report the both the mean and standard deviation for seeds 22, 32, and 42.
Table 17:
Pretraining
Aspiria
Ezhanol J
Malonaldehyde
Naphchalene
Salicylic
Tolnene
Uracil
1418
0.28
0.406
0.00
0.528
:: 0.00
0.908 + 0.04
0.613 = 0.06
0.854
0.08
0.575
0.01
0.717
Supervised
1,714
0.21
0.423
0.04
0.517
0.01
1.127 4: 6.166
0.713
0.14
L.114
0.08
0.578
0.832
0.002
Distance Prediction
1,756
0.24
0.483
0.02
0.813
0.02
1.458 0.08
0.795
0.15
1.074
2: 0.19
0.691
0.08
1.116
Geo8SL RRR
2,082
0.20
0.563
0.09
0.740
0.05
1.798 0:35
0.910
0.07
1,525
0.24
0.847
0.14
1.199
0.08
CooSSL-InfeNCE
1.375
0.07
0.432
0.03
0.560
0.05
1.101 + 0.12
0.797
0.02
1,020
0.02
0.706
0.934
0.00
SeoSSL-EBMNN
1.297
0.03
0.404
:: 0.00
0.569
:: 0.00
1.008
0.04
0.580
0.02
0.840
0.02
0.581
0.02
0.839
GeoSSL-DDM (ours)
1.333
0.25
0.379 +: 0.01
0.466
0.04
0.732 + 0.03
0.566 0.13
0,824
0.16
0.566 0.05
0,682
0.07
26

---

Published as a conference paper at ICLR 2023
G COMPARISON WITH A PARALLEL WORK
We note that there is parallel work introduced in [76], which also explores the effect of denoising for geometric data pretraining. That work is different from GeoSSL-DDM and we here summarize the main differences as follows:
The parallel work as presented in [76] is similar that of denoising score matching (DSM) as introduced in [69]. i.e., with only one layer of denoising in score matching, On the contrary, our model has multiple denoising layers, which is much closer the NCSN [58]. where the number of noise layers has been proven to be important to the effectiveness of the denoising score matching models. We here also empirically verify the above analysis. That is, we present the experimental results in Table 13, where L = 1 is equivalent to the method [76]. We can observe that with layer number L = 1 (namely the third row the table), the performance does increase in some cases, which matches with the observation in [76]. Nevertheless, the results in Table 13 clearly indicate that with larger L. the model can attain further enror reduction and improve model robustness.
Theoretically, the work in [76] specifically aims at the application task of representation learning in geometric pretraining, through straightforward adaption of denoising score matching from vision. In contrast, our GeoSSL-DDM approach indeed provides a very general framework that leverages energy-based model (EBM) for mutual information (MI) maximization for geometric data pretraining. As such, GeoSSL-DDM can be easily replaced by other EBM models such the GFlowNet network [3] to better capture the multi-mode distributions in geometric data during pretraining (please see Section 6 for more discussion).
27


Reviews:
[{"review_id": "V00zL-IazW2", "reviewer": "Reviewer_wxWw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to use denoising as pretraining for SE(3)-invariant neural network models. The main idea is to add some noise to the coordinates, and use a score model to predict the added noise. The pretrained model are used in several different tasks, including molecule property prediction, force field and binding affinity prediction. The proposed pretraining method was proved better than other pretraining methods in the tasks.", "summary_of_the_review": "This paper proposed to use denoising in training 3D roto-translation invariant neural networks, and proved its superiority in molecular property prediction, force field and binding affinity tasks. However, the experiments can be more convincing if ablation study and error bars are provided", "strength_and_weaknesses": "Strength: It is a good idea to use denoising in pretraining neural networks in molecular tasks, and the results showed that the idea is valid. The paper is written clearly.\n\nWeakness: The advantage of GeoSSL compared to other pretraining methods seems minor, as seen in Table 1, 2 and 3. I think it is necessary to include two more experiments: (1) ablation study on hyperparameters (2) error bars over multiple random seeds. With these two experiments, it will be more convincing that the advantages are really from pretraining, not hyperparameter-tuning or randomness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SQHwZoHkFvg", "reviewer": "Reviewer_a9bJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self-supervised method for 3d molecular data. The authors started with maximizing mutual information, and obtained an objective loss (denoising score matching). The idea is supported by several 3d geometric pretraining experiments.", "summary_of_the_review": "The paper proposed an interesting SSL method. But I am concerned about the computing resources needed due to the quadratic complexity.", "strength_and_weaknesses": "\nStrengths\n1. The derivation from mutual information maximization to denoising score matching is novel.\n2. The experiments are convincing.\n\nWeakness\n1. In Sec 4.3.1 Eq.(5), the definition of $d$ and $r$ are missing\n2. The calculation of $d$ is pariwise distances of all points (or atoms). If the number of points is very large, it could be very slow (quadratic complexity)?\n3. It seems that the method can also be used with general point cloud data (no edges between vertices), can the authors elaborate more about this? The complexity seems to be an important problem if we want to apply the method to large point clouds.\n4. The main contribution is the denoising distance matching. Is it appropriate to put \"SE(3)-invariant\" in the title?\n5. Why is this method superior to a vanilla denoising autoencoder (the simplest method we can come up with when dealing with SSL)? More discussions are needed\n6. How is the memory consumption given different sizes of input graphs?\n7. If I am understanding this correctly, the row marked with a dash (-) means not using any pretraining method (Table 1-3). It is just equivalent to training from scratch.\n8. In D.1, how many v100 cards do you use for a single experiment?\n9. Can the authors discuss a bit more about the limitations of the proposed method?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0tI84BuZqI", "reviewer": "Reviewer_WmAR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the presented work, the authors present an approach to perform self supervised learning on tasks where molecular 3D geometry is relevant. They do so by taking advantage of equivariant neural networks in combination with a denoising score matching method on molecular distances. ", "summary_of_the_review": "Overall, the work is original and in my belief deserves publication. However, I think the authors should probably reconsider some of the claims presented in the study, specifically those regarding them being the first to use pretraining strategies on 3D molecular data. Specifically, I believe that claim should be adapted based on the following works:\n\nhttps://arxiv.org/abs/2206.00133?context=q-bio\n\nThe authors should either compare to these approaches or alternatively discuss why their approach is theoretically different, or whether there are computational disadvantages in either of them.\n\nSome other comments/questions:\n\n•\tIt is not clear how the perturbed geometry g2 is obtained from g1 in Algorithm 1. \n\n•\tDo the authors have any suggestions on how to choose the different noise levels \\sigma_l in Algorithm 1?\n\n•\tIt is not clear from the text whether the Molecule3D dataset is a subset of the PubChemQC database.\n\n•\tIt is also not clear what the authors are referring to when they refer to the so-callsed “supervised pretraining baseline”. In particular I do not see where the pretraining is done here. From the text it appears that this is simply a supervised baseline?\n\n•\tIt would be fantastic if the authors could provide standard deviations for the numbers reported in Tables 1, 2, as they are reported in 3.\n\n•\tIn the downstream tasks on binding affinity prediction, is the pretraining done on both the pocket + ligand graph, or only on the ligand? Have the authors experimented with both of these approaches? In this section too, it is not clear to me what the second task (LEP) is. What is an active/inactive conformer of a protein? Are the authors referring to an active/inactive classification tasks for ligands (i.e. as in virtual screening)? Are the same molecules considered in both tasks?\n\n•\tSome references do not seem to be present in the main manuscript (e.g. 20)\n", "strength_and_weaknesses": "Strengths:\n* Strong empirical results\n* Novel idea\n* Clear methodology\n* Evaluation on several relevant biophysical datasets\n\nWeaknesses:\n* Hard to read at times\n* Does not reference a very related work, while claiming to be the first study to perform pretraining on 3D molecular data.\n* Does not provide code to support analyses\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster