Sliced Denoising: A Physics-Informed Molecular Pre-Training Method
Authors: Yuyan Ni
Shikun Feng
Wei-Ying Ma
Zhi-Ming Ma
Yanyan Lan
Venue: ICLR 2024 poster (2024)
Keywords: Molecule Representation; Pretraining; Denoising; Force Field
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: While molecular pre-training has shown great potential in enhancing drug discovery, the lack of a solid physical interpretation in current methods raises concerns about whether the learned representation truly captures the underlying explanatory factors in observed data, ultimately resulting in limited generalization and robustness. Although denoising methods offer a physical interpretation, their accuracy is often compromised by ad-hoc noise design, leading to inaccurate learned force fields. To address this limitation, this paper proposes a new method for molecular pre-training, called sliced denoising (SliDe), which is based on the classical mechanical intramolecular potential theory. SliDe utilizes a novel noise strategy that perturbs bond lengths, angles, and torsion angles to achieve better sampling over conformations. Additionally, it introduces a random slicing approach that circumvents the computationally expensive calculation of the Jacobian matrix, which is otherwise essential for estimating the force field. By aligning with physical principles, SliDe shows a 42\% improvement in the accuracy of estimated force fields compared to current state-of-the-art denoising methods, and thus outperforms traditional baselines on various molecular property prediction tasks.

Published as a conference paper at ICLR 2024
SLICED DENOISING: A PHYSICS-INFORMED MOLEC- ULAR
METHOD
Yuyan Nil-.+f, Shikun Feng 2; Weiying Ma², Zhiming Ma', Yanyan Lan Academy of Mathematics and Systems Science, Chinese Academy of Sciences Institute for Al Industry Research (AIR), Tsinghua University
Unniversity of Chinese Academy of Sciences
'Beijing Academy of Artificial Intelligence
ABSTRACT
While molecular pre- rre-tranning has shown great potential in enhancing drug discov- ery, the lack of a solid physical interpretation in current methods raises concerns about whether the learned representation truly captures the underlying explanatory factors in observed data, ultimately resulting in limited generalization and robust- ness. Although denoising methods offer a physical interpretation, their accuracy is often compromised by ad-hoc noise design, leading to inaccurate learned force fields. To address this limitation, this paper proposes a new method for molecular pre-training, called sliced denoising (SliDe), which is based chanical intramolecular potential theory. SliDe utilizes a novel noise strategy that perturbs bond lengths, angles, and torsion angles achieve better sampling over conformations. Additionally, it introduces random slicing approach that circum- vents the computationally expensive calculation of the Jacobian matrix, which otherwise essential for estimating the force field. By aligning with physical prin- ciples, SliDe shows a 42% improvement in the accuracy of estimated force fields compared to current state-of-the-arr denoising methods, and thus outperforms tra- ditional baselines on various molecular property prediction tasks]
the classical me-
INTRODUCTION
Molecular representation learning plays a crucial role variety drug discovery tasks, includ- ing molecular property prediction (Shhit et al. 2018 2021; Thölke & Fabritis 2022). molecular generation (Bilodeau et al. 2022, Jing et al.] 2022), and protein-ligand binding (Gao et al. 2023]
Zheng et al. 2019). To overcome the challenge of insufficient labeled data, various molecular pre- training methods have been proposed to obtain a universal molccular representation, including the
contrastive approaches (Fang et al.) 2022 Wang al. 2022 Stärk al.
2022 Li al. 2022) and the predictive approaches (Rong et al. 2020 Fang et al. 2021 Zhu et al.
2022 Liu et al. 2023aa)
According to [engio al. (2013), good representation is often that captures the posterior distribution of the underlying explanatory factors for the observed input data. Regarding molec- ular representation, we posit that an ideal representation must adhere to the underlying physical principles that can accurately and universally illustrate molecular patterns. However, the majority of existing pre-training methods draw inspiration from pre-training tasks in computer vision and natural language processing and thus overlook the underlying physical principles.
Nevertheless, designing self supervised tasks that align with physical principles remains challeng- ing. To the best of our knowledge, only one kind of the unsupervised molecular pre- trainining method has an explicit physical interpretation, i.e. the 3D denoising approach (Zaidi al. [023) Feng al., 2023), which aims to learn approximate force field for molecules. However, we have found that this approximate force field largely deviates from the true force field, due inappropriate assump- tions such assuming molecular force field is isotropic in coordinate denoising (Zaidi et al.
* 'Equal contribution. Work was done while Vuyan was a research intern at AIR, Correspondence to 1 a lanyan mayanyan@aii r. ts wet.tinnnnii inguua wwghua.edd.com edu.cn. Thee code is released publicly at attttps: httt:::://tthubb.cmm///gggg/h kut iie

---

Published as a conference paper at ICLR 2024
treating certain parts being isotropically in fractional denoising (Feng et al. 2023). Con- sequently, existing denoising methods still harbor a significant bias from physical laws, which can hinder downstream results, depicted by the experiments conducted in Feng et al. (2023) and our own experiments in Appendix [B.5] Therefore, remains an essential issue to design a denoising method that better aligns with physical principles.
2023)
should be noted that energy function is a pivotal factor in determining the quality of represen- tation learning in denoising methods. Firstly, the Boltzmann distribution used for noise sampling, which determines the conformations on which the network learns its force field, is derived from the energy function. Secondly, the learned force field, which aims align regression targets with the true molecular force field, is designated by the gradient of the energy function. As a result, a precise energy function facilitates the network to acquire accurate force fields for typical molecules, consequently enhancing the physical consistency of the representation.
Following the aforementioned analysis, we suggest utilizing the classical mechanical intramolecular potential energy function and approximating it in the quadratic form using relative coordinates, ie. bond lengths, angles, and torsion angles, with certain parameters. Inspired by the previous theoretical findings that associate the quadratic energy function with a Gaussian distribution through the Boltzmann distribution, we then propose a novel noise strategy, called BAT noise. Specifically, BAT noise introduces Gaussian noise to bond lengths, angles, and torsion angles, and their respective variances are predetermined by parameters within the energy function. This approach allows BAT noise to better approximate the true molecular distribution when compared to other existing methods. The resulting conformations from our strategy are closer to common low-energy structures than previous approaches, providing an advantage for effective representation learning.
The objective of the denoising target is to regress the molecular force field, i.e. the gradient of the energy function w.r.t. Cartesian coordinates. However, the energy function defined relative coordinates, thus requiring change of variables in the differential. Specifically, the gradient of the energy function in relation relative coordinates is readily acquirable in the form of the by-term product of the BAT noise and the parameter. Applying a variable change requires estimation of the Jacobian matrix of the coordinate transformation function, which nevertheless computationally expensive. To address this issue, we introduce a random slicing technique that converts the Jacobian estimation into simple operations coordinate noise additions and BAT noise acquisitions.
Thus we have developed a novel and efficient method, known as sliced denoising (SliDe). which is equivalent to learning the force field of the utilized energy function. Consequently, SliDe possesses the ability to align better with physical principles by estimating more precise force field. To facilitate the learning process, we introduce a Transformer- based network architecture that explicitly encodes relative coordinate information and generates equivariant atom-wise features tailored for the sliced denoising task. Our contributions are summarized as follows:
1) Methodologicallly we suggest the use of physical consistency a guiding principle for molecular representation learning, and under this principle, we develop a novel sliced denoising method and corresponding network architecture.
2) Theoretically, we derive BAT noise from the classical mechanical energy function and establish the equivalence between learning the force field and our sliced denoising method.
3) Experimentally, we demonstrate that SliDe outperforms existing pre-training methods in terms of physical consistency and downstream performance on QM9, MD17 and ANI-Ix datasets.
2 BACKGROUND
Denoising is a kind of self-supe learning task in molecular representation learning and has achieved outstanding results in many downstream tasks (Zhou et al.) 2023 Feng al. 2023 Zaidi ctt al.) 2023 Luo et al.| 2023 Liu et al., 2023b Jiao et al. 2023). It refers to corrupting original molecules by specific noise and training the neural networks predict the noise, thus reconstructing the molecules. Significant benefit of denoising over other pre-training methods is that it has been proven to be equivalent learning molecular force field, which is physically interpretable.
Coordinate denoising (Coord) (Zaddi et al. 2023) involves the addition of Gaussian noise to atomic coordinates of equilibrium structures, with subsequent training of the model to predict the noise
2

---

Published as a conference paper at ICLR 2024
Coordinate Denoising
Fractional Denoising
Sliced Denoising
Bond stretching
Geeeo Allo
AB
RE
Angle bending
types V Bond types V
Atom types x Bond types
Atom types Rotatable bonds
104 roll
(x (a zo.
Merue(te) to f(( 20) Eijjay mi)
Moll d.]] ( du)
Boad tarsion
Figure a. Illustrations of bond stretching, bond angle bending, and bond torsion interactions. b. Comparison of the three denoising methods in terms of energy functions. Coordinate denoising learns an isotropic energy in Cartesian coordinates that does not discriminate different atom types and bond types. Based coordinate denoising, fractional denoising treats the rotatable bonds in special. In contrast, sliced denoising performs fine-grained treatment for different atom types and bond types, enabling the most physically consistent description of the molecule.
from the noisy input. They establish the equivalence between coordinate denoising and force field learning, under the assumption of isotropic Gaussian noise. For given sampled molecule M, perturb the equilibrium structure 2g by p( (2)) ~ (x0. l3x) where æ denotes the noisy con- formation, N denotes the number of atoms in the molecule, and IaN the identity matrix of size 3N, the subscript stands for the coordinate denoising approach. Assume the molecular distribution satisfies the energy-biad based Boltzmann distribution w.r.t the energy function ECoord, then
LCoora(M) = (p((llo)p((po)) (ll.aaeeeeGGGGNNeee ||GNNN((x)) - (æ zn)ll
(1)
E ( (- (z)
(2)
where GN refers to a graph neural network with parameters 0 that takes the conformation
a as input and returns node-level predictions. The notation indicates the equivalence between different optimization objectives for the GNN. The proof is supplemented in the appendix A
To account for the anisotropic molecular distribution, fractional denoising (Frad) (Feng et al. 2023) proposes introducing hybrid noise the dihedral angles of rotatable bonds and atomic coordinates and fractionally denoising the coordinate noise. This specially designed denoising task allows for physical interpretation of learning force field. For a given sampled molecule M. the equilibrium structure Dg is perturbed by p(tal ebb) ~ N (N of and p(z xa)) ~ (( IaN) ). where sb and tho correpond to the dihedral angles of rotatable bonds in structures v, and æn respectively, with representing the number of rotatable bonds, and the subscript standing for Frad. Assume the molccular distribution satisfies the energy-base based Boltzmann distribution w.r. the energy function Errad, have
LFrad(M) = (tne - (x xa)ll
(3)
Em(m) |NN - a
(4)
The proof is also supplemented in the appendix A summary of denoising pre-training methods is provided in Appendix D.I
The aforementioned work has made efforts to learn physically interpretable molecular representa- tions by designing noise distributions and their corresponding energy functions based on certain chemical priors. However, their energy function is saare-grained as shown in Figure [Tp., lack- ing the capability to capture highly complex interaction information, such as bond stretching, angle bending, and bond torsion composed of different bond types and atom types. contrast, our noise distributions and force fields derived from classical mechanical energy function, which is more consistent with the characteristics of true molccules.
OUR APPROACH
Inspired by the aforementioned deduction, conclude that in pursuit of designing an effective and interpretable denoising pre-training task, physical consistency can be achieved by developing energy function that accurately approximates the true molecular energy. This, in turn, leads to
3

---

Published as a conference paper at ICLR 2024
better noise distribution capable of sampling low-energy molecules, and correspondingly improved force field learned through denoising. Following this guiding principle, we first establish a physical informed energy function in section followed by the design of a noise distribution in section 3.2 In section 3.3 we present a denoising task aimed at learning the force field of the aforementioned energy function. Finally, in section 3.4 we introduce the network architecture developed for denoising method.
3.1 ENERGY FUNCTION
According to classical molecular potential energy theory (Mol 2020 Zhou Liu| 2022). the total intramolecular potential energy can be attributed to five types of interactions: bond stretching, bond angle bending, bond torsion, electrostatic, and van der Waals interactions. Figure Dh. depicts the first three interactions. The energy function, in its general form, can be expressed as follows:
E(r,0,)) =
Σ
k (r: T2,0)² + Σ iEA
(0; 02,0)
iEB
(5)
Σ
b co-ccossu(( o,,)))) + Eetec + EvdW,
where r. 0, and & represent vectors of the bond lengths, bond angles, and bond torsion angles of the molecule, respectively. The index i corresponds to the element in the vector. To. On. and do correspond to the respective equilibrium values. The parameter vectors k kA, and kz determine the interaction strength, while the parameter vectors determine the torsion periodicity. The index set IB, A, correspond to the bonds, angles, and torsion angles in the molecule, respectively.
In order to approximate it as quadratic form, which is often required to enable the equivalence based on previous proof, we put forward two approximation operations. Firstly, when Φ + do, Taylor expansion utilized to express the bond torsion interaction in the quadratic form:
1 [wi(o )0,)))² o((o: - x0.0) 2 _ui(es 21.0)²
1
cos(wi(0 ds,o))
[1- 2
The approximation is reasonable since the noise scale in denoising methods is usually small. Sec- ondly, we drop the last two terms in order to get quadratic form of energy function in equation 6 Despite these long-range electrostatic and van der Waals interactions are important in classical sim- ulations, we find the accuracy of the approximated energy function is much higher than existing self-supervised pre-training methods in section 4.1
EBAT(r, 0)
Σ (r Tx.0)² 18
Σ ZEA
- -0,0 EMT - 0x.0)². (6)
1ET
In Figure [p., we compare our energy function with that of Coord and Frad. Their formulations provide a general outline of the energy function in averaged manner using only rameters. Unfortunately, they fail to capture the nuanced energetic characteristics of molecules. In contrast, our energy function carefully describes the impact of different atomic types and bond types on energy using specific parameters. Therefore, our approach is more closely aligned with the true physical properties of molecules.
two pa-
3.2
NOISE DESIGN
With the common assumption that the conformation distribution of a molecule follows the Boltz- mann distribution (Boltzmann) 1868), i.e. p xx ezp(-EE, we can derive the conformation distribu- tion corresponding to our quadratic energy function.
p(r, O, Φ)
(F,A-EET 0,Φ))
2B erp( B.B (Ti 1E3
r 2
Z cap( cop(-kA11 (0, 0,,0)
(b.
&r,0) )
II
II ET
exp( 17 w/
2
Z
2
KEA
(8)
where Z, - _ 2,T are normalization factors. According equation the conformation dis- tribution can be expressed as joint distribution of independent Gaussian on bond lengths, bond angles, and torsion angles. Therefore outline the following noise strategy.
4

---

Published as a conference paper at ICLR 2024
Definition 3.1 (BAT noise). The BAT noise strategy refers to perturbing the equilibrium structure by adding independent Gaussian noise on every bond length, angle and torsion angle:
~~~ Nro, diag( 1 1) k NN diag( ka )). Φ NN(ou, diag( kT @ w
(9)
KB
where @ means multiply item by item, diag(.) represents diagonal matrix whose diagonal elements are the elements of the vector. The variances are determined by the parameters that can be obtained in prior, such as from the parameter files of molecular simulation tools.
Detail implementations of the BAT noise can be found in Appendix C.I Since EвAТ approxi- mates the true molecular energy function, the sampling distribution of BAT noise resembles the true molecular distribution. This guarantees realistic sampled conformations that are beneficial for learning effective representations.
3.3 SLICED DENOISING
Since the energy function is based on bond lengths, angles, and torsion angles, the gradient of the energy function can be represented as simple form with respect to the relative coordinate:
VaERar(d) [k @ (r rr), A
0o), d2d
(Φ (h>
(10)
where d (r. 0, Φ) However, we need to derive simple expression for the gradient of our energy function with respect to Cartesian coordinates and ensure the learning of the force field of EBAT by minimizing
Ep(m(/z)) (G - V=EBAT (((BAT(((x)))))) (d(x))11²
(11)
For this purpose, we propose expanding the gradient using the chain rule and expressing the force field target as the gradient of the energy function with respect to the relative coordinates and Jacobian matrix of the coordinate transformation. A rigorous formulation is presented as follows.
Firstly, we define coordinate transformation function for a molecule M that maps from Cartesian coordinates relative coordinates:
f. R - (Rzo) m1 x (10,2 22 x ([0,2T) 2x)
(12)
x + (r,
where mi, m2, and my are numbers of bonds, angles, and torsion angles respectively. The mapping is well-defined, as these values can be uniquely determined by the Cartesian coordi- nates. Although 0, Φ are defined on torus, we can establish homeomorphism between the Euclidean space I8 +ma and ((0, Br)M(pt } ma x ((0, Prr)(r)) }mms where Pi.P are any points in m2. 1....m mg (Zorich 2016). As the denoising method only involves confor- mations in small neighborhood V around the equilibrium conformation do. transformation such that V E ((0, 2x) (pr)) x ([0, 2 [ {p,})m² function defined
[0, 2 i =
select p., PA
Consequently, the coordinate neighborhood can be regarded as mapping between the Euclidean spaces R3N RM, where M mi + m2 + m3-
Assume that the coordinate transformation function fM is continuously differentiable with contin- partial derivatives. Then the force field can be expressed by
V a FBAT((())) VaFnAr(d) J(x).
(13)
(m)
bz
BzaM
where J(x)=
R MxaN is the Jacobian matrix. Then the target in
(an)
ay (n)) BeaN
equation 11 can be written as
(14) For each noisy conformation, the Jacobian matrix can be estimated via numerical differentiation, which is time-consuming. To efficiently learn the force field, we devise cleverly designed asymp- totically unbiased estimator that does not require the computation the Jacobian matrix, by utilizing two computational techniques.
Ep(=)s)) |GNNN()) - -
Firstly, a random slicing technique is introduced to estimate the target regression loss through the projection of the GNN and force field onto random vectors, as illustrated by lemma
5

---

Published as a conference paper at ICLR 2024
Lemma 3.2 (Random Slicing). Va, b. R > 0, 2 NNO, (IN)) then
a - bll 0² Evl(a b) vl?
(15)
An intuitive illustration of lemma 3.2 is that the L2 norm of the vector b equals the expectations of the vector projected onto Gaussian random vectors. Our random slicing technique, which can be seen as a more general approach, has been inspired by the idea of Sliced Wasserstein distance(Rabin ct al. 2012).
After that, the dot product of the Jacobian matrix and the random vector can be efficiently calculated with the assistance of lemma 3.3
Lemma 3.3 (Differential of Coordinate Transformation Function). Va, R3N molecular conformation and Cartesian coordinate noise respectively, then
+ J(x) v = (a + (x) a(x;)),
(16)
where a<x; v) = o(v) as ||0|| +
Therefore, the ultimate loss function can be defined as follows.
No
CSNDD(MM = Epp(a) mo) Σ
(NNNo(=)
ws 1 [ (IM -
(17)
where v, (O, TaN), is parameter. and d are the Cartesian coordinates and relative coordinates of the structure after adding the BAT noise to the equilibrium structure, respectively. VallBAT is given in equation 10 and GNN)(w) E 13N denotes the prediction output of GNN for each atomic Cartesian coordinate.
The computation of the scalar target can be performed rapidly by leveraging the relative coordinate obtained after adding the Cartesian noise v through the utilization of RDKit (Landrum et al. 2013),
readily available cheminformatics tool.
Consequently, the total loss is averaged on every sample in the 3D equilibrium molecular dataset Mi:
ctotal StiDe
1
Σ
LsuDe(M).
(18)
M
MEM
In practise, we utilize stochastic gradient descent and approximate the total loss by batch loss. For reference, the pseudo-code outlining the approach for performing SliDe denoising pre-training is presented in Appendix C.2
Furthermore, have proven its equivalence learning the force field of EBAT shown in the following theorem, with the proof provided in appendix
Theorem 3.4 (Interpretation of Sliced Denoising). Given equilibrium structures, when ap- proaches O and N, approaches oo, minimizing (suDe(M) is equivalent to learning the force field of EBAT in Cartesian coordinate in equation 17
3.4 NETWORK ARCHITECTURE
Compared to previous denoising methods, our approach defines energy and noise w.r.t. relative coor- dinates. Relative coordinates provide a complete representation of molecular structure and conform the molecular symmetry, thereby offering advantages for molecular modeling. Further details about related work on 3D molecular modeling in relative coordinates can be found in Appendix [.2]]
While TorchMD-NET (Thölke & Fabritiis, 2022) has achieved competitive results when applied in denoising tasks, as shown (Zaidi et al. [0023] Feng et al.] 2023), employs Cartesian coordinates to inject geometry information and does not explicitly model the angles and torsion angles. Since our method explicitly utilizes relative coordinates to model energy and noise, we believe angular information is important for learning our force field target. Therefore, in addition to the vertex update in TorchMD-NET, we also incorporate edge update and introduce angular information in the edge embeddings. These edge embeddings are then utilized in the attention layer, which impacts the vertex update. Our network is denoted as the Geometric Equivariant Transformer (GET), and further details are outlined in Appendix

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
Our first experiment in section 4.1 is concerned with whether our approach achieves better physical consistency, specifically in terms of force field accuracy, as compared coordinate denoising and fractional denoising methods. Then in section 4.2 we evaluate the performance of SliDe in com- parison to state-of-the-an 3D pre-training methods on the benchmark datasets QM9 and MD17, in order to assess our model's 's ability for molecular property prediction. Furthermore, in section 4.3 we conduct ablation studies concerning fine-tuning regularization and network architecture. Additional experiments on the benchmark datasets ANI-1x, the impact of physical consistency on downstream tasks, and ablation studies related to the hyperparameters in SliDe loss, robustness to noisy pre- training data and effect of pre-training data scale can be found in the Appendix B Implementation details of the experiments can be found in the Appendix
4.1 EVALUATIONS ON PHYSICAL CONSISTENCY
To estimate the learned force field in SliDe, we calculate the Cartesian force field for each molecule
M by solving a least square estimation problem A b, where A [01, .UN.] E R NNN.3N X3N
b; = V AT (d) T (f ov;) fM)()) Vi = 1 ... No, b = [b, bx.
T
E R N. xaN
We can prove that the regression loss EM(M)M) [GNN x 1 is asymptotically an equivalent
optimization problem to SliDe. Therefore y can be viewed as the learned force field target in SliDe. Details can be found in appendix proposition A.
To verify the accuracy of the learned force field in various denoising methods, we compare the corre- lation coefficient between the learned force field and the ground-trute force field calculated by den- sity functional theory (DFT). Since obtaining the true force field label can be time-consuming, the experiment is carried out on 1000 molecules randomly selected from dataset PCQM4My2 (Nakata & Shimazaki] 2017). The noisy conformations generated according to each denoising method and the learned force fields Frad and Coord are estimated as the approach in [Feng et al. (2023).
Table 1: Correlation coefficient between the learned force field and the ground-truth force field of the three methods. The standard deviation is shown in parentheses. The top results are in bold.
Denoising method
Coord
Frad
SliDe
Correlation coefficient
0.616(0.047)
0.631 (0.046) 0.895 (0.071)
The experimental results in Table [ indicate that SliDe increases the correlation coefficient of the estimated force field by 42%, compared Frad and Coord. This confirms that the design of our energy function and sliced denoising can help the model learn more accurate force field than other denoising methods, which is consistent with our theoretical analysis. In addition, our result on Frad and Coord is in line with the experimental results in Feng et al. (2023), although the experiments are carried out on different datasets. It has been verified Feng et al. (2023) that learning an accurate force field in denoising can improve downstream tasks. We also conduct a supplementary experiment in Appendix B.5 to confirm the conclusion. As a result, SliDe greatly improves the physical consistency of the denoising method, enabling the learned representations to have better performance on downstream tasks.
The validation of force field accuracy can also help us choose hyperparameters without training neural networks. Details are in Appendix B.2
4.2 EVALUATIONS ON DOWNSTREAM TASKS
Our model is pre-trained on PCQM4Mv2 dataset (Nakata Shimazaki] 2017). which contains 3.4 million organic molecules and provides one equilibrium conformation for each molecule. Follow- ing previous denoising methods, we apply the widely-used Noisy Nodes technique (Godwin et al.) 2021), which incorporates coordinate denoising as auxiliary task in addition to the original prop- erty prediction objective in the fine-tuning phase. Nevertheless, we observe the hard optimization of Noisy Nodes in SliDe. To get the most out of the fine-tuning technique, we add regularization term in pre-training loss, i.e. IGN No(x + +D)
TU]² An ablation study on the regularization term is

---

Published as a conference paper at ICLR 2024
Table 2: Performance (MAE 1) on QM9. The best results are in bold.
homo (meV)
Jumo (meV)
gap (meV)
R²
ZPVE
Up (mie)
U
H
G
C
(a))
(o))
(meV)
(mcV)
(mcV)
weali
SchNet
0.033
0.235
41.0
34.0
63.0
0.07
1.70
14.00
19.00
14.00
14.00
0.033
H(n)-GNN
0.029
0.071
29.0
25.0
48.0
0.11
1.55
11.00
12.00
12.00
12.00
0.031
DimeNet++
0.030
0.044
24.6
19.5
32.6
0.33
1.21
6.32
6.28
6.53
7.56
0.023
PaiNN
0.012
0.045
27.6
20.4
45.7
0.07
1.28
5.85
5.83
5.98
7.35
0.024
SphereNet
0.025
0.045
22.8
18.9
31.1
0.27
1.120
6.26
6.36
6.33
7.78
0.022
ET
0.011
0.059
20.3
17.5
36.1
0.033
1.840
6.15
6.38
6.16
7.62
0.026
TM
0.037
0.041
17.5
16.2
27.4
0.075
1.18
9.37
9.41
9.39
9.63
0.022
SH(3)-DDM
0.015
0.046
23.5
19.5
40.2
0.122
1.31
6.92
6.99
7.09
7.65
0.024
3D-EMGP
0.020
0.057
21.3
18.2
37.1
0.092
1.38
8,60
8.60
8.70
9.30
0.026
Coord
0.012
0.0517
17.7
14.3
31.8
0.4496
1.71
6.57
6.11
6,45
6.91
0.020
Frad
0.010
0.0374
15.3
13.7
27.8
0.3419
1.418
5.33
5.62
5.55
6.19
0.020
SliDe
0.0087
0.0366
13.6
12.3
26.2
0.3405
1.521
4.28
4.29
4.26
5.37
0.019
provided in section 4.3.1
Hyperparameter settings for pre-training and finetuning summarized
in Appendix C.4
Our baselines include both 3D pre-training approaches, such as fractional denoising (Frad), coordi- nate denoising (Coord), 3D-EMGP (Jiao et al. 2023), SE(3)-DDM (Liu et al. 2023b), Transformer-
M (TM) (Luo et al. 2023), well supervised models such TorchMD-NEL (ET) (Thölke & Fabritiis 2022). SphereNet (Liu et al. 2002). PaiNN (Schitt et al. (021). (in)-NN((Sttoran et al.]
2021), DimeNet (Gasteiger et al.] 2020b), DimeNet++ (Gasteiger et al., 2020a), SchNet (Schümt)
et al. 2018). The results for these baselines are directly taken from the referred papers, except for Coord on MD17, which is produced by us due to its absence in their paper.
4.2.1 QM9
QM9 (Ramakrishnan et al. 2014 is quantum chemistry dataset providing one equilibrium con- formation and 12 labels of geometric, energetic, electronic, and thermodynamic properties for 134k stable small organic molecules made up of CHONF atoms. The data splitting follows standard set- tings which have a training set with 1 110,000 samples, a validation set with 10,000 samples, and a test with the remaining 10,831 samples. The performance 12 properties is measured by mean absolute eттor (MAE, lower is better) and the results are summarized in Table 2
First of all, our model achieves new state-of-the-art performance on 10 out of 12 tasks in QM9, reducing the mean absolute error (MAE) by 12.4% compared to the existing state-of-the-art. Among them, SliDe performs particularly well on challenging energetic and thermodynamic tasks. We speculate that this is because these two tasks are more closely related to molecular potential energy and force fields that we focus on during pre-training, for instance, the potential energy related to thermodynamic quantities as illustrated in (Saggion et al. 2019). It is worth noting that the downstream performance of the three interpretable methods, SliDe, Frad, and Coord, is in agreement with the result of learned force field accuracy in section 4.1 i.c. SliDe demonstrates the strongest performance while Frad outperforms Coord. These experimental findings once again confirm the importance of physical consistency to molecular representations.
4.2.2
MD17
MD17 (Chmiela et al. 2017) is a dataset of molecular dynamics trajectories of 8 small organic molecules. For each molecule, 150k nearly 1M conformations, corresponding total energy and force labels are provided. We choose the challenging force prediction as our downstream task. The data splitting follows standard limited data setting, where the model is trained on only 1000 samples, from which 50 are used for validation and the remaining data is used for testing. The performance is also measured by mean absolute error and the results are summarized in Table] 3
Despite the fact that the downstream task is closely related to our pre-training task, the input con- formations in MD17 are far from equilibrium and the limited training data setting makes even more challenging. In this case, we still outperform achieve comparable results as compared with recent baselines, indicating that the force field knowledge learned in SliDe pre-traine ge-trrining is effectively transferred to the downstream force field task.
8

---

Published as a conference paper at ICLR 2024
Table 3:
Performance (MAE 1) on MD17 force prediction (kcal/mol/ A). The best results are in bold. *: PaiNN does not provide the result for Benzene, and SE(3)-DDM utilizes the dataset for Benzene from Chmiela et al. (018). which is different version from ours CChmiela al. 2017).
Malonal dehyde
Naphtha -lene
Salicy -lic Acid
Aspirin
Benzene
Ethanol
Toluene
Uracil
SphereNet
0.430
0.178
0.208
0.340
0.178
0.360
0.155
0.267
SchNet
1.35
0.31
0.39
0.66
0.58
0.85
0.57
0.56
DimeNet
0.499
0.187
0.230
0.383
0.215
0.374
0.216
0.301
PaiNN*
0.338
0.224
0.319
0.077
0.195
0.094
0.139
ET
0.2450
0.2187
0.1067
0.1667
0.0593
0.1284
0,0644
0.0887
SB(3)-DDM*
0.453
0.166
0.288
0.129
0.266
0.122
0.183
Coord
0.2108
0.1692
0.0959
0.1392
0.0529
0.1087
0.0582
0.0742
Frad
0.2087
0.1994
0.0910
0.1415
0.0530
0.1081
0.0540
0.0760
SliDe
0.1740
0.1691
0.0882
0.1538
0.0483
0.1006
0.0540
0.0825
4.3
ABLATION STUDY
We conduct an ablation study to examine the impact of the regularization term introduced for better fine-tuning and to evaluate the performance of our modified network architectures.
4.3.1 REGULARIZATION TERM
To assess the effectiveness of the regularization term proposed for pre-training SliDe, we conduct pre-training with and without regularization and subsequently fine-tuned the models on three QM9 tasks. The network architecture remains consistent across all three setups, and the Noisy Nodes are implemented with the same configuration. The result is shown in the bottom three rows of Table Our findings indicate that the regularization term can effectively improve the performance of downstream tasks. Notably, SliDe without regularization still outperforms training from scratch and yields similar performance to Frad. Moreover, we observe in experiment that the regularization reduces the downstream Noisy Nodes loss, suggesting that the regularization term contributes to optimizing Noisy Nodes.
Table 4: Ablation study for regularization term.
Table 5: Ablation study for network design.
QM9
homo
lumo
gap
MD17 force prediction
Aspirin
Benzene
Train from scratch
17.6
16.7
31.3
SliDe w/o regularization
15.0
14.8
27.7
SliDe (ET)
0.2045
0.1810
SliDe (GET)
0.1740
0.1691
SliDe w/ regularization
13.6
12.3
26.2
4.3.2 NETWORK DESIGN
To show the advantage of the improved network to our SliDe, we pre-train the geometric equivariant Transformer (GET) and TorchMD-NET (ET) by sliced denoising and fine-tune them on MD17. As shown in T our network further improves the performance, indicating the excellence of our novel network in depicting more intricate geometric features, such as angles and torsional angles.
5 CONCLUSION
This paper proposes novel pre-training method, called sliced denoising, for molecular represen- tation learning. Theoretically, harbors a solid physical interpretation learning force fields on molecular samples. The sampling distribution and regression targets are derived from classical me- chanical molecular potential, ensuring more realistic input conformations and precise force field estimation than other denoising methods. Empirically, SliDe has shown significant improvements in force field estimation accuracy and various downstream tasks, including QM9 and MD17, as compared with previous supervised learning and pre-training methods.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work is supported by National Key R&D Program of China No.2021 and Beijing Academy of Artificial Intelligence (BAAI).
We thank anonymous reviewers for constructive and helpful discussions.
REFERENCES
Intra- and Intermolecular Potentials in Simulations, chapter 3, pp. 39-71. John Wiley & Sons, Ltd, 2020. ISBN 9783527699452. doi: http:://wouugg/////0000222222225 URL https:
//online httpa//lwwwa.a...........ooooo//
ch3
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, (588)) 2013. doi: 10.1109//PAM..2013355
Camille Bilodeau, Wengong Jin, Tommi Jaakkola, Regina Barzilay, and Klavs F. Jensen. Gen- erative models for molecular discovery: Recent advances and challenges. WIREs Computa- tional Molecular Science, 12(5):11688, 2022. doi: https://doi.orr/10.. 1002//wmss 1608. URL
online www.linnenbbrr....m http://eee..ooiinellaaaaaaaaaaar wttp://o//d/ooooooceceoooo 100.:00022wwaaa 1608
Ludwig Boltzmann. Studien uber das gleichgewicht der lebenden kraft. Wissenschaafiliche Abhand- lungen, 1:49-96, 1868.
Simon Boothroyd, Pavan Kumar Behara, Owen C. Madin, David Hahn, Hyesu Jang, Vytau- tas Gapsys, Jeffrey R. Wagner, Joshua T. Horton, David L. Dotson, Matthew W. Thompson, Jessica Maat, Trevor Gokey, Lee-Ping Wang, Daniel Cole, Michael K. Gilson, John D. Chodera, Christopher I. Bayly, Michael R. Shirts, and David L. Mobley. Development and benchmarking of open force field 2.0.0: The sage small molecule force field. Journal of Chem- ical Theory and Computation, 19(1 (M11))3551-3275 2023. doi: 1al.///cc.cct...eee
URL
httpps://de http://ooo.oggg//// wor./orggo... 1021/ acs jotc. 3cC0039
PMID: 37167319.
Stefan Chmicla, Alexandre Tkatchenko, Huziel E Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, 3(5):e1 2017.
Stefan Chmiela, Huziel E. Sauceda, Klaus- Müller, and Alexandre Tkatchenko. Towards ex- act molecular dynamics simulations with machine-learned force fields. Nature Communications, 9(1). sep 2018. doi:
467-018-666--1 URL https: ///d// http:s:://co.or///..././ 1038/0.1033 10388
17541467-011--6111
Vicky Choi. On updating torsion angles of molecular conformations. Journal of chemical informa- fion and modeling, 46:438 44, 01 2006. doi: 10.1021/ci050253h.
Xiaomin Fang. Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geometry--nhanced molecular representation learning for property prediction. Nature Machine Intelligence, 4:127 134, 2021. URL https:
orq wwt.scccooo....oooooous.......ee
Yin Fang, Qiang Zhang, Haihong Yang, Xiang Zhuang. Shumin Deng, Wen Zhang, Ming Qin, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Molecular contrastive learning with chemical element knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 3968-3976, 2022.
Shikun Feng, Yuyan Ni, Yanyan Lan, Zhi-Ming Ma, and Wei- Wing Ma. Fractional denoising for 3D molecular pre-training. In Proceedings of the 40th International Conference Machine Learning. volume 202 Proceedings of Machine Learning Research, pp. 9938 19381.996.. PMLR, 23-29 Jul 2023. URL httpttps:://pooceeddeio http:/gg.......//eeseee
wwl..pprss//ee p22ppsss/2222. 2ttp2//en222333 html
10

---

Published as a conference paper at ICLR 2024
Zhangyang Gao, Cheng Tan, Jun Xia, and Stan Z. Li. Co-supervised pre-training of pocket and ligand. In Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, and Francesco Bonchi (eds.), Machine Learning and Knowledge Discovery in Databases: Research Track, pp- 405-421, Cham, 2023. Springer Nature Switzerland. ISBN
Johannes Gasteiger, Shankari Giri, Johannes Margraf, and Stephan Günnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. In Machine Learn- ing for Molecules Workshop, NeurIPS, 2020a.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molec- ular graphs. In International Conference on Learning Representations (ICLR), 2020b.
Jonathan Godwin, Michael Schaarschmidt, Alex Gaunt, Alvaro Sanchez- Yulia Rubanova, Petar Velivckovi'c, c, James Kirkpatrick, and Peter W. Battaglia. Simple gnn regularisation for 3d molecular property prediction and beyond. In International Conference on Learning Representa- tions, 2021. URL https:://ppt semant iescholar. wwtp/orrgssaoooooo 10: 247450503
Tim Hsu, Tuan Anh Pham, Nathan Daniel Keilbart, Stephen E. Weitzner, James Chapman, Peng- hao Xiao, S. Roger Qiu, Xiao Chen, and Brandon C. Wood. Efficient and interpretable graph network representation for angle-dependent properties applied to optical spectroscopy. npj Computational Materials, 8:1-9, 2022. URL. Sittps://apt www.nticsccooooo..oo...oooocoooe
Corpus TD: 250535082
Rui Jiao, Jiaqi Han, Wenbing Huang, Yu Rong, and Yang Liu. Energy-motivated equivariant pre- training for 3d molecular graphs. Proceedings of the AAAI Conference on Artificial Intelligence,
37(7):8096 8104, Jun. 2023. doi: 10.1609/aaii v37i7 25977.. URL https: /ttp://js..co aaai.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Tor- sional diffusion for molecular conformer generation. In Advances in Neural In- formation Processing Systems, volume 35, pp. 24240-24253, Curran Associates, Inc., wtwppsrrrcpppppppppepeeeeeee pttpppprrrrfllleppap///22222 File/ E1
2022. URL httpp:://prroceeddnss...oe
Johannes Klicpera, Florian Becker, and Stephan Gunnemann. Gemnet: Universal directional graph neural networks for molecules. In Neural Information Processing Systems, 2021. URL https:
Greg Landrum et al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 2013.
Jie Li, Oufan Zhang, Seokyoung Lee, Ashley Namini, Hao Liu, João M. C. Teixeira, Julie D. Forman-Kan Kay, and Teresa Head-Gordon. Learning correlations between internal coordinates to improve 3d cartesian coordinates for proteins. Journal of Chemical Theory and Computation, 19 (14):4689-4700) 4700, 2023. doi: 10.102 1/acs. h/tacs/cccc.......ooeoee 2:01270. RRLL:ttpp://ee...r//IID...
acs. jct. 3cs. 2c01270 PMID: 36749957.
Shuangli Li, Jingbo Zhou, Tong Xu, Dejing Dou, and Hui Xiong. Geomgel: Geometric graph contrastive learning for molecular property prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4541-4549, 2022.
Shengehao Liu, Weitao Du, Zhi-Ming Ma, Hongyu Guo, and Jian Tang. group symmet- ric stochastic differential equation model for molecule multi-modal pretraining. In Proceed- ings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 21497-21526. PMLR, 23-29 Jul 2023a. URL https: //proceedings.. mlr. press
www.s.sssl.........mmmmmmmm.meee v202 / v2trss//222/11223h html
Shengchao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with SE(3)-invariant denoising distance matching, In The Eleventh International Conference on Learning Representa- tions, 2023b.
Yi Liu, Limei Wang, Meng Liu, Yu- Yu-Ching Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spheri- cal message passing for 3D molecular graphs. In International Conference on Learning Represen- fations, 2022. URL https: //api..ppp semanticscho ar htt.a//..........ooooooooooooooa

---

Published as a conference paper at ICLR 2024
Shengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie- Yan Liu, Liwei Wang, and Di He. One transformer can understand both 2D & 3D molecular data. In The Eleventh International Conference Learning Representations, 2023.
id=vZTplcPV3PG
Maho Nakata and Tomomi Shimazaki. Pubchemqe project: large-scale first-principles electronic structure database for data-driven chemistry. Journal of chemical information and modeling, 57 (6):1300-1308, 2017.
Julien Rabin, Gabriel Peyré, Julie Delon, and Mare Bernot. Wasserstein barycenter and its applica- tion to texture mixing. In Scale Space and Variational Methods in Computer Vision, pp. 435 446, Berlin, Heidelherg. 2012. Springer Berlin Heidelberg. ISBN 978-3-642-24785-9
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer large-scale molecular data. Advances Neural Information Processing Systems, 33: 1559-11257,, 2020.
Antonio Saggion, Rossella Faraldo, and Matteo Pierno. The Fundamental Relation and the Ther- modynamic Potentials, pp. 55-79. Springer International Publishing, Cham, 2019, ISBN 978- 3-030-26976-0. doi: 10. 000077978833030-267777 4. URL httphhttp///cci http://0...rr/////// 00.010.0007 978-33000--2676-8
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International conference on machine learning, pp.
PMLR, 2021.
Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning. pp. 9377-9388. PMLR, 2021.
Kristof T Schütt, Huziel Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(2))224 2018.
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Lid. 3D infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pp. 20479-20502. PMLR, 2022.
Qiming Sun, Xing Zhang, Samragni Banerjee, Peng Bao, Marc Barbry, Nick S. Blunt. Nikolay A. Bogdanov, George H. Booth, Jia Chen, Zhi-Hao Cui, Janus J. Briksen, Yang Gao, Sheng Guo, Jan Hermann, Matthew R. Hermes, Kevin Koh, Peter Koval, Susi Lehtola, Zhendong Li, Junzi Liu, Narbe Mardirossian, James D. McClain, Mario Motta, Bastien Mussard, Hung Q. Pham, Artem Pulkin, Wirawan Purwanto, Paul Robinson, Enrico Ronca, Elvira R. Sayfutyarova, Maximil- ian Scheurer, Henry F. Schurkus, James E. T. Smith, Chong Sun, Shi-Ning Sun, Shiv Upadhyay, Lucas K. Wagner, Xiao Wang. Alec White, James Daniel Whitfield, Mark Williamson, Sebas- tian Wouters, Jun Yang, Jason M. Yu, Tianyu Zhu, Timothy C. Berkelbach, Sandeep Sharma, Alexander Yu. Sokolov, and Garnet Kin-Lic Chan. Recent developments the PySCF program package. The Journal of Chemical Physics, 153(2)0024109 (77 2020. 0006074
ISSN 00021-6606. doi:
00.1063/5.0006074 URL
Philipp Thölke and Gianni De Fabritiis. Equivariant transformers for neural network based molec- ular potentials. In International Conference on Learning Representations. 2022. URL https:
//openreviee ew. met.fforte wetmmmiid==Hzzaa d=2NHzz222WWRR
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compu- tation, 2011.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. International Conference on Learning Representations, 2008
12

---

Published as a conference paper at ICLR 2024
Xu Wang, Huan Zhao, Weiwei Tu, and Quanming Yao. Automated 3D pre-training for molecular property prediction. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining, 2023a.
2591 44785
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks, Nature Machine Intelligence, 4(3):279-
287. 2022.
Yuyang Wang, Changwen Xu, Zijie Li, and Amir Barati Farimani. Denoise pretraining nonequi- librium molecules for accurate and transferable neural potentials. Journal of Chemical Theory and Computation, 19(15):5077-5087, June 2023b. ISSN 1549-9626. doi: 10. 10.1021/acss La..00//ccssjccc.300003393
URL /ttp::///...d...oo/11.........
jctc. 3c00289
Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via de- noising for molecular property prediction. In International Conference on Learning Representa- tions, 2023. URL https: /attps::/openreviee. htt.eviewwwweeeeeeeeeeeeeee mett/Eoorumm.ddd..M..o....
Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshann Aurélie C. Lozano, Payel Das, and Jian Tang. Protein structure representation learning by geometric pretraining. In The Eleventh International Conference on Learning Representations, 2023. URL
n Bttt//oorunnIddttt3[33333
Liangzhen Zheng, Jingrong Fan, and Yuguang Mu. Onionnet: a multiple-layer intermolecular- contact-based convolutional neural network for protein-ligand binding affinity prediction. ACS
Omega, (5111)))19555-199555 2019. doi: 10. 1l..////soomea..b/011 9b019977. URL htttps://doi.
org/ 10 10w.11011110 hatp://ccccaaaaaaaaaa/oaa 9b01997
PMID: 31592466.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: universal 3D molecular representation learning framework. In The Eleventh International Conference Learning Representations, 2023. URL https:
7 Tooenrevvew. net. www.mmiioiiiiooiooo....r..ro.imm 6X2MM6WVQkK
Kun Zhou and Bo Liu. Chapter 2 potential energy functions. In Molecular Dynamics Sim-
ulation, pp. 41-65. Elsevier, 2022. ISBN 978-0-12-816411988
doi: https://doi.org//00 1016/ B978-0-12-88 122--8111111180000007--0111 URL httpp::///// /www sci encedi com sci ence/
Jinhua Zhu, Yingee Xia, Lijun Wu, Shufang Xie, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-
Yan Liu. Unified 2D and 3D pre-training of molecular representations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2626-2636, 2022.
Vladimir A. Zorich. The Differential Calculus of Functions of Several Variables, pp. 427-543. Springer Berlin Heidelberg, Berlin, Heidelberg, 2015. ISBN 978-3-662-48792-11 doi: 10.1007/ 978-3-662-48792-1 8. URL http:///......ooo..........03333
Vladimir A. Zorich. *Integration of Differential Forms on Manifolds, pp. 313-362. Springer Berlin Heidelberg, Berlin, Heidelberg, 2016. ISBN 978 978-3-662-48993322
doi: 10.1007/
118-3-662-488997 /ttp://////aaaao//////////////33
13

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
Proof of Theorefical Results
15
Proof of Lemma 3.3
15
Proof of Lemma 3.2
15
Prool of heorem
15
Proposition A.
Interpretation Coordinate Denoising Zaidi al. 2023
Interpretation of Practional Denoising (Feng al. 2023
16
B
Supplementary Experiments
16
B.
More Downstream Results
16
B.2
Hyperparameter Analysis of the ShiDe Loss Function
17
B.3
Robustness to Noisy pre-training Datal
18
Effect of Pre-training Data Scale on Downstream Tasks
B.5
Effect of Pre-training Physical Consistency on Downstream Tasks
C
Detail Implementations
19
Noise Design
19
Pscudocode for Algorithms and Complexity analysis
19
Architecture Details
20
Hyperparameter Settings
21
D
Related Work
22
D.I Denoising for Molecular Pre-training
................................
22
D.2 3D Molecular Modeling in Relative Coordinates
24
E Supplementary Background Knowledge
24
Boltzmann Distribution
24
Classical Intramolecular Potential Energy Functions
24
14

---

Published as a conference paper at ICLR 2024
A PROOF OF THEORETICAL RESULTS
Proof of Lemma 3.3 Since fM is differentiable at the point æ, equation 16 is the definition of the
differential of function of several variables(Zorich 2015).
Proof of Lemma 3.2
Eul(a ((a b) u/²
E. [(a b) ww (a 6)] (a b) Es [e 1(a b) (a b) a² TaN (a b) =² b11
(19)
Divide both sides by o², then the proof is completed.
Proof of Theorem 3.4 Let v()) A B0, 1 N(0, 13N).
(M) = Pp( Ev GNNo(2)) v [
(20)
1
GNNo(w)) (p() VaEnar(d) (NM ( (
-M)))
- )
(21)
σ²
1
[NNN(x)) 000) VdEBAr(d) (J
ppp Bylo)
(22)
0²
1
(GNNo(2) - J(x) VdEBar(d)) T v(0))) ?
Fp(a) ( Byyt)
(23)
σ²
F IGNNNo()) (d)))))
(24)
The first step of approximation holds in the sense that lim LSuDe equation 20 The
second step holds by substituting ov with v(r) of The third step holds because of Lemma 3.3 and approximation holds the sense that lim ax v(0)) O. The fourth step
of the equation is the associative and distributive law of vector multiplication. The last step uses
Lemma 3.2
Proposition A.1. When No - and the least square estimation referred in section 4.1 gives the ground-trutt force field, i.e. Ax = b for any sampled v, the following regression loss is an equivalent optimization problem to SliDe.
c (M) Ep(MIM) (GNNo(M) =x,1²
(25)
Proof. When N2 + co, by the law of large numbers,
1 Σ [A GNNo(x) bj
lim (suDe(M) lim Ep(a) N00
(26)
No-000
No i=1
Ep(mn E. [A GNN((x) - b)2 ((aymm)) (M)
SliDe
(27)
(reg) SliDe (M) Eptcr ).) (GNNo(x) xx) (GNNo(2)
(28)
SliDe (M) Eppceees) E.2 (A G.N - b) AVGNN,
(asymp)
(29)
By assumption, Ar = b for any sampled v, then equation
F mo) E.2 (GNNo(x) 2Ty" A AVGNN((z)
(30)
Ep(a (GNNo(z) -x;) E. [ [ATA]
A] VGNNo(#)
(31)
Since v, N(, ISN). every element in U uN.] i..d standard normal distribution.
Therefore E, [ A N, IaN. i.e. equation
31
E 2N. (GNNN(() x,,) (GNNo(z)
(32)
V. (M)
(reg)
(33)
15

---

Published as a conference paper at ICLR 2024
Consequently, V (esymp) the same minima. SliDe
No V. (reg)
stuDe N, is constant, then the two optimization target share
Theorem A.2 (Interpretation of Coordinate Denoising (Zaidi et al. 2023)). ) Assume the conforma-
tion distribution is a mixture of Gaussian distribution centered at the equilibriums:
p(a) p(zz)aa))((0)) p(a(ær) ( laN)
(34)
Noo E RRN are equilibrium conformations and noisy conformation respectively, N is the num- ber of atoms the molecule. It relates to molecular energy by Boltzmann distribution p(æ) x exp(- (cp(--Ccora(x)),
Then given a sampled molecule M, the coordinate denoising loss is an equivalent optimization larget to force field regression:
LCoora(M) = F [GNNN(())
- -a | zn)l
(35)
PPtta) (GNN)))) (- [- FCoora(a:))1
(36)
where GNNo( denotes a graph neural network with parameters 0 which takes conformation æ as an input and returns node-level noise predictions. denotes equivalent optimization objectives for GNN.
ep GNNo(=) - V~ log p(x) By using conditional score matching lemma (Vincent, 2011), the equation above (p(e)a.))(a.)) No<)) - v log p( [2)]= + Ti, where Ti is constant independent of 0. Then (age
Proof. According to Boltzmann distribution, equation 36
---=" + T1. Finally, since coefficients do not rely on the input æ, it can be absorbed into GNNo, thus obtaining equa- tion 35
with the Gaussian assumption, it becomes
Theorem A.3 (Interpretation of Fractional Denoising (Feng al. 2023) ). Assume the conformation
distribution is a mixture distribution centered at the equilibriums:
p(x)= / plu p( ~ NNvoo Im), p(æla.) ~ N(w., , rJ Iaw),
(37)
where Fo, Fay RR3N are equilibrium conformation and noisy conformations respectively, +b and tho are the dihedral angles of rotatable bonds in conformation x and æn, m is the number of the rotatable bonds. It relates to molecular energy by Boltzmann distribution p(æ) x (ep(-EFrea(T)
Then given a sampled molecule M, the fractional denoising loss is equivalent optimizution target to force field regression:
LFrad(M) Eep(ae http://www...ooocceelll//./1//1 - (xc x.)ll2
(38)
(-v
(39)
Proof. According to Boltzmann distribution, equation 39
E GNNo(=) V. ||2
By using conditional score matching lemma (Vincent, 2011). the equation above Ep(z...) (G Va p(x: x.. +12 where 72 is constant independent of Since the part in the expectation does not contain Øo. it equals to Ep(a,.....0 ( V log p(
Finally, - can be absorbed into GNN and obtain equation 38
B SUPPLEMENTARY EXPERIMENTS
B. MORE DOWNSTREAM RESULTS
We evaluate SliDe on ANI-Ix that provides large numbers of molecules with multiple equilibrium and nonequilibrimm conformations. A summary of datasets is shown in table 6 The result is compared with baseline nonequilibrium denoising (Noneq) WWan et al. (2023b) that performs coor- dinate denoising Coord Zaidi al. (2023), while Noneq pre-trained ANI-1 and ANI-Ix that
16

---

Published as a conference paper at ICLR 2024
Table 6: Summary of datasets.
Dataset
Molecule size
Number of molecules
Number of conformations
PCQM4Mv2
~00 30
3,378,606
3,378, 1.378,606(onc equilibrium for cach molecule)
QM9
18 (3-29)
133,885
133,885(one equilibrium for each molecule)
MD17
~ (9-21)
8
3,611, 15includinng nonequilibriums)
ANI-Ix
~v 15 (4-63)
63,865
5,496,77 (including nonequilibriums
contain nonequilibrium structures. Noneq also adopts TorchMD-NET as the backbone model. The result of experiment ANI-Ix energy prediction is shown in table SliDe performs better in both pre-train improvement and without pre-train setting, indicating the superiority of SliDe pre-training method over nonequilibrim coordiante denoising as well the improved backbone model GET over TorchMD-NET.
Table 7: bold.
Performance (MAE 4) on ANI-Ix energy prediction (kcal/mol). The best results are in
Noneg
SliDe
w/o pre-train
1.50
1.362
pre-train pre-train improvement
1.01
0.786
32.7%
42.3%
B.2 HYPERPARAMETER ANALYSIS OF THE SLIDE Loss FUNCTION
Table 8: Force field accuracy in different settings of N, and a. The top results are in bold.
Settings
MSE
Scale
N. 32, o 0.001
0.536(0.067)
2.3e-4(1e-4)
0.73(0. 13))
N. 61, 0.001
0.753(0.079)
1.5e-4(8e-5)
0.98 (0.14)
N. -128, 0.001
0.895 (0.071)
7.5e-5(2o-4)
1.05(0.14)
N 512, a 0.001
0.896 (0.067)
7.6e-5(7e-5)
1.06(0.15)
N. -512. = 0.01
0.893(0.072)
0.53(0.20)
41.97(5.70)
Since an accurate force field target contributes to learning effective representations, we can choose hyperparameters by utilizing the least square estimation of learned force field accuracy introduced in section[4.1 This parameter selection strategy obviates the need for training neural networks, thereby rendering the process efficient and principled. Accordingly, we validated the accuracy of the learned force field for several combinations of hyperparameters No, and a. The results are shown in Table The accuracy measured by Pearson correlation coefficient (p, the larger the better), mean squared error (MSE, the smaller the better), and "scale". which is the quotient of the mean absolute values between learned force fields and DFT force fields, and the best value 1. The value in the bracket is the standard deviation.
In Theorem 3.4 the best hyperparameter in theory is N₂ oo and o - However, a large sampling size N, leads to slow pre-training, and a small sampling standard deviation results in higher numerical accuracy required. In experimental results, larger N, leads to better force field accuracy but the trend tends to saturate when N,, > 512. This is mainly because in the pre-training dataset, the atomic numbers of the molecules are generally distributed between 20 and 40, ie. N, 128 3.N for most molecules and the least square error is small in this case. As for the standard deviation a, it has a small impact on the correlation coefficient, but significantly affects the MSE and scale. After considering both accuracy and efficiency, choose N,, 128 and 0.001.
17

---

Published as a conference paper at ICLR 2024
B.3 ROBUSTNESS TO NOISY PRE-TRAINING DATA
In practical applications, the data quality can vary significantly. Most of the existing methods rely on equilibrium structures, limiting their effectiveness in scenarios where such structures are readily available or accurate. To investigate the robustness of SliDe to noisy conformation, we pre-train on RDKit generated conformations, which can be generated subsets of 10W and 50W molecules from PCQM4Mv2 dataset and generate cheap conformations by RDKit. We pre-train on the dataset we construct and test them on homo(QM9), as shown in When compared with training from scratch MAE=17.6 meV, pre-training with RDKit data
large scale. We extract
table 9
Table 9: Performance (MAE 4) on homo (meV) from QM9 with SliDe pre-trained on accurate and noisy conformations.
Training size
PCQ(DFT)
PCQ(RDKit)
10W
16.05
16.13
50W
14.53
15.39
notably effective. When compared with pre-training with accurate conformations, less accurate conformations compromise the performance. Overall, SliDe robust to inaccurate conformations, revealing the potential of SliDe in larger scale of pre-training.
B.4 EFFECT OF PRE-TRAINING DATA SCALE DOWNSTREAM TASKS
To investigate how the scale of the pre-training data affects performance, we randomly extract 10W, 50W, 100W data from PCQM4Mv2 dataset. pre-train on these subsets and finetune on hQM)(QM9). The results is provided in table TO] We find more pre-train data contributes to better downstream performance, whereas SliDe able to achieve competitive results when data scarce, such 50W.
Table 10: Performance (MAE () on homo (meV) from QM9 with SliDe pre- rreitraieed on different data scales.
Number of pre-training data
w/o pretrain
10W
50W
100W
300W(whole)
homo(meV)
17.6
16.05
14.53
14.21
13.6
B.5 EFFECT OF PRE-TRAINING PHYSICAL CONSISTENCY ON DOWNSTREAM TASKS
Table 11:
Performance (MAE 1) MD17 force prediction (kcal/mol/ Å). The models are pre- trained on a subset of PCQM4Mv2 dataset.
DFT label
Train from Scratch
Coord pre-training
Frad pre-training supervised pre-training
Aspirin (Force)
0.253
0.250
0.248
0.236
To verify whether learning an accurate force field in denoising can improve downstream tasks, we compare the existing denoising method with supervised pre-training on precise force field labels by DFT. Since DFT calculation can be time-consuming, we randomly select 10,000 molecules with fewer than atoms from the PCQM4Mv2 dataset and calculate their precise force field label using DFT. We pre-train the model by learning the DFT force labels. We also pre-train the same backbone model by Coord and Frad methods respectively. Then the pre-trained models are finetuned on MD17 datasets. The results are shown in Table [I The comparison between the pre-training methods indicates that as the accuracy of the force field in pre-training tasks increases, the performance on downstream tasks improves. Note that there is large gap from Frad to "DFT label supervised"
18

---

Published as a conference paper at ICLR 2024
compared to the improvement from training from Scratch to Coord and Coord Frad, indicating that there still a large room for downstream improvement along the idea learning force fields. These findings motivate to design denoising pre- vee-traiiing task to learn accurate force fields.
C DETAIL IMPLEMENTATIONS
C.I NOISE DESIGN
The BAT noise strategy refers to perturbing the equilibrium structure by adding independent Gaus- sian noise on every bond length, angle and torsion angle, shown in equation 9 The variances are determined by the parameters that can be obtained from the parameter files molecular sim- ulation tools. Our parameters are obtained from the parameter files of Open Force Field v.2.0.0 (Sage) (Boothroyd et al. 2023). Examples of BAT noise on acetaldehyde and 2-methylpyridine are provided in figure 2 illustrate the noise scales.
In some cases, the independence is unable to be achieved. We make special treatments for these situations. Firstly, when atom is connected more than two atoms, the angles centered this atom are dependent. In this case, we randomly select one edge and add noise to the angles involving this edge. For example, when adding angle noise the methyl group "-CH3" in figure the bond "C-C" is selected to be fixed and add noise on three angles of "C-C-H". Secondly, when there are ring structures the molecule, the bond lengths, bond angles, and dihedral angles formed by the atoms in the ring are dependent. As solution, we do not add noise to the bonds, angles, and torsion angles that are inside the ring. For example, in figure p. no noise is added on the pyridine backbone. One reason is that RDkit does not support modifying bond lengths, bond angles, and dihedral angles inside rings. The other reason is that we attempt to add low-level independent Gaussian Cartesian coordinate noise to the atoms inside the ring to perturb them sufficiently. However, its force field accuracy on certain molecules is much lower than without adding noise inside the ring. We speculate that this is because perturbing the atomic coordinates in the ring affects the surrounding angles and torsion angles.
find that
C.2 PSEUDOCODE FOR ALGORIHTM AND COMPLEXITY ANALYSIS
In this section, we present pseudocode to illustrate the pre-training algorithm of SliDe in Algorithm Besides, to show the ability to apply to large molecules, we discuss the time complexity of SliDe algorithm and provide comparison between Frad and SliDe.
First of all, please note that all the terms in equation 17 except for G.N are calculated in data preprocessing. As the deep learning package (specifically PyTorch in this case) generates multiple workers in the DataLoader through parallel processes, each responsible for loading and processing individual data samples before adding the processed batches to queue, this approach ensures that the time-consuming training process remains unimpeded, as long as the queue is consistently filled with batches. Therefore, will not become the bottleneck the training process, as G. (o))) what takes most of the time.
(a)
SDÉ..
(b)
(c)
SDA
SDC -0. -0.121
SD
SUA 097
SDS.c-arO.11
00.121
sD =0.00
=0.093
SDD
SD
SD =0
SDB +0.055
So!
CD
#0.8 -0,857
Figure 2: Some examples of BAT noise molecules. The scale of noise can distinguish different bond types and atom types, resulting in better molecular modeling. (a)(b) Standard deviations of BAT noise for acetaldehyde. (c) Standard deviations of BAT noise for 2- 2-methylpyridine.
19

---

Published as a conference paper at ICLR 2024
Algorithm 1 Sliced Denoising Pre-training Algorithm
Require:
GNN: Graph Neural Network
MI: Unlabeled 3D molecular pre-training dataset
T: Training steps
BS: Batch size
Standard deviation of the sampled coordinate noise
Ng: Sample number coordinate noise
N: Gaussian distribution
1: while T * do
2: Randomly sample batch of molecules from the dataset M.
3: for each molecule M with equilibrium structure Zo the batch do
4: Get the bond lengths, angles, torsion angles of M, denoted (ro. On, Φo).
5: Get the parameters of M, denoted as (k AA, ,",
6:
Add BAT noise on the structure according to equation 9 and get the perturbed structure
and
7:
Calculate VaFBAT (d) according to equation 10 for 1 1, N₂ do
8:
9:
Sample coordinate noise e, (0, Tax)
10:
Calculate its corresponding relative coordinate changes (M(+ooy) (x) Loss = [e os VoEBAr(d) (fM(x ov,) - (x))] +
11:
(o + rT,) rU,]
12:
end for
13:
end for
Optimise Loss = N.XBS Emebatch Nv Loss, and update GNN T
14:
15:
16:
end while
In theory, the computational complexity of the regression target for one molecule is ONN, x max(N, M,, F}) for SliDe, where N is the number of atoms, M is the dimension of relative co- ordinates (total number of bond lengths, bond angles, and bond torsions), F is the complexity of coordinate transformation between Cartesian coordinates and relative coordinates.
Two factors affecting the complexity is N, and F. As for coordinate transformation, the complexity F == Ommxx.N NV .M)) Choi (2006), which is executed efficiently by RDKit. As for the No times sampling procedure, in fact, for every E {1,..., N. the regression target can be calculated in parallel. Therefore the time complexity above can be further reduced to O(max N, M, F). This makes pre-training on large molecular datasets possible.
As for comparison to existing method, SliDe and Frad are thcoretically in the same seale. The computational complexity of the regression target for one molecule is ( MMr, N, Fra)) for Frad, where Mr the number of rotatable bonds and Fra is the coordinate transformation be- tween Cartesian coordiantes and diheral angles of rotatable bonds. It is in the same scale to SliDe's (mmxx(N, M, F}).
In practise, although do not process all N,, targets in parallel and incorporate edge update in network architecture, the training time of Frad and SliDe does not vary significantly. Frad pre- training takes 1d1h14m on 8 NVIDIA A100 GPU, and SliDe pre-training takes 1d17hlm on 8 Tesla V100 GPU.
C.3 ARCHITECTURE DETAILS
Our network is an equivariant graph neural network that recursively updates vertex and edge features. An illustration of the network architecture is shown in Figure The vertex feature V 333FV and S FV are respectively vector and scalar features for each vertex, Fy is the vertex feature dimension. E Re denotes the edge feature of each edge, Fg is the edge feature dimension. The edge vector x; - æz denoted by R. is the atomic type. r. 0, Φ are bond lengths, bond angles and torsion angles.
20

---

Published as a conference paper at ICLR 2024
(a)
V==0
S)=emb(Z) Eg
(b)
Edge Update:
Vertex Update
Edge Update
Message Passing
Emb(r. 0.
Vertex Update
Edge
Update
1
Vertex Update:
Vertex Update
Edge
Update
R
Attention Layer
E
Output Network
Predicted Force Field
Figure 3: Overview of the network architecture. (a) The whole architecture includes initialization, several update layers and an output network. (b) The update layer consists of edge and vertex updates. The updated edge feature will be used in vertex updates.
For the edge update, the invariant edge feature E is updated by the embeddings of the bond length r, the neighbor edge features and the embeddings of their angles 0, Φ. Specifically, it incorporates Bessel functions to embed bond length and spherical harmonics to embed angle and torsion angles, which are shown to be effective geometry embeddings for molecular representation (Liu et al. 2022 Klicpera al., 2021 Gasteiger al. 2020b). For the vertex update, the invariant S and equivariant V are updated by an attention layer, whose architecture is based on TorchMD-NET. The updated edge features are projected into two filters and are later used to calculate attention weights in vertex update.
C.4 HYPERPARAMETER SETTINGS
Table 12: Hyperparameters for pre-training.
Parameter
Value or description
Train Dataset
PCQM4MV2
Batch size
128
Optimizer
Adam
Warm up steps
10000
Max Learning rate
0.0004
Learning rate decay policy
Cosine
Learning factor
0.8
Cosine eycle length
240000
Keep aligned with downstream settings respectively on QM9 and MD17
Network structure
N, a
128
0.001
Regression target Regularization
Least square results yes 0.04
Hyperparameters for tra-ingin listed in Table 12 Details about Learning rate decay policy can be refered http://swwwsoaattt////c//eeeeeaa hatprp/opaeeaattttttooooocooccco
explained-strong
In previous denoising methods, normalizing the regression target, such as noise in Coord, is widely applied technique stabilize the training process. However, in SliDe loss equation 17 the
21

---

Published as a conference paper at ICLR 2024
normalization is hard implement. Instead, chose utilize ((re)) StiDe and normalize the regression
target f. We find the least squares estimation does not incur significant additional computational costs on the current dataset molecular sizes.
i: order to align with the downstream Noisy Node task that involves coordinate denoising, we add regularization term in pre-training. So the total pre-training loss is given by
1
[p( mo) No
Σ (GNNo(x+TO. - -uu,?
(40)
+ CNNo(1)) o [aEBar(d)
(Ma)))
Table 13: Hyperparameters for fine-tuning on MD17.
Parameter
Value or description
Train/Val/Test Splitting
950/50rremining data
Batch size
8
Optimizer
Adam
Warm up steps
1000
Max Learning rate
0.0005
Learning rate decay policy
Reducel. ROnPlateau (Reduce Learning Rate on Plateau) scheduler
Learning factor
0.8
Patience
30
Min learning rate
1.00E-07
Network structure
Geometric Equivariant Transformer
Head number
8
Layer number
6
RBF number
32
Activation function
SiLU
Embedding dimension
128
Force weight
0.8
Energy weight
0.2
Noisy Nodes(NN) denoise weight Dihedral angle noise scale in NN Coordinate noise scale in NN
0.1
20
0.005
Hyperparameters for fine-tuning MD17 are listed in Table 13
Hyperparameters fine-tuning on QM9 are listed in Table 14 The cosine cycle length is set to be 500000 for a, ZPVE, E, Uo. U, H. G and 300000 for other tasks for fully converge. Following previous literature (Liu et al., 2023b Feng et al., 2023 Liu et al.] (022), we fix seed= on QM9 and MD17, the performance is quite stable for random seeds (Schütt al. 2018 2021 Liu al.,
2022]. To further validate the issue, we finetune with seed= 0, 1, on (MM9, lumo(QM9) and Aspirin(MD17). The results is quite stable, shown in Table 15
For experiment in section 4.1 the force field calculated by DFT method implemented by PySCF tool (Sun al. 2020), with basis '6-31g' 'b3lyp'.
Noisy Nodes is implemented following (Godwin et al. 2021 Feng et al. 2023
D RELATED WORK
D.1 DENOISING FOR MOLECULAR PRE-TRAINING
Denoising self-supervise learning task originates from denoising generative models in com- puter vision Vinneet et al.) 2008). In molecular pre-training. it refers to corrupting and recon- structing the 3D structure of the molecule. Denoising is self-supervised learning task designed
22

---

Published as a conference paper at ICLR 2024
Table 14: Hyperparameters for fine-tuning QM9.
Parameter
Value or description
Train/Val/Test Splitting
11000//0000/emmiiiinng data
Batch size
128
Optimizer
Adam W
Warm up steps
10000
Max Learning rate
0.0004
Learning rate decay policy
Cosine
Learning rate factor
0.8
Cosine cycle length
300000 (500000)
Network structure
Geometric Equivariant Transformer
Head number
8
Layer number
8
RBF number
64
Activation function
SiLU
Embedding dimension
256
Head
Standardize
Applied according to
Thölke & Fabritiss 2022)
AtomRef
Label weight
Noisy Nodes denoïse weight
0.1(0.2) 0.005
Coordinate noise scale
Table 15: Random seeds have little effect on the finetuning of QM9 and MD17.
homo(QM9)
lumo(QM9)
Aspirin(MD17)
sced=0
12.3
0.1714
scod=1
13.6
12.3
0.1740
seed=2
13.3
12.2
0.1744
Mean(Standard deviation)
13.63(0.35)
12.27(0.06)
0.1733(0.0016)
specifically for 3D geometry data and achieves outstanding results in many downstream tasks for 3D molecules (Zhou et al. 2023 Feng et al. (023).
The existing denoising methods mainly differ in the noise distribution and denoise tasks. Uni- Mol (Zhou et al., 2023) adds uniform noises of [-1À, 1Å] to the random 15% atom coordinates.
The model is trained to recover the correct atom coordinates and pair distance. They combine denoising with atom-type masking to make the masking task more challenging.
Coordinate denoising (Coord) (Zaidi et al. 2023 adds Gaussian noise to atomic coordinates of cquilibrium structures and trains the model to predict the noise from the noisy input. They establish the equivalence between coordinate denoising and force field learning. Transformer-M (Luo et al., 2023) utilizes Coord to train the 3D model they proposed.
To capture the anisotropic molecular probability, fractional denoising (Frad) (Feng et al. 2023) proposes to add a hybrid noise on the dihedral angles of rotatable bonds and atomic coordinates. and fractionally denoise the coordinate noise. In this specially designed denoising task, the physical interpretation of learning force field also holds.
Compared to the aforementioned methods, our work most closely aligns with physical principles because our energy function better describes the true molecular energy landscape. This leads to more realistic molecular force field and sampling distribution that is beneficial for representation learning.
On the other hand, to make the molecular energy invariant to rotation and translation. 3D- EMGP (Liu et al. 2023b) denoises the Gaussian noise on the pairwise atomic distances and SE(3)- DDM (Jiao et al. 2023) exploits the Riesaan-Gaussian distribution for coordinate denoising. Our
23

---

Published as a conference paper at ICLR 2024
method naturally satisfies the symmetric prior because energy function is defined on bond length, bond angle and dihedral angle, which are invariant to rotation and translation.
D.2 3D MOLECULAR MODELING IN RELATIVE COORDINATES
The geometric information contained in 3D conformers is crucial for molecular representation learn- ing. Though most 3D structures are represented in Cartesian coordinates, recently many works have focused on utilizing 3D information relative coordinates i.e. bond length, bond angle, torsion angle, also called as internal coordinates or local coordinates. The relative coordinates capture the complete geometry of atomic structures and are widely used because they are invariant to rotation and translation, making them convenient for molecular description in many scenarios (Li et al.) 2023).
For one thing, relative coordinates are used to enhance the expressiveness of graph neural networks. For molecular property prediction, SphereNet (Liu et al. 2022) and GemNet (Klicpera et al.) 2021) encode bond length, bond angle and dihedral angle information by spherical Bessel functions and spherical harmonics functions. ALIGNN-d (Hsu et al. 2022) encode relative coordinates informa- tion by Radial Bessel basis and Gaussian basis and learn representations for optical spectroscopy prediction.
For the other thing, the prediction of relative 3D information is found effective in pre-training task design. ChemRL-GEEM(Fang et al., (20TT) propose to predict bond lengths and bond angles to de- scribe the local spatial structures. 3D PGT (Wang et al.) 2023a) and GearNet (Zhang et al., (202)) also incorporate the prediction of bond length, bond angle and dihedral angle. They differ signifi- cantly from BAT-denoising that their input structures remain unperturbed.
E SUPPLEMENTARY BACKGROUND KNOWLEDGE
E BOLTZMANN DISTRIBUTION
In statistical physics, the distribution of a collection of identical but distinguishable particles con- forms to the Boltzmann distribution Boltzmann (1868). We assume the distribution of the confor- mations of molecule satisfies the Boltzmann distribution:
F(x) "B"
p(x) Z
(41)
where E(x) is the (potential) energy function, æ E RN is the position of the atoms, i.e. conforma- tion, N is the number of atoms in the molecule, T is the temperature, ky is the Boltzmann constant and Z is the normalization factor. When employing neural networks to fit the energy function or its gradient, the constant T can be absorbed in the energy function, i.e. p(æ) exp(-E(xr).
E.2 CLASSICAL INTRAMOLECULAR POTENTIAL ENERGY FUNCTIONS
Here we provide more information on the classical intramolecular potential according to (Mol) 2020). The classical intramolecular potentials consider N - 1)/2 interactions between pairs of atoms in molecule with N atoms. According to the pair distance, the interactions fall into four categories: 1-2, 1-3, 1-4 and farther interactions. Specifically, 1-2 interactions denote interactions between covalently bonded atoms. 1-3 interactions denote interactions between atoms i and k, where i-j and j-k are covalently bonded but i-k is not. Similarly, 1-4 interactions denote interaction where the two atoms are separated by a chain of three covalent bonds. Farther interactions refer to 1-5, 1-6, etc. interactions.
E(r, 0,
Σ
k (r. r2,0)
+
Σ
& (0,
kkp)2
2
2
iER
iEA
1-2 interactions
1-3 interactions
(42)
k (1 - cos(wi(o - o(,)))++ +
Σ
(( clec
+ Σ
E WdW
TET
(.E)
(J
1-4 interactions
1-4 and farther interactions
24

---

Published as a conference paper at ICLR 2024
In equation 5 as is rewritten in detail in equation 42 above, the 1-2 interactions described by bond stretch potentials and the 1-3 interactions are modeled by the angle bend potentials. As for 1-4 interactions, it is modeled by bond torsion potentials, 1-4 electrostatic (elec) and 1-4 van der Waals (vdW) interactions. Farther interactions are described by electrostatic and van der Waals interactions only. (j, E IF denotes atom j and atom are separated by three covalent bonds more. The potential energy function in the form of equation [42 together with its modification are widely adopted in molecular simulations. The parameters obtained through fitting experimental data or quantum chemical calculations.
25


Reviews:
[{"review_id": "F7kwnH2Vc7", "reviewer": "Reviewer_ft62", "summary": "the paper introduces \"sliced denoising\" (slide), a novel molecular pre-training method that enhances the physical interpretation of molecular representation learning. traditional denoising methods, though physically interpretable, can suffer from inaccuracies due to ad-hoc noise design, leading to imprecise force fields. slide addresses this by utilizing classical mechanical intramolecular potential theory, leading to a 42% improvement in force field estimation accuracy over existing methods.", "questions": "1. regarding computational efficiency: can the authors provide more details on the computational requirements of slide, especially when applied to large molecular datasets? how does its computational efficiency compare to existing methods?\n\n2. on generalizability and applicability: what are low-hanging fruits to test the generalizability of slide to other types of geometric data or applications beyond molecular science? how might the method need to be adapted for such scenarios?\n\n3. empirical validation across diverse datasets: could the authors elaborate on potential plans to validate slide on a broader range of datasets, particularly those that may present different challenges than qm9 and md17, such as des15k or oc20 as in the coord paper https://arxiv.org/abs/2206.00133?\n\na curiosity question:\n1. dependence on equilibrium structures: the method's reliance on equilibrium structures, to be clear same as most other methods in this space, for training may limit its effectiveness in scenarios where such structures are not readily available or accurate. are there ways to advance molecular representation learning in such a setting?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. innovative approach: slide introduces an innovative noise strategy (bat noise) and a random slicing technique. this approach significantly enhances the accuracy of force field estimations, making it a pioneering method in the field.\n\n1. alignment with physical principles: the method closely aligns with classical mechanical intramolecular potential theory. it appears to improve the realism of molecular representations as well as help that learned representations are physically interpretable, a critical aspect in molecular sciences.\n\n1. empirical validation: slide demonstrates empirically strong results in force field estimation accuracy and downstream task performance on benchmark datasets qm9 and md17.\n\n1. methodology: the paper combines theoretical soundness with methodological innovations effectively. the use of a quadratic energy function approximation and the consequent noise strategy is interesting.\n\n1. network architecture integration: integrating a transformer-based network architecture that encodes relative coordinate information is a notable strength. this architectural choice complements the novel denoising method, enhancing its adaptability to other works using transformer backbones.", "weaknesses": "1. computational complexity: while the random slicing technique addresses computational challenges associated with jacobian matrix estimation, the overall computational demand and efficiency, especially in large-scale applications, are not comprehensively addressed​​​​​​.\n\n1. robustness to noisy data: the robustness of slide to noisy or imperfect real-world data is not thoroughly examined. this aspect is crucial for practical applications where data quality can vary significantly​​.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OVKjUEnThM", "reviewer": "Reviewer_mZU2", "summary": "The paper proposes a novel approach to molecular pre-training called Sliced Denoising (SliDe) that leverages physical principles to improve molecular property prediction. The authors introduce a new noise distribution strategy that improves sampling over conformations and a denoising task that learns the force field of the energy function. They evaluate SliDe on benchmark datasets QM9 and MD17 and show that it outperforms traditional baselines in terms of physical consistency and molecular property prediction accuracy.", "questions": "NA", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The main contribution of the paper, the pre-training algorithm, seems to be extremely relevant to the drug discovery domain with the magnitude of improvement achieved across all benchmark tasks. The paper could be of great interest to scientists in this area. \n\nThe pre-training method introduced leverages physical principles and is more interpretable. In addition, the experimental results seem very thorough.", "weaknesses": "While the paper is interesting, and makes an important contribution to the field of drug discovery, I would like to raise the question of if ICLR is the correct venue for this submission. This is an important area, and there will be a subset of audience interested in the field, but I would assume that a broader audience will have trouble understanding the paper due to the about of domain knowledge involved. I will leave it to the AC to determine this.\n\nI found the paper hard to read and understand due to the amount of domain knowledge involved. I understand that it is not possible to introduce all the background information in 8 pages, but I would urge the authors to rewrite the paper in a more accessible way for non-domain but ML experts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xUhOZhPhtk", "reviewer": "Reviewer_j2Vc", "summary": "This paper proposes a new pre-training method, Slide, that is based on intramolecular potential theory. To lower the computational expense of Slide, the authors introduce a random slicing approach. In addition, a new MLFF architecture is introduced (GET).", "questions": "Question 1: How does your slicing method related to sliced score matching [1]?\n\n[1] Song, Yang, et al. \"Sliced score matching: A scalable approach to density and score estimation.\" Uncertainty in Artificial Intelligence. PMLR, 2020.\n\nQuestion 2: How does the scale of the pre-training data effect performance?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Strength 1: The writing and motivation of this paper is very clear, with a good description of related works.\n\nStrength 2: Molecular pre-training is an important problem which the authors propose a novel approach for.", "weaknesses": "Weakness 1 (Major): The baselines used in this paper are not up to date. In fact, Gemnet (2021) and Nequip (2022) can outperform Slide on basically all of the MD17 and require no pre-training. In addition, the authors claim they set a new state of the art on these benchmarks, which is incorrect.\n\nWeakness 2 (Major): For the downstream task only one random seed is used and the gains over other methods are relatively minor (i.e. compared to Coord and Frad). This makes me doubt that the results are really significant or if they are just due to tuning. I think that multiple random seeds should be reported.\n\nWeakness 3 (Minor): Showing the best result over random seeds in table 1 is kind of strange. I think that the mean result should be shown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SKYKOmF21e", "reviewer": "Reviewer_nTFT", "summary": "The paper proposed a improved denoising pretraining method for 3D GNN in molecular machine learning tasks. The proposed method, called sliced denoising (SliDe), is inspired by classical intramolecular potential theory. SliDe adds different levels of noise to the length of bonds, the magnitude of angles, and the magnitude of torsion angles. Through a series of theoreticall derivation, the authors demonstrate the equivalence of SliDe method and learning the molecular force field. Lastly, the authors show that the SliDe has outperformed previous denoising training methods on the QM9 and MD17 dataset.", "questions": "1. Table 7 is confusing. My understanding is that “Training from Scratch” meaning no pretraining, and Coord and Frad meas pretrained with different method and then fine-tuned on MD17-Aspirin. What does DFT label supervised mean? Isn’t “Training from Scratch” also supervised? The authors should elaborate. The unit of the prediction MAE should also be included in the table.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Comparing to previous work such as the Fractional Denoising, the noise designing of this work is a more \"physical\" formulation because changing either one of the bond length, angle degree, or torsion angle degree does not affect the rest. \n\n2. The writing and proof process are of high clarity. \n\n3. Some efforts to the direction of explainablity are interesting. For example, the correlation coefficient between the learned force field and ground-truth force field (Table 1) is a nice way to quantified the learned force field.", "weaknesses": "1. In section 3.1, the claim above Eq. 6 is not correct. Modeling long-range electrostatic (coulomb) interaction is critical in many areas including energy calculation and md simulation. For large system, long-range vdW outside of a cutoff distance may be neglected, but for small molecules in QM9 and MD17, it should not be neglected. When using GNN that takes 3D coordinate and atom type as input approximate energy function, the model should be able to learn those two terms, thus not affecting the approximation of Eq.6. However, the authors should rephrase the sentence. The two citations associated with the sentence does support the claim so they should be removed.\n\n2. The noise design is very import in this work. In the BAT noise (Eq. 9), the parameter vectors are critical but there isn't detailed explaination of them. The authors briefly discussed in the section C.1, but I do think more details and example of those parameter vectors can substentially help reader in understanding the noise design. \n\n3. Missing parenthesis in the second exponential term of Eq. 8. $(\\theta_{i} - \\theta_{i, 0})^2$\n\n4. I do think the superiority of the SliDe method can be strengthen by more downstream experiments, especially energy prediction. For example, the ANI-1x dataset (www.nature.com/articles/s41597-020-0473-z) is a excellent dataset for such task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)