DenoiseVAE: Learning Molecule-Adaptive Noise Distributions for Denoising-based 3D Molecular Pre-training
Authors: Yurou Liu
Jiahao Chen
Rui Jiao
Jiangmeng Li
Wenbing Huang
Bing Su
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Molecular pre-training via denoising
Molecular property prediction
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose a novel denoising method for 3D molecular pre-training.

Abstract: Denoising learning of 3D molecules learns molecular representations by imposing noises into the equilibrium conformation and predicting the added noises to recover the equilibrium conformation, which essentially captures the information of molecular force fields. Due to the specificity of Potential Energy Surfaces, the probabilities of physically reasonable noises for each atom in different molecules are different. However, existing methods apply the shared heuristic hand-crafted noise sampling strategy to all molecules, resulting in inaccurate force field learning. In this paper, we propose a novel 3D molecular pre-training method, namely DenoiseVAE, which employs a Noise Generator to acquire atom-specific noise distributions for different molecules. It utilizes the stochastic reparameterization technique to sample noisy conformations from the generated distributions, which are inputted into a Denoising Module for denoising. The Noise Generator and the Denoising Module are jointly learned in a manner conforming with the paradigm of Variational Auto Encoder. Consequently, the sampled noisy conformations can be more diverse, adaptive, and informative, and thus DenoiseVAE can learn representations that better reveal the molecular force fields. Extensive experiments show that DenoiseVAE outperforms the current state-of-the-art methods on various molecular property prediction tasks, demonstrating the effectiveness of it.

Published as a conference paper at ICLR 2025
DENOISEVAE: LEARNING MOLECULE-ADAPTIVE NOISE DISTRIBUTIONS FOR DENOISING-BASED 3D
MOLECULAR PRE-TNAINING
Yurou Liu--r, Jiahao Chen Rui Jiao2.3.1, Jiangmeng Li', Wenbing Huang' Bing Su
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
2 Department of Computer Science and Technology. Tsinghua University, Beijing, China
3 Institute for AI Industry Research, Tsinghua University, Beijing, China
Institute of Software, Chinese Academy of Sciences, Beijing, China prurouliugg nicelemon6e 6 @@gmail com, jiaor21 @eaiiemmall..... hemmals..siinhhuee... tsinghua. edu. cn, j ianggengg0109000 90 scass ac. on
ABSTRACT
Denoising learning of 3D molecules learns molecular representations by impos- ing noises into the equilibrium conformation and predicting the added noises recover the equilibrium conformation, which essentially captures the information of molecular force fields. Due to the specificity of Potential Energy Surfaces, the probabilities of physically reasonable noises for each atom in different molecules different. However, existing methods apply the shared heuristic hand-crafted noise sampling strategy to all molecules, resulting in inaccurate force field learn- ing. In this paper, we propose a novel 3D molecular pre-training method, namely Denoise which employs Noise Generator to acquire atom- spenifpeifiic noise distributions for different molecules. It utilizes the stochastic reparameterization technique to sample noisy conformations from the generated distributions, which are inputted into Denoising Module for denoising. The Noise Generator and the Denoising Module are jointly learned in manner conforming with the paradigm Variational Auto Encoder. Consequently, the sampled noisy conformations can be more diverse, adaptive, and informative, and thus Denoise VAE can learn rep- resentations that better reveal the molecular force fields. Extensive experiments show that DenoiseVAB outperforms the current state-of-the-ar methods vari- ous molecular property prediction tasks, demonstrating the effectiveness of it.
INTRODUCTION
Molecular representation learning plays an important part in many areas, including material de- sign (Bishara et al. 2023: Ha al., 2023), ) life science (Eslami et al. 2022 Goshisht 2024) and
drug discovery (Blanco-Gonzalez Gonzalez et al. 2023 Mak et al. 2023). Due to the critical role of 3D structures in determining molecular properties and the severe insufficiency of labeled data, various studies have explored 3D molecular pre-training (Zhou et al.) 2023 Luo et al.) 2022 Liu et al. 2022a). Among them, denoising has been proven be one of the most effective approaches (Ni et al. 2023 Liu et al. 22022a] Feng et al., 2023 Jiao et al. 2024), which adds noises to atomic coordinates the equilibrium conformation and reconstruct the equilibrium conformation from the noisy structure. Since denoising is equivalent to learning the molecular force fields (Vincent (201T), can capture the dynamic properties of molecules, which beneficial for various downstream tasks.
Previous denoising methods mainly focus on exploring the noise-adding strategies for sampling noisy conformations of molecules. 3D-EMGP (Tiao al., (023) and Coordinate denoising (Co-
ord) ZZadd et al., (2022) both treat the 3D conformation of the molecule pled from the Boltzmann distribution and apply invariant-scale Gaussian noises to all atoms among
random quantity sam-
different molecules. They ignore that the molecular force fields are anisotropic (Feng et al. 2023
"Correspondence to: Bing Su <subingats gmaalgl.smmmmm..mmmm........ Wenbing Huang <hwenbing @ 126.com... indicates equal contribution.

---

Published as a conference paper at ICLR 2025
Polmte mergy
storg anfermitons
Pocantil
Malcedie cuder
Mermallonn sparce
2.0
3.5
Squibren conberake
Interatunic distanon A)
(b)
Figure 1: (a) Variation curves of potential energy with interatomic distance for different types of atoms. When the interatomic distance changes the same, the energy changes of the two atoms are different, which illustrates the anisotropy of the force fields. (b) The Potential Energy Surface (PES) of Remdesivir. Sampling from positions near the equilibrium conformation on the PES can yield conformations with different structures but also low energies, which is our learning objective.
Ni et al., 2023), which can be demonstrated by Fig. [[],, i.c. the reasonable position changes of different atoms in physics are different. To tackle this issue, Frad (Feng al. 2023 proposes hybrid noise generation strategy that combines dihedral angle noise and coordinate noise. SliDe (Ni al. 2023) develops more fne-graine strategy on this basis. Although these hand-crafted strate- gies allow for the application of different noises to different atoms, they are designed by expertise heuristics and treat all molecules equally.
As shown in Fig. bb), molecules are dynamically variant, continuously moving within 3D Eu- clidean space and creating Potential Energy Surface (PES). The minima on the PES corresponds to the equilibrium conformation of the molecule, indicating that the molecular system has reached equilibrium. However, beyond the equilibrium state, the low energy conformations in the figure are all observed when the molecular system is in equilibrium. In essence, the continuous changes in conformation are the equilibrium state of this molecular system. Therefore, generating physically reasonable noises to sample diverse and meaningful conformations around the minima of the PES is crucial for the denoising task to learn accurate force fields (Schlegel) 2003 Liu et al. 2022a Zheng al. 2024). The PESs for different molecules are highly specific and the feasible variation ranges for different atomic positions in different molecules may vary greatly (Sato 1955). For example, atoms connected by single bonds may have a larger range of variation, while if the position of atoms in conserved functional groups undergoes significant changes, the properties of the molecule may change, or even the molecule may become another molecule; the variation range of the same atom or functional group varies in different molecule environments. It is unreasonable to share the same noise design among different molecules, as in existing state-of-the-an methods including Frad (Feng al., 2023) and SliDe (Ni et al.) 2023). Moreover, hand-crafted strategies these complex and ever-changing situations. Therefore, we locate the dilemma that sharing the same hand-crafted noise generation strategies for all molecules leads to inaccurate learning of molecule- specific force fields.
difficult to adapt to
To simultaneously take into account the anisotropy of the force fields within the molecule and the PES differences among different molecules, we propose a new pre-training method, named De- VAF, that generates specific noise distributions for each atom within different molecules to adaptively sample molecular conformations with a Noise Generator. The adaptive noises and the randomness brought by the reparameterization process largely expand the sampling space of molec- ular conformations. The sampled conformations are fed into a Denoising Module for denoising. Denoise VAE jointly trains the Noise Generator and the Denoising Module in a Variational Auto Encoder (VAE) (Kingma & Welling 2013) manner due to its superior representational capacity and the solid theoretical guarantee. Through better aligning with physical principles, Denoise VAE out- performs existing denoising methods in variety of downstream tasks. Our main contributions are summarized as follows:
We propose denoising based 3D molecular pre-training method, i.e., Denoise which employs a learnable noise generating strategy instead of existing hand-crafted strategics to adaptively acquiring atom-specific noise distributions for different molecules.
2

---

Published as a conference paper at ICLR 2025
We propose variational method to jointly optimize the Denoising Module and the Noise Gen- crator of Denoise VAE, where the denoising objective encourages the generated noises to be more consistent with the physical constraints that equilibrium conformations can be recov- ered, while a KI. divergence- based regularization term is imposed to prevent low noise intensity and increase the diversity of sampled conformations.
Theoretically, we prove that optimizing our pre-training objective is equivalent to improving the evidence lower bound (ELBO) of the log-likelihood.
We conduct extensive experiments to demonstrate the effectiveness of Denoise VAE. Results show that DenoiseVAB outperforms existing denoising methods on various datasets for both molecular and complex property prediction.
2 RELATED WORK
Molecular pre-training via denoising Pre-training is an important method for molecular repre- sentation learning. Inspired by the field of computer vision (Vincent et al.) 2008), denoising as a self-supervised learning task is widely used in 3D molecular pre-training and achieves excellent
results in many downstream tasks (Jiao et al. 2023 Zhou et al. 2023 Ni et al. 2023). Current de- noising methods mainly differ in the added noise distribution and denoising tasks. Uni-Mol (Zhou) et al. 2023) combines the coordinate denoising task with the atom-level masking strategy. They add uniform noise within fixed scale the atom coordinates and reconstruct the original coordi- nates and the atom-pair distances. Transformer-M (Luo et al., (022) performs multi-view learning of molecules and jointly trains 2D graph structure and 3D coordinates. Coord (Zaidi et al., 2022) focuses more on the denoising task itself. They add Gaussian noise to the equilibrium conforma- tion and predict the noise to learn the molecular force fields. Considering that the molecular energy should be invariant to rotation and translation, GeoSSL-DDM (Liu et al.) (022a) utilizes Gaussian noise to denoise the distance between atomic pairs. 3D-EMGP (Jiao et al. 2023) uses fixed multi- scale Slemann-Gaussinn the force fields in different parts of the molccule, Frad (Feng et (023) proposes a hybrid noise that combines dihedral angle noise and coordinate noise but only denoises the coordinate part. SliDe (Ni cct al., 2023) optimizes the noise strategy on this basis, which achieves better conformation sampling by perturbing bond lengths, angles, and torsion angles. Compared with previous methods, our work is an attempt at how to sample more significant and various molecular conformations.
noise to perform coordinate denoising tasks. To satisfy the anisotropy of
Molecular property prediction with 3D information Understanding 3D molecular structures and properties crucial for material design and drug discovery (Han et al. 2024). While some stud- ies use 3D data during training, they rely solely on 2D representations for inference, limiting their ability to learn 3D conformations. (Stärk et al., 2022 Zhu et al.] 2022] Liu al. 2021). In order to fully utilize the 3D information of molecules, Transformer-M (Luo et al. 2022) performs multi-view learning by jointly training the 2D graph and 3D structure of the molecule, which explicitly uses the 3D coordinates of the molecule as input. Based on this, various subsequent works further use 3D DFT conformations to train the model for more accurate 3D molccular property prediction (Jiao)
ct al. 2023 Zaidi et al. 2022: Zhou et al. 2023 Lu et al. 2023 Feng et al. 2023; Ni et al. 2023
Jiao et al. 2024). Graphormer-31 (Shi et al. 2022) utilizes the initial 3D conformations provided by
the OC20 dataset (Chanussot et al. 2021) predict the equilibrium energy. 3D-EMGP (Jiao al.
2023) and Coord (Zaidi et al. 2202)) design pre-training tasks for approximate molecular force field learning, and perform molecular property prediction on the molecular dynamics dataset (Ramakrish- nnnn et al.) 2014). Considering the anisotropy of the molecular force fields, Frad (Feng et al.) 2023) and SliDe (Ni et al.) (023) improve the noise design in the denoising tasks, which are more accu- rately aligned with physical principles. Models such as EGHN (Han et al. (022). FastEGNN (Zhang al. 2024). and EquiRNA (Li et al. (025a] better model molecules through hierarchical or virtual node approaches. On the other hand, spherical-scalarizztio models (e.g. SO3KRATES (Frank ct al., 2024). HEGNN (Cen et al. 2024]. GotenNet (Aykent & Xial 2025)) untilize invariant in- formation (inner product or modulus) from high-degree representations to enhance the expressive power of models. More recently, EquiLLM (Li et al.) 2025b) further enhances the representation effect through knowledge injection from large language models. EPT (Jiao et al. 2024) conducts cross-domain joint training in the hope of learning more complementary knowledge across domains to optimize the results of molecular property prediction.
3

---

Published as a conference paper at ICLR 2025
3 METHOD
3.1 PRELIMINARY
According to the prior knowledge in the field of statistical physics, the probability of the occurrence of 3D molecular conformation described by Boltzmann distribution (Boltzmann) 1868):
1 Z exp(
E(X) KT
Pphysical X)
(1)
where X denotes the 3D conformation randomly sampled from the Boltzmann distribution and ( represents the potential energy the given molecule. k, T, and denote the Boltzmann constant, temperature coefficient, and normalized factor, respectively.
Due to the challenge and high cost of measuring the true force fields within molecules, it is appeal- ing to learn the force field for each atom within molecule, represented - V (xX). By taking the logarithm of Eq. I and computing the gradient with respect to I, we obtain Vx logp(X)
E X) where V x log p( X) the score function of conformation X. From previous stud- ies (Vincent 2011), we typically solve for p( X X) approximate the true p( X since the potential energy distribution E E and the true conformation distribution Pphysical ( of molecules are un- obtainable. Here, X represents molecular conformations sampled after noise perturbation. Previous studies (Vincent, 2011 Zaidi et al., [022] Feng et al.] 2023) have demonstrated the equivalence between the denoising task and learning force fields, specifically:
E ||NNN (X) V E
(ix)|/GNN,
x(x)
E
2(x)
||INNNN (X) V log p( X)]
xE
p(X
|GNN (X) V log p(X x)|
X X σ²
=Ex(x,x)
|| (X)
where GNN (X) denotes the atom-level noise predictions using graph neural network (GNN) parameterized by V, which takes the noisy conformation X input.
Therefore, 3D molecular representation learning adopting the denoising task as the training objective can learn molecular force fields. Formally, given set of unlabelled 3D molecular conformations S = [ where M is the size of the set, the objective of denoising is to train Denoising Module, which constituted by an encoder and a prediction head, to predict the added noises for the input noisy conformation, so that the acquired representations can capture the dynamic information of the potential force fields. As shown 3 can be obtained by:
the optimal parameters T of the Denoising Module
(X,
X σ²
T arg min
Σ M
1GNN (X,,
From Eq. 2 we can that better noise generation strategy for conformation sampling is the key to more accurate learning molecular force fields.
3.2 DEN
Previous research in computational chemistry (Schlegel 2003) has shown that molecules are not static but exist in a continuous state within 3D Euclidean space. These states form PES, as il- lustrated in Fig Fig [b). Since conformations corresponding to lower potential energy are more stable and have higher existence in the real world, sampling more conformations around the minima of molecule's PES is crucial for learning better molecular representations.
Previous methods (Feng et al. 2023 Ni et al. 2023) has considered the anisotropy of force fields within a molecule and added mixed noises by perturbing dihedral angle and bond angle, etc. How- ever, such complex hand-crafted strategies ignore that different molecules should not share the same
4

---

Published as a conference paper at ICLR 2025
Denoise
(0,of)
Noisy
(O, on
Add noise
Noise Generator
X
(0, (
X
Anisoraayy v whi uxdecale v
Original
Previous methods (Cound,
etc))
Add noise (hand-crafted)
400 Seawd
Denoising Module
Mavenile-aaae ptaptice X
Hand-ctaflod fisslondle -
Figure 2: The overview of the comparison between our method and hand-crafted scale-invariant noise-based method (Zaidi et al. 2022] Feng et al. 2023 Ni et al. 2023). Different from denoising after adding hand-crafted noise, our method adopts a Noise Generator- Generator-Denoising Module paradigm, which generates atom-specific noise distributions through a trainable Noise Generator, and then samples the noises and reconstructs the original conformations. Our reconstruction task is equivalent learning the force fields.
perturbing principle due the difference in PES among different molecules. As shown in Fig. in our method, we use a Noise Generator to generate an atom-specifie noise distribution for each atom in different molecules. The noisy conformation formed by a of noisy atoms, each sampled from its corresponding distribution. This noisy conformation is then fed into the Denoising Module to produce an output. We use the reconstruction loss to guide the learning process of both the De- noising Module and the Noise Generator. In addition, we keep the similarity between the generated distribution and the pre-designed prior distribution, to avoid Noise Generator from converging to trivial solution. Compared with previous methods, we introduce learnable module named Noise Generator, which is jointly trained with the Denoising Module. We provide the pseudocode in the Appendix A.6
Formally, we define = f011.... ,NN where X E 3 and æ, E R denote the original conformation and the position of the i-th atom in the molecule, respectively, and N is the number of total atoms in the molecule. To obtain the corresponding noise for each atom, we model the Noise Generator Gep, which takes the original molecular conformation X input and generates corresponding noise distribution for each atom within the molecule, as shown in Eq. 4 denotes the parameters of the Noise Generator.
(N(11, NN(NN,
22x) (X).
The noisy atom ã; should be randomly sampled from the corresponding generated distribution, where 2; (Nx,,, 22))) Following Eq. 5 achieve sampling by applying the reparameteri- zation, where Ex, represents random noise sampled from Standard Gaussian distribution.
t; = 2; + Ca,, Ozz-
(5)
According to previous research (Vincent 201 I). the denoising task is equivalent to learning the molecular foree fields. Therefore, establish Denoising Module D, to reconstruct the original molecular conformation, where 0 denotes the corresponding parameters. This module takes the perturbed molecular conformation X, consisting input and predicts the corresponding noise for each atom. Through the Denoising Module, acquire the output and demote it as (x = previous studies, we perform scaling operations using the atom-specifis variance to obtain the loss for reconstructing the original molecular conformation, LDemsise: as follows:
noise-perturbed atoms (......, NN), as
{#1,.-. . Following the recommendations from
5

---

Published as a conference paper at ICLR 2025
N
(x
Denoise = E
0
(6)
i=1
However, optimizing Eq (/66ww will lead the model converge trivial solution, preventing the learn- ing of meaningful noise. Therefore, we introduce prior assumption, which is defined as follows:
LKI. Σ DRa. o2)00,2))ll
where Pw; is the prior distribution for atom x,- Therefore, our final optimization objective for jointly training the Noise Generator and the Denoising Module is shown in Eq.
DenuiseVAE DDemoise ALKL:
where denotes the controlling paperrparameter.. In Eq. 8 the denoising objective constrains the scales of the generated voleculc-adaptics noise distributions. If the applied noises exceed the phys- ically permissible spheres of atomic activities under their structural contexts, the original confor- mation cannot be reconstructed. The KI. divergence- based regularization term avoids generating extremely small noises for all atoms and increases the diversity of the sampled conformations.
4 THEORETICAL ANALYSIS UPON RATIONALE BEHIND DENOISE VAE
Our pretraining process constitutes Variational Auto Encoder (VAE). Under modeling ap- proach, optimizing the pretraining objective is equivalent to improving the evidence lower bound (ELBO) of the log-likelihood. Below, provide detailed proof of this.
As shown in Fig. we compare the energy changes in molecules before and after applying noise under three different noise addi- tion modes. We randomly sample 10k molecules from the pre- training dataset and sMAPE (Flores 1986) for measurement.
96.84
MMPP
80
It can be observed that as the scale of the added noise decreases, the energy change the molecule also significantly diminishes. Although our Denoise VAE is constrained by prior distribution with a larger variance than the Fixed scale-0.001, can sample conformations with lower energy. Since the energy is closely re- lated to molecular properties, our method successfully field con- formations that most closely approximate the equilibrium molec- ular properties under different noise addition modes, thus rein- forcing original design intention. We formulate the phenom- ena in the following assumption.
60
40
32.17
20
10.36
Fixed scale-0.1
Fixed scale-0.001
DenaiseVAE
Assumption Applying a limited noise perturbation to a molecule X essentially leaves the molecular energy E changed:
Figure 3: Comparison of molec- ular potential energy changes before and after noise pertur- bation under different modes. Fixed scale-0.1 and Fixed scale- 0.001 denote the noises are sampled from 0.10 and (0 0.001) for all atoms, re- spectively. For our Denoise- VAE, we set our prior distribu- tion to (0 0.1).
E(X + 0)= E(X).
(9)
For DenoiseVAE goal is to better approximate p(X), which represents the distribution molecular conformations that essen- tially preserve the equilibrium state energy. Here, X denotes the molecular conformation after noise perturbation, M denotes the pre-training dataset. Considering the ELBO of the log-likelihood p(X):
Σ logp(X,)2 2 Σ (BaC(iix)) [ogpe(X; xx.)
DKI (x))))((X.)) (10)
X.EM X.EM

---

Published as a conference paper at ICLR 2025
where (x X2) represents the process of inputting the original molecular conformation into the Noise Generator to obtain the corresponding perturbed noisy molecule, (x(,,) follows our prede- fined prior distribution, and po (X; X;) denotes the denoising process by which the Denoising Module removes noise from the perturbed molecule.
Our pretraining objective is to minimize the reconstruction loss (.. log pe (e X.) with
KL divergence term described in Eq. 7 which is equivalent maximizing the ELBO of the log- likelihood. Moreover, under the assumption of the Boltzmann distribution (Boltzmann) 1868). our reconstruction process is equivalent to learning the molecular force field (Vincent 201 ). which in turn is equivalent to approximating the true distribution of molecular energy. Therefore, Eq. 10 can be rewritten as:
E logp(X,) X.EM
Σ X.EM
DenoiseVAE.
(11)
When the pre-training process ends, we ultimately achieves optimized posterior distribution, de- noted as Th (X: X:). Due to better alignment with physical principles, our approach demonstrates a higher evidence lower bound (ELBO) of p(X,)) compared to classical methods that directly sample noise from Gaussian distribution. That is:
Theorem Let L(q po; X.) denote the evidence lower bound (ELBO) of ((;;) obtained from our method, L(po; X:) denote the ELBO of p(X;) acquired from benchmark Gaussian-bused methods. We derive:
ExEEM X.)] 2 ExEEM[LPP X.)]-
(12)
Theorem indicates that our method provides a higher theoretical evidence lower bound guarantee for the real conformation distribution of isoenergetic molecules. Please refer to Appendix A..4 for the corresponding proofs.
EXPERIMENTS
5.1 SETTINGS
Datasets We leverage large-scale molecular dataset PCQM4Mv2 (Nakata Shimazaki 2017) as our pre-training dataset. It includes 3.4 million organic molecules, each represented by an equilib- rium conformation and label computed using density functional theory (DFT). We do not use this label in pre-traiiie training. For downstream tasks, we evaluate our method both on molecular and complex property prediction. For former, we test on QM9 (Ruddigkeit et al. 2012, Ramakrishnan et al.) 2014), MD17 (Chmiela et al. (017)) and PCQM4Mv2 (Nakata & Shimazaki (017). For details, QM9 contains 12 chemical properties of small molecules with stable 3D structures. We follow pre- vious work (Jiao et al.) 2023) and split the dataset the training set, validation set, and test set, which contains 100k, I 8k, and 13k conformations, respectively. MD17 contains the simulated dy- namical trajectories of 8 small organic molecules, with the recorded energy and force each frame. We select 9,500 and 500 frames as the training and validation set respectively. PCQM4Mv2 (Nakata) the
& Shimazaki 2017) has divided validation set and test set. We report the performance validation set according to formal standards, please refer to Appendix A.7 for details. For the latter, we adopt the widely recognized PDBBind dataset (v2019) for the ligand binding affinity (LBA) pre- diction, adhering to the 30% and 60% protein sequence identity splits and preprocessing methods outlined in Atom3l) (Townshend et 2020
Experimental setup We set the prior distribution Pa, as a Gaussian distribution, where x; the mean and is the standard deviation. If not specifically noted, we set 0.1 for all experi- ments. After pre-training. we discard the Noise Generator and retain the Denoising Module for downstream tasks. It is worth mentioning that our Denoise VAE is lightweight network, and our entire pre- training process does not consume too much computing resources or time, which refers to the Appendix A.7 More details of our method provided in the Appendix

---

Published as a conference paper at ICLR 2025
Performance (MAE 4) on QM9 property prediction. The best results are in bold and the second best are underlined. For detailed standard deviation, please refer to Appendix A.8
Table 1:
Method
сномо CLUMO
Ac
< R²
ZPVE
Uo
U
H
G
Cy
(D)
(og)
(meV)
(meV)
(me
(a))
(meV)
(me
(meV)
(meV)
(meV)
mel
SchNet
0.033
0.235
41.0
34,0
63.0
0.07
1.70
14.00
19.00
14.00
14.00
0.033
E(n)-GNN
0.029
0,071
29.0
25.0
48.0
0.11
1.55
11.00
12.00
12.00
12.00
0.031
DimeNet++
0.030
0.044
24.6
19.5
32,6
0.33
1.21
6.32
6.28
6.53
7.56
0.023
PaiNN
0.012
0.045
27.6
20.4
45.7
0.07
1.28
5.85
5.83
5.98
7.35
0.024
Sph
0.025
0.045
22.8
18.9
31.1
0.27
1.120
6.26
6.36
6.33
7.78
0.022
TorchMD-NET
0.011
0.059
20.3
17.5
36.1
0.033
1.840
6.15
6.38
6.16
7.62
0.026
Transformer-M M
0.037
0.041
17.5
16.2
27.4
0.075
1.18
9.37
9,41
9.39
9.63
0.022
GeoSSL-DDM
0.015
0.046
23.5
19.5
40.2
0.122
1.31
6.92
6.99
7.09
7.65
0.024
3D-EMGP
0.020
0.057
21.3
18.2
37.1
0.092
1.38
8.60
8.60
8,70
9.30
0.026
Coord
0.012 0.0517
17.7
14.3
31.8
0.4496
1.71
6.57
6.11
6.45
6,91
0.020
Frad
0.010 0.0374
15.3
13.7
27.8
0.3419
1,418
5.33
5.62
5.55
6.19
0.020
SliDe
0.0087 0.0366
13.6
12.3
26.2
0.3405
1.521
4.28
4.29
4.26
5.37
0.019
DenoiseVAF 0.0079 0.0650
14.2
11.9
26.0
0.062
1.028
4.31
4.03
4.19
5.35
0.015
Table 2: Performance (MAE J) on MD17 force prediction. The best results are in bold and the second best are underlined. For detailed standard deviation, please refer to Appendix A.9
Method Aspirin Benzene
Ethanol Malonaldchyde Naphthalene Salicylic Acid Toluene Uracil
TorchMD-NET
0.1216
0.1479
0.0492
0.0695
0.0390
0.0655
0.0393
0.0484
3D-EMGP
0.1560
0.1648
0.0389
0.0737
0,0829
0.1187
0.0619
0.0773
3D-EMGP
(TorchMD-NET)
0.1124
0.1417
0.0445
0.0618
0,0352
0.0586
0.0385
0.0477
Coord
0.0920
0.1397
0.0402
0.0661
0.0544
0.0790
0.0495
0.0507
Frad
0.0680
0.1606
0.0332
0.0427
0.0277
0.0410
0.0305
0.0323
VAF
0.0567
0.1366
0.0303
0.1012
0.0219
0.1478
0.0301
0.0757
5.2
MAIN RESULTS
We present the results on the QM9 dataset Tab. I Our method achieves the best or the second best in 11 out of 12 tasks compared with other methods. For example, Frad uses hybrid noises to sample noisy conformations but still treats all molecules equally. Our method outperforms Frad in 11 out of 12 tasks, which indicates the importance of considering the PES differences among molecules and sampling the molecule-ppecific noises. Moreover, since we focus on sampling atom- specific and molecule-adaption noise, the representations learned by our method exhibit stronger ability to fit the dynamic properties of molecules. For example, we achieve a performance gain of 21.05% in C, compared SliDe, which represents the molecule's ability to absorb and store heat at different temperatures. This result exactly confirms that our method comprehensively learns the vibrational modes among and within molecules.
QM9
The results are shown in Tab. 2 Our method achieves the best performance in 5 out 8 tasks compared with other methods. For example. Coord samples noises from the same Gaussian distribution for all atoms in the molecules during the pre-training phase, and our method outperforms it all the tasks, which indicates the importance of taking the molecular anisotropy into account. worth mentioning that Frad focuses on modeling the anisotropy of the force fields by using hand-crafted hybrid noises but we still achieve 20.94%, and 16.62% performance gains on the Naphthalene, and Aspirin tasks compared with it. It proves that our atom-specific and molecule- adaptive noise sampling strategy is more reasonable, thus we can acquire more generalizable representation for downstream tasks.
MD17
LBA
We present the results in Tab. 3 Our method offers significant advantages over existing meth- ods, especially at a sequence identity threshold of 30%, which consistently surpasses all basclines across various evaluation metrics, including RMSE, Pearson correlation, and Spearman correlation.
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance on LBA prediction. The best results are in bold and the second best are underlined. For detailed standard deviation, please refer to Appendix A.I0
Sequence Identity 30%
Sequence Identity 60%
Method
RMSE + Pearson
Spearman 1
RMSE
Pearson
Spearman
Sequence Based
DeepDDTA
1,866
0.472
0.471
1.762
0.666
0.663
B&B
1.985
0.165
0.152
1.891
0,249
0.275
TAPE
1.890
0.338
0.286
1.633
0.568
0,571
ProtTrans
1.544
0.438
0.434
1.641
0.595
0.588
Structure Based
Holoprot
1,464
0.509
0.500
1.365
0.749
0.742
IEConv
1.554
0.414
0.428
1.473
0.667
0.675
MaSIF
1,484
0.467
0,455
1.426
0.709
0.701
ATOM3D-3DCNN
1,416
0,550
0,553
1.621
0.608
0,615
ATOM3D-ENN
1.568
0.389
0.408
1.620
0.623
0.633
ATOM3D-GNN
1.601
0.545
0.533
1.408
0.743
0.743
ProNet
1.463
0.551
0.551
1.343
0.765
0.761
Pretraining Based
GeoSSL
1.451
0.577
0.572
DeepAffinity
1.893
0.415
0.426
EGNN-PLM
1.403
0.565
0.544
1.559
0.644
0.646
Uni-Mol
1.520
0.558
0.540
1.619
0.645
0.653
Denoise VAE
1.401
0.588
0.574
1.497
0.769
0.746
At the 60% threshold, Denoise maintains lead the Pearson correlation and secures second place in the Spearman correlation. It worth mentioning that since we pre-train small molecules, our model is not supplemented by knowledge in the field complex macromolecules. However, our method still performs better at the more stringent 30% threshold, which shows that Denoise VAE has better robustness and generalization ability, especially when there is significant difference between the training and test data distributions.
5.3 ABLATION STUDIES
The influence of the hyper-parameter To prevent our Denoising Module from learning trivial solutions that result in meaningless noise, we introduce a prior distribution to constrain the model. As shown in Tab. distribution is set to 0.1, our pre-trained model achieves opti- mal performance on both the QM9 and MD17 downstream tasks. This finding validates
when the standard deviation of the prior
initial intention behind the design of learnable noise: excessive or insufficient noise leads to subopti- mal conformation sampling. If the noise is too large, molecular conformations deviate from the minima on the PES, resulting the sampling of conformations with extremely low probabilities in quantum chemistry and molecular mechanics, such as saddle points transition states. Conversely, if the noise is too small, the sampling search range of molecular conformations becomes narrow, preventing adequate conformation sampling and po- tentially missing additional minima on the energy surface, i.e., stable conformations, leading to insufficient force field learning. As shown in Fig. I under the constraints of the prior distribution, our model adaptively samples from noise distributions with varying variances for atoms situated in different chemical environments.
6.00
Figure 4: The distribution of the variances of our learned noise distributions.
The influence the hyper-parameter From the discussion in the previous section, we learn that the constraint on the prior part is crucial for our model. Therefore. conduct ablation study on the constraint strength À this optimization objective. As shown in Tab. 5 when is set to 1, model performs satisfactorily Generator- Denerisin-- Module paradigm, where we encode the original molecular conformations into corresponding distributions and sample from them reconstruct the original conformations.
downstream tasks. This aligns with Noise

---

Published as a conference paper at ICLR 2025
Table 4: Performance (MAE D) with different
Table 5: Performance (MAE 1) with different X.
QMY
MD17
CHOMO
FLUMO
Aspirin
Benzene
QM9
MD17
16.9
0.2692
0.3883
нномо
ELUMO
Aspirin
Benzene
0.5
17.7
17.0
0.1423
0.1981
À
14.0
11.7
0.0559
0.1265
0.4
12.1
0.0722
0.1725
12.2
0.0722
0.1597
0.1
14.0
11.7
0.0559
0.1265
A 0.2
18.7
20.6
0,2802
0,1773
0.05
26.5
22.4
0.1219
0.2927
5.4
CASE STUDIES
We select two typical examples for visual analysis. Our learnable noise strategy involves adaptive noise sampling for each atom within different molecules, as illustrated in Fig. 5 For both the two molecules, the noise intensity sampled by our model for each atom varies.
Specifically, the molecule in Fig. (a) is an active chiral compound with polar functional groups, in which the amino group (-NH2) can react with acids, the hydroxyl group (-OH) can form hydro- gen bonds and react with acids or bases, and the cyano group (-CN) is polar and can participate in various organic reac- tions such as addition and reduction. For the molecule in Fig. [5] (b), its internal bicyclic structure endows it with high rigidity, making it less prone to struc- tural changes. Therefore, compared to the molecule in Fig. 5 (a), the overall noise perturbation applied to it should be smaller.
(a) Cyclopropane derivative
(b) Norcamphor derivative
Figure 5: Visualization of the intensity rationality of the learnable atom-level noise added to different molecules. deeper green color indicates higher noise values. the right tables, we follow the atom index order from the left molecular graphs and apply noises of the same scale to only one atom in the molecule at a time, calculating the overall energy change of the molecule before and after the perturbation.
Comparing the two molecules in Fig. we can see that they contain some iden- tical functional groups, such as hydroxyl groups. However, is clear from Fig. 5 that the noise scale applicd by our model to the oxygen atoms (O) in the hydroxyl molecules differs.
groups of the This further supports that our model adaptively performs meaningful sampling the PESs of different molecules based their dy- namic characteristics. The two molecules presented in Fig. exxibii significantly different proper- ties. Through our design of learnable noise, can adaptively sample conformations for different molecules. thereby learning better molecular representations and achieving superior performance in downstream tasks such as molecular property prediction.
CONCLUSION
In this paper, we explore the denoising method for 3D molecular pre-training. We propose an atom- specific and moleculc-adaptive molecular pre- pre-training method, named DenoiseVAE. which employs a Noise Generator to generate atom-specific noise distributions for different molecules and use Denoising Module to reconstruct the original conformations. Since the sampling is more consistent with physical principles, our method can better capture the meaningful conformations molecules, leading to more accurate force field learning which has significant influence downstream tasks. Furthermore, we provide a theoretical analysis of the rationale underlying Denoise VAE, enhancing the interpretabilityy of our method. The experimental results demonstrate the effectiveness of method. It is worth noting that our method is based on classical force field assumptions. The poten- tial integration with more accurate force fields represents promising direction for future research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported in part by the National Natural Science Foundation of China No. 62376277 and No. 61976206, Doubao large model fund and the Beijing Nova Program (No. 20230484278).
REFERENCES
Sarp Aykent and Tian Xia. Gotennet: Rethinking efficient 3d equivariant graph neural networks.
In The Thirteenth International Conference on Learning Representations, 2025. URL https:
7 //openreviee pppenneview.net/forromee httt://formmm.05555x00000000
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. XrXiv preprint arXiv:1902.08661. 2019.
Dana Bishara, Yuxi Xie, Wing Kam Liu, and Shaofan Li. A state-of-the-art review ma- chine learning-base based multiscale modeling, simulation, homogenization and design materials. Archives of computational methods in engineering. 30(1):191-22))) 2023.
Alexandre Blanco-Gonzaler Alfonso Cabezon, Alejandro Seco-Gonzalez, Daniel Corde-Torres, Paula Antelo- Riveeir, Angel Pineiro, and Rebeca Garcia-Fandino. The role ai in drug discov- ery: challenges, opportunities, and strategies. Pharmaceuticals, 16(6):891, 2023..
Ludwig Boltzmann. Studien uber das gleichgewicht der lebenden kraft. Wissenschafiliche Abhand- lungen, 1:49-96, 1868.
Laurie J Butler. Chemical reaction dynamics beyond the born-oppenheimer approximation. Annual review of physical chemistry, 49(1): 125-171, 1998.
Jiacheng Cen, Anyi Li, Ning Lin, Yuxiang Ren, Zihe Wang, and Wenbing Huang. Are high- degree representations really unnecessary in equivariant graph neural networks? In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:
//peenzeview http://forumm..ccooonnnnCCC
Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane Riviere, Kevin Tran, Javier Heras-Domingos Caleb Ho, Weihua Hu, al. Open catalyst 2020 (oc20) dataset and community challenges. Acs Catalysis, (1(10):65996600) 2021.
Stefan Chmiela, Alexandre Tkatchenko, Huziel E Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Robert Müller. Machine learning of accurate energy mergy-conserrvng molecular force fields. Science advances, 3603s165 2017.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, al. Prottrans: Toward un- derstanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):7112-7122
2021.
Mohammed Eslami, Aaron Adler, Rajmonda S Caceres, Joshua G Dunn, Nancy Loughnn,, Vanessa A Varaljay, and Hector Garcia Martin. Artificial intelligence for synthetic biology. Com- munications the ACM, 65(5):88 97, 2022.
Shikun Feng. Yuyan Ni, Yanyan Lan, Zhi-Ming Ma, and Wei-Ying Ma. Fractional denoising for 3d molecular pre-training. In International Conference on Machine Learning, pp. 9938-9961. PMLR, 2023.
Benito E Flores. A pragmatic view of accuracy measurement in forecasting. Omega, 14(2):93-99 1986.
J Thorben Frank, Oliver T Unke, Klaus-Robers Müller, and Stefan Chmiela. A euclidean transformer for fast and stable machine learned force fields. Nature Communications, 15(1):6539, 2024.
Gainza, Sverrisson, F. Monti, E. Rodolà, D. Boscaini, M. M. Bronstein, and B. E. Cor- reia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, pp. 184-192 Feb 2020. doi: 10. 09010008441159--1---06666 http::/ds doi org 10. 1038 49922419992999 01 000-992211--6666-10--11 9-0666-66
URL

---

Published as a conference paper at ICLR 2025
Johannes Gasteiger. Janek Groß, and Stephan Günnemann. Directional message passing for molec- ular graphs. arXiv preprint arXiv:2003.0311 2020.
Manoj Kumar Goshisht. Machine learning and deep learning in synthetic biology: Key architectures, applications, and challenges. ACS omega, 9(9):9921-99)) -945, 2024.
Chan Soo Ha, Desheng Yao, Zhenpeng Xu, Chenang Liu, Han Liu, Daniel Elkins, Matthew Kile, Vikram Deshpande, Zhenyu Kong, Mathieu Bauchy, et al. Rapid inverse design of metamaterials based on prescribed mechanical behavior through machine learning. Nature Communications, 14 (1):5765, 2023.
Jiaqi Han, Wenbing Huang, Tingyang Xu, and Yu Rong. Equivariant graph hierarchy-basee neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URL Eoormmid=yyuu Barum2id=yyxttm1ind
Jiaqi Han, Jiacheng Cen, Liming Wu, Zongzhao Li, Xiangzhe Kong, Rui Jiao, Ziyang Yu, Tingyang Xu, Fandi Wu. Zihe Wang, et al. A survey of geometric graph neural networks: Data structures, models and applications. arXiv preprint arXiv:2403. 00485, 2024.
Pedro Hermosilla, Marco Schäfer, Matëj Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora Kozlíková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsie convolution and pooling for learning on 3d protein structures. arXiv preprint arXiv:2007.06252, 2020.
Rui Jiao, Jiaqi Han, Wenbing Huang, Yu Rong, and Yang Liu. Energy-motivated equivariant pre- training for 3d molecular graphs. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.
2023.
Rui Jiao, Xiangzhe Kong, Ziyang Yu, Wenbing Huang, and Yang Liu. Equivariant pretrained transformer for unified geometric learning on multi-domain 3d molecules. arXiv preprint arXiv:2402.12774. 2024.
Mostafa Karimi, Di Wu, Zhangyang Wang, and Yang shen. Deepaffinity: Interpretable deep learning of compound-protein affinity through unified recurrent and convolutional neural networks. Jun 2018. URLhttpp: / //xx doi.ooi.ooooooooo www.ooog......o..... 1101/351661
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. 13Xiv:1312.611,,22 2013.
arXiv preprint
H Köuppel, Wolfgang Domeke, and Lorenz S Cederbaum. Multimode molecular dynamics beyond the born-oppenheimer approximation. Advances in chemical physics, pp. 59-246, 1984.
Greg Landrum. Rdkit: Open-source cheminformatics. 2006. Google Scholar, 2006.
Zongzhao Li, Jiacheng Cen, Wenbing Huang, Taifeng Wang, and Le Song. Size Size- generalizable RNA structure evaluation by exploring hierarchical geometries. The Thirteenth International Con- ference on Learning Representations, 2025a. URL https: httpps:/openrrvieet opwprevieeeee...ee/o/ee net. b forum?
[d=QaTBHSqmHH
Zongzhao Li, Jiacheng Cen, Bing Su, Wenbing Huang, Tingyang Xu, Yu Rong, and Deli Zhao. Large language-geometry model: When Ilm meets equivariance. arXiv preprint arXiv:2502.111499 2025b..
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. arXiv preprint arX.v:2110.07728, 2021.
Shengehao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with se (3)-invariant denoising distance matching. arXiv preprint arXiv:2206. 36002, 2022a.
Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. In International Conference on Learning Representa- fions (ICI.), 2022b.
12

---

Published as a conference paper at ICLR 2025
Shuqi Lu, Zhifeng Gao, Di He, Linfeng Zhang, and Guolin Ke. Highly accurate quantum chemical property prediction with uni-mol+. arXiv preprint 1rXi::2303.16983 2023.
Shengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie- Yin Liu, Liwei Wang, and Di He. One transformer understand both 2d & 3d molecular data. In The Eleventh International Conference on Learning Representations, 2022.
Kit-Kay Mak, Yi-Hang Wong, and Mallikarjuna Rao Pichika. Artificial intelligence in drug discov- ery and development. Drug Discovery and Evaluation: Safety and Pharmacokinetic Assays, pp- 1-38, 2023.
Maho Nakata and Tomomi Shimazaki. Pubchemqe project: A large-scale first-principles electronic structure datahase for data-driven chemistry. Journal of Chemical Information and Modeling, pp. 1300-1308, Jun 2017. doi: 10. 10.100//assjcimm7700088. URL http: doi. http://txxxoi.oo//.....
1021/acs jaim. 7b00083
Yuyan Ni, Shikun Feng, Wei-Ying Ma, Zhi-Ming Ma, and Yanyan Lan. Sliced denoising: A physics- informed molecular pre-training method. arXiv preprint arXiv:2311.02124 2023.
Anders MN Niklasson. Extended born-oppenheimer molecular dynamics. Physical review letters, 100(12): 233004, 2008
Hakime Özüürk, Arzucan Özgür, and Elif Ozkirimli. Deepdta: deep drug- trugetarget binding affinity prediction. Bioinformatics, 34(17):1821-822) 2018.
Krzysztof Pachucki. Bom-oppenheimer potential for 2. Physical Review A, 82(3):032509, 2010.
Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7,201
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Picter Abbeel, and Yun S. Song. Evaluating protein transfer learning with tape. Jun 2019. URL http: //dx. doi. 10 101/ 676825
Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of chemical information and modeling, 52(1 1):2864 2875, 2012.
Shin Sato. A new method of drawing the potential energy surface. Bulletin of the Chemical Society of Japan, 28(7))450-4433 1955.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International conference on machine learning, pp. 9323 2232-933.. PMLR, 2021.
H Bernhard Schlegel. Exploring potential energy surfaces for chemical reactions: overview of some practical methods. Journal of computational chemistry, 24(12):1514-1527, 2003.
Kristof Schüitt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Roburt Müller. Schnet: A continuous-filee convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference Machine Learning, pp. 337779388.. PMLR, 2021.
Yu Shi, Shuxin Zheng, Guolin Ke, Yifei Shen, Jiacheng You, Jiyan Hc, Shengjic Luo, Chang Liu, Di He, and Tie Yan Liu. Benchmarking graphormer large-scale molecular modeling datasets. arXiv preprint arXiv:2203.04810 2022.
Vignesh Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning on proteins.
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. In International Conference Machine Learning, pp. 20502, PMLR, 2022.
13

---

Published as a conference paper at ICLR 2025
Philipp Thölke and Gianni De Fabritiis. Torchmd-net: Equivariant transformers for neural network based molecular potentials. arXiv preprint arXiv:2202.02541, 2022.
Raphael JL Townshend, Martin Vögele, Patricia Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing, Brandon Anderson, Stephan Eismann, al. Atom3d: Tasks on molecules in three dimensions. arXiv preprint arXiv:20/2.0400 2020.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1677 2011.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Picrre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Limei Wang. Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning hierarchical protein representations via complete 3d graph networks. Jul 2022.
RG Woolley and BT Sutcliffe. Molecular structure and the born bom-ooppeneiimer approximation. Chemical Physics Lefters, 45(2):393-398, 1977.
Fang Wu, Siyuan Li, Lirong Wu, Dragomir Radev, Qiang Zhang, and Stan Z Li. Discovering the representation bottleneck of graph neural networks from multi-order interactions. 2022.
Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Tch, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via de- noising for molecular property prediction. arXiv preprint arXiv:2206.0011 2022.
Yuelin Zhang, Jiacheng Cen, Jiaqi Han, Zhiqiang Zhang, JUN ZHOU, and Wenbing Huang. Im- proving equivariant graph neural networks on large geometric graphs via virtual nodes learn-
ing. In Forty-first International Conference Machine Learning, 2024. URL https:
//openreview.
Shuxin Zheng, Jiyan He, Chang Liu, Yu Shi, Ziheng Lu, Weitao Feng, Fusong Ju, Jiaxi Wang, Jianwei Zhu, Yaosen Min, et al. Predicting equilibrium distributions for molecular systems with deep learning. Nature Machine Intelligence, pp. 1-10, 2024.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. 2023.
Jinhua Zhu, Yingee Xia, Lijun Wu, Shufang Xie, Tao Qin. Wengang Zhou, Houqiang Li, and Tic- Yan Liu. Unified 2d and 3d pre-training of molecular representations. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp. 2022.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 POTENTIAL ENERGY SURFACE
The Born- Com-Oppenheimer approximation (BO approximation, also known the adiabatic approxima- tion) is a widely used method for solving the quantum mechanical equations of systems that include both electrons and atomic nuclei (Woolley & Sutcliffe 1977). According the Oom--pppeneeier approximation, the energy of the molecular ground electronic state can be regarded as a function of the nuclear coordinates. In molecular mechanics, all defined functions can be considered as functions of the nuclear coordinates, and the energy changes of the system can be viewed as move- ments on multidimensional surface (Köuppel al., 1984] Butler 1998 Pachucki] 2010 Niklasson 2008). The molecular mechanics force field function is shown in 13 ha
ki (0: 0,00)
V (1 cos(nw -7)
V(n))
(1: 4,0)²+
Σ angels
Σ
2
bond
torsion
(13)
N
N
Oij
12
Oij
9:95
Tij
Tij
ATEOTI
Therefore, the Potential Energy Surface (PES) of a molecule is a hypersurface formed by the po- tential energy with respect to all possible positions of the atoms. The positions of all atoms can be represented by wddgree-of-freedoee Cartesian coordinate system comprising translations and rotations.
On PES of the molecule, all minima are the primary focus of our work. At these points, moving in any direction on the PES, even with slight perturbations, will cause changes the molecular struc- ture, resulting in an increase in the molecular potential energy. The molecular PES may have multi- ple minima, each corresponding to an equilibrium structure of the system. For the same molecule, different minima the PES correspond to different conformations or structural isomers. In reac- tion system, these minima correspond to reactants, intermediates, products, and so on. Therefore, considering that the minima represent the true properties of the system, work focuses on these points for study.
In the fields of quantum chemistry and molecular dynamics, rigorously determine whether a given molecular conformation is sampled from a true minima on the PES, a frequency analysis is required, and the calculated frequencies should all be positive. If negative frequencies appear, may be due to the constraints imposed by molecular symmetry. This occurs because if a molecule deviates from minima position, it will experience restoring force in the opposite direction, which allows the calculation of the molecule's 's vibrational frequencies. These vibrational frequencies correspond to the molecular spectrum.
For the stable conformations of molecules, we can employ several theoretical methods to obtain them. If the molecule is not very large, we can perform a conformation search by rotating all rotatable chemical bonds and then searching for the lowest energy points. Molecular dynamics sim- ulations and other methods can also be used search for conformations. However, if the molecule is very large, the aforementioned methods may not be able to find the true minimum; they can only find approximate local minima. Our work aims to provide solutions to the high computational cost of sampling equilibrium and stable conformations, exploring and attempting approaches that benefit practical drug discovery and new material design.
A.2 MOLECULAR FORCE FIELD LEARNING
According to the prior knowledge in the field of statistical physics, the probability of the occurrence of 3D molecular conformation is described by Boltzmann distribution (Boltzmann, 1868). Thereffrr,, can obtain Eq. 14 through calculation.
V a log - FzF(
(14)
where å denotes the perturbed conformation.
Learning the conformational distribution of molecules is equivalent to learning the molecular force field. Due the lack of energy and force field labels, we generally learn approximate molecular
15

---

Published as a conference paper at ICLR 2025
force fields by adding noise to the molecules and performing denoising tasks. Previous research has proven the equivalence of denoising tasks and learning molecular force fields (Vincent 2011). as shown in Eq. [5] More details are shown in Sec. A.3 in the Appendix.
D2
V l(gl(
(15)
72
where T constant which can absorbed into graph neural network (GNN) (Feng al. 2023 Previous work has proved this (Feng et al. 2023). as shown in Eq.
16
Exaa)|GNNN( (- V()
= =EMa)]lGNNN((1)) - V log |
= =B (G.N
V log Vzlogp(i (():p +T
(16)
H テ- - 1-
= =E =ENAA,))IGNNN(((
A.3 EQUIVALENCE OF DENOISING AND FORCE FIELD LEARNING
Since have emphasized many times before that the denoising task is equivalent to learning the molecular force field, here we prove this hypothesis. The proof process refers to Frad (Feng et al. 2023).
Proof (1) J.()).
Ji(0) = Ep(e) (GNN)()) - Va Vzlog((2)113,
(17)
J2(0) E F logp($/<)11
(18)
(1) = [ GNN,(E), V logp(t) >} Ts.
(19)
J2( = (BaaaacclllllNWee]]]-]2222-a-- G GNNo(T), Va (ogp(T())>] T4, (20)
Ep(a|<< (N), V logp(T) >
p(ā) < GN..(E.)) Vz di
Var(i),
(GNN
di
p(ž)
< (NN,, VaP(T) di (CNN.(E), V (((l=))(=)dxx))d
(21)
(ONNo(C) / (())Van((lo)dd)aaa (GNNN((), plaaaiiplaa
(((l))(()) < <NN, Vz logp(()r)) > drdž
Ep((.0)) GN)(( V logn(E)z)
16

---

Published as a conference paper at ICLR 2025
A.4 PROOF OF THE EVIDENCE LOWER BOUND OF DENOISEVAE
Proof of Theorem
logp(X,)) = logp(X;)) / (x(x)) dx,
(22)
= - (x((x(())) logg(())) dx,
log p(X)) = log px,,, Xx)
p( xx()
= log (X,, Xx.) - log p( (Xi
(23)
logg p(X;, Xz) log p(X;
9: (Xx(x:)
9o 9p (X:
Multiply both sides of the above formula by 9, (x:) and take the integral:
/uo(X]1x.) log X() dx, - f (x((x)) log (X), dX,
log logp(X.)=
(x,)
-xx.(uo(X(Xxi)IlXXX (x:)) + (x((1))))) log Po (o) d.:
(X (1,)
= (X, (x,) 1pp( (x (x))) + (x(()) (x, dx;
(24)
+ (xx(xx log
po po dx,
go (X,
D (xx])x Xx)) + E Exingo x.)
- (gp (X, 1|ps (X,))
Since any KL divergence is non-negative, we can derive:
logr(X;) 2 E (X,x.) Po (X,
DxL (2 (Xx(xx)) (X.))
(25)
Let Pe; denote the evidence lower bound (ELBO) of p(X,) obtained from our method, Lpp; X,)) denote the ELBO of p(X,) acquired from benchmark classical methods which directly sample noise from a Gaussian distribution. That is:
L(ăp; P6; X.) E logg
-KL (X,1x)
(26)
Our pretraining objective is to minimize the reconstruction loss -E
q. (X)1x)) log Po X.) with
a KL divergence term as described in Eq. 7 which is equivalent to maximizing the ELBO of the log- lieeiiiiihood. When the praitnni process ends, ultimately achieves optimized posterior distribution, denoted (X, X,). Other benchmark Gaussian-based methods approximate the true noise distribution by directly using a Gaussian distribution. In contrast, our approach learns the noise distribution from the model based on Gaussian prior. As result, the noise we learn is more flexible and adaptive, aligning better with the physical principle of anisotropy in force fields. Con- sequently, the learned distribution 7.p (X; X,) is expected to be closer to the true noise distribution compared to other classical methods.
Let ELBO denote the ELBO of the true noise distribution Pnoise (X)) obtained from our method, ELBOnoiseG denote the ELBO of Pnoise (X;) acquired from benchmark Gaussian-based methods. We have:
ELBO RoSeeDD DoiseD
ELBO noiseG
(27)
Therefore, the ELBO of x(X,) of our method is also higher than that of benchmark Gaussian-batt based methods. That is:
17

---

Published as a conference paper at ICLR 2025
Algorithm Algorithm of our Denoise VAE
Require:
1: Dg: Denoising Module
2: Gg: Noise Generator
3: T.Training steps
4: S: Training set
5: Pa.: Prior distribution
6: N: Gaussian distribution
Ensure: Da.
7: for in I do
8:
for X in S do
9:
F1. .N X
10:
[N(x1, Fe ) ... W(xN, (X)
for X
12:
2; 8i + Cx, om;
13:
end for
14:
X [ Ni
15:
{81,
(NN) Do (X)
16:
Densise
E x(x Ei= oz
8;
o3.
LKL. = 1 E DKL(NN (Ni
18:
Optimize the object Denoise XL1I.
19:
end for
20:
end for
21:
return Dg
L(ãp: Pe; X.) 2 L(p;; Xi) Extending from one molecule to the entire dataset M, we have
(28)
Ex,eM|L(gp Po: X,)] 2 (xp:
(29)
A.5 THE PROBABILISTIC INVARIANCE OF DENOISEVAE
Our method satisfies 0(3) probabilistic invariance, which also provides explanation for the good performance of our method. We give the corresponding proof below:
Given p(r' (z) = (x (ar, T²T), we need to prove p(Qx' t/Qx + t) = p(x' p(x'(x)) for VQ E 0(3),
t E 3
Proof
p(Qr' + 1Qx+1))
=N(Q2 + +t
LIQQi'Qu, =V =QQ
(30)
=N(x'|2,0'1
A.6 PSEUDOCODE OF OUR METHOD
In order to show our training process more clearly, here we give the pscudocode of our entire method pipeline, shown in Alg.
A.7 PERFORMANCE ON PCQM4MV2
Due to the limitations in the length of the main paper, we present the performance of our method on the PCQM4Mv2 dataset here. As shown in Tab. our method achieves optimal performance on the
18

---

Published as a conference paper at ICLR 2025
validation set. Notably, despite having significantly fewer parameters compared to the other methods listed in the table, our model still demonstrates superior performance. This strongly validates the effectiveness of our approach.
In addition, we introduce the PCQM4My2 dataset in more details here. The PCQM4Mv2 dataset, part of the Open Graph Benchmark (OGB), is large-scale quantum chemistry dataset designed for molecular property prediction and graph representation learning. contains approximately 3.4 million organic molecules, each represented as molecular graphs with detailed atomic and bond- ing features. The primary task is to predict the HOMO-LUMO gap, crucial property in quantum chemistry that reflects molecular stability and electronic behavior, with the target values derived from Density Functional Theory (DFT) calculations. The dataset is divided into training, valida- tion, and test sets, supporting both supervised and pre-training tasks. PCQM4Mv2 particularly valuable for training graph neural networks (GNNs) due to its scale and high-quality annotations. Its application spans self seff-supervise pretraining, molecular property prediction, and transfer learn- ing. Despite its computationally demanding nature, it serves as benchmark for models aiming to generalize across large and diverse molecular datasets.
Table 6: Performance on the PCQM4Mv2 dataset. The best results are in bold and the second best are underlined. We perform 3 replicates on the dataset to obtain the mean and standard deviation.
Method
#param.
Valid MAE
MLP-Fingerprint
16.1M
0.1735
GCN
2.0M
0.1379
GIN
3.8M
0.1195
GINE-VN
13.2M
0.1167
GCN-VN
4.9M
0.1153
GIN-vN
6.7M
0.1083
DeeperGCN-vN
25.5M
0.1021
GraphGPS
6.2M
0.0938
TokenGT
48.5M
0.0910
GRPEBASE
46.2M
0.0890
EGT
89.3M
0.0869
GRPELARGE
46.2M
0.0867
Graphormer
47.1M
0.0864
GraphGPSBASE
19.4M
0.0858
GraphGPSDEEP
13.8M
0.0852
GEM-2
32.1M
0.0793
GPS++
44.3M
0.0778
Transformer-M
47.1M
0.0787
Denoise VAI
1.44M
0.0777 + 0.0005
A.8
PERFORMANCE ON QM9
For QM9, we compare our method with SchNet (Schütt et al., 2017). E(n)-GNN (Saooras et al.
2021 4 DimeNet++ (Gstteiger et al. 2020). PaiNN (Schütt et al. (021), SpererNet (Lu et al.
2022b), TorchMD-NE NET (Thölke & De Fabritiis 2022). Transformer-h M (Luo et al. 2022). GeoSSL-
DDM
Liu et al. 2022a), 3D-EMGP (Jiao et al. 2023 Coord (Zaidi al. 2022). Frad (Feng al.)
2023
SliDE (Ni al. (003).
In order to verify the stability of our method, we randomly select random seeds and run times on the QM9 dataset to obtain the mean and standard deviation of our method, as shown in Tab. 7
Moreover, we supply more information about the QM9 dataset here. The QM9 dataset is a widely used benchmark for quantum chemistry tasks, providing high-quality data for the prediction of includes 33,000 small organic molecules. each containing up to 9 heavy atoms from elements H, C, O, and F. The dataset is derived from exhaustive quantum mechanical calculations using Density Functional Theory (DFT), ensuring accurate molecular descriptors. QM9 offers 12 molecular properties, including dipole moment, isotropic polarizability, HOMO/LUMO
molecular properties.
19

---

Published as a conference paper at ICLR 2025
Table 7: Performance (MAE 4) on QM9 property prediction. The best results are in bold and the second best are underlined. Standard deviation is in parentheses.
Method
сиомо CLUMO
Ac
< RP
ZPVE (meV)
Ug (me
U
H
G
Ce
(D)
(ag)
(meV)
(meV)
(meV)
(ag)
(me
V)
(me (me V)
mork
SchNet
0.033
0.235
41.0
34.0
63.0
0.07
1.70
14.00
19.00
14.00
14.00
0.033
E(n)-GNN
0.029
0.071
29.0
25.0
48.0
0.11
1.55
11.00
12.00
12.00
12.00
0.031
DimeNet++
0.030
0.044
24.6
19.5
32.6
0.33
1.21
6.32
6.28
6.53
7.56
0.023
PaiNN
0.012
0.045
27.6
20.4
45.7
0.07
1.28
5.85
5.83
5.98
7.35
0.024
Seh
0.025
0.045
22.8
18.9
31.1
0.27
1.120
6.26
6.36
6.33
7.78
0.022
TorchMD-NET
0.011
0.059
20.3
17.5
36.1
0.033
1.840
6.15
6.38
6.16
7.62
0.026
Transformer-M M
0.037
0.041
17.5
16.2
27.4
0.075
1.18
9.37
9.41
9.39
9.63
0.022
GeoSSL-DDM
0.015
0.046
23.5
19.5
40.2
0.122
1.31
6,92
6.99
7.09
7.65
0.024
3D-EMGP
0.020
0.057
21.3
18.2
37.1
0.092
1.38
8.60
8.60
8.70
9.30
0.026
Coord
0.012
0.0517
17.7
14.3
31.8
0.4496
1.71
6.57
6.11
6.45
6.91
0.020
Frad
0.010
0.0374
15.3
13.7
27.8
0.3419
1.418
5.33
5.62
5.55
6.19
0.020
SliDe
0.0087
0.0366
13.6
12.3
26.2
0.3405
1.521
4.28
4.29
4.26
5.37
0.019
DenoiseVAB
0.0079
0.0650
14.2
11.9
26.0
0.062
1.028
4.31
4.03
4.19
5.35
0.015
(0.00008)
(0.002)
(0.12)
(0.07)
(0.24)
(0.00077
(0.002)
(0.009)
(0.02)
(0.03)
(0.04)
(0.0001)
energy levels, and heat capacity, covering essential features for chemical and physical modeling. Each molecule is represented as molecular graph, making particularly suitable for developing and benchmarking graph neural networks (GNNs). With its standardized structure and focus on small molecules, QM9 widely adopted for tasks such as molecular property prediction, represen- tation learning, and generative modeling in computational chemistry and materials science.
A.9 PERFORMANCE ON MD17
For MD17, we compare our method with TorchMD-NET Thölke & De Fabritiis 2022 3D- EMGP (Jiao et al., 2023), 3D-EMGP(TorchMD-NNTT Coord Zaidii et al. 2022 and Frad (Feng et al. 2023).
In order to verify the stability of our method, randomly select random seeds and run times on the MD17 dataset to obtain the mean and standard deviation of our method, as shown in Tab. Bab.
Regarding the MD17 dataset, it is benchmark for molecular dynamics and force field prediction tasks, providing time-series data of molecular geometries and corresponding energy and force labels. Derived from high-accuracy Density Functional Theory (DFT) simulations, MD17 includes 8 small organic molecules, such as benzene, toluene, and ethanol, spanning diverse chemical structures. Each molecule's dataset contains thousands of molecular conformations sampled along dynamic trajectories, with energy values and atomic forces calculated for each conformation. The primary tasks involve predicting the potential energy and forces (kealmol -A- to model molecular inter- actions, enabling applications in force field learning and molecular dynamics simulations. Unlike static datasets like QM9, MD17 captures temporal dependencies, making ideal for developing ma- chine learning models that understand molecular behavior over time. The dataset challenges models to generalize across unseen conformations and accurately predict properties, providing testbed for equivariant graph neural networks and other advanced architectures.
A.10 PERFORMANCE ON LBA
For LBA, we compare our method with DeepDTA (ötürrk et al.. 2018). B&B (Bepler Berger
2019), TAPE (Rao et al. 2019). ProtTrans (Elnaggar et al. 2021). Holoprot (Somnath et al.)
IEConv (Hermosilla et al. 2020). MaSIF (Gainaa et al. 2020), STOMDD-3DCND ATOM3D ENN,
STOM3D-GNN (Townshend et al. 2020), ProNet (Wang al. 2022). GeoSSL (Liu et al.
2022a
EGNN-PLM et al. DeepAffinity (Karimi et al. 2018) and Uni-Mol (Zhou et al.
2023
We run three times under the two splittings to obtain the corresponding mean and standard deviation. The experimental results are shown in Tab, 9 and Tab. I0 which further proves the stability of our method.
20

---

Published as a conference paper at ICLR 2025
Table 8: Performance (MAE 4) on MD17 force prediction. The best results are in bold and the second best are underlined. Standard deviation is in parentheses.
Method
Aspirin
Benzene
Ethanol Malonaldehyde Naphthalene Salicylic Acid
Toluene
Uracil
TorchMD-NET
0.1216
0.1479
0,0492
0.0695
0.0390
0.0655
0.0393
0.0484
3D-EMGP
0.1560
0.1648
0.0389
0.0737
0,0829
0.1187
0.0619
0,0773
3D-EMGP
0.1124
0.1417
0.0445
0.0618
0.0352
0.0586
0.0385
0.0477
(TorchMD-NET)
Coord
0.0920
0.1397
0.0402
0.0661
0.0544
0.0790
0.0495
0.0507
Frad
0.0680
0.1606
0.0332
0.0427
0.0277
0.0410
0.0305
0.0323
DenoiseVA VAE
0.0567
0.1366
0.0303
0.1012 (0.015)
0.0219 (0.001)
0.1478
0.0301
0.0757
(0.006)
(0.011)
(0.008)
(0.018)
(0.007)
(0.006)
Table 9: Performance of sequence identity 30% on LBA prediction with standard deviation.
Sequence Identity 30%
Method
RMSE
+
Pearson ?
Spearman 1
Sequence Based
DeepDTAA B&B
1.866+0008
0.472+0002
0.471=0.00
1.985+0000
0.165++0.01
0.152+0.02
TAPE
1.890=0.04
0.338=10.04
0.286=10112
ProtTrans
1.544=0.02
0.438+10.05
0.434410006
Structure Based
Holoprot
1.464=0.01
0.509=1.00
0.500
IEConv
1.554++0.02
0.414++0.05
0.428+0.03
MaSIF
1.484+0002
0.467++002
0.455+0.01
XTOM3DD33DCNN
1.416++0.02
0.550++0.02
0.553+0001
ATOM3D-ENN
1.568=0.01
0.389-1002
0.408=0.02
ATOM3D-GNN
1.601
0.545-10003
0.533+0.03
ProNet
1.463=0.00
0.551+0.01
0.551
Pretraining Based
GeoSSL
1.45
0.577+0.002
0.572+00.1
DeepAffintty
1.893+00.55
0.415
0.426
EGNN-PLM
1.403+0.01
0.565+10.02
0.544=0.01
Uni-Mol
1.520=0.03
0.558=10.00
0.540=1.000
Denoise VAE
1.401=0.001
0.588++0.03
0.574+1.01
In addition, the LBA (Ligand Binding Affinity) dataset is benchmark for predicting the binding affinity between protein-ligand complexes, a critical task in drug discovery and computational biol- ogy. It contains 4,463 protein-ligand complex structures, each annotated with experimental binding affinity values in units of (od),,), derived from the PDBBind database. Each complex includes detailed 3D spatial information of the protein and ligand, capturing key structural and chemical interactions. The primary goal is to predict binding affinity, helping assess molecular docking accu- racy and protein-ligand interaction modeling. I.A's diverse dataset spans a range of protein sizes, ligand structures, and binding strengths, making it suitable for evaluating models' generalization across chemical and structural variations. is widely used for training and benchmarking graph neural networks (GNNs) and deep learning models incorporating 3D spatial features. LBA is instru- mental for advancing computational approaches precision medicine and ligand interaction analysis.
A.11 VISUALIZATION OF ENERGY LANDSCAPE
To investigate the learned representation space of our pre-trained model, we visualize the local en- ergy landscape around a given molecular conformation sampled from MD17 dataset. Specifically, following 3D-EMGP (Jiao et al. 2023), select a random conformation X from the dataset and utilize the pre-traine trained Noise Generator to obtain the atom-specific noise distributions for X. Ac- cording to the noise distributions, randomly generate two directions G1, G2 R Then
21

---

Published as a conference paper at ICLR 2025
Table 10: Performance of sequence identity 60% on LBA prediction with standard deviation.
Sequence Identity 60%
Method
RMSE
Pearson
Spearman 1
Sequence Based
DeepDTA B&B
1.762:10.26
0.666#0.01
0.663=0.02
1.891=0.00
0.249:10.01
0.275=0001
TAPE
1.633+0.02
0.568=10.03
0.501
ProtTrans
1.641
0.595-10.001
0.588+0.01
Structure Based
Holoprot
1.365=0004
0.74910.01
0.742
IEConv
1.473+0.02
0.667+0.01
0.675+0.02
MaSIF
1.426=0.02
0.709
0.701
ATOM3D-3DDCN
1.621+0.03
0.608110.22
0.615
ATOM3D-ENN
1.620+10.05
0.623+1.02
0.63
ATOM3D-GNN
1..08
0.743+10.02
0.743+
ProNet
1.343+0.03
0.765+0.01
0.761=
Pretraining
Based
EGNN-PLM
1.559=00.02
0.644+10022
0.646
Uni-Mol
1.619#0.04
0.645++0.2
0.653=#002
Denoise VAE
1.497=1.002
0.769-10.01
0.746
75
70
60
50
45
0.5
0.5
10
0.0
0.0
-03
-2.5
1.0
(a) Aspirin
(b) Benzene
(c) Salicylic Acid
Figure 6: Visualization of energy landscape on MD17 dataset.
construct a 2D conformation plane as (XI,,) the plane, by varying and j. we calculate the corresponding energy as E,,; E(X E(X(i,)) (i, where denotes the energy calculation function in RDKit library (Landrum) 2006). As shown Fig, 6 we select Aspirin, Benzene, and Salicylic Acid to plot the energy landscape (i. E. It is evident that the molecular force field learned by our model can identify the equilibrium conformation as local energy minimum point on the energy landscape. Moreover, the energy surfaces of different molecules vary greatly. Among them, the energy surface of the Benzene converges to its steady- state conformation at a significantly slower rate than the Salicylic Acid, which is consistent with real chemical constraints and also verifies our motivation.
x + + +GG2). For each point in
A. VISUALIZATION OF THE STABILITY OF OUR METHOD
Our method demonstrates strong training stability during both the pretraining and finetuning stages, achieving consistent convergence under various initialization and optimization settings. Since model optimization is primarily influenced by data initialization, learning rate, and the number of iter- ations and the learning rates used by existing methods in the field are largely consistent (Zaidi et al. 2022 Feng et al. 2023 Ni al. 2023). we validate the stability of our method by setting dif- ferent random seeds to vary data initialization and visualizing the loss at different iteration steps. As shown in Fig. 7 whether during pre-training the PCQM4Mv2 dataset supervised fine-tuning
22

---

Published as a conference paper at ICLR 2025
(a) Pre-training
(b) Fine-tuning QM9
Fine-tuning MD17
Figure 7: Loss curves of our method.
on downstream datasets such as QM9 and MD17, the loss of our method converges quickly to relatively low value. This provides strong evidence of the stability of our method.
A.13 MORE ABLATION STUDIES
we perform ablation studies on the number of Equivariant Graph Neural Network (EGNN) layers used by the Noise Generator in our DenoiseVAE. We find that when the number of layers is 4, model performs better on downstream tasks. This shows that appropriately increasing the depth of the model can learn better molecular representations, but at the same time, will bring about increase in computing resources.
As shown in Tab. 1
Moreover, we conduct an ablation study on the noise sampling strategies. Our method is designed focus solely learning the diagonal elements of the covariance matrix and perform independent noise sampling for each atom. Here, we extend the model to account for atomic relationships by learning the full covariance matrix, thereby capturing the dependencies between different atoms within the molecule when sampling the noises. Formally, we denote X E 1 where N is the number of atoms in the molecule X. We use the Noise Generator Gs model the noise distribution for each atom in a given molecule. That is: x(X,E)=
(31)
where N The noisy molecular conformation X; should be sampled from the correspond- ing generated distribution. Similarly, we achieve our sampling by applying reparameterization as follows, where e represents random noise sampled from the Standard Gaussian distribution.
X = X Ei.
(32)
As shown in Eq. 33 and Eq. 34 we compare the three noise sampling strategies in different methods. E,, and E,, denote the noise variance of the i-th atom and covariance between the i-th and the j-th atoms. 'Traditional method refers to adding pre-defined and constant noise to all atoms for each molecule. 'Ours (independent)' represents our Denoise VAE method, in which our model learns the noise distribution for each atom individually and performs independent noise sampling during the sampling phase. 'Ours (non-independent)) highlights an enhancement our Denoise VAE, where inter-atomic relationships incorporated by considering the off off-diagonal elements of the covari- ance matrix, Σi,,, rather than setting them directly to zero. fact, our method is a subcase this design.
2 Traditional method
E
Ours (independent)
(33)
Ours (non-independent)
0
Traditional method
0
Ours (independent)
(34)
o
Ours (non-independent)
Through the Denoising Module, we acquire the output and denote it as X Dx) The final denoising loss is shown follows, where X; and X; denote the i-th atom in the molecule X and
23

---

Published as a conference paper at ICLR 2025
output X, respectively. Σ,, represents the corresponding noise variance of i-th atom.
N
(X;
X.)
Denoise
E X
(35)
M. x)
Since directly optimizing the denoising loss will lead the model trivial solution, we introduce prior distribution to binding the learned distribution as follows:
LK = DEL (N(0, E)(px))
(36)
where px the pre-defined distribution. Then the final optimization target is:
Denoise VAB
Denoosee ACKL.
(37)
The experimental results are shown in Tab. 12 We observe that the experimental results under these two noise sampling settings are comparable. This may be attributed to the fact that our Noise Gen- erator is equivariant graph neural network, which takes into account and integrates information from neighboring atoms around each atom during the pre-training process. Although independent perturbations applied each atom during the final noise sampling stage, the scale of these per- turbations is, in fact, highly correlated with the surrounding atoms.
Table 11: Ablation studies on the number of EGNN layers of the Noise Generator.
QM9
MD17
сномо
ELUMO
Aspirin
Benzene
layer=
14.0
11.7
0.0559
0.1265
layer=2
15.6
12.0
0.0761
0.1589
Table 12: Ablation studies on the noise sampling strategy (abon--ieepeaaeerwoooonnndnnenn
QM9
MD17 Benzene Ethanol Malonaldehyde
a
ЕHОMO
ELUMO
Aspirin
Ours
0.0082
0.0639
14.5
12.2
0.0590
0.1389
0.0302
0.0989
(non-independent) Ours (independent)
0.0079
0.0650
14.2
11.9
0.0567
0.1366
0.0303
0.1012
A. DETAILS ABOUT EXPERIMENTAL SETTINGS
We present the details about the hyper- parameters of our experiments in Tab.
For training resources, all experiments are conducted on Intel(R) Xeon(R) Gold 5318Y CPU @ 10GHZ with single RTX A3090 GPU, Normally, 6 GPUs with total of 144GB of memory is sufficient for the reproduction.
Table 13: Hyper-parameter for Pre-training dataset.
Dataset
PCQM4Mv2
Batch size
128
Optimizer
Adam W 0.0005 Cosine
Max learning rate
Learning rate decay policy
Network archecture
Equivariant Graph Neural (EGNN)
Noise Generator layers
4
Denoising Module layers
24


Reviews:
[{"review_id": "XZJalC7pkV", "reviewer": "Reviewer_D6p5", "summary": "This paper introduces NoiseVAE, a novel molecular pre-training method, for learning atom-specific noise distributions in different molecules. The variational autoencoder (VAE) based model consists of a Noise Generator and a Denoising Module that are jointly trained. The pretraining objective is to learn atom-specific noise distributions that align better with quantum mechanical theories than empirically derived strategies. The results show that the pretraining DenoiseVAE can generate noises specific to atomistic environments and improve the prediction accuracy on molecular properties such as the HOMO-LUMO gap and potential energy.", "questions": "The authors are encouraged to provide more details concerning the above-mentioned weaknesses. For example,\n1. What is the significance of the PCQM4Mv2 dataset in the pretraining of the NoiseVAE model?\n   - How does the dataset help in learning atom-specific noise distributions?\n1. How are the noise distributions generated by DenoiseVAE used in the downstream molecular property prediction tasks?\n   - What is the architecture of the property predictor(s) used in the downstream tasks? What type of model is used?\n   - How are the noise distributions used in the property prediction tasks, in addition to the input 3D molecular geometry?\n1. What are the correct metrics/criteria for the results in Tables 1 and 2?\n\nAdditionally, the authors may consider addressing the following questions:\n1. The paper provides proof that the DenoiseVAE provides a higher theoretical evidence lower bound (ELBO) guarantee for the real conformation distribution of isoenergetic molecules. Is there a more direct way to show the correctness of the noise distributions generated by DenoiseVAE with the ground truth by first-principles calculations?\n   - For example, in Figure 4, the authors showed the generated noise distributions for two molecules. How close are these distributions when benchmarked against first-principles simulations?\n1. How does the DenoiseVAE model help with conformational sampling in quantum chemical simulations?\n   - The DenoiseVAE model generates noise distributions that are specific to a 3D molecular geometry (to the best of my understanding). By sampling from these noise distributions, are the sampled conformations more likely to be energetically favorable/stable than randomly sampling from a preset Gaussian distribution?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "### Originality\n*My expertise in related literature is limited.*\n1. The NoiseVAE model leverages a VAE for a more robust atom-specific and physics-related generation of noise distribution.\n\n### Quality\n1. The experiments/results are very comprehensive.\n1. The shown results on the downstream tasks suggest the robustness of the generated noise in providing quantum chemical information for accurately predicting molecular properties.\n\n### Clarity\n1. The pretraining objective and method for NoiseVAE are well introduced with details.\n\n### Significance\n1. Exploring the conformational space efficiently is of significance in quantum chemical simulations. NoiseVAE provides a way to efficiently sample conformations around a given geometry.", "weaknesses": "While I appreciate the amount of information and results presented in the paper, I find the experiments and results a bit difficult to follow.\n\n- The authors showed proof that the proposed DenoiseVAE provides a higher theoretical evidence lower bound (ELBO) guarantee for the real conformation distribution of isoenergetic molecules.\n\n- Subsequently, to show the effectiveness of the DenoiseVAE, the authors conducted downstream tasks with the pre-trained DenoiseVAE for predicting molecular properties.\n\n- However, the authors did not provide a clear explanation of how DenoiseVAE was used in the downstream molecular property prediction tasks. To the best of my understanding, the DenoiseVAE was used to generate noise distributions for the input molecules. However, how were these noise distributions leveraged was not clear to me.\n\n- The tables with results are also a bit misleading. For example, tables 1 and 2 mentioned \"force prediction\", but the 12 properties in Table 1 were not force-related and the values in Table 2 seemed more energy-related than force-related.\n\n- In addition, the paper lacks details on the training process of the NoiseVAE model. Specifically, the pretraining is performed with the PCQM4Mv2 dataset, but the paper only briefly mentions that the dataset has 3.4 million organic molecules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hB58253PT0", "reviewer": "Reviewer_Ho1Z", "summary": "The This work proposes a 3D molecule pretraining method with learnable atom-specific noise generation, as opposed to the common practice of hand-crafted and shared noise. The pre-trained representation shows higher performance than baselines in most of the downstream energy prediction tasks. Furthermore, the authors show the learned noise intensities correspond with the atomic reactivity and rigidity.", "questions": "Major:\nIt’s a bit unclear how the downstream predictive tasks are performed after pretraining. Specifically:\n\n(1) Is the force prediction achieved by a task-specific prediction head? If so, what is its architecture and how is it trained (e.g. end-to-end fine tuning)?\n\n(2) What are the features used for the prediction (e.g. pooled EGNN outputs or latent embedding)?\n\nMinor:\n(1) Following the previous questions, as the model is considered a VAE, how is the latent space obtained?\n\n(2) Is it possible to visualize the learned energy landscape (e.g. for MD17 samples)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This work addresses an important problem in molecular representation learning. The learnable and molecule-specific per-atom noise distribution is a reasonable setup and aligns better with the physical intuitions.\n2. The authors provide comprehensive theoretical analysis of the rationale of their method.\n3. The proposed method shows consistent improvement over the baselines in the majority of the tasks, while also being more parameter efficient.\n4. The method has good interpretability and the learned noise patterns are physically relevant.", "weaknesses": "The results are overall solid. However, some additional experimental and implementation details about the downstream tasks should be provided to help better understand and assess the results. See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DE2K0IBf0U", "reviewer": "Reviewer_NLii", "summary": "This paper presents DenoiseVAE for 3D molecular pre-training that adapts to the anisotropic nature of molecular force fields. The authors proposed Noise Generator, which learns molecule-specific noise distributions for each atom, allowing DenoiseVAE to generate conformations that align closely with a molecule's potential energy surface. The network architecture is a variational autoencoder, where noisy molecular conformations are denoised using a denoising module. The authors claim that this method leads to more accurate representations for downstream molecular tasks. Experiments on molecular property prediction tasks including QM9 and MD17 are conducted to demonstrate the effectiveness of the proposed method.", "questions": "1. How can different formulations of noise distributions affect the result? Currently, the noises are Gaussians with diagonal covariance, what if the covariance is non-zero at off-diagonal positions? E.g., the neighboring atoms connected by a bond will have non-zero covariance.\n\n2. How well will the model perform if the training and testing are on molecules of different sizes? Say the model is trained on small molecules but is tested for large molecular structures such as proteins.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of adaptive noise generation is novel. Previous works use hand-crafted or uniform noise across molecules, but DenoiseVAE uses an automated, atom-specific noise generation method with stochastic parameterization, which respects the unique structural and chemical properties of each molecule.\n\n2. The paper proposes theoretical derivations based on PES which aligns with quantum chemistry principles. The derivation for Theorem 1 based on ELBO offers theoretical insights into the effectiveness of the proposed method.\n\n3. Experimental results are promising and convincing: the proposed method achieved state-of-the-art performance across all benchmarks.", "weaknesses": "1. The use of DenoiseVAE introduces additional computational complexity. The training of the VAE with molecule-specific noise sampling could be computationally expensive or impossible for large datasets or complicated molecules. A more thorough analysis of training time and resource requirements could be more helpful.\n\n2. It is unclear to me how the proposed DenoiseVAE can be adapted across different dataset scales. It would be great if the authors could offer a series of studies on the performance of DenoiseVAE on different sizes of datasets.\n\n3. I can imagine the training of DenoiseVAE can be very unstable and will be sensitive to initialization and optimization settings. It would be great if the authors could show convergence of loss curves across all datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8wwupApXBI", "reviewer": "Reviewer_R3gr", "summary": "<Summary>\n\n - The authors propose a denoising-based 3D molecular pre-training method, called DenoiseVAE, which employs a learnable noise generation strategy instead of existing hand-crafted strategies. This allows for adaptive, atom-specific noise distributions across different molecules.\n\n - The authors introduce a variational approach to jointly optimize the Denoising Module and Noise Generator of DenoiseVAE. Here, the denoising objective encourages generated noise to adhere more closely to physical constraints, enabling the recovery of equilibrium conformations. Additionally, a KL divergence-based regularization term is applied to prevent low noise intensity and increase the diversity of sampled conformations.\n\n - Theoretically, the authors demonstrate that optimizing their pre-training objective is equivalent to maximizing the evidence lower bound (ELBO) of the log-likelihood.\n\n - Extensive experiments reveal that DenoiseVAE outperforms existing denoising methods across various datasets for both molecular and complex property predictions.", "questions": "<Question>\n\n - The prior distribution constrains the atom coordinate distribution to follow a predefined Gaussian form. But what is the range of these coordinate distributions? Are they confined within a narrow range close to the prior distribution, or are they spread across a finite range? If so, what is that range?\n\n - The article addresses 3D coordinate noise by adding Gaussian noise based on a learnable distribution, but what about rotations?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "<Strengths>\n\n - DenoiseVAE employs a learnable noise generation strategy that allows for adaptive, atom-specific noise distributions across different molecules.\n\n - The authors introduce a variational approach to jointly optimize the Denoising Module and Noise Generator of DenoiseVAE. Here, the denoising objective encourages generated noise to adhere more closely to physical constraints.\n\n - A KL divergence-based regularization term is applied to prevent low noise intensity and increase the diversity of sampled conformations.\n\n - Optimizing pre-training objective is proven to be equivalent to maximizing the evidence lower bound (ELBO) of the log-likelihood.\n\n - DenoiseVAE outperforms existing denoising methods across various datasets for both molecular and complex property predictions.", "weaknesses": "<Limitations>\n\n  - The Boltzmann distribution is commonly used in the classical regime. However, for precise coordinate computation of interacting atoms, a quantum approach is necessary. This involves, for example, treating the energy function as a quantized operator and finding energy eigenvalues for a given basis. However, this article does not carefully consider such approaches.\n\n - This approach essentially adopts a force field method, which may be inadequate for precise atomic-level computations.\n\n - Additionally, the energy is modeled as a simple harmonic potential, lacking any exchange correlation or other accurate potential forms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)