May the Force be with You: Unified Force-Centric Pre-Training for 3D Molecular Conformations
Authors: Rui Feng
Qi Zhu
Huan Tran
Binghong Chen
Aubrey Toland
Rampi Ramprasad
Chao Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: molecular pretraining
molecular representation learning
TL;DR: we propose a unified force-centric pre-training framework for 3D molecular conformations.

Abstract: Recent works have shown the promise of learning pre-trained models for 3D molecular representation.
However, existing pre-training models focus predominantly on equilibrium data and largely overlook off-equilibrium conformations.
It is challenging to extend these methods to off-equilibrium data because their training objective relies on assumptions of
conformations being the local energy minima. We address this gap by proposing a force-centric pretraining model for 3D molecular conformations covering both equilibrium and off-equilibrium data.
For off-equilibrium data, our model learns directly from their atomic forces. For equilibrium data, we introduce zero-force regularization and forced-based denoising techniques to approximate near-equilibrium forces.
We obtain a unified pre-trained model for 3D molecular representation with over 15 million diverse conformations. Experiments show that, with our pre-training objective, we increase forces accuracy by around 3 times compared to the un-pre-trained Equivariant Transformer model. By incorporating regularizations on equilibrium data, we solved the problem of unstable MD simulations in vanilla Equivariant Transformers, achieving state-of-the-art simulation performance with 2.45 times faster inference time than NequIP.  As a powerful molecular encoder, our pre-trained model achieves on-par performance with state-of-the-art property prediction tasks.

May the Force be with You: Unified Force-Centric Pre- Pre-Training for 3D Molecular Conformations
Rui Feng
Zhu
Computational Science and Engineering Georgia Institute Technology rfengegatech. edu
Computer Science Department University Illinois, Urbana- -haaaa-ha zhu. ckc@gmail com
Huan Tran
Binghong Chen
Materials Science and Engineering Georgia Institute of Technology huan. tranemse. gatech. edu
Computational Science and Engineering Georgia Institute of Technology
binghong@gatech. edu edu
Aubrey Toland
Rampi Ramprasad
Materials Science and Engineering Georgia Institute of Technology artoland@gatech. edu
Materials Science and Engineering Georgia Institute of Technology
rampi ramprasadmrass gatechh edu
Chao Zhang
Computational Science and Engineering
Georgia Institute of Technology
chaozhangggatech edu
Abstract
Recent works have shown the promise of learning pre-trained models for 3D molec- ular representation. However, existing pre-training models focus predominantly equilibrium data and largely overlook off eefiliiiuiiumm conformations. It is challeng- ing to extend these methods aff-equilibrium data because their training objective relies on assumptions of conformations being the local energy minima. We address this gap by proposing force-centric pretraining model for 3D molecular confor- mations covering both equilibrium and off-equilibrium data. For off-equilibrium data, our model learns directly from their atomic forces. For equilibrium data, we introduce zero-force regularization and forced- based denoising techniques to approximate near-equilibrium forces. We obtain unified pre-trained model for 3D molecular representation with over 15 million diverse conformations. Experiments show that, with our pre-training objective, we increase forces accuracy by around 3 times compared to the un un-pre- tre-trained Equivariant Transformer model. By incor- porating regularizations on equilibrium data, we solved the problem of unstable MD simulations in vanilla Equivariant Transformers, achieving state-of-the-ard simulation performance with 2.45 times faster inference time than NequlP. As a powerful molecular encoder, our pre-trained model achieves on on-par performance with state-of-the-an property prediction tasks.
Introduction
Representation learning for 3D molecules is erucial task in the field AI for science. Recent years have seen significant advancements in Graph Neural Networks [24, 19, 12] and Equivariant Graph Neural Networks architectures [30, 9, 1. 11, 23] capture interatomic features in 3D molccular 37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

conformations, thereby laying foundations for learning powerful representations. Leveraging such ar- chitectures, several existing works [33, 10. 17] have explored pre-training on 3D molecular structures on equílibrium data (PCQM4MV2 [16, 15]). Specifically, [33] developed de-noising approach that learns meaningful information about the equilibrium conformations and their surroundings. [17, 10] leveraged the potential energy of equilibrium conformations to derive supervised pretraining objec- tive. Such pre-trained models have achieved promising performance on diverse molecular property prediction tasks.
Nonetheless, these existing 3D molecule pre-training models are based on the equilibrium assumption and predominantly overlook off-equilibrium conformations. In this paper, we consider efuiiiiii conformations to be those whose atomic forces significantly deviate from zero. In this case, existing work cannot be indiscriminately extended to non-zero forces, because their modeling assumption fundamentally tied to equilibrium conformations being the local energy minima. The de-noising technique in [33], which is theoretically equivalent to learning forces around equilibrium conforma- tions, may potentially be ill ill-posed for off-equilibrium conformations. Similarly, the energy- supervised pretraining objectives in [17, 10] are based on the local minimality of these conformations and cannot be straightforwardly extended to off-equilibrumm data. To date, the pre-training of a representation model for 3D molecular conformations that encompasses unifying both equilibrium and off offlequilirrium molecular conformations remains largely underexplored. On the other hand, off-equil off-equilibrium conformations represent significant portion of the chemical space, which is crucial for comprehensive modeling of the molecular representation space. For instance, applications such as molecular dynamics (MD) simulations are heavily dependent on the model's ability to accurately represent off-equilibrium data.
In this paper, we incorporate both off- effilibiiibr and equilibrium conformations into unified representation learner. We propose a new pre-training model, "Equivariant Transformer Optimization and Representations for Equilibrium and Off-equilibrium Molecules" (ET-OREO) for 3D molecular conformations. This model integrates both equilibrium and off-equilibrium data from multiple large-scale datasets. Its training objective hinges on atomic forces, defined as the negative gradient of a molecule's potential energy with respect to atomic coordinates. Atomic forces exhibit several notable properties:(1) they are physically well-defined observable, i.e., the force acting on atom is determined solely and uniquely from its local environment, defined as the real-space distribution of its neighboring atoms; (2) they are generalizable across various molecules in the sense that atoms from different molecules that have the same local environment should experience the same atomic forces; and (3) they can unify equilibrium and off-equilibrium data, equilibrium data can be conceptualized as local minima in the latent (configuration) space with zero forces, while off-equilibrium data aid the model in more accurately characterizing the high-energy chemical space beyond equilibrium. Among these points, (2), which is the direct consequence of (1), makes atomic forces fundamentally different from potential energy which is defined for the whole system under consideration (molecules) and can only be determined up to an additional constant. Therefore. predictive model of atomic forces transferable, i.c., in principle, can be used directly for molecules of any size (number of atoms). This advantage makes the "learning atomic forces" fundamentally different from the traditional approach in which fictitious concept of "atomic energy" must be defined, predicted, and combined to obtain the total potential energy of the whole system [2].
Inspired by this, we develop our force-centric pretraining model that trains on both off-equilibrium and equilibrium molecular conformations. For off-equilibrium data, our model learns directly from their atomic forces, aligning the model gradient with respect input coordinates with atomic forces. A conceived obstacle in leveraging forces lies in data acquisition: high-accuracy forces require the application of ab-initio methods such as Density Functional Theory (DFT) [14, 20]. However, we argue that DFT, despite its time complexity of O(N*), remains tractable for moderately sized molccules [31]. Additionally, the major computational cost overlap with the potential energy, which is common in existing 3D conformation datasets.
For equilibrium data, we impose zero-force regularizations, reflecting their status as local minima of the potential energy surface. We also introduce random noise into equilibrium conformations and consider the random noise directions as approximate forces on the perturbed conformation coordinates. This allows the model to "denoise" the perturbed conformations by gradients. Our approach draws inspiration from the recent success of denoising techniques in molecular learning [33, 13], energy- based supervised pretraining [10, 17], and wore-matching generative models [27, 28, 25]. We integrate our gradient-bas based denoising and zero-force regularization on equilibrium conformations

---

with off eefieiiuiiiium force optimization, thereby providing the model with a unified landscape of molecular structures and the potential energy surface.
For model pre-training, we collated more than 15 million 31) molecular conformations. Our pre- training data leverages three open-domain datasets, including PCQM4My2 [16, 15], MD17 [6, 5], ANII-x [26]. Moreover, we contribute a new dataset, poly24, consisting of simulation trajectories of diverse family of polymers. Pretrained on these diverse sources of equilibrium and off-equilibrium data, our model attained state-of-the- achieving efficient and accurate simulations terms of distributional properties and accuracy relative to DFT calculations. Our model also serves as a potent representation learner for equilibrium data, attaining performance on par with state-of -ete-----h-errrrr molecular learning methods that focus exclusively on equilibrium data.
simulation performance on both MD17 and polymers,
In summary, contributions are as follows:
We introduce novel force-centric molecular conformation pretraining paradigm that trains a unified conformation representation encompassing both equilibrium and off-equilibrium molecules. Our paradigm enables the representation learner portray not only the equilib- rium conformation space but also the extensive off-equilibrium spaces between them.
Our model achieves highly accurate molecular dynamics (MD) simulations, by efficiently fine-tuning its parameters for with molecules and polymers. This allows for fast and reliable MD simulation, as our model is able to accurately replicate the ab initio forces and distributional properties of conformations. Furthermore, our model has demonstrated comparable performance to state-of-the models that are solely focused on equilibrium data in molecular property prediction.
We provide the community with diverse set of DFT simulation data comprising varied set of polymers, which are valuable not only for studying polymer properties such as ring-opening enthalpies but also for the general modeling of molecular forces.
2
Related Work
Machine Learning Forcefield In recent years, there has been a surge in the development of deep learning models for molecule forcefields. Two prominent groups of models have emerged: geometric message passing models [24, 19, 12] and the Tensor Field Network (TFN) family [30, 9, 1, 11]. Geometric message-passing models [24, 19, 12] utilize traditional graph neural networks or message- passing networks on pairwise radial features that are translational and rotational invariant. On the other hand, the TFN family models learn SEu()-equivariant features by leveraging harmonic functions and the irreducible representations of the SE(3) group. NequlP [1] is the state-of-the-art model in this family and has achieved high stability and fidelity on the MD17 dataset.
Along another line, EGNN [23] learns equivariant features by integrating directional distance vectors between atom pairs in their implicit 3-dimensional coordinates. This approach allows for the learning of equivariant features without incurring the computational cost of the TFN family. Our method's backbone model is MDrchMDNet [29], which we view embeds the implicit 3-dimensional coordinates in a latent high- high-dimensional space, enhancing the model's sss capacity to represent 3-dimensional equivariant features. In addition, TorchMDNet leverages the concept of continuous distance filters from SchNet [24] to further increase its representation power.
extension of EGNN. TorchMDNet
Molecular Pretraining Inspired by the success of pre-training foundation models in the fields of NLP and CV, there have been several attempts to pretraining for 3D molecular structures [33, 10, 17]. Specifically, the NoisyNode [33] work adopted the denoising regularization technique of graph neural networks [13] as a pretraining method for equilibrium conformations, achieving -fate-of-the-art performance on molecular property predictions. [10] and [17] both base their pre-training on the supervised energy data from equilibrium conformations with forces regularization. However, as aforementioned, these existing methods predominantly rely equilibrium data and cannot be easily extended to off-equilibrium data. While achieving high performance for property prediction tasks, they leave the vast chemical space of offfequilibrium conformations unexplored.
3

---

porce-baset training over offlequilibrimm molecular conformations has been explored in literature. [6, 5, 7, 4, 3] all focus on learning atomic forces and predict energy based on numeric integration. based on the claims that the potential energy and forces have different noise distribution pattern(6, $] and forces being local and immediate atomic features [4, 3]. Their methods still focus on training on off-equilibrium simulation data from single molecules. In supplementary materials, we showed that the joint optimization of potential energy and forces is problematic for multimoolecule settings. We instead propose force-centered and energy-free objectives to learn unified pretraining model for both off-equilibrium and equilibrium data; and we show that force-centered pre-training improves multi- optimization and generalization.
3 Methodology
3.1 Problem Setup
Consider a molecule type x that can exist in various 3D) conformations. Let n, represent the number of atoms in molecule x. The distribution of the molecule's conformations is represented as ry RR and its atom types are represented as 2, E 2"x The potential energy of 3D molecule is determined by its conformation ~ r, and atom types z, denoted E = E(r,z) E R. The forces applied to the atoms, which are defined as the negative gradient of the potential energy with respect to atomic coordinates, are given by F = dE I_ In theory, a stable conformation is one where the potential energy achieves a local minimum with zero forces. In molecular dynamies (MD) simulations, the molecular conformations are moved according to forces and the thermostat of the simulation.
Our goal is learn equivariant model (r, parameterized by 0 that learns molecular represen- tations from both equilibrium and off-equilibrium conformations. In this paper, Φo needs to predict mergyyconseevvttvv atomic forces. To achieve this, we follow the standard machine learning forcefield paradigm. For simplicity, we omit z in E and F, and we ignore the molecule type index x when referring to general molecules. We use c; (R**3) x - to output the potential energy and take - - as forces. To distinguish between equilibrium and off-equilibrium conformations, we define the equilibrium of conformations as & == (r F(r) 0} and the off-equilibrium set _ as the complementary set where the forces are non-zero.
3.2 Joint Training of Forces on Equilibrium and Off-equilibrium Data
Our methodology is centered on forces optimization. For off-equilibrium conformations x J with non-zero forces, we directly optimize on the atomic forces by minimizing || V ( For equilibrium molecular conformations X 6, assume their forces to be zero and impose zero-force regularization: || - v (r.) However, this objective gives the model little knowledge of the conformation structures in the neighborhood of ry. To better inform the model of the forcefield around equilibrium conformations, we further use a de-noising equilibrium regularization, where we add Gaussian noise equilibrium conformations and use the noise direction as noisy forces:
(0.aa V D(r, E) -E12]
X~
(1)
The rationale of the above siice-guieed& denoising objective is as follows. The perturbed confor- mation, denoted as r, E, could either approximate high-energy, off-equilibrium confirmation or, alternatively, represent an unphysical state. In the former scenario, the conformation is anticipated to relax back to the proximate local minimum, т,, thereby yielding forces that are consistent with the direction E. Conversely, in the latter situation, the learning model is still sufficiently equipped to maintain robust representations, thereby enabling return to a stable conformation from any unphysical deviations. More formally, (I) can be interpreted as learning approximate forcefield around equilibrium conformations, will be discussed in Section 3.3.
Combining forces optimization on eufiibbiiii data and zero-force regularization and de-noising objective on cquilibrium data, we have the unified force-centric pre-training objective
Exres zeroo zero-forre force regularization
(r.))
+ +E | (r, e1
E [F V,, ((,
forces optimization
de de-noising equilibrium
(2)

---

where the first expectation over x & samples equilibrium conformations and imposes zero-force regularization and the denoising objectives on the atomic coordinates. The expectation over y samples off-equilibrium conformations and optimizes the model gradient with forces.
3.3 On the De-noising Equilibrium Objective
Objective function (I) can be viewed as learning an approximate forcefield around equilibrium data. In fact, by dell-known results in de- de-noising score-matching [32, 27, 28], the objective
c || - -e/o² -
(3)
is equivalent to the score-matchin wore-matching-like like objective
Byo(t) -VV (f)
go (o(r)) E
(4)
which by [33] is equivalent to learning implicit forces around r, assuming r is an local minimizer of the energy. Hence, (1) is equivalent to learning forces around equilibrium conformations up to scaling constant. This proof, however, cannot be naïvely extended to molecules whose forces are non-zero, which the driving motivation for us to supplement the de-noising objective with forces from off-equilibrium conformations.
While [33] used the same argument for their de-noising objective, our implementation is different. They used a prediction head for predicting the noises on top of an Equivariant Transformer, while we directly predicted the noise with forces predicted by the model gradient. Our design has several unique advantages: 1) Forces principally govern atomic movements. By formulating the de-noising objective with predicted forces, our model a physical interpretation where perturbed conformations, be high-energy off-equilibrium conformations and physical conformations. This property especially helpful in MD simulations, as will be shown in Section 4. 2) We can unify the prediction of the forces for both equilibrium and off-equilibrium data, providing the model with a consistent energy landscape across conformations of different molecules and states.
unphysical conformations, could relax back to stable
3.4 Model Pre- Pre-Training and Fine-Tuning
Model Architecture. We use TorchMDNet [29] (also known as Equivariant Transformer) as our molecule encoder 4b. following [33]. TorchMDNet is one of the best best-performing models in terms of predicting molecular properties and atomic forcefields. While [1] achieves higher accuracy in molecular prediction tasks, find TorchMDNet more favorable for pre-training due to its expressivity and better computational efficiency.
Pre-training. Our model has 8 layers and 256 embedding sizes, 8 attention heads, and radial basis function (RBF) dimension of 64, consistent with [29] and [33]'s best-performing model. The model parameters optimized with the Ad2mW[22, 18] optimizer with a peak learning rate set as le 4. The learning rate is scheduled with 10,000 warmup steps that gradually increase the learning rate from 1e- le- 4, and afterward will decrease with multiplier of 0.8 after 30 validation epochs of non-decreasing validation loss. On every 500 training steps, one validation epoch is performed. The model is trained with batch size of 32 samples for 3 epochs for 468,750 gradient steps. For the denoising objective, the variance of noise o is set to be 0.04.
Model Fine-tuning. The pre-training stage provides the model with good initial representation of molecules in general but is not necessarily optimal for each specific dataset or task. For our experiments, the model further fine-tuned on target dataset to optimize task-specific performance. For ET-OREO, the training/testing split of overlapping datasets in both the pre-training and fine- tuning stages are consistent so that during fine-tuning, the test set will not include any data seen during training.
3.5 Pre-Training Data
Our paper focuses primarily on the structures of organic molecules and polymers vacuum. This approach enables to maintain consistent learning of quantum mechanical interactions between
5

---

atoms within a uniform environment. For this purpose, we use three public 3D molecular conformation datasets MD17, ANII-x, and PCQM4Mv2; and we also create new polymer simulation dataset, detailed as follows.
Poly24: MD Simulations for Polymers. We contribute poly24, DFT-based MD simulation dataset for polymers. We generated DFT simulation data for 24 types of polymer families, broadly categorized into cycloalkanes, lactones, ethers, and others. Each polymer family consists a cyclic monomer and its iin--ppenin polymerizations. Within ring-opening polymerization, ring of the cyclic monomers is broken and then the "opened" monomer is added to a long chain, forming a polymer chain. The details data generation be found in the Appendix. In total, run generally 10 DFT simulations for different initialization of each 1.-lop (I. 1,3,4,, 4,5,6) 6) polymer across the 24 types of polymers. We have 1311 DFT trajectories and 6,552,624 molecular conformations. Only polymers with less than or equal to 64 atoms are used for pre-training, totaling 3,851,540 conformations. The remaining larger-polymer conformations are used for fine-tuning and testing.
MD17, ANII-x, and PCQM4Mv2.
Dataset # Conformations
Equilibrium
Off-equilibrium
PCQM4Mv2 ANIIx
3,378,606
addition to our own poly24, we have uti- lized three existing public datasets, namely MD1716, 5], AN11-x(26), and PCQM4Mv2[16, 15] for our model pre-tree pre-training. These datasets contain small organic molecules in a vacuum, and property prediction for such molecules is area of great interest to the cheminformatics community. The machine learning for molecules community has also extensively studied and benchmarked these datasets. Table summarizes all the dataset used in the pre-training stage. In total, have more than 15 million samples from MD17, ANII-x, PCQM4Mv2, and Poly24, covering equilibrium and off-equilibrium conformations for diverse organic molecules.
4,956,005
MD17
3,611,115
poly24
3,851,540
x
Total
15,718,279
Table 1: Datasets used in our model pre- pre-training process.
Experiments
4.1 MD Simulations for Small Molecules on MD17
Setup After pre-training our model ET-OREO, we further finetune on the MD17 dataset to validate the performance of molecular simulations using the forces predicted by our fine-tuned model. We run simulations with our model's 's predicted forces with a Nose-Hoover thermostat, with 500K temperature and 0.5fs timestep for 600k steps. The simulation setting is the same [8], which benchmarked popular deep learning forcefields for MD simulations. We use the ASE package for the implementation of molecular simulation environment [21] with characteristic time for the Nose-Hoover thermostat set to 25fs.
Baselines and Metrics.
Following [8], we report metrics:
Forces MAE, which is the mean absolute enror of forces on the DFT trajectories;
Stability, which measures how long the simulation can run before blowing up. According to [8], this is detected when the radial distance function (RDF) deviates from the reference simulation by threshold (0. (0A).
h(r) MAE, with (((r) representing the distribution of interatomic distances during simulation. According [8], MAE here is calculated as the /l-norm between the reference distribution and the predicted distribution.
We compare ET-OREO with the best cest-performing models reported by [8]. Furthermore, ET-OREO is compared with TorchMDNet [29] trained MD17 from scratch, and ET-ORE, which is an ablation version of ET-OREO without the zero-force and de-noising regularizations. Consistent with previous
DDatsset will be made available upon publication.

---

Molecule
Metric
DimeNet
GemNer-T
GemNet-d1
NequiP
TorchMDDNe
ET-ORO
KTFORFO
Aspirin
Force (1)
1O.0
3.3
5.1
2.3
7.4
4.2
1.0
Stability (1)
14
72:50)
192)
300,;;
102,a5:
94(421
300(o)
h(r) (L)
0.04
(.04)
(.04
(.02
0.04
(.04
0.0240.00)
Ethanol
Force
4.2
2.1
1.7
1.3
5.6
3.1
1.0
Stability
26:10)
169)
g00
500
121
300ga)
300
b(r)
0.15
0. (.000)))))
(.09
0.08 (o.00).00)
0.12
0. (00000
0.03
Naphthalene
Force
5.7
1.5
1.9
1.1
3.3
2.0
0,9
Stability
(5,68)
8())
25((0)
300,0;
50:20)
25(y)
300
h(r)
0.10
(.0000
0.12
0.12
0.12;0.00
g0.00)
0.03x0.00
Salicylic Acid
Force
9.6
4.0
4.0
1.6
4.7
2,5
0.9
Stability
(3
26:24)
94 (100
500
60) (o)
(8)
500
h(r)
0.06
0.08
0.07 0.05)
0.03
0.06
Q.05
(.02
Table 2: Simulation results on MD17. For all results, force MAE is reported in the unit of [meV/AS
and stability is reported in the unit of [ps]. The distribution of interatomic distances h(r) MAE is
unitless. FPS stands for frames per second. For all metrics (L) indicates the lower the better, and (f)
indicates the higher the better. The first group of methods is taken from [8]. The second group of methods is our new baselines, including TorchMDNet [8]. ET-ORO, and ET-OREO. These models share the same architecture and have the same FPS.
Aspirin
Ethanol
Naphthalene
Salicylic Acid
0.8
Reference Prediction
1.5
Referwnce Prodiction
Beference Prodiction
Bee Refveence Prediction
Denety
3.22
0.5
hi::: 0.02
0.5
1.0
P6::: 0.09
Denaty
0.4
0.6
ht))
0.08
Density
0.6
0.2
0.4
htrh 0.02
0.0
0.0
2.5
5.0
7.5
10.0
10.0
0.0
25
5.0
7.5
10.5
2.0
5,5
10.0
Interatomic Distanoes (/4
Interatomic Distances ((A)
Interatomic Distances
(
Interatomic Distances (A)
(a) Interatomic distance distribution during simulations.
-2.53
-2.5
-0.63
..83
Eneegg (mee))
15.83
Enegy (med)
1,.00
Emerge meeg
9..9
6.69
6.63
-3.03
-0.33
-0.03
0.83
302
(ps]
Time
Time [pai
(b) Potential energy curve during simulations.
Figure 1: Illustration of simulation statistics for ET-OREO on MD17 dataset. Figure (a): distributions of interatomic distances during reference simulation and simulation with ET-OREO predictions finctuned on MD17. Figure (b): the curve of potential energy predicted by ET-OREO during simulation.
simulation benchmark [8], TorchMDNet, ET-ORE, and ET-OREO are fine-tuned with 9,500 confor- mations for training and 500 for validation. The training and validation conformations are sampled from the same training data used in the pre- pre-training of ET-OREO, and the metrics are reported on the rest of MD17 data unseen to both the fine-tuming and the pre- pre-training of ET-OREO and ET-ORE. Furthermore, in our implementation, we focus solely on forces optimization for TorchMDNet. We show in Appendix that this improves forces accuracy without sacrificing the model's ability to predict the potential energy.
Results. In Table 2, we compare the performance of ET-OREO with baseline models. We make the following key observations: (1) On all of the simulations, we have achieved a lower interatomic distance distribution h(r) than the previous perforreform model NequlP reported by [8]. Particu- larly, we reduced l((r) for aspirin and salicylic acid by half, 10% for ethanol, and 18% for naphthalene. While NequlP [1] can achieve stable and accurate MD simulations by training from scratch on MD17, it suffers from lower FPS as it computationally expensive. In contrast, ET-OREO can achieve both fast and high-quality MD simulations. (2) ET-OREO improves forces accuracy over TorchMDNet by over three times, achieving state-of-the-art MAE on all four tested molecules. The major differ- ence between ET-OREO and TorchMDNet is that ET-OREO is pre-trained with our force-centric objective before fine-tuning on MD17 data. Such significant performance gains show the benefits of pre-traiie mre-training over diverse equilibrium and off-equilibrium conformations. (3) ET-OREO produces
7

---

stable simulations and accurate interatomic distance distributions. ET-OREO is able to stably run simulations on all four molecules with accurate interatomic distance distribution h:(r) compared to
the reference trajectory in MD17. Figure visualizes the close approximation of ET-OREO predicted h(()) compared with reference data, and that ET-OREO can produce energy-conserving simulations that sample around the equilibrium.
Regularization Equilibrium Conformations is Vital for Simulations. We found that training TorchMDNet itself on MD17 cannot produce stable MD simulations. Meanwhile, the performance of ET-ORE shows that pre-training with forces only helped with higher forces accuracy. However, ET-ORE still does not perform well for MD17 simulations. Except for ethanol, ET-ORE cannot produce stable MD simulations, despite consistently improved forces accuracy. This in line with [8]'s observation that higher forces accuracy does not always guarantee simulation performance. The difference between ET-ORE and ET-OREO is that the latter incorporates stable zero-force conforma- tions and a de-noising regularization objective. The additional regularization on the conformation space proves vital stable and accurate simulations.
ET-OREO achieves accurate MD simulation with 3 times inference speed. From Table 2, NequlP reported similar simulation performance to ET-OREO without need for pre-training. However, NequlP has a significantly slower inference time. The NequlP model in Table 2 has an inference time of approximately 119ms per step on NVIDIA V100 with 1.05 million model parameters. [8] In comparison, ET-OREO has an inference time of 48.5ms on the same hardware with 6,87 million parameters. Hence, have achieved state-of-the-arr simulation performance with 2.45 faster inference speed.
4.2 Simulation on Large-loop Polymers
The vast majority of our model's training data is small molecules. In this section, we investigate the model's s ability to generalize larger molecules, consisting of 15-loop polymers unseen in the training data.
Setup We finetune model on the forces of small polymers (5-loop less) in the poly24 dataset for one epoch with a learning rate 0.0001. For testing the model's MD simulation performance on large polymers, on them with TTT-OREO-fine-tee for these 15-loop polymers are, respectively: 360, 360, 180, and 240. In the training data, we have most molecules consisting of Rs 1000 atoms. Therefore, ET-OREO is required to perform accurate MD simulations unseen polymers, testing both its simulation and generalization ability. SMILES
take the larger 15-loop polymers from the poly24 dataset and run MD simulations forcefields for a maximum of 600K steps. The number of atoms
ID
Polymer Name
Monomer Polymer
CK
cyclooctane
360
McCCc000C]1
OTH
n-alkane substituted 8- 8-valcrolactone
360
[10C{CCC))CCC(())+
LACI
y butyoollacoone
180
(*)0CCOCC(())=0
LAC2
3-methyl- -.dix----l--
240
[*0CCCCC())C())000
Table 3: Details of 15-loop polymers we used for simulations, including their ID, polymer name, number of atoms for the 15-loop polymer, SMILES string, and rdkit visualizations.
Results. Figure 2 (a) visualize the comparison results between ET-OREO predicted forees and DFT reference data. For all 15 loop polymers, ET-OREO obtains highly accurate forces to DFT calculations, with 0.01 eV/ MAE energy and close to zero cosine distance. Hence, ET-OREO can perform MD simulations references. The test 15-loop polymers contain up to 360 atoms, unseen the training data. This shows that ET-OREO in practice, ET-OREO only needs training data from small polymers, which are cheap to generate ab initio data and thus greatly reduce the cost of DFT simulation.
multiple types of polymers with uniformly high correlation with DFT
extrapolate well to large unseen polymers with known monomers. Furthermore,
Figure (b) shows the potential energy curves during simulation. The potential energy curves indicate that the polymer in the simulation converges stable near-equilibrium distribution quickly after

---

initial fluctuations. Furthermore, ET-OREO can explore different possible conformation states of the polymers, as illustrated in Figure (c).
CK
OTH
LAC1
LAC2
Predicted (eViA) 1.
Predicted Force (e41A)
Prodicted Force n
Predicted
-2
Costa: Vi
MAE: 8 3-4meVIA CosD::
Cost: bLAnDeVIA
CoSt:
5.0-2.5 0.0 2.5 5.0
2.5 DFT Torce (eWIA)
DFT Force (eWA))
DFT Force (eV/A)
DFT Force (eVAA)
(a) Forces correlation of ET-OREO to DFT forces during simulations.
25.0
(ev)
33.00
Encrgy [ev]
Energy [ev]
24.0
Energy tev]
35.0
.0.0
.27.5
26.0
40.0
30.0
45.0
-45.0
33.5
200
(b) Potential energy during ET-OREO simulations.
(c) Sampling of conformation spaces during ET-OREO simulations.
Figure 2: Accuracy and Robustness of ET-OREO in MD simulations for large polymers. In Figure (a) it is observed that ET-OREO forces highly correlated with DFT forces during its own MD simulations. Figure (b) shows the RMSD and potential energy curves of ET-OREO during MD simulations. The curves suggest that ET-OREO simulations experience equilibration. Figure (c) shows the visualization of sample conformations captured at various time points during the simulation.
4.3 Property Prediction QM9
To test ET-OREO's ability to encode equilibrium conformation for property prediction tasks, we follow the same experiment setting as NoisyNode [33] on QM9. [33] exclusively pre-trained on cquilibrium conformations from PCQM4Mv2 with de-noising objective. In comparison, ET-OREO has access to larger off off-equilibrium data supervised by forces. We follow the exact same fine-tuning setup as [33] save the de- de-noising regularization in the fine-tuning stage. In Table we compare ET-OREO with NoisyNode and TorchMDNet [29] HOMO-LUMO properties prediction on QM9. We report the performance of NoisyNode trained on the TorchMDNet encoder reported in [33], hence all three models have the same encoders, except that MDNet trains from scratch, and NoisyNode and ET-OREO fine-tune from their pre-trained parameters. ET-OREO improves the performance of property prediction by ~300 compared TorchMDNet trained from scratch, implying that our pre-training paradigm provided the model with useful information about the quantum mechanical properties of equilibrium conformations. Our performance is on par with NoisyNode, and unlike in simulation experiments, we observe no accuracy improvement by the inclusion of the rich off- equilibrium conformation. We speculate that such equilibrium property prediction tasks benefit little from additional information off- aff-equilibrium conformation. We want to particularly point out that, despite the pre-training effort, NoisyNode and ET-OREO both require extensive fine-tuning with large amount of data (more than 10,000) and training epochs (roughly 150 epochs for convergence) for optimal performance. An important future improvement in molecular pre- pre-training should in improving the data and time efficiency of fine-tuning.
TorchMDNet
NoisyNode
ET-OREO
ΕНОМО
20.3
15.6
16.8
ELUMO Ae
17.5
13.2
14.5
36.1
24.5
26.4
Table 4: Fine-tuning on HOMO-LUMO properties on QM9. Metrics are MAE in meV.

---

Discussion
We presented ET-OREO, a pre-trained model for 3D molecules on =15M 3D conformations of both equilibrium and off- off-eqqulibrium states. We unified the pre-training of molecular conformations of different sources and states with force-centric pre-training objective. With our pre-training model, we achieved offi-oof-thearrt performance on MD simulations, with both high force accuracy and simulation efficiency. As a potent encoder for conformations, our model also attained a state-of-the-art level of performance property prediction on equilibrium data.
Our current model limited mostly to single and small molecules in a vacuum. leaving more complex molecular environments as future work. We also deem it a promising direction to explore more model architectures and larger model sizes. Furthermore, our de-noising objective on equilibrium data does not leverage the model's is learned information on atomic forces. It will be interesting to study the possibility of more sophisticated coupling of equilibrium and off-equilibrium optimization, make the model progressively leverage its quantum mechanical knowledge for equilibrium data.
References
[1] Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):1-
11, 2022.
[2] Jörg Behler and Michele Parrinello. Generalized neural-network representation of high- dimensional potential-energy surfaces. Physical review letters, 98(14):146401, 2007.
[3] Venkatesh Botu, Rohit Batra, James Chapman, and Rampi Ramprasad. Machine learning force fields: construction, validation, and outlook. The Journal of Physical Chemistry C, 121(1):511-522 2017.
[4] Venkatesh Botu and Rampi Ramprasad. Learning scheme predict atomic forces and accelerate materials simulations. Physical Review B, 92(():094306,, 2015.
[5] Stefan Chmiela, Huziel E Sauceda, Klaus-Robert Müller, and Alexandre Tkatchenko. Towards exact molecular dynamics simulations with machine-learned force fields. Nature communica- tions, 91):38877 2018.
[6] Stefan Chmiela, Alexandre Tkatchenko, Huziel Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, 3(5):e1603015, 2017.
[7] Stefan Chmiela, Valentin Vassilev-Galinde Galindo, Oliver T Unke, Adil Kabylda, Huziel E Sauceda, Alexandre Tkatchenko, and Klbes-Robert Müller. Accurate global machine learning force fields for molecules with hundreds of atoms. arXiv preprint arXiv:2209. 14865, 2022.
[8] Xiang Fu, Zhenghao Wu, Wujie Wang, Tian Xie, Sinan Keten, Rafacl and Tommi Jaakkola. Forces are not enough: Benchmark and critical evaluation for machine learning force fields with molecular simulations. arXiv preprint arXiv:2210.07237, 2022.
[9] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se(3)-transformers: 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 13:1970-1981, 2020.
[10] Xiang Gao, Weihao Gao, Wenzhi Xiao, Zhirui Wang, Chong Wang, and Liang Xiang. Su- pervised pretraining for molecular foree fields and properties prediction. arXiv preprint 1rX1v:2211.14429, 2022.
[11] Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790-6802, 2011.
[12] Johannes Gasteiger, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123, 2020.
[13] Jonathan Godwin, Michael Schaarschmidt, Alexander Gaunt, Alvaro Rubanova, Petar Veličković, James Kirkpatrick, and Peter Battaglia. Simple gnn regularisation for 3d molecular property prediction & beyond. arXiv preprint ArXiv:2106.07971 2021.
Yulia
10

---

[14] P. Hohenberg and W Kohn. Inhomogeneous electron gas. Phys. Rev. 136:B864-B87 1964.
[15]
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb- Isc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09433 2021.
[16] Weihua Hu, Matthias Fey, Marinka Zitiik Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs
Advances in neural information processing systems, 33:22118 12-213332 2020.
[17] Rui Jiao, Jiaqi Han, Wenbing Huang, Yu Rong, and Yang Liu, Energy-motivann equivariant pretraining for 3d molecular graphs. arXiv preprint arXiv:2207.088222 2022.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprini arXiv:1412.6950, 2014.
[19] Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint PrXii:2003.03122 2020.
[20] W. Kohn and L Sham. Self-consistent equations including exchange and correlation effects. Phys. Rev., 140:A1133-A1 138, 1965.
[21] Ask Hjorth Larsen, Jens Jørgen Mortensen, Jakob Blomqvist, Ivano E Castelli, Rune Christensen, Marcin Dulak, Jesper Friss, Michael N Groves, Bjork Hammer, Cory Hargus, al. The atomic simulation environment- python library for working with atoms. Journal of Physics: Condensed Matter, 29(27):273022. 2017.
[22] llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v 101, 2017.
[23] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. (n) equivariant graph neural networks. In International conference on machine learning, pages 9323 9332. PMLR, 2021.
[24] Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241722, 2018.
[25] Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In International Conference Machine Learning, pages 9558-995 55689558.
PMLR, 2021.
[26] Justin S Smith, Roman Zubatyuk, Benjamin Nebgen, Nicholas Lubbers, Kipton Barros, Adrian Roitberg, Olexandr Isayev, and Sergei Tretiak. The ani-Iccx and ani-Ix data sets, coupled-cluste and density functional theory properties for molecules. Scientific data, 7(1):134, 2020.
[27] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[28] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and
Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13466, 2020.
[29] Philipp Thölke and Gianni De Fabritis. Torchmd-net. Equivariant transformers for neural network based molecular potentials. arXiv preprint arXiv:2202.00 02541, 2022.
[30] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv::8022. 082 19, 2018.
[31] Oliver T Unke, Stefan Chmiela, Huziel E Sauceda, Michael Gastegger, Igor Poltavsky, Kristof Schütt, Alexandre Tkatchenko, and Klaus-Robert Müller. Machine learning force fields. Chemi- cal Reviews, 121(16):10142--0188 2021.
[32] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23( 2011.
[33] Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206.00/33 2022.


Reviews:
[{"review_id": "qQggqb4lFC", "reviewer": "Reviewer_EQHq", "summary": "The paper propose a set of additional objectives for training machine learning interatomic potentials, in particular for equilibrum conformations as well as a denoising term. They show that this improves stability, when pretrained on a large dataset. They also introduce a new dataset of polymers.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The improvement from ET-ORE to ET-OREO is interesting and surprising. It doesn't really make sense that equilibrum forces should help The polymer data set is great!", "weaknesses": "Probably the biggest weakness is that the authors chose a weak baseline model with TorchMDNet. The entire paper hinges on work from Fu et al. that showed that in their experiments all methods except the NequIP potential were unstable in MD simulation. With many additional tricks and an enormous amount of data they now match NequIP. But will this entire improvement just go away when you apply the ideas to a stronger baseline model like NequIP? I would strongly suggest the authors to repeat the experiments with a better baseline and see if there are still improvements. Otherwise why wouldn't anyone just use NequIP to begin with? The speedup is small and as explained below mostly an artefact of hyperparameters.\n\nFinally, a central weakness it that ET-OREO is pretrained on a massive dataset while the baseline methods that are stable can do this from a much much much smaller dataset. \n\nAnother big weakness is that this unexplained improvement from ET-ORE to ET-OREO simultaneously adds the equilibrum and the denoising objective. Why no ablation study to understand which of them is the important one?\n\nThe inference speed comparison to NequIP is not thorough. NequIP has shallow pareto curves on accuracy vs efficiency which is well known, i.e. a L=2 models gets almost exactly the same accuracy as L=3, but is much faster (see original NequIP paper). The paper compares to a NequIP model that is solely optimized for accuracy, not for efficienc. This would be helpful to compare to. In addition, NequIP modern versions of NequIP now exists that are faster, e.g. Allegro or MACE. \n\nThe presentation in addition has several obvious, outlined item-by-item below: \n\n- \"they are locally well-defined for atoms and generalizable across various molecules\" --> this statement is clearly not true in this generality. In what way is a force generalizable across molecules? Needs clarification what is meant here? \n\n- following immediately: \"they reflect quantum mechanical interactions among atoms and expose the underlying quantum structures of molecules;\" --> extremely generic, if even true, atoms should be electrons here... \n\n- \"By harnessing these factors, we posit that learning from atomic forces can\n57 guide the model to learn a comprehensive and unified landscape of molecular conformations across\n58 different molecules, datasets, and equilibrium states\" --> this has been explored for decades in every single paper training interatomic potentials, leveraging atomic forces is in now way a novel idea, this needs a strong rephrasing. \n\n- \"We introduce a novel force-centric molecular conformation pretraining paradigm that trains\n87 a unified conformation representation encompassing both equilibrium and off-equilibrium\n88 molecules\" this is a vast overstatement, see above\n\n- 'The above-mentioned papers followed the same learning paradigm, where different models are\n116 trained for different molecules. Consequently, these methods cannot be easily generalized to unseen\n117 molecules\" --> there have been many efforts at building general-purpose force-fields, see e.g. the ANI series. \n\n- \". Despite their impressive accuracy in static prediction, their ability\n119 of running simulations were not reported until [26] showed that the increased static accuracy does not\n120 translate to better simulation performance\" again, horrendously false statement, many of the MLIP papers you cite ran various simulations, across diverse and difficult phenoma from ionic diffusion to heterogeneous catalysis to small molecules. \n\n- \"In molecular dynamics (MD) simulations, the molecular\n149 conformations are moved according to forces and the thermostat of the simulation.\" what about NVE simulations that don't have a thermostat? \n\n- \"For equilibrium molecular conformations x ∼ E , we assume their forces to be zero and impose a\nzero-force regularization:\" --> what is the reasoning behind this, if you *have* the forces anyway, they will be zero at equilibrium, so you're just again training on forces, again what this community has been doing since the 60s. \n\n- the denoising objective seems like it has obvious failure cases: perturbing a relaxed structure and argueing the force must go into the direction of the GS completely ignores the many-bodyness of the PES, discussing these would be helpful\n\n- a lot of the work seems to be based on improving one or two specific previously introduce pretraining objectives for equilibrium data. In the opening of the manuscript, these are cited several times, but never explained. This makes the manuscript difficult to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "d6fFY9D6xp", "reviewer": "Reviewer_8Ywg", "summary": "This paper incorporates off-equilibrium molecular conformations into consideration, in which the energy is not minimized. This paper defines pre-training objectives for force field learning on sampled off-equilibrium molecular conformations. Experimental results validate the effectiveness of pre-training on additional off-equilibrium data. ", "questions": "It seems the reported results on MD17 are not that superior. Probably this reviewer misunderstands the experimental tables but it seems the sota method is usually not the proposed method. Does the proposed method outperform the previous baseline methods?", "limitations": "The major limitation of the proposed method is that the force prediction might only play a role as a regularizer since learning force field is also a very challenging task since force is a vector feature instead of a scalar feature like energy. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The presentation of the paper is quite clear and easy to follow. The writing is well-organized;\n\n(2) The proposed idea is quite interesting and necessary. How to utilize off-equilibrium data is a useful and important research question for pre-training 3D molecular property prediction. This reviewer agrees with the significance of the proposed idea. ", "weaknesses": "(1) The novelty of the proposed method is not very high. Considering there are already published papers researching denoised pretraining strategies on 3d molecules, which lowers the novelty of the proposed method a bit. Then the major contribution of the proposed method is mainly about introducing additional data. This reviewer suggests that this paper might be more suitable for a domain journal; \n\n(2) The contribution of this paper might not be enough. Currently, the major contribution includes introducing additional data, and additional pre-training objectives. There are some important works like SCN (Spherical channel networks for modeling atomic interactions) utilizing separate force predictions. So separately predicting forces cannot be regarded as an important contribution;\n\n(3) The experimental results do not demonstrate the effectiveness of the proposed method. It seems that the proposed method seldom outperforms previous baseline methods, which makes this reviewer feel a little bit confused. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "P4Y9Loh0vG", "reviewer": "Reviewer_1y4N", "summary": "1. proposing a pretraining framework for conformation space including equilibrium and off-equilibrium.\n2. this method also shows highly accurate in MD or relax, including molecules and polymers.\n3. providing the community with a diverse set of DFT dataset of polymers.", "questions": "1. Would it be possible to incorporate the PCQM4Mv2 homo-lumo prediction results in the downstream analysis? This benchmark is well-defined and provides a competitive comparison for evaluating the performance of various models.\n\n2. About code and data access, I believe that the Poly24 dataset is of significant value to the research community for future investigations.", "limitations": "In the discussion section, it is mentioned that both the backbone and time efficiency play crucial roles in the performance of the proposed AI model. \nBesides I recommend incorporating partial charge prediction during the pretraining or downstream stages of your framework, if feasible. This addition could potentially enhance the overall effectiveness and applicability of your model.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and well-organized, effectively presenting the significance of force-centric pretraining for molecular systems. The proposed framework demonstrates comprehensive and impressive results in molecular dynamics simulations of small molecules and polymers, as well as in quantum chemistry property predictions.\n2. The authors also introduce an innovative concept in their work: the de-noising equilibrium balance and off-equilibrium co-training method.", "weaknesses": "1. As previously mentioned, I am particularly intrigued by the efficiency of the de-noising equilibrium tasks. It would be better to conduct an detail ablation study of de-noise equillibrium and force regularization.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable insight! In our separate work, we have indeed found that including partial charge in pre-training where available helps property prediction tasks. We are actively investigating this and more strategies to make the pre-training more efficient. \n\nIn the rebuttal period, we are unable to finish complete experiments on pre-training and fine-tuning without the de-noising objective. We agree that it would be valuable to investigate the ablated effects of each objective, and we will do so to make the paper more technically solid. "}]

Author rebuttal:

First, we thank the reviewers for their time and effort to provide valuable feedback. Below, we take the opportunity to discuss your points of concern.

**Our contribution is focused on the pre-training paradigm, instead of the model architecture.**
 NequIP has a great performance with limited training data and outperforms TorchMD-Net. The major novelty of this paper is orthogonal to NequIP – that we proposed a pre-trained framework that should ideally work for arbitrary deep learning models with sufficient expressivity, including NequIP. However, for practical reasons, we chose TorchMD-Net as the base model for efficiency:  we discovered that the training times for TorchMD-Net and NequIP differ significantly (Table R1), preventing us from conducting experiments using NequIP, even for lower basis. 

**Also in the paper, we focus on fastly developing an accurate forcefield from the pre-trained ET-OREO with limited training epochs. With the more epochs, ET-OREO outperforms NequIP.**

Rarely discussed in recent deep learning forcefield literature are the epochs required to achieve the reported performance. To achieve the performance reported in our paper, NequIP needs to be trained for maximally 2000 epochs for 10k data, while ET-OREO needs only be finetuned for 300 epochs. In Table R2, we compare NequIP and ET-OREO specifically at different epochs. ET-OREO, when further trained, improves ~30% over NequIP. 

The reason that we chose to report only 300 epochs in the paper is because the purpose of developing pre-trained models is that it can serve as an efficient base forcefield that can be efficiently adapted to many molecules. ET-OREO helps achieve NequIP-level performance at 300 epochs and further improved performance at 1000, at an already faster inference time per epoch.
In addition to the inference speed per training step, the number of epochs required for convergence also suggests a higher difference in time required to develop a forcefield from ET-OREO and NequIP. During the rebuttal time, we couldn’t train NequIP fully to 2000 epochs ourselves. However, for a quick reference, NequIP (l=1) takes 21 hours to train for 1000 epochs, (l=2) takes 35 hours, on a single A100 for aspirin. 

Of course, NequIP is still a better performing model compared with TorchMD-Net without pre-training, and we could reasonably assume that NequIP pre-trained with the ET-OREO framework could also have more impressive performance. In the future, we will incorporate NequIP and its subsequent more efficient variants into our work. 

**Force-centric Pre-training.**

The idea of training on atomic forces is not new; However, our motivation and contribution is using forces as a principle to unify the pre-training of molecules of many different domains. 

Previously, deep learning potentials have achieved impressive accuracy on limited data from single molecules. However, as witnessed in recent development in NLP and CV, with the massive availability of data and highly efficient model, we have the potential to push the boundary further. Models such as LLama and ChatGPT are not necessarily the best-performing chatbot model when data is limited. In fact, with limited data, I doubt if they can outperform simple rule-based models, which have been going on for decades. However, combined with efficient training strategies on massive corpora from the internet from multitudes of different domains, they transform the whole NLP landscape as we know it.

Traditionally, people develop models trained on potential energy, or the combo of potential energy and forces. Indeed, ANI-1 has an impressive performance on a large collection of molecules, trained solely on potential energy. However, we find this approach difficult when extending to multiple sources of data. Within MD17 itself, it already presents a challenge (See our Supplementary). One potential reason why training on potential energy could fail might be the software differences when obtaining the DFT predictions; The Supplementary of our paper and of [1] have some theoretical insights as to why this happens from an optimization perspective. Therefore, we resort to training exclusively on forces.  

Finally, we incorporate the data where forces are assumed zero, where we can apply the de-noising objectives to further provide useful signals to the model. In our paper, we find a way to unify the pre-training of these data under one force-centric principle in the context of deep learning models. 

**We will also make our languages and claims more concise, as the following:**
- Generalizability is in reference to the machine learning model and not the forces being trained on. We mean that one model can make predictions for many different molecules. We will make revisions to make the language more precise. 
- Indeed, electrons were the better word to use when speaking from a strictly quantum mechanics standpoint as DFT solves wavefunctions for electrons rather than atoms. This slip-up in language is because the forces calculated later in DFT and used in training here are on the atomistic scale, not the electron scale.
- Most papers, and the ones cited by the reviewer, train one model for one atomistic system (one model for one molecule), and the trained model can thus only work for force predictions for one molecule. In doing this NequIP has done extremely well. What we have done is train one pre-trained model to many different atomistic systems or molecules, with relatively little efforts on fine-tuning. In Section 4.2 we also deploy ET-OREO on large polymers unseen in the training data. We will rephrase to make our claim more precise. 
- Perturbed relaxed conformations don’t necessarily go back to the original structure, however it is the best guess we can have without further information. This comes from the Gaussian assumption from which the widely used MSE objective is also derived. 

[1] https://www.science.org/doi/10.1126/sciadv.1603015



Meta-review: The authors introduce a pre-training paradigm for molecular representation learning that incorporates both equilibrium and off-equilibrium conformations. As strengths, this paper is well-written and well-organized, effectively presenting the significance of force-centric pretraining for molecular systems. The proposed framework demonstrates comprehensive and impressive results in molecular dynamics simulations of small molecules and polymers, as well as in quantum chemistry property predictions. While other pre-training methods have achieved similar state-of-the-art results, the authors introduce a new framework for including off-equilibrium conformations in the pre-training objective. This work also contributes a new dataset of large polymers Poly24, which reviewers agreed will add value to the research community.
Decision: Accept (poster)