Physics-empowered Molecular Representation Learning
Authors: Seunghoon Yi
Youngwoo Cho
Jinhwan Sul
Seung Woo Ko
Soo Kyung Kim
Jaegul Choo
Hongkee Yoon
Joonseok Lee
Venue: Submitted to ICLR 2023 (2023)
Keywords: Physics
Transformer
Molecular representation learning
ML potential

Abstract: Estimating the energetic properties of molecular systems is a critical task in material design. With the trade-off between accuracy and computational cost, various methods have been used to predict the energy of materials, including recent neural-net-based models. However, most existing neural-net models are context-free (physics-ignoring) black-box models, limiting their applications to predict energy only within the distribution of the training set and thus preventing from being applied to the real practice of molecular design. Inspired by the physical mechanism of the interatomic potential, we propose a physics-driven energy prediction model using a Transformer. Our model is trained not only on the energy regression in the training set, but also with conditions inspired by physical insights and self-supervision based on Masked Atomic Modeling, making it adaptable to the optimization of molecular structure beyond the range observed during training, taking a step towards realizable molecular structure optimization.

Under review as a conference paper at ICLR 2023
PHYSICS-EMPOWERED MOLECULAR REPRESENTA- TION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Estimating the energetic properties of molecular systems is critical task in ma- terial design. With the trade-off between accuracy and computational cost, vari- methods have been used predict the energy of materials, including recent neural-net-based models. However, most existing neural-net models are context- free (physies-ignoring) black-box models, limiting their applications predict energy only within the distribution of the training set and thus preventing from being applied to the real practice of molecular design. Inspired by the physical mechanism of the interatomic potential, we propose a physics- diction model using Transformer. Our model trained not only on the energy regression in the training set, but also with conditions inspired by physical insights and self-supervision based on Masked Atomic Modeling, making it adaptable to the optimization of molecular structure beyond the range observed during training, taking a step towards realizable molecular structure optimization.
energy pre-
INTRODUCTION
Material simulation is vast research field that spans understanding material's optimal structure, simulating microscopic dynamics depending on time, temperature, and pressure beyond the exper- imental resolution, and reducing trial-error loops in designing new materials. The foundation of this simulation is defining the energy at the atomic level considering interactions between numerous atoms, so-called many-body problem. Advances in theory and computational capability have led to higher predictability of energy with greater accuracy than ever before. Despite the tremendous ad- vances, however, many-body interactions between atoms are exponentially complex, so increasing raw computing power alone has fundamental limitations. Thus, it is unavoidable to put an effort into reducing computational cost, as well as improving the prediction accuracy (e.g., quantum mechan- ies), which has been a grand challenge in computational material simulations.
Quantum mechanical electron structure simulation is aligned with the direction to enhance accuracy. Specifically, Density Functional Theory (DFT; Kohn & Sham (1965); Parr (1980)) is one of the most successful methods terms of accuracy, describing and predicting the structure of material, dynamics based on temperature, phase change, and chemical reaction. However, the number of atoms that DFT can practically handle is limited to a few thousand atoms. Thus, it complicated to compute the dynamics
larger scale with DFT, which demands an alternative method to scale up.
Another direction is to reduce the computational cost by expediting computation through approxima- tion while considering many-body interactions. For atoms of much larger scale, so-called classical force field approximations |arrison et al. (2018) have been developed. Various classical force field potentials have been proposed, sharing standard scheme: formulating the term-by-term energy equation using chemical intuition and performing parameterization for the targeting system. This approach also has drawbacks. First, building a classical force field requires extensive human effort to define energy terms by understanding the target system and parameterization, Also, the potential described the bonding terms of the material, so applicability restricted to non-reactive materials within similar phases, since the parameter should be changed depending on the chemical environment. Computationally heavier ReaxFF (Senftle et al. 2016 Gomzi et al., 2021 exists, but it focuses on chemical reactions that require parameters tailored to the specific reaction. The force field is challenged to secure accuracy despite a great deal of effort, and generalization to various chemical situations remains a challenging question. Nevertheless, it mimic potentials with few parameters, since it is based on equations and thus scales better than quantum simulation, which requires energy optimization by self-consistent calculations.

---

Under review as a conference paper at ICLR 2023
Recently, machine learning has made an impact the material simulation. Specifically, machine learning (ML) surrogate potentials, which predict the energies of given molecules, have drawn much attention from the scientific community as they provide an alternative solution to the aforementioned inevitable trade-off between cost and accuracy of physics-based models. Several state-of-the-ard MI surrogate potentials achieved high accuracy close to that of quantum mechanics while requiring significantly less computational resources than traditional DFT-based methods.
One might argue that we can produce an infinite training set using DFT, and fully data-driven approach would eventually work well on this problem, just like on other regression problems. How- ever, considering the exponentially increasing complexity of this problem and limited resources, is desirable to equip the surrogate potential model with domain knowledge from physics.
In this paper, we propose hybrid approach that combines the powerful expressive power of Trans- formers (Vaswani et al. 2017) with classical fiee--field-ttyle equations. In particular, our paper's main contributions are summarized as follows:
molecular representation learning method, actually preserv- ing the optimal structure instead of simply fitting the single energy value.
We propose a
Taking advantage of Transformers, our molecular representations are trained in seff-suuer manner by Masked Atomic Modeling, inspired by the approach of Masked Language Modeling. We conduct extensive experiments to evaluate the proposed model quantitatively and qualita- tively, introducing several novel approaches to actually evaluate the model's ability capture optimal structure of a molecule.
2 RELATED WORK
ML potentials can be categorized into three types based on model complexity and history: kemel- based descriptors, fixed atomic descriptors, and learnable descriptors using deep neural networks.
Kernel-based Methods. Kemee-rrerression-beeee based potentials are mainly applied to single atom or a few elemental species, where the kernel method is one of the lightest forms of ML. Gaus- sian Approximation Potential (GAP: BBrrók et al.] (2010)) Smooth Overlap of Atomic Potential (SOAP: Bartók et al. (2013)), and Spectral neighbor analysis potential (SNAP: Chen et al. (2017)) are representative examples. These models can be trained on a small amount of data, but it difficult to be extended to chemically complex cases.
Fixed deseriptors. Behler & Parrinello (2007) introduced an ML potential model that uses the atom-centered symmetry function (descriptor) to describe the local environment of cach atom and passes each descriptor value to the simple feed-forward neural network to map map the total energy. These descriptors (symmetry functions) process distance and angle information between paired atoms within a specific cutoff and produce single value for each descriptor. Behler-Parinello, Neu- ral Network (BPNN; [Behler & Parrinello (2007)) series are the representative practical examples that increase model complexity for high-dimensional Potential Energy Surface (PES) compared to previous kernel-based methods. BPNN was the realistic and the first attempt to decompose the total energy as a sum of each individual atom's 's energy. A fundamental limitation of this approach is that insufficient to
fixed descriptors or chemical functional groups), limiting the knowledge transferability between different molecules. Also, the original symmetry function does not reflect the chemical environment outside the cutoff at all Kulichenko et al. (2021). Despite these limitations, it achieved accuracy that no previous classi- cal force field reached. has been shown to work for systems with many atoms in dense system with few species Behler 2015): Kulichenko et al. (202I).
complex spatial patterns (e.g., ring structures, bond types,
Deep Learning- Models. Recently, deep neural networks have been actively applied con- struct surrogate potentials. Most models in this category allow the chemical environmental infor- mation can be transferred between atoms over greater distance than traditional models, providing a higher degree of freedom. As a specific example, ANI (Smith et al.) (017) extended BPNN by modifying its angular function. Gilmer et al. (2017) 7) proposed a Message Passing Neural Networks (MPNN), specialized in learning from graph-structured data by updating hidden node states by com- bining messages from adjacent nodes. Since then, various graph-basee approaches (Schütt et al.
2018 Gasteiger al.; 2020 Unke & Meuwly 2019) have been proposed. MPNNs significantly
improved accuracy in mnoeecule-reltten tasks on QM9 (Ruddigkeit et al.,
2015: Ra-
makrishnan al. 2014), while the increased model capability nest risk of overtitting (Hawkins.
2

---

Under review as a conference paper at ICLR 2023
2004
Zuo et al. 2020). Recently, the Transformer Vaswanni al. 2017) is applied to this problem
as well (Cho et al. [021
Thölke & De Fabritis 2022), following its success on natural language
processing Dev lin et al. 2018 and computer vision (Dosovitskiy al. 2021: Lu et al. 2019 Sun
et al. 2019).
METHOD
3.1
PROBLEM DEFINITION AND NOTATIONS
Given molecular structure graph G (V,E), where V is set of N atoms consisting the molecule and & is a set of bonds between a pair of atoms with direct interaction, we aim at regression problem to estimate the energy Emot E IR of the molecule. The total energy at the molecule level Exual is decomposed into the atomic level, denoted by E; for each atom = 1, .... N, where Emol = 2 Es Each atom i in the molecule is represented by its atomic number 2 E IR, its position p. E R in Cartesian coordinates, and electro-negativity n2; of the atom type. We denote the pairwise 1.2 distance matrix D E N between atoms, computed from {pt). Here, the element di, is the radial distance between atoms i and j. Adjacency matrix that represents bond information of the molecule denoted by A E {0, 1
3.2 ATOM REPRESENTATIONS
Each atom first needs to be represented as a vector before we feed into our Transformer-based model. Here, we describe how we represent each atom in detail.
Atom-wise Representation. Atom i is embedded as vector x! (self) Rd based on its type and its electro-negativity nzz;:
x (sel)) = [E(2;); nz.):
where, E. is embedding layer, and [:] indicates concatenation.
Radial Basis Functions. Inspired by the localized orbitals in DFT, we start with a simple Gaussian basis represent the relationship between two atoms. For pair of two atoms i and in the molecule, we assign ni basis functions following Unke & Meuwlyy2201 (201)):
Va.ja ((dis) exp { {-Bz.k Bz,, (exp(-dd) d4.)) )
exp
(2)
where i = L,.... N is the center atom index, j = N.....NN atomic number of atom i, and = 1,2 1.2...., Th denotes the index of the basis for each center atom type 2;. For predefined distance threshold т, so(d)
N is a neighboring atom index, 2; is the
if d << and () otherwise. With reasonable n&g can enhance expressibility of the model, generating more accurate potentials. Bg.. and Hz..k are the learnable parameters for each atom type 2s, which control the center and width of each individual basis. Finally, a cosine envelope function (Thölke & De Fabritiis, 2022)
(((25)) is applied to guarantee continuity at the cutoff edges, i.e.,
(p(() Od d== 0:
(cos adw. +
if S de,s ST otherwise
p(dr.))
(3)
Neighbor Embedding. We adopt the idea of neighbor embedding Thhöke & De Fabritiis 2022).
which represents relative information from nearby atoms under the distance of some threshold т, denoted by x (neighbor) E d
(ncighbor) Σ
U
(self) x
Vo5.]
(4)
where tb.,; = [ v..) E RMa E Rdx a projection matrix from radial basis functions the atomic embedding space, and @ indicates element-wise multiplication. U E ddd another linear projection matrix. As a result, x;
(neighbor) E Rd the neighbor embedding of atom is in the same atomic embedding space. For each atom i, combine the atomic and neighbor embeddings, then they are projected back to the same dimensionality by W ddx2d That is, x; W beln,
x (nighhoor).
3

---

Under review as a conference paper at ICLR 2023
(
Molectior Attembie Block
Molecular Attention Block
FC
Molecular Ateeiion Block
(0)
II
Figure 1: (a) Our model architecture. (b) Detailed Molecular Attention Block. (c) C2H4 example.
3.3 OUR TRANSFORMER MODEL
As illustrated in Fig. our model based on a Transformer. Given molecule as set of its N atoms, encoded as X; Rd for = 1.... .N, our model adds an additional [CLS] token, denoted by X9 E IRd to explicitly learn to represent the overall molecule embedding. On this input sequence, the model stacks I, Molecular Attention Blocks (MAB) to contextualize each atom representation across the molecule (within the cutoff distance T), which will be detailed subsequently. We denote by ( the atom embedding after O, ..., L stages the MABs. After L blocks, the final sequence of atomic embeddings x(4) are produced, and we estimate the overall molecule-level energy from them in two popular ways with Transformers. First, may predict the atom-level energy E, for atom by passing x to a simple linear layer. That is, E. faum (2")), where faom IRd + IR is an atom-level energy regressor, and then, summation over all atoms 1,..,, gives the molecule- level energy; that is, Emal Σ E.
Another approach directly computing the molecule-level energy from the [CLS] by Emu (ma((x(")) where fmol : Rd - R is a molecule-level energy regressor. For fatom and fmot. we use a single fully-connected layer. Both approaches evaluated in Sec. 4 In Sec. [3.4] will introduce our main approach for this regression to take advantage of domain knowledge from physics.
Details Molecular Attention Block. Each Molecular Attention Block (MAB) level e takes O,... from the previous level. For each atom x; (f-1) as query and all atoms including as the context (keys and values), it performs self-attention as in Fig. [b). Following TorchMDNet (Thölke & De Fabritiis 2022), we modify from the vanilla Transformer (Vaswani et al. 2017) to explicitly reflect the relation arisen from the physical distance between two atoms and j. in addition the semantic relevance between them modeled by regular Transformers. Specifically, from the radial basis vh,, (Orr et al. 1996), we compute D* D Nxx where m is the embedding dimensionality used for query, key, and value. An element dY E R represents physical tendency to attract each other between atom and j for key- purpose and value-purposs respectively. These are mapped from the radial basis function vhC, by linear layer, followed by SiLU (Elfwing et al. 2018) activation. This relation represented as TR instead of scalar to reflect cimension-wise relationship.
a sequence of atomic embeddings (x(*-1)
In addition to the changes introduced by Thölke & De Fabritiss (2022). we additionally feed the adjacency matrix A E {0, N followed by a linear layer and SiLU activation. This A-mask multiplied element- wise with the inferred attention weights, in order to additionally control this semantic relevance based physical adjacency. For instance, two atoms that far away will be likely multiplied by a low value, reducing its relationship even if semantic relevance is estimated high. This part be optionally turned on off, and provide ablation study in Sec.
4

---

Under review as a conference paper at ICLR 2023
3.4 PHYSICS-DRIVER PARAMETRIC ENERGY PREDICTION
Instead of directly regressing to the atom or molecule energy as described in Sec. 3.3 we propose to design a parametric model that reflects physical insights. For this formulation, we use a simple form that can simultancously reflect the repulsive and attractive forces between two atoms i, j within the bond energy Bi namely, Coulomb's law and Lennard-Jonee Potential (LJP):
9o dig
(
(B4 dig
28
34 d
F = -81
B2
(S)
where In corresponds q:lj. influence of charges between two atoms in Coulomb potential. Ba is the equilibrium distance between atom i and j. where the repulsive and attractive forces become equivalent, and thus the atom-atom potential energy becomes zero. The energy becomes minimal at this point. 31 and B3 are linear coefficients for the Coulomb and LJP parts. It is known that Bg si 6 under the condition of following the London dispersion force London (1930p: Cornell al. (1995), but the repulsive equivalence of 2/3 = 12 is much more an approximate term (square of the attractive term), so we leave 83 as an open parameter to learned from the given data. These 5 parameters, denoted by B [S0, 31, B2, B3, Ba], are estimated by regressor Shand R2d - R$; that is, 3 foood [xx; x|))-
The overall molecule-level energy is calculated by the sum of all pair-wise bond energies and the atomic self-energies; that is, Emol = Brand + Entam, where Enont and Estom are defined as
Exoo Σ Sarom (x. (x)
Ebond
Esg: and
(6)
We summarize what to expect from modeling to predict the parameters of physical formulation as follows. First, we aim to make sure that we satisfy physical conditions when we learn from given information and extrapolate to unseen cases. Second, by observing the predicted parameters, can monitor whether the model actually captures the physical properties of the molecule. Lastly, we expect the model to predict the energy directly from Estom if the given formula is difficult to follow. In Eq. (5), for instance, if the inter-atomic potential does not fit well with LJP, the model assigns A3 Rs O, relying solely on the Coulombic potential.
Initially, the model is subjected to minimize the MSE Loss between the predicted molecule energy Emot and its ground truth Emot. In other words, the energy loss Lenergy defined as || Emol Emol 1|
3.5 MASKED ATOMIC MODELING
Masked Language Modeling (MLM), originally introduced by BERT (Devlin et al.) 2018) for lan- guage modeling, has been successfully utilized as pre-training task for various multimodal models and tasks (Lu et al.] 2 Sun et al. 2019 Zhang et al. 2020ap. The main idea is to randomly mask a subset of tokens and let the model recover them from its contexts, i.e., the other textual or visual tokens in the input sequence. This concept naturally supports self-supervised learning long the elements in the input sequence are contextually relevant, requiring no human labeling.
In this paper, we propose Masked Atomic Modeling (MAM) in similar spirit. All chemical mate- rials are composed of multiple atoms, often with more than one type. When a majority of atoms in a valid molecule is known, set of possible atoms in the rest is significantly reduced when consider- ing the properties of each atom according to the law of chemistry, e.g., the octet rule, Lewis symbol analysis. With MAM, would like train our Transformer discover such chemical restrictions purely by observing a set of valid molecules in the training examples without direct supervision.
Formally, on sequence X R. with N atoms, we randomly mask each token by probability of p (we use 0.3, twice as Devlin et al. (2018)), replacing the masked tokens to [mask]. In the resulting matrix, the rows of X corresponding to the masked tokens, are replaced with the mask embedding. The model is trained to minimize the log loss over the masked tokens:
Lmak
log p( X & ml X @ m))
where m E 10,1]* is a binary mask vector for atoms, 1 is a one-valued vector, and @ indicates row-wise multiplication. p is estimated by binary classifier, where use two-layer MLP.
5

---

Under review as a conference paper at ICLR 2023
3.6 COMBINING PHYSICAL CONSTRAINTS
Zero-Force Regularization. When a molecule is in its equilibrium state, the net force on each atom should be at zero. This condition may provide strong hint for the model find the valid and optimal molecular structure, but this has not been utilized well in existing studies. Thus, we additionally regularize to minimize the force, computed by the partial gradients of the predicted energy with respect to the 3-dimensional axis (I, y, 2). Formally,
N
OE dx
0
0 0z
Diorce Σ
F
+
Dy
where F E R is the predicted force of atom i.
Inequality Bound Condition. The definition of a stable equilibrium structure is a structure corre- sponding to the lowest energy under the given composition. Such an optimal structure can be found by estimating energy from the given structure, differentiating with respect to the position, and changing the position based on force information. Naturally, if there is any local deviation from the optimal structure, the energy is always greater than its ground energy. This sounds obvi- ous physically, but machine learning model is unaware of this and its estimation may be invalid. Therefore, we apply an additional condition to narrow down the solution space. The energy should be greater than the ground state when locally deviating from the stable structure. During training, small Gaussian random noise with an amplitude of 0.5 is applied to the optimal structure. This is implemented by additional loss term Lhound based the energy inequality condition:
{ Emol
Eiool
E mol I Emot otherwise
Lhound
0
3.7 OVERALL OBJECTIVE
Combining all together, our model minimizes three loss functions:
L Lenergy + Amask + Aforce + Abound
(10)
where Aforces Amask. and Abound are coefficients controlling relative importance of each loss term.
EXPERIMENTS AND RESULTS
We conduct experiments to answer the following questions: Q1. How does our model perform on energy estimation compared other models? (Sec. (422) Q2. How does our model perform on
basic structural optimization experiment compared to other models? (Sec. 4.3 Q3. How much physics-driven constraints affect the prediction? (Sec.
4.1 EXPERIMENTAL SETTINGS
Datasets. We use two public benchmarks to compare our model with competing models. QM9 (Ruddigkeit et al. [2012] Ramakrishnan et al. 2014) dataset is collection of optimal structures of 130,000 molecules with up to nine atoms of (C, H, N, F}, selected from GDB-17 (Ruddigkeit et al. 2012). We use 80% for training, 5% for validation, and 15% for testing. Additionally, evaluate on OC20 dataset (Chanussot* et al. 2021), which contains stable structures and relaxation trajectories for systems of 15K bulk catalysts and 82 adsorbates. We train model on two tasks, energy and force regression based on the given structure (S2EF) and relaxed energy prediction with given initial structure (IS2RE).
Baselines. We compare state-of-the-at energy prediction models on the QM9 dataset: SchNet (Schitt et 2018), DimeNet (Gasteiger al. 2020)) (orchMDNe((ET) (Thölke & De Fabritiss 2022). ForceNet (Hu et al.) 2021) and MXMNet (Zang et al., 2020b). Except for ForceNet and MXMNet, publicly available pre-trained parameters are utilized the experiment.
Evaluation Metric. We report the mean average error (MAE) between the prediction and the ground truth for the estimation energy (MAEg, in //mol)) and force (MAEp, eV/A) estimation, following existing studies.
More implementation details are provided Sec. A the Appendix.

---

Under review as a conference paper at ICLR 2023
Dataset (Task)
QM9 MABp (U)
OC20 (S2EF)
OC20 (ISZRE)
Model
MABE ())
AP (A)
MABp (L)
MAEp (L) (A)
SchNei Schittr
2018
14.00
2.64
0.47
0.0743
L,599
0.60
CGCNN Xie & Grossman
2018
0.0673
0.988
0.58
MXMN Zisang
MXMNa
20206
5,90
1.83
1.57
DimeNer Gästerger al 2020
8.02
1.79
0.58
0.0693
1.012
0.55
ForceNet Hu et (202)
18.62
0.41
0.21
TorchMDNeLLE1 boike & De Fabritiis
2022
6.15
1.15
0.32
GemNet di Gasteiper a.a. (222
2021
0.0257a
0.18
SpinConv Shuaib
12.00
0.0329
0.21
Ours (Coneg only)
8.35
1.23
Ours (full model)
15.16
0.005
0,025
0.0549
0.887
0.10
Table I: Comparison with baseline models for energy and force accuracy (in MAE) and average distortion AP after structure optimization experiment. ("indicates trained on 100x larger data.)
4.2 COMPARISON WITH BASELINES
Tab. compares the performance of our model with baselines on tasks of energy prediction, force prediction, and structure optimization. We report the performance of our model with two configura- tions; one with the full model and the other with Lenrrg only, by setting (mmss..fors::bondd) to zero.
In this line of research, the Energy MAE has been the most widely used metric. At glance to the first column, Energy MAE, we observe that our proposed model estimates the molecule energy comparably with most baselines, slightly lagging behind the current state-of-the-art, MXMNet.
We ask question here: does this mean that the models actually understand the molecular structure and estimate the energy precisely from it? We are not able to answer this question solely from the energy MAE, since one energy value in the optimal structure does not contain all information on how the PES is reflected according to the degree of freedom with the position of the atoms. Therefore, we further investigate with additional metrics which reflect the structure or physical conditions: zero-force condition on optimal structures and simple structural optimization task.
The second column of Tab. D reports the MAE in force estimation of each model by differentiating the energy with respect the position. Ideally, lower energy MAE should indicate better molecular structure, and thus it should lead to lower force MAE as well. This is the underlying expectation that the entire research community has solely focused on energy MAE optimization.
Interestingly, we observe that energy MAE and force MAE do not have strong one-to-one corre- spondence. All existing models and our Lenergy model is optimized only for the energy prediction while showing significantly worse performance on the force estimation. In other words, those pre- vious models are significantly over-optimized only for energy estimation, without consideration of basic constraints that potential models must satisfy. On the other hand, our full model dramatically reduces the force error through the additional terms based on physical insights. Although the en- ergy MAE is slightly higher than other models like MXMNet, difference of about 10 me V/mol is acceptable in terms of the overall quality.
On OC20, We experiment with both tasks, S2EF and IS2RE. The result is added to Tab. where we compare against a few baselines using scores reported in Open-Catalyse Cattaaatt-Pooooeee Projec indicates that method is competent on both tasks, outperforming all baselines.
! This table
4.3 QUALITATIVE ANALYSIS WITH STRUCTURE OPTIMIZATION
A primary application for calculating molecular energy is search for a stable structure and to perform molecular dynamics (MD) simulations of structural changes over temperature and time. All of these works are the foundation for the design and discovery of new materials (Friederich et al. 2021 Louie et al. ).
In order to see if the models actually capture the optimal structure of molecules, we conduct an ad- ditional structure optimization experiment. Starting from the stable structures in QM9, we slightly perturb each atom's position from its original optimum and optimize the structure again, expect- ing it to converge back to the original optimal one. Upon convergence, we measure the structure distortions AP, in average Euclidean distance, as each atom moves from its optimal structure.
ht httpss//githut wwtp:/ww..oo.p.ee.eeee Dat lyat-ta- pyaatPPree Project
7

---

Under review as a conference paper at ICLR 2023
(a)
(b)
SchNet
SchNet
ForceNet
DimeNet
MXMNet
0.4
ForceNet
Mapiiaa Stocurd
DimeNec
Probability
MOMVNe
0.3
OurstE.F)
0.2
0.1
10T
10
10t
10
10
(
(c)
ForceNet
0.20
DimeNet
Probability
MXMNNet
0.15
OurstE,F
0.10
0.05
10
10
10"
10"
10
AP(A)
Figure 2: (a) Structural optimization results. The left-most column is the initial stable structure in QM9, followed by recovery results by competing models sequentially. For more structural opti- mization results, see Appendix Fig. [I (b-c) Distribution of energy difference (AEg) and structural change ( P) before and after structural optimization, in log scale.
Sap
Sep
Figure 3: Molecule assembly results from (a) GDB-35 and (b) GDB-87. GT the ground truth structure with 500 relaxation steps. The discon- nected bond is well connected and recovered to its original stable struc- ture.
C
The right-most column of Tab. reports quantitative comparison results of this experiment. Our physics-driven model achieves more accurate score compared other models, indicating that our proposed model indeed understands and is capable of reproducing stable structure instead of simply over-optimizing on the energy estimation.
Fig. (a) illustrates a few examples of optimized structures by baseline models and ours. The initial stable structures the left-most column are not well-maintained by the baselines. For the simplest molecule, CH4, the Hydrogen atoms around the Carbon atom are expected to be symmetrically ranged, but the optimized structures by the baselines are not symmetric. more complex molecules, most potentials do not preserve the stable structure.
Fig. (b.c) show the distribution of the energy difference AE, before and after the reoptimization, averaged over 256 molccules (128 smallest and 128 randomly sampled molecules) from the QM9 dataset. Fig. 2 (b) demonstrates the A in log scale, which clearly shows the difference in the performance of each model. With our model, the center of AFg. is at least two orders of magnitude smaller compared to other potentials, indicating that our model recovers the optimal structure far better than all other models. Also, in Fig. 2 (c), the distance deviation AP is mostly less than O. and even compared to other potentials. it can be seen that AP is at least 10 times smaller. This structure optimization benchmark is tough task because the QM9 dataset consists only of stable structures. Furthermore, it is unreasonable to expect molecular dynamics (MD) to work appropri- ately, reflecting the temperature and dynamics on this stable structure-only dataset. On the contrary, great performance on this challenging optimization task demonstrates that our physics-driven model captures basic physical information, such as distance symmetry, even from limited information. Additional examples are presented in Sec. D in the Appendix.
4.4 MOLECULE ASSEMBLY TASK
We employ our approach for additional task, namely, the molecule assembly task. In addition to the structure optimization conducted in Sec. 4. where we start from (almost) optimal structure and optimize the energy if the model can recover the stable structure again, this molecule assembly task makes it further challenging by even breaking one or more bonds the molecule by moving some functional group far away. This task aims recover the original stable structure from this
8

---

Under review as a conference paper at ICLR 2023
No.
Base
[C18]
LJP
Mask
Force
Bound
MABe
MAE;
AP
Table Abla-
tion study results, adding or subtracting components in the loss function. Red indicates unaccept-
11.8
0.77
1.76
9.03
0,90
1.11
9,70
1.11
0.814
V
10.18
0.016
0.141
16.34
0.007
0.038
ably inferior results (MAE,, AP >> 0.1).
17.50
0.005
0.027
17.34
0.013
0.044
9.65
0.015
0.083
15.16
0.005
0.025
completely broken one. Since the QM9 dataset does not contain non-equilibrium information, it is challenging to expect accurate energy values along the pathways which molecules are combined. Therefore, our goal is to examine if the bond-broken molecule can be re-assembled to stable structure. For this, we randomly select one or two functional groups in a molecule and disconnect bonds between them by translating each towards different directions. (In experiment, move them by O.7À.) Starting from this distorted structure, we optimize the structure again using our model see if it can recover the original stable structure.
As shown in Fig. 3] only our method succeeds in recovering the structure to the original undistorted one, while others show catastrophic failure. To this end, we conclude that the strength of our ap- proach to reproduce a physically correct molecular structure can be extended to reproducing the correct reaction pathway.
4.5 ABLATION STUDY
We conduct ablation study to see which component contributes improving which metric. The 'Base' model indicates our Transformer model described in Sec. 3.3 without using [CLS] token. Tab. 2 compares multiple configurations of our model using subset of components. Comparing #1 and 42, the [CLS] token turns out to be effective, reducing the energy error. The rest compares by adding
each component separately starting from base+equation model (#3, #4, #5) and by eliminating each component from the full model (#6 #10). We observe the following:
Mask plays its role in improving the energy estimation. Comparing #7 and #10, having Mask helps the model to improve MAE; without affecting MAE₁ or A. Solely with Mask (#3), achieves nice MABE, but its structure looks suboptimal implied by inferior MAEp and AP. Bound condition is the most important component for understanding the overall structure. With- out it (#9), AP gets significantly worse than the full model (#10), while MAEg gets (probably illegally) better by focusing more on the energy like baseline models. With Bound only (#5), it achieves reasonable MAE; and AP, which not possible only with Mask (#3) or Force (#4).
Force affects all three metrics slightly at the same time. Without Force (#8), all three metrics get slightly worse compared to the full model (#10). With the Force only (#4), however, the A is suboptimal. We can conclude that the Bound condition is also needed to get the acceptable AP. Sec. B in the Appendix presents additional ablation study on model size and MAM masking ratio. Also, Sec. C provides additional qualitative analysis on MAM and physics-driven modeling.
CONCLUSION
In this work, we propose a physics-driven model and regularization scheme for molecular energy prediction. The physics-driven design enables our model to generate a physically meaningful struc- ture. We also propose self-supervised learning method inspired by masked language modeling. Our observation indicates that the state-of-the-art models may not be robust under the structure op- timization task with the QM9 dataset, which only contains optimized stable structures. We utilize inequality constraints and force information of the optimal conformations, which results in phys- ically more reasonable outcomes. With the combination of physics-driven modeling and regular- ization, our model outperforms the state-of-the-ar models in structure optimization tasks with the QM9 dataset. We also demonstrate that our approach can be used for structure optimization with a few meV scale differences in the initial structure of small molecules. Furthermore, we find that much improvements are needed to achieve a reasonable PES that not only matches a single energy value for large molecules. When constructing an ML potential, it is always challenging to have sufficient dataset; thus, our approach that maximally utilizes the information in both model design and training would shed light future research.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Albert P Bartók, Mike C Payne, Risi Kondor, and Gábor Csányi. Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons. Physical review letters, 104(13): 136403, 2010.
Albert P Bartók, Risi Kondor, and Gábor Csányi On representing chemical enviromments. Physical Review B, 87(18):184115 15, 2013.
Jörg Behler. Constructing high-dimensional neural network potentials: A tutorial review. Int. J. Quantum Chem., 115:1032-10500 2015. ISSN 1097-461X.
Jörg Behler and Michele Parrinello. Generalized Neural-Network Representation of High- Dimensional Potential-Energy Surfaces. Phys. Rev. Lett., 98: 18:144 April 2007.
Lowik Chanussot*, Abhishek Das*, Siddharth Goyal*, Thibaut Lavril*, Muhammed Shuaibi*, Mor- gane Riviere, Kevin Tran, Javier Heras- Daeras-Dmmingo... Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 2021. doi: 0c04555.
Chi Chen, Zhi Deng, Richard Tran, Hanmei Tang, lek-Heng Chu, and Shyue Ping Ong. Accurate force field for molybdenum by machine learning large materials data. Physical Review Materials, 1(4):043603, 2017.
Youngwoo Cho, Hongkee Yoon, Seunghoon Yi, Jaegul Choo, Myung Joon Han, Joonseok Lee, and Sookyung Kim. Deep-DFT: physics-ml hybrid approach to predict molecular energy using transformer. In Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2021.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv:1511.0728) 2015.
Wendy D. Cornell, Piotr Cieplak, Christopher I. Bayly, Ian R. Gould, Kenneth M. Merz, David M. Ferguson, David C. Spellmeyer, Thomas Fox, James W. Caldwell, and Peter A. Kollman. A Sec- ond Generation Force Field for the Simulation of Proteins, Nucleic Acids, and Organic Molecules. May 1995. ISSN 0002-7863.
J. Am. Chem. Soc., 11::5179-5197
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16 words: Transformers for image recognition scale. In Proc. of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighte linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.
Pascal Friederich, Florian Häse, Jonny Proppe, and Alán Aspuru-Guzik. Machine-learned potentials for next- mext-generation matter simulations. Nat. Mater,, 20:750-761, 2021.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molec- ular graphs. In Proc. of the International Conference on Learning Representations (ICLR), 2020.
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proc. of the International conference on machine learning (ICMI.), 2017.
Vjeran Gomzi, Iva Movre Sapié, and Andrej Vidak. ReaxFF Force Field Development and Appli- cation for Toluene Adsorption on MnMOx (M = Cu, Fe, Ni) Catalysts. Phys. Chem. A, 125: 10649-10656, December 2021. ISSN 1089-5639.
10

---

Under review as a conference paper at ICLR 2023
Judith A. Harrison, J. David Schall, Sabina Maskey, Paul T. Mikulski, M. Todd Knippenberg, and Brian H. Morrow. Review of force fields and intermolecular potentials used in atomistic compu- tational materials research. Applied Physics Reviews, 5:031104, August 2018.
Douglas M. Hawkins. The Problem of Overfitting. J. Chem. Inf. Comput. Sci., 44:1-12, January 2004. ISSN 0095-2338.
Weihua Hu, Muhammed Shuaibi, Abhishek Das, Siddharth Goyal, Anuroop Sriram, Jure Leskovec, Devi Parikh, and C Lawrence Zitnick. ForceNet: A graph neural network for large-scale quantum calculations. arXiv:2103.01436 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
W. Kohn and J. Sham. Self-consistent equations including exchange and correlation effects. Phys. Rev., 140:A1133-A111 November 1965.
Maksim Kulichenko, Justin Smith, Benjamin Nebgen, Ying Wai Li. Nikita Fedik, Alexander I. Boldyrev, Nicholas Lubbers, Kipton Barros, and Sergei Tretiak. The Rise of Neural Networks for Materials and Chemical Dynamics. J. Phys. Chem. Lett., 12:6227-6243, July 2021.
London. Zur Theorie und Systematik der Molekularkräfte. Z. Physik, 63:245 22224, March 1930.
ISSN 0044-3328.
Steven G. Louie, Kan Chan, Felipe H. da Jornada, Zhenglu Li, and Diana Y. Qiu. Discovering and understanding materials through computation. Nat. Mater, 20:728-7355 2738, June 2021. ISSN 1476-4660.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViBBERT: Pretraining task-agnostic visiolin- guistic representations for vision-and- vision-and-language tasks. Advances in neural information processing systems (NeurIPS), 32, 2019.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proc. 27th Int. Conf. Int. Conf. Mach. Learn., (CML'10, pp. 807-814, Madison, WI, USA, June 2010. Omnipress.
Mark JL Orr et al. Introduction to radial basis function networks, 1996.
Robert G Parr. Density functional theory of atoms and molecules. In Horizons of quantum chemistry, pp. 5-15. Springer, 1980.
Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
Jean-Louis Reymond. The Chemical Space Project. Acc. Chem. Res., 48:722-730, March 2015.
ISSN 0001-4842.
Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database GDB-17. Journal of chemical information and modeling. 52(11)2284-2875 2012.
Kristof T. Schütt, Huziel E. Sauceda, Pieter-Jan Kindermans, Alexandre Tkatchenko, and Klaus- Robert Müller. SchNet deep learning architecture for molecules and materials. J. Chem. Phys., 148:241722, June 2018. ISSN 0021-9606, 1089-7690.
Thomas P. Senftle. Sungwook Hong, Md Mahbubul Islam, Sudhir B. Kylasa, Yuanxia Zheng, Yun Kyung Shin, Chad Junkermeier, Roman Engel-Herbert, Michael J. Janik. Hasan Metin Ak- tulga, Toon Verstraelen, Ananth Grama, and Adri C. T. van Duin. The ReaxFF reactive force- freeeffied:: Development, applications and future directions. Npj Comput. Mater, 2:1-14, March 2016. ISSN 2057-3960.
Muhammed Shuaibi, Adeesh Kolluru, Abhishek Das. Aditya Grover, Anuroop Sriram, Zachary Ulissi, and C. Lawrence Zitnick. Rotation Invariant Graph Neural Networks using Spin Convolu- tions, June 2021.
11

---

Under review as a conference paper at ICLR 2023
Justin S Smith, Olexandr Isayev, and Adrian E Roitberg. ANI-1:
extensible neural network potential with dft accuracy at force field computational cost. Chemical science, 8(4):3192-3203, 2017.
Chen Sun, Austin Myers. Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT: joint model for video and language representation learning. In Proc. of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Yi Sun, Xiaogang Wang. and Xiaoou Tang. Deeply learned face representations are sparse, selective, and robust, December 2014.
Philipp Thölke and Gianni De Fabritiis. TorchMD-NET: Equivariant Transformers for Neural Net- work based Molecular Potentials. ArXiv22020254/ Phys., February 2022.
Oliver T. Unke and Markus Meuwly. PhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments and Partial Charges. Chem. Theory Comput., 15:3678-3693, June 2019. ISSN 1549-9618, 1549-9626.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Infor- mation Processing Systems (NeurIPS), 2017.
Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for accurate and interpretable prediction of material properties. Physical review letters, 120( (11)))144330) 2018.
Bowen Zhang, Hexiang Hu, Joonseok Lee, Ming Zhao, Sheide Chammas, Vihan Jain, Eugene Ie, and Fei Sha. A hierarchical multi-modal encoder for moment localization in video corpus. arXiv:2011.09045 2020a.
Shuo Zhang, Yang Liu, and Lei Xie. Molecular mechanics-driven graph neural network with mul- tiplex graph for molecular structures. Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2020b.
Yunxing Zuo, Chi Chen, Xiangguo Li, Zhi Deng, Yiming Chen, Jörg Behler, Gábor Csányi, Alexan- der V. Shapeev, Aidan P. Thompson, Mitchell A. Wood, and Shyue Ping Ong. Performance and Cost Assessment of Machine Learning Interatomic Potentials, J. Phys. Chem, A, 124:731-745. January 2020. ISSN 1089-5639.
12


Reviews:
[{"review_id": "itbHdQSxJI4", "reviewer": "Reviewer_bSWh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel masked atomic modeling framework for molecular representation learning. Their architecture combines several problem-specific designs, including Molecular Attention Block and other physics constraints as loss functions, including Zero-Force Regularization and Bound Condition. Empirical results show that the proposed method outperforms previous popular frameworks such as DimeNet. ", "summary_of_the_review": "This paper proposes Masked atomic modeling for molecular representation learning. It also incorporates several physical constraints effectively. I think this is solid contribution to the community.", "strength_and_weaknesses": "Strengths:\n1. This paper applies masked atomic modeling to molecular modeling, similar to masked language modeling. The most important part is that the model is not just naively using a standard transformer but incorporates a lot of problem-specific design.\n2. This paper shows good empirical results compared to previous methods. \n\nWeaknesses:\n1. The dataset and the molecules in the study are relatively small. Also, all experiments are conducted on the same dataset for one task. It is questionable how such learned representations can be used for other datasets/tasks.\n2. The ablation study is not clear. It only shows several extreme comparisons of removing several important loss function terms. However, it is questionable how each design component is effective. For example, what is the optimal mask ratio? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "d6leSJhDpV", "reviewer": "Reviewer_jtDm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a physical-driven model and regularization scheme to predict energy for molecular systems. Compared to other baseline models, their full model preforms much better on force prediction, while the energy MAE is slightly lagging behind the state-of-the art MXMNet model. When applying to the structure optimization task, the full model outperforms the other base line models, which demonstrates that the physical-driven model captures the basic physical information instead of simply over-optimizing on the energy estimation. Besides, inspired by the Masked Language Modeling, the authors proposed Masked Atomic Modeling to help the model to discover chemical restrictions.", "summary_of_the_review": "In summary, the paper proposed a physical-driven model, which performs better on molecular structure optimization and force prediction than the state-of-the-art models. The model may capture the true underlying physics rather than simply fitting. The ideas in the paper would pave the way to more accurate and general physical-driven models in the future research. For these reasons, I recommend receiving this article.", "strength_and_weaknesses": "Strength:\n\nThe physical-driven model outperforms the other base line models in the realistic structure optimization task, and the output structures are more physical than other models.\n\nThe loss terms are physical meaningful and effective, which shed light on future research.\n\nWeaknesses:\n\nThe authors didn’t compare the mask, force and bound terms separately, to see which is the most relevant in terms of the performance improvement.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "63yLPLNnywi", "reviewer": "Reviewer_wZ2H", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a transformer model that is trained to learn the force-field relationship of molecules. It has the ability to predict molecular energy without expensive molecular dynamics computation. To achieve this, they use a transformer with a molecular attention block to predict atom force and bond force and then calculate the energy.  A physics-driven parametric energy model, masked atomic modeling, and physics-inspired constraints are combined to improve energy prediction accuracy. On QM9, the model surpasses previous models in terms of force prediction accuracy and structural stability prediction.", "summary_of_the_review": "The paper is overall clear and well-written. The physics inductive bias seems to be effective. However, the experimental results are not strong in terms of energy prediction. Therefore, I recommend it as borderline reject and I am willing to increase the score if the authors could clarify why the force accuracy is better than energy prediction.", "strength_and_weaknesses": "Strength: \n1. Effective use of physics inductive bias. The force-field relationship is crucial for accurate quantum energy understanding. \n2. Gives a good energy landscape in terms of molecular structure space. It seems similar in principle to molecular dynamics and helps understand why this method work.\n\nWeakeness: \n1. I'm interested to know why the result in Table 1 shows that the force prediction is more accurate yet the energy prediction is not. I wonder if the LJP model gives too much constraint on the energy prediction and affects performance. \n2. The experiments are not complete enough: \n  - Only performance on QM9 is shown, which is ok since its the standard benchmark in this field. Other datasets (e.g. OC20 ) could also be used as QM9 is often criticized for its non-equilibrium structures of molecules. \n  - Strong baselines are missing. ForceNet[1] also learns force in addition to energy and reaches similar resolution to this method in terms of \nforce prediction.\n\n[1] Hu, W., Shuaibi, M., Das, A., Goyal, S., Sriram, A., Leskovec, J., ... & Zitnick, C. L. (2021). Forcenet: A graph neural network for large-scale quantum calculations. arXiv preprint arXiv:2103.01436.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2-ksfNMYlPW", "reviewer": "Reviewer_x1A4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript describes a Transformer-based neural network aimed at predicting the energies and the forces of molecules. The network is trained on the prediction of the energy of ~130,000 small molecules. The loss includes the square deviation between the ground truth energy and the predicted energy, and also terms aimed at ensuring that the \"experimental\" structures are stable.  The energy estimator includes, beside the transformer, a  two-body energy, eq 5. The model predicts the energy with an accuracy which is marginally worse than that of other approaches, but it allows recovering the correct structure in geometry optimizations started from perturbed configurations.", "summary_of_the_review": "The results presented in the manuscript are mostly technically correct, but, in my opinion,  do not advance significantly the state of the art of machine-learning based potentials, since the numerical tests are performed on tasks which are not of practical use in molecular modelling. Some of the directions which are explored , for example the BERT-style masking loss, are interesting and could bring to important advances.", "strength_and_weaknesses": "Strength:\n1) the idea of training  a network using a BERT-style censoring of the position of some atoms is, to my knowledge, original.\n\nWeaknesses:\n1) the challenge in the field is reproducing the forces (and the energies) along a *reaction pathway*, namely for sequences of configurations connecting two different molecular species, or two different configurations of the same molecule. The forces along these pathways are typically very high, and determine the chemistry of the system.  The test set used in this work is therefore mostly of academic interest: the forcefield developed in this work is good for reproducing the geometry of the molecules, but worse than other approaches in reproducing the energy difference between molecules. No relevant \"activated\" geometry is included in the test set. \n2) the idea of fitting the forces, and not only the energy, dates back to the 90ties (see wiki, \"force matching\") and is  used almost ubiquitously also in NN training (see, for example, https://pubs.acs.org/doi/10.1021/acscentsci.8b00913 eq 9). The fact that by adding a penalty on the value of the forces allows recovering the correct  geometry is correct, but trivial, since the configurations explored in these optimization are very similar to the global minimum. The relevant result would be reproducing the forces along a reaction pathway (see point 1).\n3) The idea of adding explicitly LJ or Coulomb interactions in a NN is also not new, but, more importantly, adding these terms for isolated and very small molecules is not chemically justified. In fact these terms are typically added to describe liquid environments (see for example https://www.nature.com/articles/s41467-020-20427-2 ), where different molecules interact by non-bonded interactions. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject