Energy-Inspired Molecular Conformation Optimization
Authors: Jiaqi Guan
Wesley Wei Qian
qiang liu
Wei-Ying Ma
Jianzhu Ma
Jian Peng
Venue: ICLR 2022 Poster (2022)

Abstract: This paper studies an important problem in computational chemistry: predicting a molecule's spatial atom arrangements, or a molecular conformation. We propose a neural energy minimization formulation that casts the prediction problem into an unrolled optimization process, where a neural network is parametrized to learn the gradient fields of an implicit conformational energy landscape. Assuming different forms of the underlying potential energy function, we can not only reinterpret and unify many of the existing models but also derive new variants of SE(3)-equivariant neural networks in a principled manner. In our experiments, these new variants show superior performance in molecular conformation optimization comparing to existing SE(3)-equivariant neural networks. Moreover, our energy-inspired formulation is also suitable for molecular conformation generation, where we can generate more diverse and accurate conformers comparing to existing baselines.

Published as a conference paper at ICLR 2022
ENERGY-INSPIRED MOLECULAR CONFORMATION OPTIMIZATION
Jiaqi Guan"
Wesley Wei Qian"
University of Illinois at Urbana-Chaan -haamaana j diaqie @1 1al11nnooo waliiois.c.m edu
University of Illinois at Urbana- Chammpainnn weigi an301 1111111111 nnnii iis...... edu
Qiang Liu
Wei-Ying Ma
University of Texas at Austin iqiangles. utexas edu
AIR, Tsinghua University maweiyingeair: http://ngggair.sssnaaaaa..ee tsinghua. edu en
Jianzhu Ma
Jian Peng
Peking University
University of Illinois at Urbana-Champaign AIR, Tsinghua University HeliXon Limited j jianpeng@ compeng@iilinne iiliioisiiiiiiiii edu
Beijing Institute for General Artificial Intelligence ma jijazzu@kkku edu.kuueedu.comm
ABSTRACT
This paper studies important problem in computational chemistry: predicting molecule's 's spatial atom arrangements, or a molecular conformation. We propose a neural energy minimization formulation that casts the prediction problem into unrolled optimization process, where a neural network is parametrized to learn the gradient fields of an implicit conformational energy landscape. Assuming different forms of the underlying potential energy function, we can not only reinterpret and unify many the existing models but also derive new variants of SE3)veequivarian neural networks in principled manner. In our experiments, these new variants show superior performance in molecular conformation optimization comparing to existing -qqiivaatt neural networks. Moreover, our energy-inspired formulation is also suitable for molecular conformation generation, where we can generate more diverse and accurate conformers comparing to existing baselines.
INTRODUCTION
The 3D molecular conformation is one of the most important features in determining many physico- chemical and biological properties of a molecule. The molecule's 3D charge distribution and physical shape crucial when considering the steric constraints or electronic effects for chemical reactions
and interaction. Therefore, molecular conformers are widely adopted quantitative structure-activity relationships (QSAR) prediction (Verma et al., 2010) and drug discovery (Hawkins) (017). With growing interest in virtual drug screening (Zhavoronkov et al. 2019 McCloskey et al. 2020 Stokes al. 2020) and de novo drug design (De Cao & Kipi 2018
et al. 2018 Jin et al.
2018 Zhou et al. 2019 Mercado et al.) (020). it becomes more desirable to predict the molecular conformations both quickly and accurately such that 3D geometry features can be considered.
Deep learning techniques have recently been introduced to address this problem and demonstrated promising results. For instance, generative models (Hoffmann & Noč 2019 Simm Hernandez Lobato 2020: Xu et al. 2020) are proposed to first generate pairwise distances for atom pairs, and then infer the conformer from such matrix. Since the 3D structures are generated indirectly, the second step can be sensitive towards the eтror in estimated distances, and extra model capacity could be required to encode the redundancy (Hoffmann & Noé 2019). In the meantime, many SE(()-equivariant networks emerge to work with 3D roto-translation symmetrical inputs enabling direct manipulation of the Cartesian coordinates. One of the most recent successful applications
"Equal Contributions.

---

Published as a conference paper at ICLR 2022
of such models is the AlphaFold system for protein structure prediction (Jumper et al. 2021 Despite the many benefits, however, existing -q(3)arqu networks are either derived from complicated mathematical theory involving expensive coefficient calculations (Thomas et al. 2018 Anderson et al. 2019) Finzi et al. 2020; Fuchs et al. (020), or heuristically designed (Satorras et al. 2021
based on the intuition of message passing (Gilmer et al. 2017).
This work presents a new formulation that unrolls the molecular conformer prediction into a fixed number of optimization steps, where a neural network is parametrized to learn gradient fields of the implicit conformational energy landscape. Under such formulation, the model can refine a given conformer toward a more stable and energetically preferred state. Instead of relying on complex mathematical tools or designing the model heuristically, we propose novel inspired framework that explicitly connects an underlying enregy function with an -e(3)--quivaria model. We are able to not only provide new interpretations for some existing models but also derive new variants of SE(3)-equivariant model from the assumption of a conformational energy function, aligning model's architecture with its physical assumptions. Through extensive experiments, these new variants show superior performance in conformer optimization comparing to other SE(3)- equivariant models. Moreover, the proposed optimization method is also suitable for conformer generation and capable of generating diverse and accurate conformers.
In conclusion, the main contributions of this work include:
Formulating the conformer prediction problem as a multi-step unrolled optimization where a model learns the gradient field of the conformational energy landscape.
Proposing a novel framework that explicitly connects an underlying energy function with an SE(3)-equivarian neural network such that one can derive new models that align with a physical assumption, or interpret some of the existing models from an energy perspective.
Demonstrating competitive empirical performance for molecular conformation prediction in both the optimization and generation settings.
2
RELATED WORK
2.1 MOLECUALR CONFORMATION GENERATION
In conventional computational approaches for conformer generation, we observe an accuracy and efficiency trade-off where highly accurate ab initio methods are very time-consuming while efficient methods leveraging empirical force fields are not accurate enough (Unke et al., (021). J. To efficiently and accurately estimates molecular conformations, many learning-baat based methods have been proposed. For instance, models, such as VAE (Simm& & Hernandez- Lobato 2020), GAN (Hoffmann & Noě 2019), and continuous normalizing flow (Xu et al. [2020]. have all been developed predict molecular conformers by first generating processing step for 3D structure construction can be susceptible to the error, and model capacity could be wasted to encode the redundancy in such low-rank matrix (Hoffmann & Noē, 2019). Despite the many drawbacks of such methods, the distance matrix is difficult to circumvent because the models cannot capture the 3D noto-translation symmetry nature of a conformer.
intermediate pairwise atom distance matrix. However, the post-
Besides modeling the atom distances, another solution for conformer generation is to estimate the force field with machine learning methods. For instance, sGDML (Chmiela et al.) 2019) is capable of reproducing the molecular potential energy surface given a few reference conformations and generating compelling results with MD simulation. Although such model scales well with the molecule size, it requires retraining for every new molecules (Unke et al. 202T). Similarly but more generally, methods including |Behler& & Parrinello (2007); BBartok al. (2010) 2013): /mmith al.) (2017); Schütt et al. (2017); Faber et (2018) can also be used to generate molecular conformers with their learned force fields. Recently, ConfGF (Shi et al., 202T) specifically tackled the conformer generation problem by estimating the gradient fields of interca distances (similar to force fields), where the 3D coordinates of the conformer can be generated through steps of Langevin dynamics achieving ofte-offthh-art performance. While our model also tries to estimate the gradient field, we instead employ an unrolled optimization architecture (Domke 2012: Liu et al., 2018) aiming to model the entire optimization process instead of a single force field optimization step. Such one-past end-to-end design is proved to be more effective empirically as shown in the experiment section. In addition, our approach also enjoys much faster run time.
(Shütt
2

---

Published as a conference paper at ICLR 2022
Q)
Energy Surface
403
12
-23
184
10.7
Porce Fieid
Init Coondinctes
Goer deodes
Exbadding
Reletionchip Exbedilags
Neural
Energy Minimization
Srewedds decuall
Nourd Energy Mavesized Coandiootes
Neural Energy Minimization with the SE(3)- E-quivarrnnn Network
b)
Afun Embeiding Uplates
Goandinate Gradients
45
VE(
63
+0
+2
Opttesiation
62
Grodients Extieuted Neandl Networiks
05
64
-00
Figure 1: Method Overview. a) Our energy-inspired neural optimization formulation aims optimize the initial conformer a molecule towards the most stable and field approximated by neural network. b) To minimize the conformational energy, we can perform gradient descent for the 3D coordinates along the energy landscape, where the gradient is estimated by the model. To model the gradient fields, the neural network & considers the current 3D conformation, the atom representations, and the relationship embeddings. After layers of non- non-lincar transformations the model returns the gradient updates for the 3D coordinate along with the updated atom representations.
state, following an energy gradient
SE3iivvquivariant operations,
2.2 SE(3)-EQUIVARIANT NEURAL NETWORKS
Since 3D ato-transsltion symmetry is a common property for many physical systems, various meth- ods have been proposed to model such symmetry in problems like quantum mechanics (Ramakrishnan al.) 2014), molecular design (Simm al., 2020), particle simulations (Thomas 2018)). and point clouds (Uy et al. 2019). The ability to capture the spatial symmetry makes these models quite suitable for conformer optimization. In general, there are two ways to introduce the equivariance into a model. One thread of research relies on building models with irreducible representations and spherical harmonics (Thomas et al.) 2018 Anderson et al.) 2019 Finzi et al., 20200 Fuchs et al.. 22020) Batzner al. 2021 Different from this line of work, approach aims to avoid the complexity of such mathematical tools, considering the higher-order features in these more theoretical framework
might not be practical for our purpose. The other line of research relies on rector-based methods. where equivariant filters are constructed by taking the gradient of invariant filters (Schütt et al. 2021) or designed (Satorras al. 202T) based the intuition of message passing (Gimmer et al.) 2017). By explicitly connecting an underlying energy function with an SE(3)-equivarian neural network in our framework, we can not only reinterpret some of the existing models but also derive new variants that aligns the model parametrization with underlying physical assumption shown in Section 3.3.
3 METHODS
3.1 PROBLEM DEFINITION
We represent each molecule as a graph Gmol and use a set of 3D coordinates X to denote the conformer. The molecular graph is defined as Gmol = (V, E) where = {v,} is the set nodes representing atom features, and E { is the set of edges representing the relationship between atoms: including covalent interactions and distance of their shortest path. consists of the 3D coordinate {a, we want to learn an optimization model that refines the initial conformation towards an energetically preferred state X X arg min x E(X E( where F. is some energy surface of nature.
R
} of each atom. Given molecular graph Gmoi and initial conformer X
3

---

Published as a conference paper at ICLR 2022
3.2 CONFORMATIONAL ENERGY MINIMIZATION WITH A NEURAL NETWORK
Molecular conformation represents a 3D arrangement of the atoms and shares close relationship with the conformational energy, as cach equilibrium a molecular conformation can be considered as a local minimum of the potential energy surface (Axelrod & Gomez Bombarelli 2020). One could therefore find these stable conformations through gradient descent as a process of energy minimization. Specifically, given particular energy function E, we can optimize the conformation X by updating the 3D Cartesian coordinates with gradient descent such that E can be minimized:
(1) where y is the learning rate, and we will observe X = X - x (mal) for an energetically stable conformer. Therefore, we can easily find a stable molecular conformation from an initial conformation by applying the gradient descent update iteratively if the energy function E P(X(p mma) fully characterized and differentiable with respect X. Unfortunately, this often not the case.
X X >Vx F
Instead of hand-designing the energy function and its gradient, we can take advantage of the approxi- mation power of neural network and use it to learn the gradient field from the data directly. Once the network is trained, we can then unroll the optimization (Domke) 2012 Liu et al. 2018) with the model and perform a neural version of the above optimization where the gradient term VE in Equation is not derived from an explicit energy function but estimated by neural network.
More concretely, we parameterize the energy surface as a function of the molecular graph G = (V.E) and the 3D conformer X. We can consider X as an external 3-dimensional representation of the molecule, while and represent an internal high dimensional representation of the molecule. These internal embeddings V and & not only encode the initial atom/bond information but also embed information about their neighborhood related to the evolving conformation X. For each step, we want to minimize E(X,) V.E) with the following gradient updates:
++1 = x + (G' E
0 0,+1 of + "
= dh
for all æ; E X, v. E V, and eij E E, where & are neural networks estimating their respective gradient field, and the exact parametrization of the neural network will depend on the energy function as described in the following section. In this work, we view Esj as constant representation for simplicity, and therefore, is not updated in our models.
3.3 RECIPE FOR DERIVING ENERGY-INSPIRES -QUUIARRQNA MODELS
From the perspective of implicit energy minimization, can derive various models by making different assumptions of the target energy function, resulting in a principle architectural design that aligns with the underlying physical assumption. In Appendix A we show that model derived from an -inaaannn energy function (as they should be) always enjoys the SE(()--quivariance naturally, This section instantiates three variants of -quiivaquv networks as examples.
Basic Two-Atom Model We start with simple energy formulation where the conformation energy considers atom pairs independently:
Σ u(dy, Us Us. e.y)
i.jVvitt
where diy || æ, the euclidean distance between two atoms i and j while u(-)) is some unknown the potential energy function in nature. Following the formulation, we can then derive the gradient field of atom coordinates:
OF X, V,E)
au(d? U., eig)
u,, auty au Py v2 Uy eig) dx:
Σ jEV
Σ
Ox:
Ix:
Ad?
jEV
2(x;- -x,) duld?y U2 Ug. Cij)
Σ JEV
Σ
(a, - x, fz) Uy eiy)
Ad?
jEV
where far is a Transformer-bate neural network (Vaswani et al. du with a constant scalar. The optimization step for a two-atom model therefore becomes:
2017) approximating the gradient
field 0d
x
1-
+
Σ JEV
(
( 0%, 0/ cgy)
4

---

Published as a conference paper at ICLR 2022
and since fz is (E(3)-Ein)))))) invariant and (æ) -x))) is rotational equivariant, the optimization step Equation 3 observes SE(3)-qquivariance (proof in Appendix [A). The equivariance here is consistent with the invariant nature of the energy function, as the gradient field should rotate with the input coordinate. Similarly, we can derive the following update for the internal atom representation:
++I =
E JEV.YS
foldiig vi. 0 ely)
High-Order/ Three-Atom Model While start with energy function that only considers atom pairs, the energy function can also be extended to consider atom triplets allowing us to capture the potential energy related different bond angles:
E E(X, V.8)
Σ
u uld., cl (Tij, rsk), Viske Eigk)
where rúj = 2 and (. denotes the inner product of two vectors capturing bond angle Ziue-
The dependency on internal atom and edge embeddings is also extended to atom triplets Vijk {vs Vj, t4 and their corresponding relationship representations Eijk {eij, Eik, derivation in Appendix we observe the following updates for the three-atom model:
Eik, ezk J. Following
Σ
[( - =)) dia. (rlj, rlk)) ViJae Elj
(5)
+ (af xf)g dhss (rly, rads ViJa Elj)
where -quivveequv is also achieved. Similarly, have the following update function for the internal atom representation: b. of + + ,everrseie fu(diy, dik, (rl), rik) V E.
While we only extend the framework to atom triplets, one could extend the framework to an even higher-order model to capture the energy contribution for geometric structures like torsional angles.
Geometry of the Internal Representation In the basic two-atom model, derive direct update formula as Equation 4 atoms, and therefore, the proximity (P) between two n-dim vectors also contributes to the energy:
However, we can also view v as an internal spatial organization the
8,, = Σ
uldyy. Us Ug P(v,, Uy),
ijEV
where P is some distance metric for and Vy. For instance, using cosine similarity P(u, Uy) P3 as the distance metric, we arrive at the following update by taking the gradient of E w.r.t v;;
(+1 o/ + Σ
Soldd Us, vg P(u,, vs))
Vi |,
(dt ny U3 P( m,))]
(6)
JEV
Reinterpretation for existing models Besides new variants, idea of implicit energy minimiza- tion also allows to reinterpret some existing models. For instance, in EGNN (Satorras et al. 2222). the authors propose the following updates for atom "representations" v and "coordinates" æ::
o+ Φn(of, JEN())
Σ
Seldyy 0/,00, e(,)) ei))
x
x +
Σ jEV
(xf
23)02 (6 of u/ ei)))
From the new perspective, these are equivalent to optimizing an conformational energy function that depends both the atom representations and inter-atomic distances just like basic Two-Atom model, i.e. = Eijev. u(d,, v; Ui Csy). On the other hand, the "gradient" for their atom representations are only aggregated from the corresponding neighbors (as message passing (Gimmer a learned function
et al.) (017)) instead of all atoms (as ours). In addition, this formulation Φ to update the representation of instead of simply adding the negative gradient. the context of optimization, the operation can be interpreted optimizing with learned optimizer (Metz al.
5

---

Published as a conference paper at ICLR 2022
2019) instead of performing simple gradient descent. Additional reinterpretations for existing models can be found in Appendix. C
We believe the connection between the models and underlying energy functions represents new perspective for SE(3)-qqivvarian models and expect the community to derive more interesting variants by designing different energy functions with more suitable assumptions for their applications.
EXPERIMENTS
To evaluate the proposed methods, we perform experiments in two separated settings:
The optimization setting where the goal for the model is to take a 3D conformer as input and produce a single optimized conformer with the most stable energy. Since our model is proposed to solve an optimization task, this setting is direct measure of its performance against other -qqiivaqiaa models also capable of taking 3D Cartesian coordinates as input. Our results suggest that models derived from higher-order energy function can achieve better performance, and we believe such energy-inspired perspective would also help the community to develop more powerful extensions of requivaauv models in the future.
The generation setting where the goal is to generate multiple conformers that capture a diverse set of relatively stable conformers. This setting has been studied recently, even outside the context of model (Mansimov et al. 2019 Siimm & Hernandez Lobatto 2020 Xu et al. 2020 Shi al. 202T), and represents more realistic application scenario for practitioners. Our experiment suggests that the unrolled optimization formulation can be easily extended for the generative setting. Compared to existing baselines, our models can generate a more diverse and accurate ensemble of conformers, demonstrating realistic application scenario for the proposed formulation.
4.1 MOLECULAR CONFORMER OPTIMIZATION
Data We test molecular conformer optimization on the QM9 dataset with small molecules (up to 9 heavy atoms) (Ramakrishnan et al., (014) as well as the GEOM-Drugs dataset with larger molecules (up to 91 heavy atoms) (Axelrod & only one reference conformer generated through DFT calculation is considered as the lowest-energy conformer. For the GEOM-Drugs dataset, since multiple low-energy conformers are given through a semi-empirical method and after realizing larger molecules could have multiple similarly optimal conformers, we select multiple conformers with high Boltzmann weights as reference conformers (see Appendix for details). We randomly split the QM9 dataset into 123k, 5k, 5k, and the GHOMM-Drugs dataset into 270K, 10K, and 10K for respective training, validation, and testing.
2020). For the QM9 dataset,
Baselines Since the goal for this experiment is to show the energy-deriver model is indeed capable of optimizing an existing 3D conformer towards its most energy stable state, we compare our model with two state-of-the-ar -quivvaaaaa models: EGNN (Satorras al.) (202T) and SE(3)- Transformer (Fuchs et al. 2020) that can also take 3D Cartesian coordinates as input directly. For non-machine-learning method, we also include one classical approach RDKit + MMFF (Riniker & Landrum 2015), where the predict conformer is estimated through Euclidean Distance Geometry (EDG) and further optimized with Merck Molecular Force Field (MMFF) (Halgren 1996).
we include all three variants for this experiment: (1) Ours-TwoAtom basic two-atom model where we only consider pairwise atomic distances in the energy function. (2) Ours-Ext,, which extends the two-atom model by considering the geometry of the internal representation v in the energy function. (3) Ours-ThreeAtom. which further extends the two variants above by considering the atom triplets in the energy function. To train our models and other machine learning baselines, we optimize model parameters by minimizing the L.2 loss between pairwise distance matrices of the generated conformer and its closest reference conformer (see Appendix B for more model details). Taking advantage of the SE(3) baseline and our models, we train all the models to take the RDKit MMFF predicted conformers as input and allow the models to start with a reasonable estimation at a low computation cost. We perform same number of optimization step (=9) for all models and both datasets. The results for random initialization can also be found in Appendix H.I and the results for effects of the number of optimization steps can be found in Appendix K
Setup Following Sec. 3.3,
property of the

---

Published as a conference paper at ICLR 2022
Dataset
QM9
GEOM Drugs
Model
mean RMSD (1)
median RMSD (4)
mean RMSD (1)
median RMSD (1)
RDKi++MMFF
0.3872 + 0.0029
0.2756 + 0.0075
1.7913 + 0.0030
1.6433 0.0097
-BC3)-Tr, (Fuchs al. 2020
0.2476 = 0.0021
0.1657 = 0.0022
1.0050 + 0.0022
0.9139 = 0.0041
EGNN Satorras et al. 2021
0.2101 = 0.0009
0.1356 + 0.0013
1.0405 i 0.0018
0.9598 + 0.0038
Ours- Twos-m
0.1415 = 0.0004
0.0534 + 0.0002
0.8839 + 0.0014
0.7733 0.0026
Ours-Ext,
0.1383 0.0005
0.0505 + 0.0001
0.8691 + 0.0015
0.7535
0.0028
Ours- Atom
0.1374 = 0.0004
0.0522 + 0.0002
0.8567 1 0.0014
0.7192 0.0024
Table 1: Molecular conformer optimization the QM9 dataset and GEOM-Drugs dataset. The mean and median RMSDs (Unit Á) hetween reference and predicted conformers reported. The confidence interval is calculated by inference with ten different initialization from RDKit+MMFE, (1) denotes lower scores better. More studies on model ablations and variations can be found Appendix
Molecule
RDKit MMFF
EGNN
Ours
Reference
Figure Model-Predicted Conformers. Molecule & sampled from the QM9 dataset, and the larger molecule sampled from the GEOM Drugs dataset. The reference conformer, as well as the initial conformations from RDKi++MMFF, are also shown. See Appendix L for more examples and failure cases.
Results Following Mansimov et al. (2019), we measure the difference between predicted con- former and its reference by the Root-Mean Root-Mcan-SSqare Deviation (RMSD) for all the heavy atoms. As shown in Table all the optimization models can improve upon the initial conformation estimated by RDKit+MMPF, and new variants derived from the new framework also outperform the two lfte-of-the-an art derived with more complex and higher-order energy definitions also outperform the basic model that considers inter-atomic distance. In the GEOM-Drugs dataset, where have large molecules with many rotatable bonds, we hypothesize the higher-order model can outperform the basic two-atom model because the corresponding higher-order energy function can better approximate the actual potential energy surface by considering the bond-angle explicitly. Since we only consider three variants in this work and they all have shown great performance, look forward other more powerful variants the community can develop with such perspective.
models with clear margin. More interestingly, model variants
In Figure examples mreeecpredicted conformers are visualized along with their initial conformers from
and the reference conformers in their optimal energy state. The visualization shows that our models can produce conformers that are not only realistic but also energetically stable. With intentional design, model- preei--redicted conformers components quite well (c.g., benzene forms a ring in the same plane). For larger molecules where the initial conformers (RDKit+MMFF) not folded/rotated in the most stable fashion, model can also optimize the conformers towards the stable state as reference conformers.
also capture the structure of the basic
To measure the quality of the optimized conformers beyond the RMSD metric, evaluate two downstream applications with the QM9 test set conformers predicted by different methods. In the first study, compare the HOMO and LUMO energy (Smith et al. 2020) of molecule when calculated
7

---

Published as a conference paper at ICLR 2022
HOMD-LUMO Gap
PearsonR=0.802
1.00
1.00
1.50
Reference Conformer
Reference Conformer
0.75
0.75
0.73
0.25
0.25
0.2
0.00
0.00
0.00
Predicted
Figure 3: HOMO-LUMO gap calculated with reference conformers ones calculated with the predicted
conformers. The black dotted line represents the identical values. Four hundred molecules sampled.
Task
<x bohr
CHOMO meV
CLUMO meV
C₂ cal/mol
Conformer
meV
D
Reference
0.235
63
41
34
0.033
0.033
RDKF++MMIF
0.277
112
83
76
0.394
0.110
MGGNN-Predicter
0.278
115
91
70
0.328
0.111
Ours- Murs-Predicted
0.254
0.274
0.099
Table 2: SchNet (Schütt 2017 model performance (MAE) on QM9 property prediction when trained and evaluated with predicted conformers instead the reference conformers. The best performance is bolded.
with the reference conformers vs. the model-predicted conformers. As shown in Figure 3 we find the HOMO-LUMO gap calculated from our model-ppeeicceer conformer match well with the one from reference, while the correlation between the reference and force field optimized conformers is much lower. In the second study, we train standard neural network model, SchNet (Schütt et al.) 2017) for the QM9 benchmark (Ramakrishnan et al. 2014) with the reference predieted conformers. As shown in Table 2] the model trained with our predicted conformers can outperform the ones trained with other predicted conformers. The result indicates better inductive bias in our predicted 3D structures compared to the ones estimated using other algorithms. Even though there is still a performance gap when comparing results with the model trained with the reference conformers, believe the model-predicted conformer could still be helpful for datasets where no references are not available. More experiment details can be found in Appendix
4.2
MOLECULAR CONFORMER GENERATION
Data
We test molecular conformer generation on both GEOM-QME and GEOM- GEOM-Drugs datasets
(Axelrod & Gomez- Bombarelli 2020) following the same data split as Shi al. (222 including
40k training molecules and 200 test molecules for both datasets (see Appendix D for details).
We compare our model against an EDG-based approach RDKit (Riniker & Landrum 2015) as well as four deep generative models specifically studied for the generation setting: CVGAE (Mansimov et al., 2019) is conditional VAE that directly generates atom coordinates based on molecular graphs. GraphDG (Simm & Hernandez-Lobato 2020) and CGCF (Xu et al.) 2020) are VAE- and flow-based method that only generate the pairwise distances for 3D conformer conversion. ConfGF (Shi et al. 2021) is a recently-publicn
Baselines
state-of- the-art model that learns the gradient field for molecular conformations and perform generation through iterative Langevin sampling.
Setup In order to test our optimization formulation for the generative setting, we adopt the same two-atom model from the optimization setting here with the same hyper-parameters. However, instead of training the model to optimize a single initialization towards one reference, we train the model to optimize K initialization such that these K optimized conformers can be exactly matched to K different sampled references. More concretely, we optimize the model parameters against an optimal optimal-transport like loss similar to other generative models (Genevay al. 2018
Lot min Σ T&J C(X;,X,) xeI
8

---

Published as a conference paper at ICLR 2022
Datasct
GEOM-QM9
GEOM-Drugs
(OV(S)) (t)
MIS(S) (J)
MAT(Á) (U)
COV(S)) (1)
MIS(()) (U)
MAT(A) (1)
Metric
Mean
Median
Mean
Median
Mean
Median
Mean
Median
Mcan
Median
Meun
Median
RDKit
83.26
90.78
8.13
1.00
0.3447
0.2935
60.91
65,70
27.95
12.07
1.2026
1.1252
CVGAE
0.09
0.00
1.6713
1.6088
0.00
0.00
3.0702
2.9937
GraphDG
73.33
84.21
56.09
64.66
0.4245
0.3973
8.27
0.00
97.92
100.00
1.9722
1.9845
CGCF
78.05
82.48
63.31
64.66
0,4219
0,3900
53.96
57.06
78.32
86.28
1.2487
1.2247
ConfGF
88.49
94.13
53.56
56.59
0.2673
0.2685
62.15
70.93
76.58
84.48
1.1629
1.1596
Ours Random
88.83
93.18
30.21
30.74
0.3778
0.3736
76.50
83.78
31.40
23.03
1.0694
1.0583
DDist
86.68
91.34
5.46
0.00
0.2667
0.2125
67.72
75.30
21.52
2.66
1.0739
1,0372
Table 3: Molecular conformer generation for the GEOM-QM9 and GEOM- GEOM-Drugs dataset. COVerage score reports the percentage of reference conformers that are produced by the predicted ensemble. MISmatching score reports the percentage of generated conformers that can not be matched with any reference conformer. MATching score reports the minimum RMSD between generated conformer and the references. (1)/(t) denotes a metric for which lower/higher scores are better. The best performance is bolded.
where C(X; Xj) is the same L.2 loss between the distance matrices of the reference (X;) and generated (X3) conformer, while the optimal transport plan и E {0, K is realized as minimum weight bipartite graph matching and I is the set of all permutations. For the initial conformer, we again take advantage of the -equiaaquia of our method and evaluate the generation performance with initialization from either a random set of coordinates (i.e., Ours-Randon model) or a rough RDKit estimate (i.e., Ours-RDKit model, whose training inputs are perturbed references).
Results To evaluate the diversity, precision, and structural quality of generated conformers, adopt the corresponding COVerage, MISmatching, and MATching scores following Xu et al. (2020) SndSSh et al. (202T) as defined in Appendix G As shown in Table. ] our models achieve competitive performance in all three metries, especially for the more challenging GEOM-Drugs dataset where we outperform existing machine learning methods by a clear margin. These results suggest that our optimization formulation can be easily extended to the generation setting for diverse and accurate ensemble of conformers. Ablation study in Appendix[aale also suggests the improvement is theeevemmeeee setup and the optimal transport loss. Since no Langevin
mainly contributed by the SE(())-quivarrig sampling is required in our method, the inference time (<ls per molecule) for our method is also much faster than ConfGF (~~ 10ss per molecule). We also noted the performance difference between random initialization and RDKit initialization also represents a classical trade-off between diversity and accuracy, where RDKit initialization can achieve better accuracy (MAT/ / MIS) by providing a more accurate starting point, while a random initialization can achieve better coverage (COV) by sufficient sampling of the space. We left more interesting initialization (e.g. mixture of the two) for future work, and please consult Appendix G for details of the implementations.
CONCLUSION AND DISCUSSION
This paper proposes a new formulation for molecular conformer prediction by parametrizing an -equiv-quva network to model the gradient field of the conformational energy landscape. By connecting an SE(3)-equivariant model with an underlying energy function, we also found new perspective to principally derive new variants of -Equivaquv models that align with an explicit set of underlying assumptions for the physical system. Through an extensive set of experiments, we show that the proposed method is capable of optimizing a given conformer towards its most energetic- stable state and generating an ensemble of diverse and accurate conformers efficiently. We believe such energy-insee energy-inspired perspective also represents a new direction for thinking about -Euivvaiqu model and expect the community to derive more interesting variants with energy assumptions that are more suitable for their applications.
For future work, our interest also goes beyond conformations of small molecules, as we can further scale up the system to model larger bio-molecular systems such as protein-ligand complexes. In addition, it would be interesting to explore models like DEQ (Bai et al. 2019) where the equilibrium is modeled directly. Since our training objective is blind to the chirality, our system will need to infer the chirality post model generation, and it remains an interesting question that whether we can build the system with chirality built-in (Pattanaik et al. 2020). Last but not least, with the ability to quickly generate high-quality ensembles of molecular conformers, we are also interested in molecular machine learning with conformer ensembles (Axelrod & Gómez-Bombarelli 2020).

---

Published as a conference paper at ICLR 2022
Reproducibility Statements The model implementation, experimental data and model checkpoints can be found here: bttps://github.co
wttp:/uwwoocoouuuuuuaaeeeoo.auue com guan wwwp//////aaa/aaaaaaaaaoc /ttpa//aao/oooooooooooooooo officiiat_fffiall
Acknowledgement We thank all the reviewers for their feedbacks through out the review cycles of the manuscript. We also thank Yuanyi Zhong and Yunan Luo for many helpful discussions. This work was supported by U.S. National Science Foundation under grant no. 2019897 and U.S. Department of Energy award DE-SC001 8420. J.P. acknowledges the support from the Sloan Research Fellowship and the NSF CAREER Award.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Brandon M. Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In NeurlPS, 2019.
Simon Axelrod and Rafael bles. ArXiv, abs/2012.08452, 2020.
Molecular machine learning with conformer ensem-
Simon Axelrod and Rafael Bomezz-ommarrlli. Geom: Energy-annotated molecular conformations for property prediction and molecular generation. arXiv preprint arXiv:2006.0553/ 2020.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural
Information Processing Systems, 32, 2019.
Albert Bartók, Mike C Payne, Risi Kondor, and Gábor Csányi. Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons. Physical review letters, 104(13): 136403, 2010.
Albert P Bartók, Risi Kondor, and Gábor Csányi. On representing chemical environments. Physical Review B, 87(18):184115, 2013.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. Se equvivarai graph neural networks for data-efficient and accurate interatomic potentials. arXiv preprint arXiv:2101.03164 2021.
Jörg Behler and Michele Parrinello. Generalized neural-network representation of high-dimensional potential-energy surfaces. Physical review letters, 98(14); 146401, 2007.
Stefan Chmiela, Huziel E Sauceda, Igor Poltavsky, Klaus-Roher Müller, and Alexandre Tkatchenko. sgdml: Constructing accurate and data efficient molecular force fields using machine learning. Computer Physics Communications, 240:38-45, 2019.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs arXiv preprint arXiv: 11973, 2018.
Justin Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics, pp. 318-326. PMIRR, 2012.
Felix A Faber, Anders S Christensen, Bing Huang, and O Anatole Von Lilienfeld. Alchemical and structural distribution based representation for universal quantum machine learning. The Journal of chemical physics, 148(24):241717, 2018.
Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning, pp. 3165-3176. PMLR, 2020.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 33, 2020.
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn diver- gences. In AISTATS, 2018.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pp. 1263-1272. PMLR, 2017.
Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández- Lobato, Benjamín Sánchez-L Lengeling, Dennis Sheberla, Jorge pigiilra-pparrgguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268-21) 22666 2018.
Thomas Halgren. Merck molecular force field. basis, form, scope, parameterization, and performance of mmf194. Journal of computational chemistry, 17(5-6):40-519, 1996.
11

---

Published as a conference paper at ICLR 2022
Paul CD Hawkins. Conformation generation: the state of the art. Journal of Chemical Information and Modeling, 57(8):1747-1756, 2017.
Matthew Hirn, Stéphane Mallat, and Nicolas Poilvert. Wavelet scattering regression of quantum chemical energies. Multiscale Modeling Simulation, 15(2):827-86) 863, 2017.
Moritz Hoffmann and Frank Noé. Generating valid euclidean distance matrices. arXiv preprint arXiv:1910.03333, 2019.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International Conference Machine Learning, pp. 2323-2332. PMLR, 2018.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool. Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58) 2021.
Isay Katsman, Aaron Lou, Derek Lim, Qingxuan Jiang, Ser-Nam Lim, and Christopher De Sa. Equivariant manifold flows. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/ 412.69880 2015.
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: sampling configurations for multi-body systems with symmetric energies. arXiv preprint 19110:191, 2019.
Risheng Liu, Shichao Cheng, Xiaokun Liu, Long Ma, Xin Fan, and Zhongxuan Luo. A bridging framework for model optimization and deep propagation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp.
2018.
Elman Mansimov, Omar Mahmood, Seokho Kang, and Kyunghyun Cho. Molecular geometry prediction using a deep generative graph neural network. Scientific reports, 901):1-13, 2019.
Kevin McCloskey, Eric A Sigel, Steven Kearnes, Ling Xue, Xia Tian, Dennis Moecia, Diana Gikunju, Sana Bazzaz, Betty Chan, Matthew A Clark, et al. Machine learning on dna-encoded libraries: new paradigm for hit finding. Journal of Medicinal Chemistry, 63((6))8857-8866, 2020.
Rocío Mercado, Tobias Rastemo, Edvard Lindelöf, Günter Klambauer, Ola Engkvist, Hongming Chen, and Esben Jannik Bjerrum. Graph networks for molecular design. Machine Learning: Science and Technology, 2020.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pp. 4556-4565. PMLR, 2019.
Benjamin Kurt Miller, Mario Geiger, Tess E Smidt, and Frank Noé. Relevance of rotationally cquivariant convolutions for predicting molecular properties. arXiv preprint arXiv:2008. 08461, 2020.
Lagnajit Pattanaik, Octavian-Euge Eugen Ganea, Ian Coley, Klavs F Jensen, William H Green, and Connor W Coley. Message passing networks for molecules with tetrahedral chirality. arXiv preprint arXiv:20/2.00094 2020.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7,2014,
Sereina Riniker and Gregory Landrum. Better informed distance geometry: using what we know to improve conformation generation. Journal of chemical information and modeling, 55(12): 2562-2574, 2015.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. (n) equivariant graph neural networks. In International Conference on Machine Learning. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2022
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Rober Müller. Schnet: continuous-fin convolutional neural network for modeling quantum interactions. In NIPS, 2017.
Kristof T Schütt, Oliver T Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. arXiv preprint arXiv:2102.03150, 2021.
Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In International Conference on Machine Learning. 2021.
Gregor Simm and Jose Miguel Hernandez-Lobato. A generative model for molecular distance geometry. In International Conference on Machine Learning, pp. 8949-89558 PMLR, 2020.
Gregor NC Simm, Robert Pinsler, Gábor Csányi, and José Miguel Hemándee--Lobao Symmetry- aware actor-critic for 3d molecular design. In International Conference on Learning Representa- tions, 2020.
Daniel GA Smith, Lori Burns, Andrew C Simmonett, Robert Parrish, Matthew Schieber, Raimondas Galvelis, Peter Kraus, Holger Kruse, Roberto Di Remigio, Asem Alenaizan, al. Psi4 1.4: Open-source software for high-throughput quantum chemistry. The Journal of chemical physics, 152(18):184108 2020.
Justin S Smith, Olexandr Isayev, and Adrian E Roitberg. Ani-1: extensible neural network potential with dft accuracy at force field computational cost. Chemical science, 8(4:3312,-3203, 2017.
Jonathan M Stokes, Kevin Yang. Kyle Swanson, Wengong Jin, Andres Cubillos-Ruis Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar deep learning approach to antibiotic discovery. Cell, 180(4):688-702, 2020.
et al. A
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li, Kai Kohlhoff, and Patrick Riley. neural networks for 3d point clouds.
Tensor field networks: Rotation-an arXiv preprint arXiv:180.0082111 2018.
Oliver T Unke, Stefan Chmiela, Huziel E Sauceda, Michael Gastegger, Igor Poltavsky, Kristof T Schutt, Alexandre Tkatchenko, and Klaus-Robert Muller. Machine learning force fields. Chemical Reviews, 2021.
Mikaela Angelina Uy, Quang- Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: new benchmark dataset and classification model on real- world data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Jitender Verma, Vijay M Khedkar, and Evans C Coutinho. 3d-qsar in drug design-a review. Current topics medicinal chemistry, 10(1):95-115, 2010.
Arthur G Wasserman. Equivariant differential topology. Topology, 8(2):127-150, 1969.
Minkai Xu, Shitong Luo, Yoshua Bengio, Jian Peng, and Jian Tang. Learning neural generative dynamics for molecular conformation generation. In International Conference Learning Representations, 2020.
Alex Zhavoronkov, Yan A Ivanenkov, Alex Aliper, Mark S Veselov, Vladimir A Aladinskiy, Anas- tasiya Aladinskaya, Victor Terentiev, Daniil A Polykovskiy, Maksim Kuznetsov, Arip Asadulaev, et al. Deep learning enables rapid identification of potent ddr1 kinase inhibitors. Nature biotechnology, 37(9):
7799)1036 (038-1040 2019.
Zhenpeng Zhou, Steven Kearnes, Li Li, Richard Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific reports, 9()):1100, 2019.
13

---

Published as a conference paper at ICLR 2022
A PROOF OF SE(3)-EQUIVARIANG
A. PROOF OF SEUUAR-EQUV FOR THE TWO-ATOM MODEL
One of the crucial challenges for modeling molecular conformers to capture the 3D roto-translation symmetry, rotating moving the conformer in the 3D space on its would not change the conformational energy. Therefore, to capture such symmetry for better generalization (Köhler et al. 2019), our model required to change its output accordingly if the input conformer X undergoes transformation of rotation or/and translation in the 3D space. More concretely, the model needs to satisfy SE(3) -qquivarrai:
(T((X) (()()))
where T X X is a transformation on X for the SE(3) group. Specifically, for each 3D coordinate æ, we have (ax) = Ax + where E R3x3 is the rotation matrix and b E RR the translation vector. In the rest of the section, we simply denote 2 as T(x).
Now that we have defined SE -quuvviqui,, we want to prove that the following coordinate update in Eq. equation 4 satisfies -Equivaqui
x +++
( = x +
Σ (ei-xx))e 2i vjj, eig) JEV
and we will do so by first showing that (-) is SE(3)-invariant. To show fa(-) is SE(3)-invariant, it sufficient to show that its various inputs diy, Uss and eij are SE(3)-invariant, Intuitively, di should not change with the 3D roto- roto-translation T, but formally we have:
i | 2 = || (Ax, + b) (Ax, + b)1l |
Ax, 1
= - -
A A(x, xσ) = (z: xzy) I(x: xz) = || 1, di
Since the initialization and update of in Equation only depend on variables satisfying SE(3)- invariant, v is also SEv3)-invrria Similarly, the relationship embedding is also SE(3)-invariant. Therefore, function fz satisfies SE(3)-invariant.
Now, if we look at the update for ((T(x'))
)) or o(z"), we have:
o(T(x)) = x² +
Σ ( - =f)
JEV
= x' + Σ ( (55-2))fuh f)+()
jEV,yn
(Axx + b)
Σ
((Ax) b) (Axt) b]) fz(.)
jEVitA
=AAx' +b)
Σ
A(a! - x7)(a(()
JEV,N
A x² + Σ
o of +
jEV.
Ad(wx) +
T(ő(x'))
Thus, the two-atom model update is SE(3)-equivariant
A.2 PROOF OF EEUTARRTUT FOR THE GENERAL CASE
For the more general case, we have the following equivariance theorem:
Theorem 1. Let G be SE(3)-gron which acts R If potential function + Rax3 + R a G-imvariant function (i.e. the assumed energy function E shown in Equation 7 is G G-invariant), then the gradient vector field V. D will be G-equivariant.
14

---

Published as a conference paper at ICLR 2022
Proof. For rotation transformation 4ig E G with rotation matrix Rg. have the following & is SE(3)-invariant: (2 4) = v. Now, according to the chain rule, we also have the following: Vo(Rg o ④) = (VaR (VR,* = Ry (VR, Now putting the two equations together, and multiply the orthogonal matrix R, on both sides, we have: Rg (x) = Rg o R2J V R,, ( (Rgr)= (R, Rg Similarly, for a transnational transformation g(x) = x I, one could show that V t) (x)
SE(3)-invariant. The combination of these achieve the -quuiiaqniaa of the update formula:
(Ro + 1) V R + (R + = (x v,D(z)) + I
ie.f o gg = Tg o f, where f is our neural network and T3 is the SE(3) transformation.
This shows that if the constructed energy function E is to guarantee that the gradient field VE (and estimated neural network) -quivvaanin The theorem can be generalized to any group G utilizing the Riemannian geometry. We refer Wasserman (1969) and Katsman al. (2021) to readers of interest.
as should be, it is sufficient
B DERIVATION OF THE THREE-ATOM MODEL
For the three-atom model, we have the following energy formulation:
E V,8)=
Σ
u uldty di. ra), Vijke Eggh)
and following the energy function, we can derive the following gradient update w.r.t. x::
dE X, V,
Bu(de, di (ng, rsk), Vesk, Eijk)
Σ
Ox,
Ox;
adl du(-)
+ ddh, Ou(-)
a(ri). au(-)
Σ
Oa: Jely
oa. 0dz.
8x:
Orrig, ras)
[1x: x.j) 20u(.)
+ (a:: az) 20u(.)
θu(-)
Σ
+ (2xx; 2z xx.)
ddl
a(reg rik)
[10; - (0r1-,,(( 20u(()
Oul.)
+ (ar, - -x,,)( 20uu())
Ou(-))
Σ
+ H(ris, rak)
)]
Odk
Arrig. r(k)
ddl
Σ
[x; - 2z (-) + (æ: xt xz)9(())
and therefore, we have the following updates for the three-aton models:
Σ
[ xz)f (d. dia (rlys rle), ViJae Elyx)
J.
+ (x) - x1)9 (diy. dha (rly. rich Vigk E1
C REINTERPRETATIONS OF EXISTING MODELS
C.I EQUIVARIANT BOLTZMANN GENERATOR
Equivariant Boltzmann Generator Köhler et al. 2019) introduces an E(n)-equivariant update for æ; in the following form:
++ x +
Σ (Φf - (()k))
jEV,ifj
where á is Multilayer Perceptron (MLP). It explicitly optimizes the energy function E = ijv which can be interpreted as simplified case of the energy function proposed in Equation since it only concerns about the distance between two nodes and disregards the information associated with the nodes and edges.
15

---

Published as a conference paper at ICLR 2022
C.2 TFN AND SE(3)-TRANSFORNE)
TFN (Thomas et al., 2018 and SE Transformer (Fuchs et al. 2020), the attention-based extension of TFN, are a family of equivariant model that performs the update for a representation Ø; with learned kernel utilizing spherical harmonics such that the model is equivariant with respect to æş;
(u( +
Σ JEV.53
00;-2, x., (4))
where 9i is simple projection matrix, and Pn,- -a; kernel matrix that learns project æ; based on (a, 2z)). Since kernel Pa, Pay-w; consists of a learned radial function for the distance d, and an angular preserve basis for = this formulation is in close relationship with Equation d.y
as the angular preserve basis also captures the term (a: -xty) while the learnable radial function equivalent the gradient estimator fe. Therefore, such operation can also be interpreted as optimizing the same energy function of our two-atom model, but with higher-order equivariant
representations involving special designed basis. Last but not least, since the optimized æ can only be obtained at the last step through linear projection, the whole model is effectively performance only an one-step optimization.
D DATA PREPROCESSING DETAILS
D.1 REFERENCE SELECTION
There are three datasets we used in the experiment section: QM9, GEOM-QM9, and GEOM-Drugs.
For the molecular conformation optimization task, choose because the conformers of it are computed with high fidelity DFT method. For the GEOM-Drugs dataset computed with semi-empirical method, they include multiple low-energy stable conformers. If we compare the Boltzmann weights (proxy of energy stability) for the most likely conformers and the second likely conformers, we find many of them are pretty close, as shown in Figure [ST)). In Figure STp)) we also showed an example where two conformers with very similar Boltzmann weight but drastically different conformation. Therefore, for GEOM-Drugs in the optimization setting, we use multiple reference conformers during the training and evaluation for GEOM- -rugg,, but the model predicted conformers only need to match one of the references. In order to select similarly energetically preferred conformers, select the set of target conformers {X Px 0.5 * Px where Px. is the Boltzmann weight for the most likely conformers.
QM9 instead of GEOM-QM9
For the molecular conformation generation task, dataset since they contain multiple references for each molecule. However, to filter out some of the rare conformers, of the sum Boltzmann weight 0.95 in the training set: 1 Σn Pxx z 0.95, E Pxx < 0.95, Px, 2 ... 2 Px J. but still report metrics on all reference conformers as previous work does.
both GEOM-QM9 and GEOM- Drugg the
first sort reference conformers by their Boltzmann weights and
threshold
D.2 FEATURIZATION
We construct the molecular graph G as a fully-connected graph to represent the molecules where the nodes are featured with atom type, aromatic, hybridization, and the number of hydrogens, while the edges are featured with the bond type (including no-bond), length of the shortest path in the graph, and whether the atom pair at both ends in the same conjugated system. We perform initial feature projection with e-update in each model to get the atom embedding and relationship embedding from these features. Similar other works (Mansimov et al. 2019 Xu et al. 2020). we only model the heavy atoms for the molecular graph and contormer in this experiment.
E EXPERIMENT DETAILS FOR MOLECULAR CONFORMATION OPTIMIZATION
E. MODEL PARAMETRIZATION
(Fuchs et al. 2020 is parametrized with the following hyper-parameter,
which maximize the GPU memory: num degrees=2, num num_layers=9, hidden dim=66 64, and finally n he
16

---

Published as a conference paper at ICLR 2022
b)
Comparing Boltzmann Weights for Multt Conf--onnoomrrrs
Churt
40000
20000
Payc
Pist 0.331
0.329
Figure S1: Boltzmann Weights Conformers in GEOM-Drugs Dataset. For each molecule the GEOM- Drugs dataset, multiple conformers are generated with a Boltzmann weight associated with them, representing their energy stability. a) We divide the Boltzmann weight of the most likely conformer with the one of the second likely conformer. The distribution of the ratio shown, and many of the conformers have at least two top conformers that share similar Boltzmann weight. b) We show example where conformers very different shapes could share very similar Boltzmann weight.
EGNN (Satorras et al. 2021 is parametrized with the following hyper-parameters: num num_layers=,, hi dden_dimm2555 dim-256, and the distance square features are expanded using an RBF kernel with 50 basis between OÅ 10Å.
Ours -TwoAtom and -Ext, contain three optimization blocks sharing the parameters, and within each optimization block, there are three layers of x and updates. The neural network in these updates are featured as a Transformer (Vaswami al., 2017) with hidden hidden_didneeeeeeeeeen dim=25 and n heads=sd6 The key/value embedding and attention score are generated through 2-layer MLP with LayerNorm and ReI.U. addition, an additional interaction weight is introduced to model different energy contribution of the atom pairs which is simply modeled duj sigmoid(MLP (MLP(dd.))) Last but not least, we implement a dynamic cutoff where we ignore the energy contribution between atom pairs whose euclidean distance is less than I0Å. Similar to the EGNN implementation above, distance features are also expanded with the same RBF kernel. Figure S2
shows the overall architecture.
a)
b)
C)
X
Optimization
X Optimization
Embedding
Vi
dij eij
X;
V
VJ
d
eij
F&
V, E
Query
Key
Value
Query
Key
Value
Connectivity
Product
Op timiiationn For
Multi-Head Attention
Optimization
Multi-Head Attention
Avi
Axi
Loyer
Add
Add
Block
X;
Figure S2: Model Architecture for Ours-TwoAtom. a) Overall architecture with one optimization block consisting multiple optimization layer. b) Optimization for where diy ||æ, x; ||. c) Optimization for where r. 2, Zz.
is three-atom model extension of Ours-Ext,, and therefore follow the same model hyper-paun parameters as above. Since we consider the bond angles Zisk and Zaij the same, the neural networks Sa, Ir and fe need to satisfy permutation invariant between and ik. Therefore,
17

---

Published as a conference paper at ICLR 2022
for these three functions, construct them as the following:
f = [d(dij das, (Tij. rak), Vijk, Eigk) + [(dx, dajs (ras, riy). Viks,Eaks)] //2
In addition, we generalize the three-atom model the two-atom case by including the triplet (i. k) where k. While treating the molecule as a fully connected graph allows us to model the non-loca relationship, the challenge is that the number of atom triplets grows very quickly with the size of the molecule. Since the bond angle in a conjugated ring system with no rotatable bond is not super interesting, we trim down the number of atom triplets by only considering (i, j. k) triplet where esj or ei represents rotatable bond.
E.2 TRAINING PROCEDURES
Our proposed models and other machine learning baselines are all trained in the same procedure.
We sample a batch size of 128 molecules for the QM9 dataset and batch size 32 molecules for the GEOM-Drugs dataset during the training. In one training batch, one initial conformer is randomly sampled for each molecule from a pool of ten pre-initialized conformers from RDKi++MMFE, Since we expect the model to stop the optimization when the input conformer is already stable, we also sample reference conformer as the initial input conformer with a probability p = 0.05.
The model is trained end-to-end by the L2 loss between the distance matrices for the predicted conformer and the reference conformer via gradient descent method Adam Kingma & Ba (2013). The hyper-paramen settings for the optimizer O 999), , and grappggrrneen........ _noom=....... We also schedule to decay the learning rate exponen- tially with factor of 0.5 and minimum learning rate of 0.00001. The learning there is no improvement for the validation loss in 8 consecutive evaluations, and the training will be completely stopped if no improvement is found for 20 consecutive evaluations. The evaluation is done for every 2000 training steps/batches. We tried a small number of learning rate schedules based on the validation set make sure the training sufficient. Gradient norm is clipped make the training stable. Other
init _att___eannnng_ratte_...... 001, be tas (0. 95, Batgg_rrtee00_0
is decayed if
are set following the common choices.
E.3 COMPUTATION RESOURCE AND TRAINING TIME
We train all conformer optimization models for the QM9 dataset with one NVIDIA GeForce GTX 1080 GPU and be trained within 36-48 hours except the Turs-ThrecAtom model, which takes longer time.
GeForce GTX 3090 GPU for the GEOM-Drugs dataset, All the models can
E.4 CODE AND DATA AVAILABILITY
We train all conformer optimization models for the QM9 dataset with one NVIDIA GeForce GTX 1080 GPU and one GeForce GTX 3090 GPU for the GEOM-Drugs dataset. All the models can be trained within 36-48 hours except the Ours- ThreeAtom model, which takes longer.
F EXPERIMENT DETAILS FOR DOWNSTREAM APPLICATIONS
F.1 CORRELATION FOR THE HOMO-LUMO GAP
For this study, sample 400 molecules from the test dataset of the conformer optimization experi- ment (Sec. 4..1 for the quantum mechanic calculation. Since the dataset and the model is heavy atom only, infer the coordinate of hydrogens via RDKit (Riniker & Landrum 2015). The HOMO and LUMO energy for the reference and two-atom model predicted conformers are finally calculated via Psi4 (Smith al. 2020) with the MP2/6-31 111-6))))++((.p))) basis set.
F.2 TRAINING NEURAL NETWORKS WITH PREDICTED CONFORMERS
In this study, we train the standard SchNet (Shhitt et al. 2017) on the common QM9 property prediction benchmark with the data and splits provided by Anderson et al. (2019). However, we want to train and evaluate the model with the generated conformers instead of using the reference conformers. Since SchNet (Schütt et al. 2017) requires the hydrogen atom for its full potential and the benchmark dataset follows different split conformer optimization task, train
18

---

Published as a conference paper at ICLR 2022
two-atom model the training split of this benchmark with the hydrogen atom included and use this new model to generate conformers.
In our study, SchNet contains 7 layers and a hidden dimension of 128. To expand the distance feature, we use 50 Gaussian basis, and construct the molecular graph with a distance cutoff of 10 A. We use L1 Loss to train the model end to end with Adam (Kingma & Ba 2015) batch si 2e=96, and num epochs=100)).. The learning rate starts between 5e-4 and 1e-6 for different tasks and follows a cosine decay.
6,
Since the single reference conformer does not constrain us, we consider multiple conformers in this study. RDKit+MMFF first estimates multiple imitial conformers, and different predicted conformers generated from these initial conformers with our two-atom model and the baseline EGNN model. To select for these initial conformers, we first sample 10 conformers with RDKi++MMFF, and the top 5 conformers with the lowest energy are selected. One of the five conformers sampled randomly at each batch during training, but the average prediction from the five conformers is reported as the model estimate during inference.
G EXPERIMENT DETAILS FOR MOLECULAR CONFORMATION GENERATION
We follow the setting in Shi al. (202 and report baseline results from their paper. The metrics we used are defined as follows:
COV(S,, S,) 1
(X* S,/RMSD(X, x S X E Sy))
S
1
MTS(Sg, = |Sg]
|[X E Sy RMRD(MDXX,,, X*) > X E S.
1
MAT(S,, 5,) = |S X
min RMSD(X, X') XES,
where RMSD(X, X') min+ E -; n is the number of heavy atoms and O an alignment function which aligns two conformers with rotational and translational operations. The thresholds of COV and MIS are 5 = 0.5 and 5 = 1.25 for GEOM-QM9 and the larger GHOM-Drugs respectively, which are same to Shi et al. (202T).
We used the same setup as the two-atom model in the conformation optimization setting with the same saper-parameters for this experiment. Tor train the model, we use the optimal transport loss during the training described in Eq. a choose K = 5. In Table. 3 the conformers in both training and evaluation phases of Ours-Randon are initialized with random noise, which is drawn from Gaussian (00 σ²) and 0.028 |V]. On the other hand, Ours- are initialized with permuted reference conformers during training, where the permutation is drawn from a Gaussian (00 r and 0.5 for GEOM QM9 and 1.0 for GEOM-Drrus account for the average volume difference. In evaluation, we apply the model with initialization from RDKit since ground truth reference is not available.
H ADDITIONAL EXPERIMENTS
H.1 CONFORMER OPTIMIZATION WITH RANDOM INITIALIZATION
While our optimization experiments are done with an initial conformer estimated by RDKi++MMFE. the initial conformer can be generated elsewhere, and even from other generative models like CGCF Xu et al. (020). In the simplest case, we could also initiate the conformers from random points for the model training and evaluation where each coordinate value is drawn from Gaussian (O o²) and = 0.1 * [VD. As shown in Table S1] while the model performance is less satisfactory with the randomly initialized conformers, the predicted conformers are still quite accurate and even more accurate than the ones optimized by the hand-designed foree-field method. This also shows the potential of our methods as the model prediction can be even more accurate if better initialization is possible.
19

---

Published as a conference paper at ICLR 2022
Dataset
QM9
GEOM-Drus Druug
Model
mean RMSD (4)
median RMSD (4)
mean RMSD (L)
median RMSD (4)
RDK
0.3872 = 0.0029
0.2756 + 0.0075
1.7913 + 0.0030
1.6433 + 0.0097
Ours-TwoAtore TwoAtom
0.1415 0.0004
0.0534 0.0002
0.8839 0.0014
0.7733 = 0.0026
Murs-TwoAtom-Rande
0.2256 0.0008
0.1602 0.0020
1.3026 0.0037
1.2087 + 0.0049
Table S1: RMSD when training and evaluating our optimization models random initialization instead RDKI++MMFF. The confidence interval is calculated from the inference with ten different initializations.
H.2 PREDICTING ENSEMBLE PROPERTY
Here we perform the same ensemble property prediction as Table of Shi et al. (2021), where they measure how well the generated conformers capture the population statistics of electronic property of the reference set. The authors are very kind and provide us with the code and the 30 molecules used in their experiments. We reproduce the results here for future references.
Method
E
Emin
A
Ammin
Fmax
RDKit
1.03::0.34
0.67=0.21
0.38
0.36=0.47
1.08 2.29--22.59
ConfGF
1.67++3.48
0.16+00133
0.47++0.44 0.62+00.5
0.13=0.09
Ours-Random
2.63++6.25
1.71++3.03
0.37
1.03++1.21
Table mean and standard deviation
absolute errors of predicted ensemble properties. Unit: eV.
H.3 REPRESENTATION LEARNING
While have been focusing on optimizing molecular conformation, equivariant model for molecular representation learning, even when the native conformer is already given. In this experiment, representation under the framework of implicit energy minimization. Following the same setup as Satorras et al. (202T), we compare our model against other models the QM9 property prediction benchmark. As shown in Table S3 our e-update-only model can achieve a comparable performance with state-of-the-arr models well. The consistently good performance again demonstrates the generalizability of our framework where powerful SE(3)-equivariant and -invariant models can be derived with different definitions of the energy function.
also apply SE(3)-
treat the conformer X constant and only update the internal atom
Task
a
Ac
CHOMO meV
CLUMO meV
C₁
Model
bohr
meV
D
cal/mol K
WaveScatt (Hirn et 2017)
0.160
118
85
76
0.340
0.049
NMP Gilmer et al. 2017)
0.092
69
43
38
0.030
0.040
SchNet Schlitt et al.
2017
0.235
63
41
34
0.033
0.033
Cormorant (Anderson et al.
2019)
0.085
61
34
38
0,038
0.026
LINet (Miller et al. 20220)
0.088
68
46
35
0.043
0.031
LieConv (Finzi et al.
(2020)
0.084
49
30
25
0.032
0.038
TFN Thomas et al. 2018)
0.223
58
40
38
0.064
0.101
SE(3)- Tr. (Tuchs et al.
2020)
0.142
53
35
33
0.051
0.054
EGNN (Satorras et al.
202T)
0.071
48
29
25
0.029
0.031
Ours
0.078
53
31
27
0.019
0.037
Table S3: MAE for QM9 molecular property prediction benchmark comparing other non-equivariant (top) top-3 3 performances each are bolded.
equiviriant (bottom) models.
20

---

Published as a conference paper at ICLR 2022
1 STUDY OF MODEL ABLATION AND VARIATION
I.1 EFFECTS OF GRAPH CONNECTIVITY
While our model based on the attention mechanism on the whole graph, we can also define the graph connectivity to reduce the computational complexity. Performing attention directly on the molecular graph may be inefficient. One possible solution is to perform attention on the augmented graph (Xu al., 2020 Simm & Hermandez-Lobatt 2020 where the molecular graph is augmented with virtual bonds connecting atoms that are or 3 hops away in the original graph. However, such node distances may be close in the 3D space. An alternative way is to construct a fully connected graph to capture all long-range dependencies, but it could introduce noise the attention process. Our model dynamically calculates the graph connectivity based on the euclidean distances between atom pairs/triplets at each intermediate optimization step. Specifically, we exclude the contributions for pairs/triplets they are too far from each other in the 3D space and only consider interaction inside this radius cutoff.
sometimes
informative enough since two atoms far apart in graph distance
Table shows that dynamic connectivity is consistently better than other approaches for both the QM9 and GEOM-Drugs datasets. Although fully connected attention works better than augmented attention on the QM9 dataset, worse than the augmented graph on GFOM-Druge even with more computational complexity. The performance degradation here could be caused by the noise from unrelated atom pairs/triplets. In contrast, our dynamic connectivity can take advantage of the predicted conformers and construct more informative connectivity than the augmented approach based on the graph distance.
Dataset
QM9
GEOM GEOM-Drugs Drugs
Model
mean RMSD
median RMSD
mean RMSD
median RMSD
Augmented
0476
0.0568
0.9194
0.8136
Fully- Selle--Codeeeedd
0.1456
0.0569
0.9244
0.8140
Dynamically-Conneece
0.1415
0.0534
0.8839
0.7733
Table 54: Study of Model Ablation and Variation for Molecular Graph Connectivity.
1.2 EFFECTS OF THE PARAMETERIZATION FOR INTERACTION WEIGHT
While the dynamic connection discussed in the above section treats the interaction between atoms in binary fashion, we also consider their interaction weight fractionally. In EGNNSatorras et al. (202I), estimates the interaction weights with a linear layer which takes as inputs. In our model, we propose a prediction layer for weights of interaction, which considers the distance between an atom pair and their atom type.
In Table S5 we test different approaches to parameterize the interaction weights. While we find the Per-layer u (EGNN-like performs worse than others, the effects of distance based prediction net vary from dataset to dataset. In the models reported in the main text, we use Per-block d for all variants, but Per-layer setup could improve the performance the GEOM-Drugs dataset. Thus, we could tune this design from dataset to dataset. We leave further exploration to future work.
Dataset
QM9
GEOM-Drugs
Model
mean RMSD
median RMSD
mean RMSD
median RMSD
None
0.1456
0.0582
0.8780
0.7618
Per-layer v
0.1590
0.0693
0.8918
0.7719
Per-layer d
0.1427
0.0555
0.8529
0.7287
Per-block d
0.1415
0.0534
0.8839
0.7733
Table 85: Study of Model Ablation and Variation for Interaction Weight Parameterization. None refers no interaction prediction net; Per-layer v an EGNN-like prediction net: Per-layer and Per-block directly take intermediate atomic distances as inputs and perform prediction at each layer or each block separately.
21

---

Published as a conference paper at ICLR 2022
1.3 EFFECTS OF PARAMETER SHARING
For our models reported in the main text, we implement three optimization blocks that share their parameters while cach block contains three layers of a and v updates that do not share their parameters. However, could push this to either extreme in terms of parameter sharing. For the of total update layers, we could have either nine shared optimization blocks with only one layer per block single optimization block with total of nine layers that do not share parameters all. worth mentioning that all these parameter sharing designs make sense since they correspond to the different hypotheses of the shape of the gradient field. For example, if the gradient field relatively smooth, sharing parameters across all blocks could also perform well. We report the performance for these two extreme cases in Table S6 and found our hybrid design strikes the right balance between the two.
number
Dataset
QM9
GEOM-Drugs
Model
mean RMSD
median RMSD
mean RMSD
median RMSD
9-Block x 1-Layer
0.1505
0.0597
0.8874
0.7727
1- 1-Block x 9-Layer
0.1395
0.0527
0.9750
0.8792
3-Block x 3-Layer
0.1415
0.0534
0.8839
0.7733
Table 56: Study of Model Variation for Parameter Sharing.
1.4 EFFECTS OF ITERATIVE UPDATE
For models in the main text, the coordinate and the internal representation updated in iterative fashion, where the update for x++1 depends on wh+1, However, we can also update them in a parallel fashion as EGNN does, where the update for x++1 will depend on of instead. These both reasonable approaches depending on the optimization strategies.
Empirical results Table show that optimizing æ and v iteratively is slightly better than optimizing them in synchronized fashion.
Dataset
QM9
GEOM-Drugs
Model
mean RMSD
median RMSD
mean RMSD
median RMSD
Parallel Update
0.1483 0.1415
0.0584 0.0534
0.9588 0.8839
0.8515 0.7733
Iterative Update
Table S7: Study of Model Variation of Iterative and Parallel Update.
1.5
COMPARISON WITH EGNN
Dataset
Designs
QM9
GEOM-Drug median RMSD
Model
Tim-BB
IW
PS
IU
RMSD
median RMSD
RMSD
EGNN EGNN-Tfm
0.2101
0.1356
1.0405
0.9598
0.2049
0.1209
1.0334
0,9458
Per-layer IW
0.1590
0.0693
0.8918
0.7719
Non-Share Para.
0,1395
0.0527
0.9750
0.8792
Parallel Update
0.1483
0.0584
0.9588
0.8515
Ours(Full)
0.1415
0.0534
0.8839
0.7733
Table S8: Ablation Study Comparing EGNN. Tfm--B denotes the transformer backbone. /W denotes interaction weight, which is marked if per-block interaction weight is adopted, otherwise per-layer interaction weight is adopted. PS denotes parameter sharing. IU denotes parallel update.
Finally, we summarize our different designs and their effects comparing to EGNN in Table S8 There are couple of differences between our methods and EGNN from the unrolled optimization
22

---

Published as a conference paper at ICLR 2022
formulation including per-block interaction weight (1.2). shared parameters (1.3), and iterative updates (I.4). These differences make up some of the improvements together. On the parameterization that EGNN uses MLP architecture while uses
side, arguably the biggest differences Transformer-like architecture. However, switching their MLP architecture with a Transformer-like architecture provides little bit of lift, but it's not the major contributor.
J ABLATION STUDIES IN CONFORMATION GENERATION
Dataset
GruMsDrugs
COV(%)
(t) Median
MIS(%) (4)
MAT(Á) (4)
Metric
Mean
Mean
Median
Mean
Median
Multi-step Eval
72.96
76.73
37.78
28.88
1.1031
1.0853
EGNN-bb
77.09
82.57
23.88
11.58
1.0491
1.0514
Non OT-Loss
59.67
60.96
16.41
1.82
1.2443
1.2223
Full model 76.50
83.78
31.40
23.03
1.0694
1.0583
Table 59: Ablation Study in Conformation Generation. Multi-step Eval means the model trained learn the part optimization process evaluated with multiple steps achieve the entire optimization. EGNN-bb denotes the model uses EGNN as the backbone. Non OT-Loss means the model is trained to minimize the L2 loss between the generated conformers and their closest reference conformers, instead of using the optimal transport loss.
We introduce ablation studies to further study the importance of different components of our proposed method in the generative setting. We find the formulation of framing the generative task as a fixed-step unroll optimization process using only see a very small performance regression when switching our backbone optimization model EGNN. On the other hand, ConfGF is proposed to model one-step optimization and relies Langevin dynamics for performing the entire sequence of optimization. Additionally, the optimal transport loss also helps quite bit, as encourages the model optimize the conformers towards different directions instead of collapsing the mode.
(Eu))vequi model is the more important bit,
K INFLUENCE OF NUMBER OF UNROLLED OPTIMIZATION STEPS
We test the optimization performance on the QM9 dataset with different numbers of optimization steps. We observe diminishing return for steps >4 since the "gradient" would be small if close to the solution. Therefore, we believe the method is not sensitive to the number of steps as long a sufficient number of steps are taken. In all our experiments, we perform a fixed number of optimization steps.
0.20
1.10
QM9 Mean
Drug Mean
QM9 Median
1.05
Drug Median
RMSD (Unit: Al
0.16
0.14
0.95
0.12
RMSD
0.90
0.10
0.08
0.80
0.06
2
4
5
8
9
1
4
5
7
8
Number Optimization Steps
Number Optimization Steps
Figure S3: Influence of Number of Unrolled Optimization Steps
L MORE EXAMPLES OF MODEL-OPTIMIZED CONFORMERS
Please see next page.
23

---

Published as a conference paper at ICLR 2022
Molecule
RDKit MMFF
EGNN
Ours
Reference
Figure S4: More Examples of Model-Predicted Conformers. Molecule are sampled from QM9 dataset, and the larger molecule 7-12 are sampled from the GEOM-Drug' dataset. The reference conformer. as well as the initial conformations from RDKit+MMFF, are also shown. Green IDs are the successful cases 10% of the population. Red IDs are the failure cases where the
where the predicted conformers' RMSD predicted conformers' RMSD are 90% of the population. Yellow DDs the with RMSD between 45% and 55% population.
24


Reviews:
[{"review_id": "ctsyR5sBPcv", "reviewer": "Reviewer_nNjh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Concerns about technical parts  \n- My major concern is the formulation of \"energy minimization\". Although the main claim of the work is energy minimization, I don't find any relation between the mathematical formulation and energy minimization. \n    - The potential energy function $u()$ in eqn (2) is not specified.\n    - $f_x()$ in eqn (3) is not trained to minimize the energy function (which is not specified); instead, it is trained to minimize the L2 distance between the generated conformer and its closest reference conformer.\n- The basic idea of this work needs justifications. The authors seem to suggest that casting \"the prediction problem into an unrolled optimization process\" is better than previous approaches such as ConfGF. This is not clear to me. \n   - \"While our model is also trying to estimate the gradient field, we instead employ an unrolled optimization architecture (Domke, 2012; Liu et al., 2018) aiming to model the entire optimization process. By learning to minimize a parametric energy function in a fixed number of steps, our models enjoy a much faster inference time.\"  This claims that the proposed method is faster but not better (or more accurate). Then it is better to explain why it is better than ConfGF in Table 3, actually much much better in terms of MIS(%).\n    - Seems the number of steps is a key hyperparameter in the proposed method, which is unfortunately not studied in experiments. For example, how many steps are used in experiments? Are the final results sensitive to the number of steps?  \n\nConcerns about experiments  \n- In Section 4.1, Table 1 shows that the three atom model is better than the two atom model for conformation optimization. However, only the two atom model is studied and compared in Section 4.2 for conformation generation. I expect the comparison with the three atom model rather than the two atom model in Section 4.2.\n- As shown in Table S1, the results of the proposed method are (very) sensitive to the initialization of molecular conformation, i.e., Ours-TwoAtom is much better than Ours-TwoAtom-Random. Does this mean the proposed model can only work well with a good initialization? If yes, this may limit the application scope of the proposed model. \n- Similar concern about \"instead of training the model to optimize a single initialization towards one reference, we train the model to optimize K initialization such that these K optimized conformers can be exactly matched to K different sampled references.\" Are the K initializations different from each other? If so, how to get the K different initializations?  ", "summary_of_the_paper": "This work studies the problem of molecular conformation optimization and proposes a neural \"energy minimization\" formulation, where a neural network is parametrized to learn the gradient fields of a conformational energy landscape.", "summary_of_the_review": "I have several concerns about the technical part and empirical results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "QgaApnfQksY", "reviewer": "Reviewer_jLPL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# 1. Positives\ni. The paper has strong empirical results.\nBy considering both optimization and generation baselines, the authors are able to compare against a wide range of previous baselines over several different forms of evaluation. They obtain improved performance over these models using multiple different metrics. This improved performance also seems to be obtained with reduced compute costs (e.g., approx. 170x speedup over ConfGF).\n\nii. I found the paper easy to follow.\nFigure 1 is helpful in understanding the approach. I found Section 3 interesting and well-organized; describing previous methods through considering the implied form of the potential energy function (see also Appendix C) seems like a useful lens in which to compare and view different models.\n\n\n# 2. Negatives\ni. It would be nice to have been provided with some evidence for what aspect of the proposed approach is important for improving upon the previous SOTA ML model, ConfGF, in Table 3, given that (at a high-level) ConfGF is also predicting the gradient of the energy function. For instance, is it because of the unrolled optimization routine, different architectures, the optimal transport loss, etc.? Could this be teased out with an ablation study?\n\nii. The number of time steps (i.e. the max value of $t$) used/required in the different experiments is not very well discussed/evaluated. Questions I would like to see answered include: Is the authors' model sensitive to this value? Does the value differ for the two datasets? How can one choose this value?\n\n\n# 3. Clarification Questions to the Authors\n\ni. Table S1 indicates that the RDKIT initialization is important for performance. I just wanted to check that you also used RDKIT initializations for the baseline approaches you compared to (where applicable)? As opposed to RDKit initializations, have you thought about predicting the initial positions?\n\nii. I'm surprised that in Table 1 \"Ours-TwoAtom\" does so much better than EGNN, given on p.5 you describe the similarities between these models. Do you have any intuition for why this is?\n\niii. Sorry if I missed this, but in Table 3 are you using the two atom or three atom model?\n\niv. In Table 2 is SchNet really the best reference model? From Table S2 it seems that one can do much better with an equivariant model (when you know the conformation). \n\nv. How do you choose the step size in Eqn 1, $\\gamma$?\n \n \n\n\n\n# 4. Very minor comments (e.g., typos)\ni. The grammar/typos could be fixed in a few places, e.g.:\n- p.1 \"could be required to encode the redundancy observed in such matrix\" --> \"could be required to encode the redundancy observed in such _matrices_\" ?\n- Fig1, caption: \"the gradient updates for the 3D coordinate long with the updated atom\" --> \"the gradient updates for the 3D coordinate _along_ with the updated atom\"\n- p.5 \"perspective of neural energy minimization, these are equivalent to optimizing an energy function that depends on both the atom presentations and inter-atomic distances just like our basic\" --> \"perspective of neural energy minimization, these are equivalent to optimizing an energy function that depends on both the atom _representations_ and inter-atomic distances just like our basic\" ?\n- p.15 \"we ﬁnd many of them are pretty closed, as\" --> \"we ﬁnd many of them are pretty _close_, as\"\n\nii. On p.9 when introducing terms COV, MIS, and MAT you could refer to their definitions in Appendix G. Also what value does $\\delta$ take? (apologize if I missed this)\n\niii. The experiments in Appendix I are interesting and impressive! Maybe it would have been nice to have more explicitly referred to this section in the main paper?\n\niv. How were the hyperparameters described in Appendix E chosen?\n\n", "summary_of_the_paper": "This paper proposes a new method to generate molecular conformations (i.e., the spatial arrangements of atoms belonging to a given molecule). The method works by updating initial atom positions (which can be initialized either randomly or using an alternative conformer generation method) through an iterative process. Each iteration involves predicting the gradients associated with atoms' positions and then changing their coordinates by taking a step in this direction (Eqn. 1 & Fig. 1b). \n\nThe authors:\n1. show how the parameterization of the gradients relates to how one models the conformational energy (Section 3.3);\n2. demonstrate competitive performance on optimization (finding a single best conformer; Section 4.1) and generation (finding a set of relatively stable conformers; Section 4.2) tasks compared to previous benchmarks. This is judged in terms of (a) how well the generated conformers match up to ground truth conformers (e.g., Table 1) as well as (b) how well the generated conformers enable the prediction of molecular properties (Table 2).\n", "summary_of_the_review": "Overall, I thought this was an interesting paper and the proposed method achieves impressive empirical results. However, for now, I have gone with an overall score of 6, as I would like to see some of the experimental aspects better resolved (see 2.i-ii & 3.i-ii). Happy to consider raising my score if this gets addressed in the rebuttal!\n\nI went with a lower confidence score as I have not gone through the proofs in the appendix in detail and only have a higher-level knowledge of some of the pieces of related work.\n\n##  Rationale behind the significance scores\nI put _\"3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.\"_ for the technical contributions question below. I did not choose the top score, given that Shi et al. (2021) also consider parameterizing the gradients of the energy function directly. (I realize that there are differences however, such as the training routine and optimal transport loss).\n\nI put _\"2: The contributions are only marginally significant or novel.\"_ for the empirical novelty and significance. This paper's method is evaluated on existing datasets (Ramakrishnan et al., 2014; Axelrod & Gomez-Bombarelli, 2020) in a way that is similar to existing work. To introduce a new empirical evaluation the authors could consider evaluating how well the models trained on one dataset transfer to another or consider directly comparing the predicted gradients (from the NN) to gradients derived from traditional approaches. (Although in my opinion making more of a technical contribution and less of an empirical one (or vice versa) is fine).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MuqicKU9kK", "reviewer": "Reviewer_h2Mm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The proposed method seems to be simple and easy to implement.\n2. The proposed method outperforms the existing methods in the experiments.\n\nCons:\n1. I think the paper could use some clarifications in their writing. I point out some suggestion in what follows.\n- First, I am not sure if the proposed method can be called an \"neural energy minimization\" method and an \"unrolled optimization algorithm.\" There is no energy or optimization objective for the given DNN architecture. Without any regularization, the proposed DNN architecture is not a proper gradient field. \n- The \"energy function E()\" is defined only on the conformation X in Equation (1), but defined on the conformation X and atom features V in Equation (2).  \n- This paper uses the terminology \"V\" as the set of atom nodes and set of atom features. I think the authors should add a new terminology to denote the set of atom features. \n- I am a little concerned that the paper propose to solve \"conformation optimization\", but the proposed architecture is trained using supervised learning and the evaluation is done using RMSD. In other words, there is no optimization in the proposed task. I understand that the task is to \"predict the optimized conformation.\" Maybe the authors could make minor clarifications or changes to clarify this aspect?\n2. Additional experimental results can strengthen this paper, especially for future works to build on this paper. For example, can the authors provide their experimental results on predicting the ensemble property (Table 5 in ConfGF paper)? \n\nMinor:\n1. Given the training scheme used in the paper, it seems that the DNN-based \"molecular conformation generation\" methods are applicable to \"molecular conformation optimization\". Is this correct? \n2. How did the authors perform hyper-parameter tuning? ", "summary_of_the_paper": "This paper proposes a new deep neural network (DNN) architecture to generate molecular 3D conformation. It derives its architecture starting from gradient-based updates for minimization of the molecule's energy function. Then the architecture becomes a repetition of vertex-wise aggregation layer of SE(3)-equivariant Transformer-based DNN. The architecture takes a molecule and 3D conformation pre-calculated from RDkit as input. The authors show how the proposed architecture generalize existing DNN architectures for molecular conformation generation. Empirical results demonstrate superiority of the proposed method. ", "summary_of_the_review": "This paper shows solid empirical performance with the proposed DNN architecture for molecular conformation generation. However, there are some logic or details that was not very convincing to me, e.g., the proposed method being a neural energy minimization. I would like to see them clarified for me to raise my score.   ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DAA2PRm_qYD", "reviewer": "Reviewer_83EH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Regarding the machine learning approach and experiments, the comparison to existing experiments is unfair at points, since conformer ensembles generated with RDKit are often used as starting guesses, while other approaches rely on random positions. Since the authors have provided experiments with random starting positions in the SI (and still find good, although less impressive performance) this is only a minor issue. However, this difference between model evaluation should be clearly stated in the main text. A discussions on how hydrogen atoms are treated would also be helpful for putting the results into perspective. The presented approach appears to only generate heavy atom positions, with the hydrogens added via RDKit. How does this compare to other approaches and how does it influence the search for stable conformers (e.g. are additional optimizations with ab initio methods necessary)? Since a general correspondence between properties and structure can be expected, the downstream application experiments appear to offer no new insights on the quality of the method. Nevertheless, it would still be helpful to know how the labels for these experiments were generated. Were new computations performed for the new confomers or did the experiments reuse the orginal QM9 labels for molecules with the same graphs? There is also a slight confusion regarding the comparison to CGCF in Table 3. The numbers reported there diverge from [Xu 2020]. In addition, it was never specified which delta value was used for evaluating the COV and MIS scores.\n\nOne of the main points of criticism is the proposed neural energy minimization framework for constructing equivariant networks. It essentially recapitulates already established basics of SE(3) equivariant models and offers no new insights. It is well known, that the Cartesian gradients of functions are one way to systematically construct equivariant filters for interactions/updates. This ansatz was e.g. used in  [Schütt 2020] to derive continuous convolutions of an invariant input with an equivariant filter (order l=1). The resulting update has a strong resemblance with Eq 3. In general, updates and interaction can be expressed as a multipole expansion of the energy, and if this is done e.g. in Cartesian coordinates the interaction tensors take the form of first and higher order derivatives w.r.t. atomic positions. Irreducible representations and spherical harmonics are another way to systematically construct equivariant models frequently and are closely related. For example, the update formula in Eq. 3 could also be derived as the interaction between scalar (l=0) and vector (l=1) geometric tensors in this   framework [Batzner 2021]. The statement, that conventional SE(3) equivariant networks are either based on complex mathematical frameworks, which necessitate expensive coefficient calculations, or derived ad hoc from message passing is highly misleading. In general, there is a limited way on how equivariance can be introduced into a model properly. If these operations are to be formulated in a principled and general way, tensor algebra is helpful. This does not automatically imply, that operations are inefficient. If only lower order features/interactions (e.g. l <= 2) are used, no significant computational overhead compared to conventional message passing is introduced (see e.g. [Batzner 2021]). On the other hand, if one were to expand the proposed derivative mechanism to higher order interactions, similar problems would be encountered. E.g. interactions between two dipole vectors require second order derivative tensors (3x3), between quadrupoles a fourth order derivative (3x3x3x3) tensor, etc, thus greatly increasing computational cost.\n\nFinally, there are several problems regarding the introduction and related work sections. \nWith respect to molecular conformer generation, full ab initio molecular dynamics is neither the standard approach nor required for conformer search. Most of the time, heuristic approaches as e.g. used in RDKit followed by ab initio optimization are used. These are much more tractable and usually work well for small organic molecules, as can be seen based on the performance of RDKit in the conformer generation experiment. The main problem of such approaches is the energetic weighting of the conformers, for which said ab initio optimizations are required.\nIn the same paragraph (and introduction), a series of generative approaches is described as using distance matrices as intermediates which are then translated into 3D coordinates. However, many of the mentioned approaches do not rely on distance matrices as intermediates. G-SchNet, for example, uses an iterative approach based on factorized probabilities to directly generates a new 3D structure.\nThe description of ML based force fields is also misleading. While it is true, that sGDML requires retraining for every new molecular graph, this neglects a plethora of machine learning potentials, which do not suffer from this problem and scale linearly with molecule size (see e.g. [Unke 2021] for an overview).\nIt would also be helpful to provide an in depth discussion on the parallels between ConfGF [Shi 2021] and the present approach in the related works section.\n\n\nSchütt, Kristof T., Oliver T. Unke, and Michael Gastegger. \"Equivariant message passing for the prediction of tensorial properties and molecular spectra.\" arXiv preprint arXiv:2102.03150 (2021).\n\nBatzner, Simon, et al. \"Se (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials.\" arXiv preprint arXiv:2101.03164 (2021).\n\nUnke, Oliver T., et al. \"Machine learning force fields.\" Chemical Reviews (2021).", "summary_of_the_paper": "The authors present an equivariant graph network approach for optimizing and generating molecular conformers.", "summary_of_the_review": "While sharing conceptual similarities to the gradient fields by [Shi 2021], unrolling the structure optimization process in the form of a graph network is an interesting approach and casting the model as a fixed point search has parallels to deep equilibrium networks. Its potential is demonstrated by good performance in different experiments. The proposed theoretical framework, on the other hand, is a restatement of common concepts in equivariant architectures, which have been presented in a similar or more general form elsewhere. In addition, the different sections (primarily related work) suffer from several fundamental problems which further detract from the quality of the work. Due to the two latter points, I tend towards rejecting the work in its present form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers except one agreed that this paper should be accepted because of the strong author response during the rebuttal phase. Specifically the reviewers appreciated the new ablation study showing that improvements are not due to minor architectural changes, the new experiment on the number of time steps required for experiments, the agreement to change language around "neural energy minimization", the improvements to the related work, the novelty of the unrolled optimization approach, and the nice experimental results. Given this, I vote to accept. Authors: please carefully revise the manuscript based on the suggestions by the reviewers: they made many careful suggestions to improve the work and stressed that the paper should only be accepted once these changes are implemented. Once these are done the paper will be a nice addition to the conference!
Decision: Accept (Poster)