Learning Neural Generative Dynamics for Molecular Conformation Generation
Authors: Minkai Xu
Shitong Luo
Yoshua Bengio
Jian Peng
Jian Tang
Venue: ICLR 2021 Poster (2021)
Keywords: Molecular conformation generation
deep generative models
continuous normalizing flow
energy-based models

Abstract: We study how to generate molecule conformations (i.e., 3D structures) from a molecular graph. Traditional methods, such as molecular dynamics, sample conformations via computationally expensive simulations. Recently, machine learning methods have shown great potential by training on a large collection of conformation data. Challenges arise from the limited model capacity for capturing complex distributions of conformations and the difficulty in modeling long-range dependencies between atoms. Inspired by the recent progress in deep generative models, in this paper, we propose a novel probabilistic framework to generate valid and diverse conformations given a molecular graph. We propose a method combining the advantages of both flow-based and energy-based models, enjoying: (1) a high model capacity to estimate the multimodal conformation distribution; (2) explicitly capturing the complex long-range dependencies between atoms in the observation space. Extensive experiments demonstrate the superior performance of the proposed method on several benchmarks, including conformation generation and distance modeling tasks, with a significant improvement over existing generative models for molecular conformation sampling.

Published as a conference paper at ICLR 2021
LEARNING NEURAL GENERATIVE DYNAMICS FOR MOLECULAR CONFORMATION GENERATION
Minkai Xuito,, Luo*3, Yoshua Bengio 1.2.4 2.24, Jian Peng", Jian Tang MMila Québec AI Institute, Canada Université de Montréal, Canada
PPeking University, China
*Canadian Institute for Advanced Research (CIFAR), Canada
University of Illinois at Urbana-Champaign, USA HEEC Montréal, Canada
"xuminkats yoshua.... yoshua.bengion luostêpku. edu. cn dianpenge iereengillliioo alliniis.ede jian. tang@hec.cc.c ca
quebec
ABSTRACT
We study how generate molecule conformations (ie., 3D structures) from molecular graph. Traditional methods, such as molecular dynamics, sample con- formations via computationally expensive simulations. Recently, machine learn- ing methods have shown great potential by training on a large collection of con- formation data. Challenges arise from the limited model capacity for capturing complex distributions of conformations and the difficulty in modeling long-range dependencies between atoms. Inspired by the recent progress in deep generative models, in this paper, we propose novel probabilistic framework to generate valid and diverse conformations given a molecular graph. We propose method combining the advantages of both flow-based and energy-basee models, enjoying: (1) high model capacity to estimate the multimodal conformation distribution; (2) explicitly capturing the complex long-ran long-range dependencies between atoms in the observation space. Extensive experiments demonstrate the superior perfor- mance of the proposed method on several benchmarks, including conformation generation and distance modeling tasks, with significant improvement over ex- isting generative models for molecular conformation sampling
INTRODUCTION
Recently, we have witnessed the success of graph-based representations for molecular modeling in variety of tasks such property prediction Gilmmr et al. (017)) and molecule generation (You et al. 2018 Shi et al. 2020). However, a more natural and intrinsic representation of molecule is its 3D structure, commonly known the molecular geometry conformation, which represents each atom by its 3D coordinate. The conformation of a molecule determines its biological and physical prop- erties such as charge distribution, steric constraints, as well interactions with other molecules. Furthermore, large molecules tend to comprise a number of rotatable bonds, which may induce flex- ible conformation changes and large number of feasible conformations in nature. Generating valid and stable conformations of given molecule remains very challenging. Experimentally, such struc- tures are determined by expensive and consuminn crystallography. Computational approaches based on Markov chain Monte Carlo (MCMC) or molecular dynamics (MD) (De Vivo et al. 2016) are computationally expensive, especially for large molecules (Ballard et al. 2015)..
Machine learning methods have recently shown great potential for molecular conformation gen- cration by training on a large collection data to model the probability distribution of potential conformations R based the molecular graph G, i.e., p( R)G). For example, Mansimov et al.
"Equal contribution. Work was done during Shitong's intermship at Mila. Code available at atthtpss:/ / thub. wtw....cmm/Deeeprraaphhhiic Coc.e//eaaooecccccooooooooooooo fGen

---

Published as a conference paper at ICLR 2021
2019) proposed a Conditional Variational Graph Autoencoders (CVGAE) for molecular conforma- tion generation. A graph neural network (Gilmer et al., 2017) is first applied to the molecular graph to get the atom representations, based which 3D coordinates of such an approach is that by directly generating the 3D coordinates of atoms it fails to model the rotational and translational invariance of molecular conformations. To address this issue, instead of generating the 3D coordinates directly, /Simm & Hernández-Lobato (2020) recently proposed to first model the molecule's 's distance geometry (i.e., the distances between atoms)-which are rota- tionally and translationally invariant-any then generate the molecular conformation based on the distance geometry through post-processing algorithm (Liberti al., 2014). Similar MMansimvv] et al. (2019), a few layers of graph neural networks are applied to the molecular graph to learn the representations of different edges, which are further used to generate the distances of different edges independently. This approach is capable of more often generating valid molecular conformations.
further generated. One limitation
Although these new approaches have made tremendous progress, the problem remains very chal- lenging and far from solved. First, each molecule may have multiple stable conformations around a number of states which are thermodynamically stable. In other words, the distribution p( is very complex and multi-modal. Models with high capacity are required to model such complex dis- tributions. Second, existing approaches usually apply a few layers of graph neural networks to learn the representations of nodes (or edges) and then generate the 3D coordinates (or distances) based on their representations independently. Such approaches are necessarily limited to capturing a single mode of p( ([()) (since the coordinates or distances are sampled independently) and are incapable of modeling multimodal joint distributions and the form of the graph neural net computation makes it difficult to capture long-range dependencies between atoms, especially in large molecules.
Inspired by the recent progress with deep generative models, this paper proposes a novel and prin- cipled probabilistic framework for molecular geometry generation, which addresses the above two limitations. Our framework combines the advantages of normalizing flows (Dinh et al.) 2014) and energy-based approaches (LeCun et al., 2006), which have strong model capacity for modeling complex distributions, are flexible to model long-range dependency between atoms, and enjoy effi cient sampling and training procedures. Similar to the work of Simmm & Hernández-Lobate (2020), we also first learn the distribution of distances d given the graph G. i.e., p(d G)) and define another distribution of conformations R given the distances d, i.e., p( R d, g). Specifically, we propose novel Conditional Graph Continuous Flow (CGCF) for distance geometry (d) generation condi- tioned on the molecular graph G. Given a molecular graph G. CGCF defines an invertible mapping between base distribution (e.g.. multivariate normal distribution) and the molecular distance ge- ometry, using a virtually infinite number of graph transformation layers on atoms represented by a Neural Ordinary Differential Equations architecture (Chen et al., 2018). Such an approach enjoys very high flexibility to model complex distributions of distance geometry. Once the molecular dis- tance geometry d is generated, we further generate the 3D coordinates R by searching from the probability p( R RRd, 9).
Though the CGCF has a high capacity for modeling complex distributions, the distances of different edges are still independently updated in the transformations, which limits its capacity for model- ing long-range dependency between atoms in the sampling process. Therefore, we further propose another unnormalized probability function, i.e., an energy -based model (EBM) (Hinton & Salakhut- dinov [2066; LeCun et al., 2006; Ngiam et al. 2011). which acts as a tilting term of the flow-based distribution and directly models the joint distribution of R. Specifically, the EBM trains energy function E R. 9), which is approximated by a neural network. The flow- and energy-based models combined in a novel way for joint training and mutual enhancement. First, energy-basee meth- ods are usually difficult to train due to the slow sampling process. In addition, the distribution of conformations is usually highly multi-modal, and the sampling procedures based on Gibbs sampling or Langevin Dynamics (Bengio et al. 2013a| b) tend to get trapped around modes, making it diffi- cult mix between different modes (Bengio al. 2013a). Here we use the flow-based model as a proposal distribution for the energy model, which is capable to generate diverse samples for training energy models. Second, the flow-based model lacks the capacity to explicitly model the long-range dependencies between atoms, which we find can however be effectively modeled by an energy func- tion ERR, 9). Our sampling process can be therefore viewed as a two-stage dynamic system, where we first take the flow-based model to quickly synthesize realistic conformations and then used the learned energy E R, 9) to refine the generated conformations through Langevin Dynamics.
2

---

Published as a conference paper at ICLR 2021
We conduct comprehensive experiments several recently proposed benchmarks, includ- ing GEOM-QM9, GEOM-Drugs (Axelrod & Komez-Bombanne 2020) and ISO17 (Simm & Germández-Loaace 2020). Numerical evaluations show that our proposed framework consistently outperforms the previous state-of-the-art (GraphDG) on both conformation generation and distance modeling tasks, with clear margin.
2 PROBLEM DEFINITION AND PRELIMINARIES
2.1 PROBLEM DEFINITION
Notations. Following
existing work (Simm & Hernández-Lobato) 2020), each molecule is repre- sented as an undirected graph G = (V.E), where V is the set of nodes representing atoms and & is the set of edges representing inter-atomic bonds. Each node in V is labeled with atomic properties such as element type. The edge in & connecting u and v is denoted as Cuv. and is labeled with its bond type. We also follow the previous work (Simm & Hernández-Lobato 2020) to expand the molecular graph with auxiliary bonds, which is elaborated in Appendix B For the molecular 3D representa- tion, each atom in is assigned with a 3D position vector r E I We denote due = ||r, - r, 2 as the Euclidean distance between the th and 4d atom. Therefore, we can represent all the positions (ro hevv as matrix R. E IRIV x3 and all the distances between connected nodes [du as vector d E RIEI.
Problem Definition. The problem of molecular conformation generation is defined as conditional generation process. More specifically, our goal is to model the conditional distribution of atomic positions R given the molecular graph 9, i..., p( g)..
2.2 PRELIMINARIES
Continuous Normalizing Flow. A normalizing flow (Dinh et al. 2014; Rezende & Mohamed, 2013) defines series of invertible deterministic transformations from an initial known distribution p(z) to a more complicated one p(z)) Recently, normalizing flows have been generalized from discrete number of layers to continuous (Chen et al. 2018: Grrthhohhl et al. 2018) by defining
the transformation fe as continuous-time dynamic (a(l)
fo(=(t), t). Formally, with the latent variable =(4u) p(z) the start time, the continuous normalizing flow (CNF) defines the transfor-
mation x = 2(to)= + fhf fo(z(t),t)dt Then the exact density for po (x) can be computed by:
log (a:) logp(=((o)) 1'' Tr
f
dt
(I)
az(()
where 2(fo) can be obtained by inverting the continuous dynamic 2(tn) x + 1t, fo())t), t)dt. black-box ordinary differential equation (ODE) solver can be applied to estimate the outputs and
inputs gradients and optimize the CNF model (Chen et al. 2018 Grathwohl et al. 2018).
Energy-based Models. Energy-based models (EBMs) (Dayan et al.. 1995 Hinton & Salakhutdi-
nov 2006: LeCun et al. 2006) use scalar parametric energy function (x) to fit the data distri-
bution. Formally, the energy function induces a density function with the Boltzmann distribution (z(z) exp(-
exp(-Ex(x)) (x))) (o)), where Z z= f exp(-EE, (x)) dr denotes the partition function. EBM can be learned with Noise contrastive estimation (NCE) (Gutmann & Hyvärinen| 2010) by treating
the normalizing constant a free parameter. Given the training examples from both the dataset and
a noise distribution gzx), & can be estimated by maximizing the following objective function:
Po(x)
((x)
( Epasesl log (5( + ((x) +E,g
| log Pe(x)) ((())+0) +
(2)
which turns the estimation of EBM into a discriminative learning problem. Sampling from E.g can be done with a variety of methods such as Markov chain Monte Carlo (MCMC) or Gibbs
sampling (Hinton& & Salakhutdinov (006),, possibly accelerated using Langevin dynamics (Du & Mordatch| 2019 Song et al. 2020), which leverages the gradient of the EBM to conduct sampling:
C V. (IR-1) + Vew,w (0,
Th
-h-1
where e refers to the step size. 2o are the samples drawn from a random initial distribution and we take the r. with K Langevin dynamics steps the generated samples of the stationary distribution.
3

---

Published as a conference paper at ICLR 2021
po g
g
Eø (R,
g
Predict distances for the impul graph.
Search 3D coordinates given the distances.
Further optimize the generated structures.
Graph
Fa Flow Dynamics
P
ES MOMC
Gradient
I)
d(ta)
CGCF
d(t1)
p p(R|d 9)
ETM
R
Figure 1: Illustration of the proposed framework. Given the molecular graph, 1) first draw latent variables from Gaussian prior, transform them to the desired distance matrix through Conditional Graph Con- tinuous Flow (CGCF); 2) search the possible 3D coordinates according to the generated distances and 3) further optimize the generated conformation via a MCMC procedure with the Energy-based Tilting Model (ETM).
3
METHOD
3.1 OVERVIEW
We first present high-level description of our model. Directly learning generative model on Carte- sian coordinates heavily depends on the (arbitrary) rotation and translation (Mansimov et al. 2019). Therefore, in this paper take the atomic pairwise distances intermediate variables generate conformations, which are invariant to rotation and translation. More precisely, the cornerstone of our method is factorize the conditional distribution pa (R g) into the following formulation:
po (o(g)) = p(R)d, (o dd,
(
4)
where Po (d] models the distribution of inter-atomic distances given the graph G and p p(R|d, G) models the distribution of conformations given the distances In particular, the conditional gener- ative model Po (dlg) is parameterized as a conditional graph continuous flow, which can be seen as a continuous dynamics system to transform the random initial noise to meaningful distances. This flow model enables us to capture the long-range dependencies between atoms in the hidden space during the dynamic steps.
Though CGCF capture the dependency between atoms in the hidden space, the distances of different edges are still independently updated in the transformations, which limits the capacity of modeling the dependency between atoms in the sampling process. Therefore further propose correct Po R()) with an energy-based tilting term (R,
Pa. R|G) pa RR
(5)
The tilting term is directly defined on the joint distribution of and G. which explicitly captures the long-range interaction directly in observation space. The tilted distribution pa (R()) can be used provide refinement optimization for the conformations generated from pa Rl)). This energy function is also designed to be invariant to rotation and translation.
In the following parts, we will firstly describe our flow-based generative model po [R(g) in Sec- tion 3.2 and elaborate the energy-based tilting model E&g RR G) in Section (3.]] Then we introduce the two-stage sampling process with both deterministic and stochastic dynamics in Section 3.4 An illustration of the whole framework is given in Fig.
3.2 FLOW-BASHR GENERATIVE MODEL
Conditional Graph Continuous Flows po(d We parameterize the conditional distribution of distances g)) with the continuous normalizing flow, named Conditional Graph Continuous Flow (CGCF), CGCF defines the distribution through the following dynamics system:
d = Fo(d(to), 9) = d(to) + S. fo(d(t),t)))d d(to) N(0,
(6)
4

---

Published as a conference paper at ICLR 2021
where the dynamic fo is implemented by Message Passing Neural Networks (MPNN) (Gilmer et al. 2017). which is a widely used architecture for representation learning on molecular graphs. MPNN takes node attributes, edge attributes and the bonds lengths d(t) as input to compute the node and edge embeddings. Each message passing layer updates the node embeddings by aggregating the information from neighboring nodes according its hidden vectors of respective nodes and edges. Final features are fed into a neural network to compute the value of the dynamic fe for all distances independently. As t + oc, our dynamic can have an infinite number of steps and is capable to model ong-range dependencies. The invertibility of Fg allows us to not only conduct fast sampling, but also easily optimize the parameter set by minimizing the exact negative log likelihood:
Ofe.c ad(1)
mmle 9: -Epana log po -lEpasta
logg(d((o))
J.o TIr
dt
(7)
Closed-form pl (R|d, 9). The generated pair-wise distances can be converted into 3D structures through postprocessing methods such as the classic Euclidean Distance Geometry (EDG) algorithm. In this paper, adopt alternative way by defining the conformations
conditional distribution:
p(R, ( 9) exp {
1
Σ Eus
Obu (1/mu - rol12 20 duo))
where Z is the partition function normalize the probability and {awv} are parameters that control the variance of desired Cartesian coordinates, which can be either learned or manually designed ac- cording to the graph structure G. With the probabilistic formulation, we can conduct either sampling via MCMC or searching the local optimum with optimization methods. This simple function is fast to calculate, making the generation procedure very efficient with a negligible computational cost.
Compared with the conventional EDG algorithm adopted in GraphDG (Simm & Hernández-Looaar 2020 our probabilistic solution enjoys following advantages: 1) p( R d, 9) enables the calculation for the likelihood pa (Rlg) of Eq. 4 by approximation methods, and thus can be further combined with the mnergy-based tilting term Egg (R. G) to induce a superior distribution; 2) GraphDG suffers the drawback that when invalid sets of distances are generated, EDG will fail to construct 3D struc- ture. By contrast, our method can always be successful to generate conformations by sampling from the distribution p( [R|d, 9).
3.3 ENERGY-BASED TILTING MODEL
The last part of our framework is the Energy-based Tiling Model (ETM) E, 9), which helps model the long-range interactions between atoms explicitly the observation space. (R, G) takes the form of SchNet (Schütt et al. 2017), which is widely used to model the potential-energy surfaces and energy-conserving force fields for molecules. The continuous-filte convolutional lay- ers in SchNet allow each atom to aggregate the representations of all single, pairwise, and higher- order interactions between the atoms through non-linear functions. The final atomic representations pooled to single vector and then passed into network to produce the scalar output.
Typically the EBMs can be learned by maximum likelihood, which usually requires the lengthy MCMC procedure and is time-consuming for training. In this work, learn the ETM by Noise Contrastive Estimation (Gutmann & Hyvärinen 2010), which is much more efficient. In practice, the noise distribution is required to be close to data distribution, otherwise the classification problem would be too easy and would not guide Eg to learn much about the modality of the data. We propose take the pre-trained CGCF serve a strong noise distribution, leading the following discriminative learning objective for the ETM?
Luce RR (. Epast [log
-BEpaan
1 exp(E, (R. (R] Epu [og
1 exp(-Ea(( E R. 9
(9)
3.4 SAMPLING
We employ a two-stage dynamic system to synthesize a possible conformation given the molecular graph representation G. In the first stage, we first draw latent variable from the Gaussian prior
Detailed derivations of the training loss can he found in Appendix
5

---

Published as a conference paper at ICLR 2021
(N 1), and then pass it through the continuous detenministic dynamics model F6 defined in Eq. get dg Fo (20 G). Then an optimization procedure such as stochastic gradient descent employed to search the realistic conformations R with local maximum probability of p(R]d, d () (defined in Eq. 8). By doing this, an initial conformation R can be generated. In the second stage, we further refine the initial conformation R( with the energy-based model defined Eq. with K steps of Langevin dynamics:
Ra- RR - V Eo,s (R(g) + Vew, (O,,I),
2
(10)
where (R(g) = log log Po R = Eg R 9) log p(R]d, 9)po (d(g)dd.
where denotes the step size. The second integration term in Eg can be estimated through approx- imate methods. practice, we use Monte Carlo Integration conduct the approximation. which is simple yet effective with just a few distance samples from the CGCF model po (dlg).
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Evaluation Tasks, To evaluate the performance of proposed model, we conduct experiments by comparing with the counterparts on: (1) Conformation Generation evaluates the model's capacity to learn the distribution of conformations by measuring the diversity and accuracy of generated sam- ples (section 4.2 ; (2) Distribution over distances is first proposed in Simm & Hernández-Lood Lobato 2020), which concentrate on the distance geometry of generated conformations (section 4.2).
Benchmarks. We use the recent proposed GEOM-QM9 and GEOM-Drugs (Axelrod & Gomez- Bombarelli (020)) datasets for conformation generation task and IS017 dataset (Simm & Herááneez-Lobate 2020) for distances modeling task. The choice of different datasets is because of their distinct properties. Specifically, GEOM datasets consist of stable conformations, which is suitable to evaluate the conformation generation task. By contrast, ISO17 contains snapshots of molecular dynamics simulations, where the structures are not equilibrium conformations but can reflect the density around the equilibrium state. Therefore, it is more suitable for the assessment of similarity between the model distribution and the data distribution around equilibrium states.
More specifically, GEOM-QM9 is an extension to the QM9 (Ramakrishnan et al. 2014) dataset: it contains multiple conformations for most molecules while the original QM9 only contains one. This dataset limited to 9 heavy atoms (29 total atoms), with small molecular mass and few ro- tatable bonds. We randomly draw 50000 conformation- moleccle pairs from GEOM RMM be the training set, and take another 17813 conformations covering 150 molecular graphs as the test set. GEOM-Drugs dataset consists of much larger drug molecules, up maximum of 181 atoms (91 heavy atoms). It also contains multiple conformations for each molecule, with a larger variance in structures, e.g., there are the 6.5 rotatable bonds average. We randomly take 50000 conformation- molecule pairs from GEOM-Drugs as the training set, and another 9161 conformations (covering 100 molecular graphs) as the test split. IS017 dataset also built upon QM9 datasets, which consists of 197 molecules, each with 5000 conformations. Following Simm & Hernández-Lobato (2000), we also split ISO17 into the training set with 167 molecules and the test set with another 30 molecules.
Baselines. We compared our proposed method with the following state-of-the-art conformation generation methods. CVGAE (Mansimov et al., (209)) uses conditional version of VAE to directly generate the 3D coordinates of atoms given the molecular graph. GraphDG (Simm & Hernández- Lobato 2020) also employs the conditional VAE framework. Instead of directly modeling the 3D structure, they propose to learn the distribution over distances. Then the distances are converted into conformations with EDG algorithm. Furthermore, also take RDKit (Riniker & Landrum, 2013) as a baseline model, which is a classical EDG approach built upon extensive calculation collections in computational chemistry.

---

Published as a conference paper at ICLR 2021
Table I: Comparison of different methods the COV and MAT scores. Top 4 rows: deep generative models for molecular conformation generation. Bottom rows: different methods that involve an additional rule-based force field to further optimize the generated structures.
Dataset
GEOM QM9
GEOM -rugg
COV* (%)
MAT (À)
COV* (%)
MAT (Á)
Metric
Mean
Median
Mean
Median
Mcan
Median
Mcan
Median
CVGAE
8.52
5.62
0.7810
0.7811
0.00
0.00
2.5225
2.4680
GraphDG
55.09
56.47
0.4649
0.4298
7.76
0.00
1.9840
2.0108
CGCF CGCF ETM
69.60
70.64
0.3915
0.3986
49.92
41.07
1.2698
1.3064
72.43
74.38
0.3807
0.3955
53.29
47.06
1.2392
1.2480
RDKit
79.94
87.20
0.3238
0.3195
65.43
70.00
1.0962
1.0877
CVGAE + FF
63.10
60.95
0.3939
0.4297
83.08
95.21
0.9829
0.9177
GraphDG+ +F FF 70.67
70.82
0.4168
0.3609
84.68
93.94
0.9129
0,9090
CGCF + FF
73.52
72.75
0,3131
0.3251
92.28
98.15
0.7740
0.7338
CGCF ETM+ FF
73.54
72.58
0.3088
0.3210
92.41
98.57
0.7737
0.7616
For the reported COV score, the threshold is set 0.5A for QM9 and 1.25A for Drugs More results of COV scores with different threshold are given in Appendix H
Cra
CigPOC
Our
Rsforroce
Visualization generated conformations from state-of-the-ant baseline (GraphDG), our method and the ground-truth, based on four random molecular graphs from the test set of GEOM-Drugs. C, O, H, and Cl are colored gray, red, white, yellow and green respectively.
Figure 2:
4.2 CONFORMATION GENERATION
In this section, we evaluate the ability of the proposed method to model the equilibrium conforma- tions. We focus both the diversity and accuracy of the generated samples. More specifically, diversity measures the model's capacity to generate multi-modal conformations, which is essential for discovering new conformations, while accuracy concentrates on the similarity between generated conformations and the equilibrium conformations.
Evaluation. For numerical evaluations, we follow previous work (Hawkins) 2017 Mansimov al. 2019) to calculate the Root-Mean- Reon-Mean-Square Deviation (RMSD) of the heavy atoms between generated samples and reference ones. Precisely, given the generated conformation R and the reference R", we obtain by translating and rotating R' to minimize the following predefined RMSD metric:
RMSD R, R
R, -R.11)
(11)
where n is the number of heavy atoms. Then the smallest distance is taken the evaluation metric. Built upon the RMSD metric, we define Coverage (COV) and Matching (MAT) score to measure the diversity and quality respectively. Intuitively, COV measures the fraction of conformations in the reference set that are matched by at least one conformation the generated set. For each conformation in the generated set, its neighbors in the reference set within given RMSD threshold

---

Published as a conference paper at ICLR 2021
Table 2: Comparison of distances density modeling with different methods. We compare the marginal distri- bution of single (p(d...9)) pair (p(due, d:yl9)) and all (p(d(g)) edges between and atoms. Molecular graphs 9 are taken from the test set of IS017. We take two metrics into consideration: 1) median MMD between the ground truth and generated ones, and 2) mean ranking (1 to 3) based on the MMD metric.
Single
Pair
All
Mean
Median
Mean
Median
Mean
Median
RDKit
3,4513
3,1602
3,8452
3.6287
4.0866
3,7519
CVGAE
4.1789
4.1762
4,9184
5.1856
5.9747
5,9928
GraphDG
0.7645
0.2346
0.8920
0.3287
1.1949
0.5485
CGCF
0.4490
0.1786
0.5509
0.2734
0.8703
0.4447
CGCF ETM
0.5703
0.2411
0.6901
0.3482
1.0706
0.5411
ô are marked matched:
COV(S,(0),8.(0)) 1 | R S, RMSD(R, R\) < 6, E E S,
(12)
where (g) denotes the generated conformations set for molecular graph G, and (g) denotes the reference set. In practice, the number of samples in the generated set is two times of the reference set. Typically, a higher COV score means the a better diversity performance. The COV score is able to evaluate whether the generated conformations are diverse enough to cover the ground-truth.
While COV effective to measure the diversity and detect the mode-collapse case, it still possible for the model to achieve high COV with high threshold tolerance. Here we define the MAT score as a complement to measure the quality of generated samples. For each conformation the reference set, the RMSD distance to its nearest neighbor in the generated set is computed and averaged:
(9), =
Σ min RMSD(R. R').
(13)
RES,
R R''S,
This metric concentrate on the accuracy of generated conformations. More realistic generated sam- ples lead to lower matching score.
Results. Tab. M shows that compared with the existing kfae-of-thhe-r baselines, our CGCF model can already achieve superior performance on all four metrics (top 4 rows). As CNF-based model, CGCF holds much the higher generative capacity for both diversity and quality compared than VAE approaches. The results are further improved when combined with ETM to explicitly incorporate the long-range correlations. We visualize several representative examples in Fig. [ and leave more examples in Appendix G A meaningful observation is that though competitive over other neural models, the base- RDKit method occasionally shows better performance than our model, which indicates that RDKit generate more realistic structures. We argue that this is because after generating the initial coordinates, RDKit involves additional hand-designed molecular force field (FF) energy functions (Rappé al., 1992; Halgren 1996a) to find the stable conformations with local minimal energy. By contrast, instead of finding the local minimums, our deep generative models aim to model and sample from the potential distribution of structures. To yield a better comparison. we further test our model by taking the generated structures as initial states and utilize the Merck Molecular Force Field (MMFF) (Halgren 1996a) find the local stable points. more precise description of about the MMFF Force Field algorithms in RDKit is given in Appendix This postprocessing procedure is also employed in the previous work (Mansimov et al. (2019). Additional results in Tab. verify our conjecture that FF plays vital role in generating more realistic structures, and demonstrate the capacity of our method to generate high-quality initial coordinates.
4.3 DISTRIBUTIONS OVER DISTANCES
Tough primarily designed for 3D coordinates, also following Simm & Bernández-Lobatt (2020) to evaluate the generated distributions of pairwise distance, which can be viewed as a representative element of the model capacity to model the inter-atomic interactions.
Evaluation. Let p(d 99) denote the conditional distribution of distances on each edge cur given molecular graph G. The set of distances are computed from the generated conformations R. We
8

---

Published as a conference paper at ICLR 2021
Table 3: Conformation Diversity, Mean and Std represent the corresponding and standard deviation of pairwise RMSD between the generated conformations per molecule.
RDKit
CVGAE
GraphDG
CGCF
CGCF +ETM
Mean
0.083
0.207
0.249
0.810
0.741
Std
0.054
0.187
0.104
0.223
0.206
calculate maximum mean discrepancy (MMD) (Gretton et al. 2012) to compare the generated dis- tributions and the ground-truth distributions. Specifically, we evaluate the distribution of individual distances p(dwo g). pair distances p(daagr dejl9) and all distances p(d(g). For this benchmark, the number of samples in the generated set is the same as the reference set.
Results. The results of MMD are summarized Tab. 2 The statisties show that RDKit suffers the worst performance, which is because it just aims to generate the most stable structures illustrated in Section 4.2 For CGCF, the generated samples are significantly closer to the ground-truth distri- bution than baseline methods, where consistently achieve the best numerical results. Besides, we notice that ETM will slightly hurt the performance this task. However, one should note that this phenomenon is natural because typically ETM will sharpen the generated distribution towards the stable conformations with local minimal energy. By contrast, the IS017 dataset consists of snapshots of molecular dynamies where the structures are not equilibrium conformations but sam- ples from the density around the equilibrium state. Therefore, ETM will slightly hurt the results. This phenomenon is also consistent with the observations for RDKit. Instead of generating unbi- ased samples from the underlying distribution, RDKit will only generate the stable ones with local minimal energy by involving the hand-designed molecular force field (Simm Hernández- 2020). And as shown in the results, though highly competitive in Tab. D RDKit also suffers much weaker results in Tab. 2 The marginal distributions (9) for pairwise distances in visualized in Appendix K which Turther demonstrate the superior capacity of our proposed method.
We also follow Mansimov et al. (019)) to calculate the diversity of conformations generated by all compared methods, which measured by calculating the mean and standard deviation of the pairwise RMSD between each pair of generated conformations per molecule. The results shown in Tab. 3 demonstrate that while method can achieve the lowest MMD, it does not collapse to generating extremely similar conformations. Besides, we observe that ET' will slightly hurt the diversity of CGCF, which verifies our statement that ETM will sharpen the generated distribution towards the stable conformations with local minimal energy.
5 CONCLUSION AND FUTURE WORK
In this paper, we propose a novel probabilistic framework for molecular conformation generation. Our generative model combines the advantage of both flow-based and energy-based models, which is capable of modeling the complex multi-moda geometric distribution and highly branched atomic correlations. Experimental results show that our method outperforms all previous state-of-the-an baselines on the standard benchmarks. Future work includes applying our framework on much larger datasets and extending more challenging structures (e.g., proteins).
ACKNOWLEDGMENTS
This project is supported by the Natural Sciences and Engineering Research Council (NSERC) Dis- covery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft Re- search and Mila, Samsung Electronics Co., Ldt., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bird Gift Fund and a NRC Collaborative R&D Project also partially funded by IVADO Fundamental Research Project grant PRF-2019-35831197277
This project was

---

Published as a conference paper at ICLR 2021
REFERENCES
Mohammed AlQuraishi. End-to-end differentiable learning of protein structure. Cell systems, 8(4):
292-301, 2019.
Simon Axelrod and Rafael Domez-Bombarreli.. Geom:
anmotated molecular conformations for property prediction and molecular generation. arXiv preprint 2rXi::2006.05531,, 2020.
Andrew Ballard, Stefano Martiniani, Jacob D Stevenson, Sandeep Somani, and David J Wales. Exploiting the potential energy landscape to sample free energy. Wiley Interdisciplinary Reviews: Computational Molecular Science, 5(3):273-289 2015.
Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep represen- tations. International conference on machine learning, pp. 552-560, 2013a.
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. In Advances in neural information processing systems, pp. 899-907, 2013b.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Gordon M Crippen, Timothy F Havel, al. Distance geomelry and molecular conformation, vol- ume 74, Research Studies Press Taunton, 1988.
Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel. The helmholtz machine. Neural computation, 7(5):889-904, 1995.
Marco De Vivo, Matteo Masetti, Giovanni Bottegoni, and Andrea Cavalli. Role of molecular dy- namics and related methods in drug discovery. Journal of medicinal chemistry, 59(9):4035-4061, 2016.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv: 10.85566 2014.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv: 1903. 08689, 2019.
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation 3d point sets for the targeted discovery of molecules. In Advances in Neural Information Processing Sys- tems, pp. 7566-7578, 2019.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. arXiv preprint arXiv: 2017.
Will Grathwohi, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01366. 2018.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch. Berhhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Inielligence and Statistics, pp. 297-304, 2010.
Thomas A Halgren. Merck molecular force field. i. basis, form, scope, parameterization, and per- formance of mmf194. Journal of computational chemistry, 17(5-6):40-519, 1996a.
Thomas A Halgren. Merck molccular force field. v. extension of mmff94 using experimental data, additional computational data, and empirical rules. Journal of Computational Chemistry, 17(5-6): 616-641, 1996b.
Paul CD Hawkins, Conformation generation: the state of the art. Journal of Chemical Information and Modeling, 177--1777 2017.
10

---

Published as a conference paper at ICLR 2021
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-007, 2006.
Moritz Hoffmann and Frank Noé. Generating valid euclidean distance matrices. arXiv preprint 1910003900,,111,, 2019.
John Ingraham, Adam Riesselman, Chris Sander, and Debora $ Marks. Learning protein structure with a differentiable simulator. In International Conference on Learning Representations, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. A tutorial on energy-based learning. Predicting structured data, 100), 2006.
Tobias Lemke and Christine Peter. Encodermap: Dimensionality reduction and generation of
molecule conformations. Journal of chemical theory and computation, 15(2):1209 1215, 2019.
Leo Liberti, Carlile Lavor, Nelson Maculan, and Antonio Mucherino. Euclidean distance geometry
and applications. SIAM review, 56(1):3-66 69, 2014.
Elman Mansimov, Omar Mahmood, Seokho Kang, and Kyunghyun Cho. Molecular geometry pre- diction using a deep generative graph neural network. arXiv preprint arXiv:1904.0033 2019.
Jiquan Ngiam, Zhenghao Chen, Pang W Koh, and Andrew Y Ng. Learning deep energy models. In Proceedings of the 28th international conference machine learning (ICMI--I pp- 1105- 1112, 2011.
Frank Noé. Simon Olsson, Jonas Köhler, and Hao Wu. Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning. Science, 365(6457) 147, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):177, 2014.
Anthony K Rappé. Carla J Casewit, KS Colwell, William A Goddard III, and W Mason Skiff. Uff, a full periodic table force field for molecular mechanics and molecular dynamics simulations. Journal of the American chemical society, 114(25):10024-10 100355 1992.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770 2015.
Sereina Riniker and Gregory A Landrum. Better informed distance geometry: using what we know to improve conformation generation. Journal of chemical information and modeling, 55(12): 2562-2574, 2015.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: continuous-fitte convolutional neural network for modeling quantum interactions. In Advances in neural information processing systems, pp- 991-1001,
Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Zidek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning, Nature, 577(7792):06-7110. 2020.
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang- Graphaf: a flow- autoregressive model for molecular graph generation. arXiv preprint arXiv:2001.09382. 2020.
Gregor NC Simm and José Miguel Heránndez-Lobaee A generative model for molecular distance geometry. In International Conference on Machine Learning, 2020.

---

Published as a conference paper at ICLR 2021
Justin S Smith, Olexandr Isayev, and Adrian E Roitberg. Ani-1: extensible neural network potential with dft accuracy at force field computational cost. Chemical science, (44::3122-3203, 2017.
Yuxuan Song, Qiwei Ye, Minkai Xu, and Tie-Yan Liu. Discriminator contrastive divergence: Semi-amortized generative modeling by exploring energy of the discriminator. arXiv preprint 1rXiv:2004.01704. 2020.
Jianwen Xie, Yang Lu, Song-Chur Zhu, and Yingnian Wu. theory of generative convnet. In International Conference Machine Learning, pp. 2635-2644, 2016.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional pol- icy network for goal-directed molecular graph generation. In Advances in neural information processing systems, pp. 6410-6421 2018.
12

---

Published as a conference paper at ICLR 2021
A RELATED WORKS
Conformation Generation. There have been results showing deep learning speeding up molecular dynamics simulation by learning efficient alternatives to quantum mechanics-based energy calcu- lations (Schütt et al., 2017, Smith et al,, 2017). However, though accelerated by neural networks, still time-consuming due the lengthy MCMC process. Recently, /ebauer et al. (219)) and Hoffmann & Noë (2019) propose to directly generate 3D structures with deep gen- erative models. However, these models can hardly capture graph- or bond-based structure, which is typically complex and highly branched. Some other works (Lemke & Peter 2019) [AQuuaishhi 2019 Ingrahaam et al. 2019, Noé al. 2019) Senior et al. 2020) also focus on learning models to directly generate 3D structure, but focus on the protein folding problem. Unfortunately, proteins are linear structures while general molecules are highly branched, making these methods not naturally transferable to general molecular conformation generation tasks.
these approaches
Energy-based Generative Model. There has been long history for energy-based generative mod- els. Xie et al. (2016) proposes to train an energy-basee model parameterized by modern deep neural network and learned it by Langevin based MLE. The model is called generative ConvNet since be derived from the discriminative ConvNet. In particular, this paper is the first formulate modern ConvNet--arametrian it to discriminative ConvNet classifier. More recently, Du & Mordatch (2019) implemented the deep EBMs with ConvNet as energy function and achieved impressive results on image generation.
EBM as exponential tilting of a reference distribution, and connect
Different from the previous works, we concentrate on molecular geometry generation, and propose a novel and principled probabilistic framework to address the domain-specific problems. More specifically, we first predict the atomic distances through the continuous normalizing flow, and then convert them to the desired 3D conformation and optimize it with the energy-based model. This procedure enables us to keep the rotational and translational invariance property. Besides, to the best of our knowledge, we are the first one to combine neural ODE with EBMs. We take the ODE model to improve the training of EBM, and combine both to conduct the two-stage sampling dynamics.
B DATA PREPROCESS
Inspired by classic molecular distance geometry (Crippen et al. 1988), we also generate the con- firmations by firstly predicting all the pairwise distances, which enjoys the invariant property to rotation and translation. Since the bonds existing the molecular graph are not sufficient to char- acterize a conformation, we pre-process the graphs by extending auxiliary edges. Specifically, the atoms that are bonds of the original graph. These extra edges contribute to reducing the degrees of freedom in the 3D coordinates, with the edges between second neighbors helping to fix the angles between atoms, and those between third neighbors fixing dihedral angles.
hops away are connected with virtual bonds, labeled differently from the real
C NETWORK ARCHITECTURE
In this section, we elaborate on the network architecture details of CGCF and ETM.
C.I CONTINUOUS GRAPH FLOW
In CGCF, the dynamic function fo defined in Eq. 6 is instanced with a message passing neural networks. Given the node attributes, edge attributes and intermediate edge lengths input, first embed them into the feature space through feedforward networks:
h (0) NodeeFnbeckding(v), V V V,
(14)
heg. EdgeEmbedding((en Eunry duo (lo)), Cuv E
Then, the node and edge features along are passed sequentially into layers message passing net- works with the graph structure g:
h = MLP ((-11)
Σ
a(h((-t) + hew!),, f= 1_....A ...LT
(15)
MENo(1)
13

---

Published as a conference paper at ICLR 2021
where Ng(u) denotes the first neighbors in the graph and is the activation function. After L message passing layers,
the final hidden representation (L)) as the node representations.
Then for each bond, the corresponding node features are aggregated along with the edge feature to be fed into neural network to compute the value of the dynamic fo:
Odwr NN(ha: Res hewn t).
(16)
C.2 ENERGY-BASED TILTING MODEL
The ETM implemented with SchNet. takes both the graph and conformation information as input and output a scalar to indicate the energy level. Let the atoms are described by tuple of features X (x).... xx,,, where denote the number of atoms and denote the layer. Then given the positions R, the node embeddings are updated by the convolution with all surrounding atoms:
x (x' W
Σ
(r,-;))
(17)
where "o" represents the element-wise multiplication. It is straightforward that the above function enables to include translational and rotational invariance by computing pairwise distances instead of using relative positions. After L convolutional layers. we perform sum-pooling operator over the node embeddings to calculate the global embedding for the whole molecular structure. Then the global embedding fed into feedforward network compute the scalar of the energy level.
D TWO-STAGE DYNAMIC SYSTEM FOR SAMPLING
Algorithm Sampling Procedure of the Proposed Method
molecular graph G. CGCF model with parameter 0 ETM with parameter Φ, the number of optimization steps for p( R,d, G) M and its step size the number of MCMC steps for Eg, N and its step size
Input:
Output: molecular conformation R
1: Sample d(to) Z)
2: d = FF((t)))) G)
3: for m L,..., M do
4: Rs, R + rv log p( R|d,9)
5: end for
6: for 1,..., N do
7: R, Rn-1 1 V n. Ea. (Rg) Vew,w ~ NOO,
8: end for
E IMPLEMENTATION DETAILS
Our model is implemented in PyTorch (Paskke et al., 2017). The MPNN in CGCF implemented with 3 layers, and the embedding dimension set as 128. And the SchNet in ETM is implemented with layers with the embedding dimension set as 128. We train our CGCF with a batch size 128 and learning rate of 0.001 until convergence. After obtaining the CGCF, train the ETM with batch size of 384 and a learning rate of 0.001 until convergence. For all experimental settings, we Adam (Kingma & Ba 2014) optimize model.
14

---

Published as a conference paper at ICLR 2021
F DETAILED DERIVATIONS OF ENERGY-BASED MODEL
Here we present the detailed derivations of the training objective function of thergy-basee Tilting Model (ETM) Eq.
g
Pa.e (Rl
pa [
Lace R. () = Epasi log
Ep log
p8,0 [ + Po R (g)
Pe.0
[RG)) pe (Rl
Po (o( exp( (R)
E pussa
log
po (RR () cxp( Eg R2 9)) Po (R(g)
(18)
Pa R
Epo [log
po (R(g) exp( - E R, 9)) po (R()) 1
Epases log
Eps log
+ exp((=Es( 9))
1 exp (R)
Es
G MORE GENERATED SAMPLES
We present more visualizations of generated 3D structures in Fig. which generated from model (CGCF + ETM) learned on both GEOM-QM9 and GEOM- Drugs datasets. The visualizations demonstrate that our proposed framework holds the high capacity model the chemical structures in the 3D coordinates.
Graph
Conformations
Figure 3: Visualizations of generated graphs from our proposed method. In each row, we show multiple generated conformations for one molecular graph. For the top 5 rows, the graphs are chosen from the small molecules in GEOM QQ9 test dataset; and for the bottom 4 rows, graphs chosen from the larger molecules GEOM- Drugs test dataset. O, and are colored gray, red, white, yellow and green respectively.
15

---

Published as a conference paper at ICLR 2021
H MORE RESULTS OF COVERAGE SCORE
We give more results of the coverage (COV) score with different threshold ó in Fig. As shown in the figure, proposed method can consistently outperform the previous state-of-the-an baselines CVGAE and GraphDG, which demonstrate the effectiveness of our model.
GEOM-QM9
GEOM-QM9 (FF)
GEOM-Drugs
GEOM-Drugs (FF)
CGCF+ET
CGCF-ETT
CBCF+ETN
0.8
GrapeDd CUGAE
0.8
Grap-DG
0.8
GraPNDG
0.8
GraphDD
CYGAE
CNGAE
CUSAE
Coverage 56
0.6
KOKA
ROKA
0.4
0.2
0.2
0.2
0.2
0.00
0.25
0.50
0.75
1.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
3.0
1.5
2.0
0.0
0.5
3.0
1.5
2.0
RMSD
RMSD
RMSD
RMSD
Figure 4: Curves of the averaged coverage score with different RMSD thresholds on GEOM-QM9 (left two) and GEOM-Drugs (right two) datasets. The first and third curves are results of only the generative models, while the other two results when further optimized with rule-based force fields.
I IMPLEMENTATION FOR MMFF
In this section, we give more precise description of the MMFF Force Field implementation in the RDKit toolkit (Riniker & Landrum 2015).
In MMFF, the energy expression is constituted by seven terms: bond stretching, angle bending, stretch-bend, out-of-plane bending, torsional, van der Waals and electrostatic. The detailed func- tional form of individual terms can found in the original literature (Halgren 1996a). To build the force field for given molecular system, the first step is to assign correct types to each atom. At the second step, atom-centered partial charges are computed according to the MMFF charge model (Hal-) gren 1996b). Then, all bonded and non-bonded interactions in the molecular system under study, depending on its structure and connectivity, are loaded into the energy expression. Optionally, exter- nal restraining terms be added to the MMFF energy expression, with the purpose of constraining selected internal coordinates during geometry optimizations. Once all bonded and non-bonded in- teractions, plus optional restraints, have been loaded into the MMFF energy expression, potential gradients of the system under study can be computed to minimize the energy.
J MORE EVALUATIONS FOR CONFORMATION GENERATION
Junk Rate. The COV and MAT score in Section 4.2 do not appear to explicitly measure the gener- ated false samples. Here we additionally define Junk rate measurement. Intuitively. JUNK measures the fraction of generated conformations that far away from all the conformations in the reference set. For each conformation in the generated set, it will be marked as false sample if its RMSD to all the conformations of reference set are above a given threshold 8:
(S) ())
S {
R { S, RMSD R. R') > S E e..
(19)
Typically, a lower JUNK rate means better generated quality. The results are shown Tab. 4 As shown in the table, our CGCF model can already outperform the existing state-of-the-art baselines with an obvious margin. The results are further improved when combined with ETM to explicitly correlations.
incorporate the
K DISTANCE DISTRIBUTION VISUALIZATION
In Fig. 5, we plot the marginal distributions p(d,vll) for all pairwise distances between C and O atoms of molecular graph in the IS017 test set. As shown in the figure, though primarily designed
16

---

Published as a conference paper at ICLR 2021
Table 4: Comparison of different methods the JUNK Top 4 rows: deep generative models for molecular conformation generation. Bottom rows: different methods that involve an additional rule-based force field to further optimize the generated structures.
Dataset
GEOM-QMA
GEOM-Drugs
JUNK (%)
JUNK (%)
Metric
Mean
Median
Mean
Median
CVGAE
71.59
100.00
100.00
100.00
GraphDG
61.25
66.26
97.83
100.00
CGCF
55.24
57.24
77.82
90.00
CGCF ETM
52.15
54.23
75,81
88.64
RDKit
17.07
5.90
45.51
45.94
CVGAE + FF
62.92
71.21
72.01
78.44
GraphDG + FF
45.53
46.35
55.50
61.54
CGCF + FF
43.01
46.69
37.48
36.63
CGCF + ETM FF
41.63
43.97
36.16
33.05
For the reported JUNK score, the threshold 5 is set as 0.5A for QM9 and 1.25A for Drugs.
for 3D structure generation, our method can make much better estimation of the distances than GraphDG, which is the state-of-the-arr model for molecular geometry prediction. As a representative element of the pairwise property between atoms, the inter-atomic distances demonstrate the capacity of our model to capture the inter-atomic interactions.
DE
23
Jec 0-4
Meni
Mesil
GraphDG
Ours
275 2.50
Figure Marginal distributions p(duv G) ground-truth and generated conformations between C and atoms given a molecular graph from the test set of ISO17. each subplot, the annotation (u v) indicates atoms connected by the corresponding bond duv- We concentrate on the heavy atoms (C and O) and omit the atoms for elarity.
17


Reviews:
[{"review_id": "TgmbFYL30tU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper combines flow-based and energy-based models to generate molecular conformations from a molecular graph.\n\nStrengths\n- The modeling is well-motivated and creative. A continuous flow model maps a graph-based molecular representation into a distribution over conformations. The continuous flow is well-motivated; it factors out the distance calculations to resolve concerns in previous works where pairwise distances are modeled independently. \n- Additionally, an energy-based model (EBM) is used to further help the model capture long-range atomic interactions. Recent advancements in EBMs are deployed for stable training, such as a lavengin dynamics.\n- The paper is well-written, clear and easy to follow, despite the complexity of the model\n- Strong baselines: CVGAE, GraphDG, and RDKit. Results are slightly better than CVGAE for conformation generation, and slightly better for distance distribution. \n\nWeaknesses\n- While the model performs better than neural baselines, it does not yet approach RDKit, which is based on expert knowledge features. This means the proposed model is not yet practically useful.\n- Only two tasks were investigated, and only one or two datasets within each task. The empirical evaluation of this paper could be improved by including more experiments (e.g. along the lines of Mansimov et al. 2019)\n- The paper attributes the advantage in RDKit to its inclusion of a force field. Therefore, the authors combine a force field with their model and assess the difference (good!) However, the conclusion in the paper seems misguided - looking at Table 1, including the force field does not result in a significant change in their model's performance. In fact, this experiment was also done in the CVGAE paper (Mansimov et al. 2019).\n- In Table 1, the EBM improves performance on the conformation generation tasks. However, in Table 2, the EBM decreases performance on the distribution over distances task. I would have liked to see more investigation or discussion of why this is the case.\n- The RDKit baseline is not included in Table 2. It is included in the GraphDG paper and should be included here as well.\n\nOverall, this paper presents a creative model that performs better than existing neural models. However, the evaluation feels preliminary compared to what we see in earlier works.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PeQs67uJBHa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this manuscript proposed a generative dynamics system for the modelling and generation of 3D conformations of molecules. Specifically, there are three components: (1) conditional graph continuous flow (CGCF) to transform random noise to distances,  (2)a closed-form distribution p(R|d, G), and (3) an energy-based tilting model (ETM) to capture long-range interactions and correct the position matrix distribution. The proposed framework was compared with two deep learning methods for conformation generations -- CVGAE & GraphDG, as well as the computational chemistry tool RDKit on GEOM-QM9, GEOM-Drugs, and ISO17 data sets. Comparisons in terms of COV and MAT scores show that the proposed method (particularly the one enhanced with ETM) can outperform baselines. Further comparisons of distances densities show that CGCF (but without ETM) worked best over baselines. \n\nOverall, I think it is an interesting work. The major novelty is the use of continuous flow to model the conditional distribution of the distances and an energy-based model to correct the conditional distribution of positions. However, I have the following concerns.\n\n1. The presentation of this paper can be significantly improved. A few typos need to be corrected:\nsection x -? Section X\na optimization -> an optimization\nwhich a ...-> which is a...\ndemotes -> denotes\nreferences should be further formatted\n\n2. A bit more precise description about the force-fields algorithms in RDKit is needed.\n \n3. From results in Table 2, CGCF combined with the ETM component does work better than GraphDG, although the authors state that it is because the sharpness of the distance distribution. Clear justifications should be given to show the benefits, if any, of this phenomenon. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "LnQoFGQTj5y", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to generate diverse small molecule conformations given its graph by combining a conditional flow-based model with an energy-based model. Sampling is performed in two separate stages: 1) a normalizing flow produces a distribution over interatomic distances (which is then postprocessed into cartesian coordinates), 2) sampled coordinates are refined by Langevin dynamics with gradient signal produced from an energy-based model. The models are trained separately.\n\nI thought that this was an interesting, principled approach that advances the state of the art in generative models for molecular conformation sampling.  This approach (CGCF+ETM) appears to generate better samples than the VAE-based baselines, though all methods are still improved by a final refinement step with traditional forcefields (esp. for larger drug-like molecules) which suggests room for improvement. The general sampling strategy is potentially relevant in many other domains.\n\nThere were some aspects of the evaluation that could be improved/clarified:\n- COV and MAT do not appear to measure false positives — the CGCF+ETM approach could be generating many junk conformations and this would not be captured by the COV/MAT.  Can the authors report a statistic over the generated distribution (instead of over the test set)?\n- The abstract/introduction focuses on the computational expense of competing approaches. Can the authors comment on the computationally expense for generating each conformation?\n- How do the molecular properties of the generated distribution compare to the test distribution (e.g. small molecule properties in Simm & Hernandez-Lobato Table 2)?\n- Can the authors clarify if any of the test set molecules from GEOM-QM9 or GEOM-DRUGS are in the training set? It was unclear from the writing.\n- Which bonds are in the set of interatomic distances? The authors defines distances as the set of all covalent bonds in the molecule in the methods section, but later mention auxiliary angle/dihedral bonds. This was confusing in the first read.\n- Typo in Eq (9) ‘==‘", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper combines flow-based and energy-based models to generate molecular conformations given a molecular graph.
For this, a continuous flow model is used to map the graph-based molecular representation into a distribution over conformations. 
An energy-based model (EBM) is used to further help the model capture long-range atomic interactions. The proposed method is compared with strong baselines: CVGAE, GraphDG, and RDKit.

The authors addressed most of the reviewers' concerns in the rebuttal.

All the reviewers agree on acceptance.
Decision: Accept (Poster)