LEARNING NEURAL GENERATIVE DYNAMICS FOR MOLECULAR CONFORMATION GENERATION
Year: 2021

Abstract:
We study how to generate molecule conformations (i.e., 3D structures) from a molecular graph. Traditional methods, such as molecular dynamics, sample conformations via computationally expensive simulations. Recently, machine learning methods have shown great potential by training on a large collection of conformation data. Challenges arise from the limited model capacity for capturing complex distributions of conformations and the difficulty in modeling long-range dependencies between atoms. Inspired by the recent progress in deep generative models, in this paper, we propose a novel probabilistic framework to generate valid and diverse conformations given a molecular graph. We propose a method combining the advantages of both flow-based and energy-based models, enjoying: (1) a high model capacity to estimate the multimodal conformation distribution; (2) explicitly capturing the complex long-range dependencies between atoms in the observation space. Extensive experiments demonstrate the superior performance of the proposed method on several benchmarks, including conformation generation and distance modeling tasks, with a significant improvement over existing generative models for molecular conformation sampling1.

heading: 1 INTRODUCTION
text: Recently, we have witnessed the success of graph-based representations for molecular modeling in a variety of tasks such as property prediction (Gilmer et al., 2017) and molecule generation (You et al., 2018; Shi et al., 2020). However, a more natural and intrinsic representation of a molecule is its 3D structure, commonly known as the molecular geometry or conformation, which represents each atom by its 3D coordinate. The conformation of a molecule determines its biological and physical properties such as charge distribution, steric constraints, as well as interactions with other molecules. Furthermore, large molecules tend to comprise a number of rotatable bonds, which may induce flexible conformation changes and a large number of feasible conformations in nature. Generating valid and stable conformations of a given molecule remains very challenging. Experimentally, such structures are determined by expensive and time-consuming crystallography. Computational approaches based on Markov chain Monte Carlo (MCMC) or molecular dynamics (MD) (De Vivo et al., 2016) are computationally expensive, especially for large molecules (Ballard et al., 2015).
Machine learning methods have recently shown great potential for molecular conformation generation by training on a large collection of data to model the probability distribution of potential conformations R based on the molecular graph G, i.e., p(R|G). For example, Mansimov et al.
∗Equal contribution. Work was done during Shitong’s internship at Mila. 1Code is available at https://github.com/DeepGraphLearning/CGCF-ConfGen.
(2019) proposed a Conditional Variational Graph Autoencoders (CVGAE) for molecular conformation generation. A graph neural network (Gilmer et al., 2017) is first applied to the molecular graph to get the atom representations, based on which 3D coordinates are further generated. One limitation of such an approach is that by directly generating the 3D coordinates of atoms it fails to model the rotational and translational invariance of molecular conformations. To address this issue, instead of generating the 3D coordinates directly, Simm & Hernández-Lobato (2020) recently proposed to first model the molecule’s distance geometry (i.e., the distances between atoms)—which are rotationally and translationally invariant—and then generate the molecular conformation based on the distance geometry through a post-processing algorithm (Liberti et al., 2014). Similar to Mansimov et al. (2019), a few layers of graph neural networks are applied to the molecular graph to learn the representations of different edges, which are further used to generate the distances of different edges independently. This approach is capable of more often generating valid molecular conformations.
Although these new approaches have made tremendous progress, the problem remains very challenging and far from solved. First, each molecule may have multiple stable conformations around a number of states which are thermodynamically stable. In other words, the distribution p(R|G) is very complex and multi-modal. Models with high capacity are required to model such complex distributions. Second, existing approaches usually apply a few layers of graph neural networks to learn the representations of nodes (or edges) and then generate the 3D coordinates (or distances) based on their representations independently. Such approaches are necessarily limited to capturing a single mode of p(R|G) (since the coordinates or distances are sampled independently) and are incapable of modeling multimodal joint distributions and the form of the graph neural net computation makes it difficult to capture long-range dependencies between atoms, especially in large molecules.
Inspired by the recent progress with deep generative models, this paper proposes a novel and principled probabilistic framework for molecular geometry generation, which addresses the above two limitations. Our framework combines the advantages of normalizing flows (Dinh et al., 2014) and energy-based approaches (LeCun et al., 2006), which have a strong model capacity for modeling complex distributions, are flexible to model long-range dependency between atoms, and enjoy efficient sampling and training procedures. Similar to the work of Simm & Hernández-Lobato (2020), we also first learn the distribution of distances d given the graph G, i.e., p(d|G), and define another distribution of conformations R given the distances d, i.e., p(R|d,G). Specifically, we propose a novel Conditional Graph Continuous Flow (CGCF) for distance geometry (d) generation conditioned on the molecular graph G. Given a molecular graph G, CGCF defines an invertible mapping between a base distribution (e.g., a multivariate normal distribution) and the molecular distance geometry, using a virtually infinite number of graph transformation layers on atoms represented by a Neural Ordinary Differential Equations architecture (Chen et al., 2018). Such an approach enjoys very high flexibility to model complex distributions of distance geometry. Once the molecular distance geometry d is generated, we further generate the 3D coordinates R by searching from the probability p(R|d,G). Though the CGCF has a high capacity for modeling complex distributions, the distances of different edges are still independently updated in the transformations, which limits its capacity for modeling long-range dependency between atoms in the sampling process. Therefore, we further propose another unnormalized probability function, i.e., an energy-based model (EBM) (Hinton & Salakhutdinov, 2006; LeCun et al., 2006; Ngiam et al., 2011), which acts as a tilting term of the flow-based distribution and directly models the joint distribution of R. Specifically, the EBM trains an energy function E(R,G), which is approximated by a neural network. The flow- and energy-based models are combined in a novel way for joint training and mutual enhancement. First, energy-based methods are usually difficult to train due to the slow sampling process. In addition, the distribution of conformations is usually highly multi-modal, and the sampling procedures based on Gibbs sampling or Langevin Dynamics (Bengio et al., 2013a;b) tend to get trapped around modes, making it difficult to mix between different modes (Bengio et al., 2013a). Here we use the flow-based model as a proposal distribution for the energy model, which is capable to generate diverse samples for training energy models. Second, the flow-based model lacks the capacity to explicitly model the long-range dependencies between atoms, which we find can however be effectively modeled by an energy function E(R,G). Our sampling process can be therefore viewed as a two-stage dynamic system, where we first take the flow-based model to quickly synthesize realistic conformations and then used the learned energy E(R,G) to refine the generated conformations through Langevin Dynamics.
We conduct comprehensive experiments on several recently proposed benchmarks, including GEOM-QM9, GEOM-Drugs (Axelrod & Gomez-Bombarelli, 2020) and ISO17 (Simm & Hernández-Lobato, 2020). Numerical evaluations show that our proposed framework consistently outperforms the previous state-of-the-art (GraphDG) on both conformation generation and distance modeling tasks, with a clear margin.

heading: 2 PROBLEM DEFINITION AND PRELIMINARIES

heading: 2.1 PROBLEM DEFINITION
text: Notations. Following existing work (Simm & Hernández-Lobato, 2020), each molecule is represented as an undirected graph G = 〈V, E〉, where V is the set of nodes representing atoms and E is the set of edges representing inter-atomic bonds. Each node v in V is labeled with atomic properties such as element type. The edge in E connecting u and v is denoted as euv , and is labeled with its bond type. We also follow the previous work (Simm & Hernández-Lobato, 2020) to expand the molecular graph with auxiliary bonds, which is elaborated in Appendix B. For the molecular 3D representation, each atom in V is assigned with a 3D position vector r ∈ R3. We denote duv = ‖ru − rv‖2 as the Euclidean distance between the uth and vth atom. Therefore, we can represent all the positions {rv}v∈V as a matrix R ∈ R|V|×3 and all the distances between connected nodes {duv}euv∈E as a vector d ∈ R|E|. Problem Definition. The problem of molecular conformation generation is defined as a conditional generation process. More specifically, our goal is to model the conditional distribution of atomic positions R given the molecular graph G, i.e., p(R|G).

heading: 2.2 PRELIMINARIES
text: Continuous Normalizing Flow. A normalizing flow (Dinh et al., 2014; Rezende & Mohamed, 2015) defines a series of invertible deterministic transformations from an initial known distribution p(z) to a more complicated one p(x). Recently, normalizing flows have been generalized from discrete number of layers to continuous (Chen et al., 2018; Grathwohl et al., 2018) by defining the transformation fθ as a continuous-time dynamic ∂z(t) ∂t = fθ(z(t), t). Formally, with the latent variable z(t0) ∼ p(z) at the start time, the continuous normalizing flow (CNF) defines the transformation x = z(t0) + ∫ t1 t0 fθ(z(t), t)dt. Then the exact density for pθ(x) can be computed by:
log pθ(x) = log p(z(t0))− ∫ t1 t0 Tr ( ∂fθ ∂z(t) ) dt (1)
where z(t0) can be obtained by inverting the continuous dynamic z(t0) = x + ∫ t0 t1 fθ(z(t), t)dt. A black-box ordinary differential equation (ODE) solver can be applied to estimate the outputs and inputs gradients and optimize the CNF model (Chen et al., 2018; Grathwohl et al., 2018).
Energy-based Models. Energy-based models (EBMs) (Dayan et al., 1995; Hinton & Salakhutdinov, 2006; LeCun et al., 2006) use a scalar parametric energy function Eφ(x) to fit the data distribution. Formally, the energy function induces a density function with the Boltzmann distribution pφ(x) = exp(−Eφ(x))/Z(φ), where Z = ∫ exp(−Eφ(x)) dx denotes the partition function. EBM can be learned with Noise contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010) by treating the normalizing constant as a free parameter. Given the training examples from both the dataset and a noise distribution q(x), φ can be estimated by maximizing the following objective function:
J(φ) = Epdata [ log
pφ(x)
pφ(x) + q(x)
] + Eq [ log
q(x)
pφ(x) + q(x)
] , (2)
which turns the estimation of EBM into a discriminative learning problem. Sampling from Eφ can be done with a variety of methods such as Markov chain Monte Carlo (MCMC) or Gibbs sampling (Hinton & Salakhutdinov, 2006), possibly accelerated using Langevin dynamics (Du & Mordatch, 2019; Song et al., 2020), which leverages the gradient of the EBM to conduct sampling:
xk = xk−1 − 2 ∇xEφ (xk−1) +
√ ω, ω ∼ N (0, I), (3)
where refers to the step size. x0 are the samples drawn from a random initial distribution and we take the xK withK Langevin dynamics steps as the generated samples of the stationary distribution.
Published as a conference paper at ICLR 2021
<latexit sha1_base64="vIs0DtZMVgyrXYZyWvMVVQo2iJ8=">AAAB8nicbVDLSgMxFM3UV62vqks3wSK4kDIjBXVXcKHLCvYB06Fk0kwbmkmG5I5Qhn6GGxeKuPVr3Pk3ZtpZaOuBwOGce8m5J0wEN+C6305pbX1jc6u8XdnZ3ds/qB4edYxKNWVtqoTSvZAYJrhkbeAgWC/RjMShYN1wcpv73SemDVfyEaYJC2IykjzilICV/H5MYEyJyO5mg2rNrbtz4FXiFaSGCrQG1a/+UNE0ZhKoIMb4nptAkBENnAo2q/RTwxJCJ2TEfEsliZkJsnnkGT6zyhBHStsnAc/V3xsZiY2ZxqGdzCOaZS8X//P8FKLrIOMySYFJuvgoSgUGhfP78ZBrRkFMLSFUc5sV0zHRhIJtqWJL8JZPXiWdy7rXqN88NGrNi6KOMjpBp+gceegKNdE9aqE2okihZ/SK3hxwXpx352MxWnKKnWP0B87nD3aukVM=</latexit>
<latexit sha1_base64="ZURxyFRkumcoZhtLB9tuZDzUxpQ=">AAACFHicbVDLSgMxFM34rPU16rKbYBEqSJmRgroruNGNVLAP6JSSSTNtaJIZkoxQhln4E/6CW927E7fu3folZtpZ2NYDIYdz7uXee/yIUaUd59taWV1b39gsbBW3d3b39u2Dw5YKY4lJE4cslB0fKcKoIE1NNSOdSBLEfUba/vg689uPRCoaigc9iUiPo6GgAcVIG6lvlzyO9AgjltylFc/niZOewey/TU/7dtmpOlPAZeLmpAxyNPr2jzcIccyJ0JghpbquE+legqSmmJG06MWKRAiP0ZB0DRWIE9VLpkek8MQoAxiE0jyh4VT925EgrtSE+6YyW1ktepn4n9eNdXDZS6iIYk0Eng0KYgZ1CLNE4IBKgjWbGIKwpGZXiEdIIqxNbnNTfJ4WTSjuYgTLpHVedWvVq/tauV7J4ymAEjgGFeCCC1AHN6ABmgCDJ/ACXsGb9Wy9Wx/W56x0xcp7jsAcrK9foQCeKQ==</latexit>
<latexit sha1_base64="S5vzONPPGcE2Bnxqdal57hBhNUE=">AAACAnicbVDLSsNAFJ3UV62vqks3g0Wom5JIQd0V3LisYB+QhjKZTNqh8wgzE6GE7PwFt7p3J279Ebd+idM2C9t64MLhnHu5954wYVQb1/12ShubW9s75d3K3v7B4VH1+KSrZaow6WDJpOqHSBNGBekYahjpJ4ogHjLSCyd3M7/3RJSmUjyaaUICjkaCxhQjYyV/EPIsyutm6F4OqzW34c4B14lXkBoo0B5WfwaRxCknwmCGtPY9NzFBhpShmJG8Mkg1SRCeoBHxLRWIEx1k85NzeGGVCMZS2RIGztW/ExniWk95aDs5MmO96s3E/zw/NfFNkFGRpIYIvFgUpwwaCWf/w4gqgg2bWoKwovZWiMdIIWxsSktbQp5XbCjeagTrpHvV8JqN24dmrVUv4imDM3AO6sAD16AF7kEbdAAGEryAV/DmPDvvzofzuWgtOcXMKViC8/UL8JyXWw==</latexit>
<latexit sha1_base64="F31lh5CKHaiA3aXYW2ZI+Y7ONAQ=">AAAB/3icbVA9SwNBEJ2LXzF+RS1tFoNgIeFOAmoXEMQygvmA5Ah7m71kye7dsTsnhJDCv2CrvZ3Y+lNs/SVukitM4oOBx3szzMwLEikMuu63k1tb39jcym8Xdnb39g+Kh0cNE6ea8TqLZaxbATVciojXUaDkrURzqgLJm8Hwduo3n7g2Io4ecZRwX9F+JELBKFqpddft4IAj7RZLbtmdgawSLyMlyFDrFn86vZilikfIJDWm7bkJ+mOqUTDJJ4VOanhC2ZD2edvSiCpu/PHs3gk5s0qPhLG2FSGZqX8nxlQZM1KB7VQUB2bZm4r/ee0Uw2t/LKIkRR6x+aIwlQRjMn2e9ITmDOXIEsq0sLcSNqCaMrQRLWwJ1KRgQ/GWI1gljcuyVynfPFRK1YssnjycwCmcgwdXUIV7qEEdGEh4gVd4c56dd+fD+Zy35pxs5hgW4Hz9AkIZln0=</latexit>
<latexit sha1_base64="3uFAD5t20r5hkITaCaR1FRhMgvY=">AAACAnicbVDLSsNAFJ3UV62vqks3g0Wom5JIQd0V3LisYB+QhjKZTNqh8wgzE6GE7PwFt7p3J279Ebd+idM2C9t64MLhnHu5954wYVQb1/12ShubW9s75d3K3v7B4VH1+KSrZaow6WDJpOqHSBNGBekYahjpJ4ogHjLSCyd3M7/3RJSmUjyaaUICjkaCxhQjYyV/EPIsyutm6F0OqzW34c4B14lXkBoo0B5WfwaRxCknwmCGtPY9NzFBhpShmJG8Mkg1SRCeoBHxLRWIEx1k85NzeGGVCMZS2RIGztW/ExniWk95aDs5MmO96s3E/zw/NfFNkFGRpIYIvFgUpwwaCWf/w4gqgg2bWoKwovZWiMdIIWxsSktbQp5XbCjeagTrpHvV8JqN24dmrVUv4imDM3AO6sAD16AF7kEbdAAGEryAV/DmPDvvzofzuWgtOcXMKViC8/UL8jGXXA==</latexit>
<latexit sha1_base64="BSPQokcnd67Ox4bSGWxni9WSepw=">AAACF3icbVBNS8NAEN3Ur1q/qh4FWSxCBSmJFNRbwYMeK9gPaErZbDft0t0k7E6EEnPzT/gXvOrdm3j16NVf4rbNwVYfDDzem2FmnhcJrsG2v6zc0vLK6lp+vbCxubW9U9zda+owVpQ1aChC1faIZoIHrAEcBGtHihHpCdbyRlcTv3XPlOZhcAfjiHUlGQTc55SAkXrFw6jnwpABKbueTPopfsCuJDCkRCTX6UmvWLIr9hT4L3EyUkIZ6r3it9sPaSxZAFQQrTuOHUE3IQo4FSwtuLFmEaEjMmAdQwMime4m0z9SfGyUPvZDZSoAPFV/TyREaj2Wnumc3KgXvYn4n9eJwb/oJjyIYmABnS3yY4EhxJNQcJ8rRkGMDSFUcXMrpkOiCAUT3dwWT6YFE4qzGMFf0jyrONXK5W21VDvN4smjA3SEyshB56iGblAdNRBFj+gZvaBX68l6s96tj1lrzspm9tEcrM8feuyfvQ==</latexit>
<latexit sha1_base64="dhDz+mg0jfbxnERYd0jVT19v2HI=">AAACGHicbZDLSsNAFIYnXmu9RV26cLAIFUpJpKDuCi50WcVeoAllMpm2Q2eSMDMRSszSl/AV3Orenbh159YncdJmYVt/GPj4zzmcM78XMSqVZX0bS8srq2vrhY3i5tb2zq65t9+SYSwwaeKQhaLjIUkYDUhTUcVIJxIEcY+Rtje6yurtByIkDYN7NY6Iy9EgoH2KkdJWzzyKyo7Hk7sUPsIM/LQCHY7UECOWXKenPbNkVa2J4CLYOZRArkbP/HH8EMecBAozJGXXtiLlJkgoihlJi04sSYTwCA1IV2OAOJFuMvlICk+048N+KPQLFJy4fycSxKUcc093ZjfK+Vpm/lfrxqp/4SY0iGJFAjxd1I8ZVCHMUoE+FQQrNtaAsKD6VoiHSCCsdHYzWzyeFnUo9nwEi9A6q9q16uVtrVSv5PEUwCE4BmVgg3NQBzegAZoAgyfwAl7Bm/FsvBsfxue0dcnIZw7AjIyvX4fin7c=</latexit>
<latexit sha1_base64="gQ1W5BUBQuSs4llOR92XD3cLttw=">AAAB/XicbVA9SwNBEJ2LXzF+RS1tFoOQKtyJoHYBG8so5gOSI+xt9pI1u3vH7p4QjsO/YKu9ndj6W2z9JW6SK0zig4HHezPMzAtizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+Obqd9+okqzSD6YSUx9gYeShYxgY6VWLxDpfdYvV9yaOwNaJV5OKpCj0S//9AYRSQSVhnCsdddzY+OnWBlGOM1KvUTTGJMxHtKupRILqv10dm2GzqwyQGGkbEmDZurfiRQLrScisJ0Cm5Fe9qbif143MeGVnzIZJ4ZKMl8UJhyZCE1fRwOmKDF8YgkmitlbERlhhYmxAS1sCURWsqF4yxGsktZ5zbuoXd9dVOrVPJ4inMApVMGDS6jDLTSgCQQe4QVe4c15dt6dD+dz3lpw8pljWIDz9QvyzpXD</latexit>
<latexit sha1_base64="OpcR7prbb0BxuXfNQa1FkN+WMYo=">AAACGHicbZDLSsNAFIYn9VbrLerShYNFqCAlkYK6K7jQZRV7gSaUyWTaDp1JwsxEKDFLX8JXcKt7d+LWnVufxEmbhW39YeDjP+dwzvxexKhUlvVtFJaWV1bXiuuljc2t7R1zd68lw1hg0sQhC0XHQ5IwGpCmooqRTiQI4h4jbW90ldXbD0RIGgb3ahwRl6NBQPsUI6WtnnkYVRyPJ3cpfIQZ+OkpdDhSQ4xYcp2e9MyyVbUmgotg51AGuRo988fxQxxzEijMkJRd24qUmyChKGYkLTmxJBHCIzQgXY0B4kS6yeQjKTzWjg/7odAvUHDi/p1IEJdyzD3dmd0o52uZ+V+tG6v+hZvQIIoVCfB0UT9mUIUwSwX6VBCs2FgDwoLqWyEeIoGw0tnNbPF4WtKh2PMRLELrrGrXqpe3tXK9ksdTBAfgCFSADc5BHdyABmgCDJ7AC3gFb8az8W58GJ/T1oKRz+yDGRlfv4aun7M=</latexit>
<latexit sha1_base64="lhItiocMBRqGObaIUKv7lTDqZ9g=">AAAB/XicbVBNSwMxEJ2tX7V+VT16CRbBg5RdKVRvBRE8VrAf0C4lm2bb2CS7JFmhLMW/4FXv3sSrv8Wrv8S03YNtfTDweG+GmXlBzJk2rvvt5NbWNza38tuFnd29/YPi4VFTR4kitEEiHql2gDXlTNKGYYbTdqwoFgGnrWB0M/VbT1RpFskHM46pL/BAspARbKzUvO114yHrFUtu2Z0BrRIvIyXIUO8Vf7r9iCSCSkM41rrjubHxU6wMI5xOCt1E0xiTER7QjqUSC6r9dHbtBJ1ZpY/CSNmSBs3UvxMpFlqPRWA7BTZDvexNxf+8TmLCKz9lMk4MlWS+KEw4MhGavo76TFFi+NgSTBSztyIyxAoTYwNa2BKIScGG4i1HsEqal2WvUr6+r5RqF1k8eTiBUzgHD6pQgzuoQwMIPMILvMKb8+y8Ox/O57w152Qzx7AA5+sXoUCVkw==</latexit>
<latexit sha1_base64="8NEpmi0zn1Gl+TfONZYwYrCHrJU=">AAACFHicbVDLSsNAFJ34rPUVddnNYBEqlJJIQd0VRHRZxT6gCWUynbRDZ5IwMxFKyMKf8Bfc6t6duHXv1i9x0mZhWw9cOJxzL/fe40WMSmVZ38bK6tr6xmZhq7i9s7u3bx4ctmUYC0xaOGSh6HpIEkYD0lJUMdKNBEHcY6Tjja8yv/NIhKRh8KAmEXE5GgbUpxgpLfXN0nXfiUa04ng8uU+r0OFIjTBiyU162jfLVs2aAi4TOydlkKPZN3+cQYhjTgKFGZKyZ1uRchMkFMWMpEUnliRCeIyGpKdpgDiRbjJ9IoUnWhlAPxS6AgWn6t+JBHEpJ9zTndmNctHLxP+8Xqz8CzehQRQrEuDZIj9mUIUwSwQOqCBYsYkmCAuqb4V4hATCSuc2t8XjaVGHYi9GsEzaZzW7Xru8q5cb1TyeAiiBY1ABNjgHDXALmqAFMHgCL+AVvBnPxrvxYXzOWleMfOYIzMH4+gWBw54d</latexit>
<latexit sha1_base64="gQ1W5BUBQuSs4llOR92XD3cLttw=">AAAB/XicbVA9SwNBEJ2LXzF+RS1tFoOQKtyJoHYBG8so5gOSI+xt9pI1u3vH7p4QjsO/YKu9ndj6W2z9JW6SK0zig4HHezPMzAtizrRx3W+nsLa+sblV3C7t7O7tH5QPj1o6ShShTRLxSHUCrClnkjYNM5x2YkWxCDhtB+Obqd9+okqzSD6YSUx9gYeShYxgY6VWLxDpfdYvV9yaOwNaJV5OKpCj0S//9AYRSQSVhnCsdddzY+OnWBlGOM1KvUTTGJMxHtKupRILqv10dm2GzqwyQGGkbEmDZurfiRQLrScisJ0Cm5Fe9qbif143MeGVnzIZJ4ZKMl8UJhyZCE1fRwOmKDF8YgkmitlbERlhhYmxAS1sCURWsqF4yxGsktZ5zbuoXd9dVOrVPJ4inMApVMGDS6jDLTSgCQQe4QVe4c15dt6dD+dz3lpw8pljWIDz9QvyzpXD</latexit>
<latexit sha1_base64="ORiRlP3d8Ck3qImxG8IgZjn4WBQ=">AAAB+HicbVA9SwNBEJ2LXzF+RS1tFoNgIeFOAmoXsLFMwHxAcoS9zSRZsnt37O4J8cgvsNXeTmz9N7b+EjfJFSbxwcDjvRlm5gWx4Nq47reT29jc2t7J7xb29g8Oj4rHJ00dJYphg0UiUu2AahQ8xIbhRmA7VkhlILAVjO9nfusJleZR+GgmMfqSDkM+4IwaK9XjXrHklt05yDrxMlKCDLVe8afbj1giMTRMUK07nhsbP6XKcCZwWugmGmPKxnSIHUtDKlH76fzQKbmwSp8MImUrNGSu/p1IqdR6IgPbKakZ6VVvJv7ndRIzuPVTHsaJwZAtFg0SQUxEZl+TPlfIjJhYQpni9lbCRlRRZmw2S1sCOS3YULzVCNZJ87rsVcp39UqpepXFk4czOIdL8OAGqvAANWgAA4QXeIU359l5dz6cz0VrzslmTmEJztcvBTaTkA==</latexit>

heading: 3 METHOD

heading: 3.1 OVERVIEW
text: We first present a high-level description of our model. Directly learning a generative model on Cartesian coordinates heavily depends on the (arbitrary) rotation and translation (Mansimov et al., 2019). Therefore, in this paper we take the atomic pairwise distances as intermediate variables to generate conformations, which are invariant to rotation and translation. More precisely, the cornerstone of our method is to factorize the conditional distribution pθ(R|G) into the following formulation:
pθ(R|G) = ∫ p(R|d,G) · pθ(d|G) dd, (4)
where pθ(d|G) models the distribution of inter-atomic distances given the graph G and p(R|d,G) models the distribution of conformations given the distances d. In particular, the conditional generative model pθ(d|G) is parameterized as a conditional graph continuous flow, which can be seen as a continuous dynamics system to transform the random initial noise to meaningful distances. This flow model enables us to capture the long-range dependencies between atoms in the hidden space during the dynamic steps.
Though CGCF can capture the dependency between atoms in the hidden space, the distances of different edges are still independently updated in the transformations, which limits the capacity of modeling the dependency between atoms in the sampling process. Therefore we further propose to correct pθ(R|G) with an energy-based tilting term Eφ(R,G):
pθ,φ(R|G) ∝ pθ(R|G) · exp(−Eφ(R,G)). (5) The tilting term is directly defined on the joint distribution of R and G, which explicitly captures the long-range interaction directly in observation space. The tilted distribution pθ,φ(R|G) can be used to provide refinement or optimization for the conformations generated from pθ(R|G). This energy function is also designed to be invariant to rotation and translation.
In the following parts, we will firstly describe our flow-based generative model pθ(R|G) in Section 3.2 and elaborate the energy-based tilting model Eφ(R,G) in Section 3.3. Then we introduce the two-stage sampling process with both deterministic and stochastic dynamics in Section 3.4. An illustration of the whole framework is given in Fig. 1.

heading: 3.2 FLOW-BASED GENERATIVE MODEL
text: Conditional Graph Continuous Flows pθ(d|G). We parameterize the conditional distribution of distances pθ(d|G) with the continuous normalizing flow, named Conditional Graph Continuous Flow (CGCF). CGCF defines the distribution through the following dynamics system:
d = Fθ(d(t0),G) = d(t0) + ∫ t1 t0 fθ(d(t), t;G)dt, d(t0) ∼ N (0, I) (6)
where the dynamic fθ is implemented by Message Passing Neural Networks (MPNN) (Gilmer et al., 2017), which is a widely used architecture for representation learning on molecular graphs. MPNN takes node attributes, edge attributes and the bonds lengths d(t) as input to compute the node and edge embeddings. Each message passing layer updates the node embeddings by aggregating the information from neighboring nodes according to its hidden vectors of respective nodes and edges. Final features are fed into a neural network to compute the value of the dynamic fθ for all distances independently. As t1 → ∞, our dynamic can have an infinite number of steps and is capable to model long-range dependencies. The invertibility of Fθ allows us to not only conduct fast sampling, but also easily optimize the parameter set θ by minimizing the exact negative log-likelihood:
Lmle(d,G; θ) = −Epdata log pθ(d|G) = −Epdata [ log p(d(t0)) + ∫ t1 t0 Tr ( ∂fθ,G ∂d(t) ) dt ] . (7)
Closed-form p(R|d,G). The generated pair-wise distances can be converted into 3D structures through postprocessing methods such as the classic Euclidean Distance Geometry (EDG) algorithm. In this paper, we adopt an alternative way by defining the conformations as a conditional distribution:
p(R|d,G) = 1 Z exp { − ∑ euv∈E αuv ( ‖ru − rv‖2 − duv )2} , (8)
where Z is the partition function to normalize the probability and {αuv} are parameters that control the variance of desired Cartesian coordinates, which can be either learned or manually designed according to the graph structure G. With the probabilistic formulation, we can conduct either sampling via MCMC or searching the local optimum with optimization methods. This simple function is fast to calculate, making the generation procedure very efficient with a negligible computational cost.
Compared with the conventional EDG algorithm adopted in GraphDG (Simm & Hernández-Lobato, 2020), our probabilistic solution enjoys following advantages: 1) p(R|d,G) enables the calculation for the likelihood pθ(R|G) of Eq. 4 by approximation methods, and thus can be further combined with the energy-based tilting term Eφ(R,G) to induce a superior distribution; 2) GraphDG suffers the drawback that when invalid sets of distances are generated, EDG will fail to construct 3D structure. By contrast, our method can always be successful to generate conformations by sampling from the distribution p(R|d,G).

heading: 3.3 ENERGY-BASED TILTING MODEL
text: The last part of our framework is the Energy-based Tiling Model (ETM) Eφ(R,G), which helps model the long-range interactions between atoms explicitly in the observation space. Eφ(R,G) takes the form of SchNet (Schütt et al., 2017), which is widely used to model the potential-energy surfaces and energy-conserving force fields for molecules. The continuous-filter convolutional layers in SchNet allow each atom to aggregate the representations of all single, pairwise, and higherorder interactions between the atoms through non-linear functions. The final atomic representations are pooled to a single vector and then passed into a network to produce the scalar output.
Typically the EBMs can be learned by maximum likelihood, which usually requires the lengthy MCMC procedure and is time-consuming for training. In this work, we learn the ETM by Noise Contrastive Estimation (Gutmann & Hyvärinen, 2010), which is much more efficient. In practice, the noise distribution is required to be close to data distribution, otherwise the classification problem would be too easy and would not guide Eφ to learn much about the modality of the data. We propose to take the pre-trained CGCF to serve as a strong noise distribution, leading to the following discriminative learning objective for the ETM2:
Lnce(R,G;φ) =− Epdata [ log
1 1 + exp(Eφ(R,G)) ] − Epθ [ log
1 1 + exp(−Eφ(R,G)) ] . (9)

heading: 3.4 SAMPLING
text: We employ a two-stage dynamic system to synthesize a possible conformation given the molecular graph representation G. In the first stage, we first draw a latent variable ẑ from the Gaussian prior
2Detailed derivations of the training loss can be found in Appendix F.
N (0, I), and then pass it through the continuous deterministic dynamics model Fθ defined in Eq. 6 to get d̂0 = Fθ(ẑ0, G). Then an optimization procedure such as stochastic gradient descent is employed to search the realistic conformations R with local maximum probability of p(R|d,G) (defined in Eq. 8). By doing this, an initial conformation R(0) can be generated. In the second stage, we further refine the initial conformation R(0) with the energy-based model defined in Eq. 5 with K steps of Langevin dynamics:
Rk = Rk−1 − 2 ∇REθ,φ (R|G) +
√ ω, ω ∼ N (0, I),
where Eθ,φ(R|G) = − log pθ,φ(R|G) = Eφ(R,G)− log ∫ p(R|d,G)pθ(d|G)dd.
(10)
where denotes the step size. The second integration term in Eθ,φ can be estimated through approximate methods. In practice, we use Monte Carlo Integration to conduct the approximation, which is simple yet effective with just a few distance samples from the CGCF model pθ(d|G).

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENT SETUP
text: Evaluation Tasks. To evaluate the performance of proposed model, we conduct experiments by comparing with the counterparts on: (1) Conformation Generation evaluates the model’s capacity to learn the distribution of conformations by measuring the diversity and accuracy of generated samples (section 4.2); (2) Distribution over distances is first proposed in Simm & Hernández-Lobato (2020), which concentrate on the distance geometry of generated conformations (section 4.2).
Benchmarks. We use the recent proposed GEOM-QM9 and GEOM-Drugs (Axelrod & GomezBombarelli, 2020) datasets for conformation generation task and ISO17 dataset (Simm & Hernández-Lobato, 2020) for distances modeling task. The choice of different datasets is because of their distinct properties. Specifically, GEOM datasets consist of stable conformations, which is suitable to evaluate the conformation generation task. By contrast, ISO17 contains snapshots of molecular dynamics simulations, where the structures are not equilibrium conformations but can reflect the density around the equilibrium state. Therefore, it is more suitable for the assessment of similarity between the model distribution and the data distribution around equilibrium states.
More specifically, GEOM-QM9 is an extension to the QM9 (Ramakrishnan et al., 2014) dataset: it contains multiple conformations for most molecules while the original QM9 only contains one. This dataset is limited to 9 heavy atoms (29 total atoms), with small molecular mass and few rotatable bonds. We randomly draw 50000 conformation-molecule pairs from GEOM-QM9 to be the training set, and take another 17813 conformations covering 150 molecular graphs as the test set. GEOM-Drugs dataset consists of much larger drug molecules, up to a maximum of 181 atoms (91 heavy atoms). It also contains multiple conformations for each molecule, with a larger variance in structures, e.g., there are the 6.5 rotatable bonds in average. We randomly take 50000 conformationmolecule pairs from GEOM-Drugs as the training set, and another 9161 conformations (covering 100 molecular graphs) as the test split. ISO17 dataset is also built upon QM9 datasets, which consists of 197 molecules, each with 5000 conformations. Following Simm & Hernández-Lobato (2020), we also split ISO17 into the training set with 167 molecules and the test set with another 30 molecules.
Baselines. We compared our proposed method with the following state-of-the-art conformation generation methods. CVGAE (Mansimov et al., 2019) uses a conditional version of VAE to directly generate the 3D coordinates of atoms given the molecular graph. GraphDG (Simm & HernándezLobato, 2020) also employs the conditional VAE framework. Instead of directly modeling the 3D structure, they propose to learn the distribution over distances. Then the distances are converted into conformations with an EDG algorithm. Furthermore, we also take RDKit (Riniker & Landrum, 2015) as a baseline model, which is a classical EDG approach built upon extensive calculation collections in computational chemistry.

heading: 4.2 CONFORMATION GENERATION
text: In this section, we evaluate the ability of the proposed method to model the equilibrium conformations. We focus on both the diversity and accuracy of the generated samples. More specifically, diversity measures the model’s capacity to generate multi-modal conformations, which is essential for discovering new conformations, while accuracy concentrates on the similarity between generated conformations and the equilibrium conformations.
Evaluation. For numerical evaluations, we follow previous work (Hawkins, 2017; Mansimov et al., 2019) to calculate the Root-Mean-Square Deviation (RMSD) of the heavy atoms between generated samples and reference ones. Precisely, given the generated conformation R and the reference R∗, we obtain R̂ by translating and rotating R∗ to minimize the following predefined RMSD metric:
RMSD(R, R̂) = ( 1 n n∑ i=1 ‖Ri − R̂i‖2 ) 1 2 , (11)
where n is the number of heavy atoms. Then the smallest distance is taken as the evaluation metric. Built upon the RMSD metric, we define Coverage (COV) and Matching (MAT) score to measure the diversity and quality respectively. Intuitively, COV measures the fraction of conformations in the reference set that are matched by at least one conformation in the generated set. For each conformation in the generated set, its neighbors in the reference set within a given RMSD threshold
Table 2: Comparison of distances density modeling with different methods. We compare the marginal distribution of single (p(duv|G)), pair (p(duv, dij |G)) and all (p(d|G)) edges between C and O atoms. Molecular graphs G are taken from the test set of ISO17. We take two metrics into consideration: 1) median MMD between the ground truth and generated ones, and 2) mean ranking (1 to 3) based on the MMD metric.
Single Pair All Mean Median Mean Median Mean Median
RDKit 3.4513 3.1602 3.8452 3.6287 4.0866 3.7519 CVGAE 4.1789 4.1762 4.9184 5.1856 5.9747 5.9928 GraphDG 0.7645 0.2346 0.8920 0.3287 1.1949 0.5485
CGCF 0.4490 0.1786 0.5509 0.2734 0.8703 0.4447 CGCF + ETM 0.5703 0.2411 0.6901 0.3482 1.0706 0.5411
δ are marked as matched:
COV(Sg(G),Sr(G)) = 1 |Sr| ∣∣∣{R ∈ Sr∣∣RMSD(R,R′) < δ, ∃R′ ∈ Sg}∣∣∣, (12)
where Sg(G) denotes the generated conformations set for molecular graph G, and Sr(G) denotes the reference set. In practice, the number of samples in the generated set is two times of the reference set. Typically, a higher COV score means the a better diversity performance. The COV score is able to evaluate whether the generated conformations are diverse enough to cover the ground-truth.
While COV is effective to measure the diversity and detect the mode-collapse case, it is still possible for the model to achieve high COV with a high threshold tolerance. Here we define the MAT score as a complement to measure the quality of generated samples. For each conformation in the reference set, the RMSD distance to its nearest neighbor in the generated set is computed and averaged:
MAT(Sg(G),Sr(G)) = 1 |Sr| ∑
R′∈Sr
min R∈Sg
RMSD(R,R′). (13)
This metric concentrate on the accuracy of generated conformations. More realistic generated samples lead to a lower matching score.
Results. Tab. 1 shows that compared with the existing state-of-the-art baselines, our CGCF model can already achieve superior performance on all four metrics (top 4 rows). As a CNF-based model, CGCF holds much the higher generative capacity for both diversity and quality compared than VAE approaches. The results are further improved when combined with ETM to explicitly incorporate the long-range correlations. We visualize several representative examples in Fig. 2, and leave more examples in Appendix G. A meaningful observation is that though competitive over other neural models, the rule-based RDKit method occasionally shows better performance than our model, which indicates that RDKit can generate more realistic structures. We argue that this is because after generating the initial coordinates, RDKit involves additional hand-designed molecular force field (FF) energy functions (Rappé et al., 1992; Halgren, 1996a) to find the stable conformations with local minimal energy. By contrast, instead of finding the local minimums, our deep generative models aim to model and sample from the potential distribution of structures. To yield a better comparison, we further test our model by taking the generated structures as initial states and utilize the Merck Molecular Force Field (MMFF) (Halgren, 1996a) to find the local stable points. A more precise description of about the MMFF Force Field algorithms in RDKit is given in Appendix I. This postprocessing procedure is also employed in the previous work (Mansimov et al., 2019). Additional results in Tab. 1 verify our conjecture that FF plays a vital role in generating more realistic structures, and demonstrate the capacity of our method to generate high-quality initial coordinates.

heading: 4.3 DISTRIBUTIONS OVER DISTANCES
text: Tough primarily designed for 3D coordinates, we also following Simm & Hernández-Lobato (2020) to evaluate the generated distributions of pairwise distance, which can be viewed as a representative element of the model capacity to model the inter-atomic interactions.
Evaluation. Let p(duv|G) denote the conditional distribution of distances on each edge euv given a molecular graph G. The set of distances are computed from the generated conformations R. We
calculate maximum mean discrepancy (MMD) (Gretton et al., 2012) to compare the generated distributions and the ground-truth distributions. Specifically, we evaluate the distribution of individual distances p(duv|G), pair distances p(duv, dij |G) and all distances p(d|G). For this benchmark, the number of samples in the generated set is the same as the reference set.
Results. The results of MMD are summarized in Tab. 2. The statistics show that RDKit suffers the worst performance, which is because it just aims to generate the most stable structures as illustrated in Section 4.2. For CGCF, the generated samples are significantly closer to the ground-truth distribution than baseline methods, where we consistently achieve the best numerical results. Besides, we notice that ETM will slightly hurt the performance in this task. However, one should note that this phenomenon is natural because typically ETM will sharpen the generated distribution towards the stable conformations with local minimal energy. By contrast, the ISO17 dataset consists of snapshots of molecular dynamics where the structures are not equilibrium conformations but samples from the density around the equilibrium state. Therefore, ETM will slightly hurt the results. This phenomenon is also consistent with the observations for RDKit. Instead of generating unbiased samples from the underlying distribution, RDKit will only generate the stable ones with local minimal energy by involving the hand-designed molecular force field (Simm & Hernández-Lobato, 2020). And as shown in the results, though highly competitive in Tab. 1, RDKit also suffers much weaker results in Tab. 2. The marginal distributions P (duv|G) for pairwise distances in visualized in Appendix K, which further demonstrate the superior capacity of our proposed method.
We also follow Mansimov et al. (2019) to calculate the diversity of conformations generated by all compared methods, which is measured by calculating the mean and standard deviation of the pairwise RMSD between each pair of generated conformations per molecule. The results shown in Tab. 3 demonstrate that while our method can achieve the lowest MMD, it does not collapse to generating extremely similar conformations. Besides, we observe that ETM will slightly hurt the diversity of CGCF, which verifies our statement that ETM will sharpen the generated distribution towards the stable conformations with local minimal energy.

heading: 5 CONCLUSION AND FUTURE WORK
text: In this paper, we propose a novel probabilistic framework for molecular conformation generation. Our generative model combines the advantage of both flow-based and energy-based models, which is capable of modeling the complex multi-modal geometric distribution and highly branched atomic correlations. Experimental results show that our method outperforms all previous state-of-the-art baselines on the standard benchmarks. Future work includes applying our framework on much larger datasets and extending it to more challenging structures (e.g., proteins).

heading: ACKNOWLEDGMENTS
text: This project is supported by the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronics Co., Ldt., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bird Gift Fund and a NRC Collaborative R&D Project (AI4D-CORE-06). This project was also partially funded by IVADO Fundamental Research Project grant PRF-2019-3583139727.

heading: A RELATED WORKS
text: Conformation Generation. There have been results showing deep learning speeding up molecular dynamics simulation by learning efficient alternatives to quantum mechanics-based energy calculations (Schütt et al., 2017; Smith et al., 2017). However, though accelerated by neural networks, these approaches are still time-consuming due to the lengthy MCMC process. Recently, Gebauer et al. (2019) and Hoffmann & Noé (2019) propose to directly generate 3D structures with deep generative models. However, these models can hardly capture graph- or bond-based structure, which is typically complex and highly branched. Some other works (Lemke & Peter, 2019; AlQuraishi, 2019; Ingraham et al., 2019; Noé et al., 2019; Senior et al., 2020) also focus on learning models to directly generate 3D structure, but focus on the protein folding problem. Unfortunately, proteins are linear structures while general molecules are highly branched, making these methods not naturally transferable to general molecular conformation generation tasks.
Energy-based Generative Model. There has been a long history for energy-based generative models. Xie et al. (2016) proposes to train an energy-based model parameterized by modern deep neural network and learned it by Langevin based MLE. The model is called generative ConvNet since it can be derived from the discriminative ConvNet. In particular, this paper is the first to formulate modern ConvNet-parametrized EBM as exponential tilting of a reference distribution, and connect it to discriminative ConvNet classifier. More recently, Du & Mordatch (2019) implemented the deep EBMs with ConvNet as energy function and achieved impressive results on image generation.
Different from the previous works, we concentrate on molecular geometry generation, and propose a novel and principled probabilistic framework to address the domain-specific problems. More specifically, we first predict the atomic distances through the continuous normalizing flow, and then convert them to the desired 3D conformation and optimize it with the energy-based model. This procedure enables us to keep the rotational and translational invariance property. Besides, to the best of our knowledge, we are the first one to combine neural ODE with EBMs. We take the ODE model to improve the training of EBM, and combine both to conduct the two-stage sampling dynamics.

heading: B DATA PREPROCESS
text: Inspired by classic molecular distance geometry (Crippen et al., 1988), we also generate the confirmations by firstly predicting all the pairwise distances, which enjoys the invariant property to rotation and translation. Since the bonds existing in the molecular graph are not sufficient to characterize a conformation, we pre-process the graphs by extending auxiliary edges. Specifically, the atoms that are 2 or 3 hops away are connected with virtual bonds, labeled differently from the real bonds of the original graph. These extra edges contribute to reducing the degrees of freedom in the 3D coordinates, with the edges between second neighbors helping to fix the angles between atoms, and those between third neighbors fixing dihedral angles.

heading: C NETWORK ARCHITECTURE
text: In this section, we elaborate on the network architecture details of CGCF and ETM.
C.1 CONTINUOUS GRAPH FLOW
In CGCF, the dynamic function fθ defined in Eq. 6 is instanced with a message passing neural networks. Given the node attributes, edge attributes and intermediate edge lengths as input, we first embed them into the feature space through feedforward networks:
h(0)v = NodeEmbedding(v), v ∈ V, heuv = EdgeEmbedding(euv, duv(t0)), euv ∈ E .
(14)
Then, the node and edge features along are passed sequentially into L layers message passing networks with the graph structure G:
h(`)v = MLP ( h(`−1)v + ∑ u∈NG(v) σ(h(`−1)u + heuv ) ) , ` = 1 . . . L, (15)
where NG(v) denotes the first neighbors in the graph G and σ is the activation function. After L message passing layers, we use the final hidden representation h(L) as the node representations. Then for each bond, the corresponding node features are aggregated along with the edge feature to be fed into a neural network to compute the value of the dynamic fθ:
∂duv ∂t = NN(hu,hv,heuv , t). (16)
C.2 ENERGY-BASED TILTING MODEL
The ETM is implemented with SchNet. It takes both the graph and conformation information as input and output a scalar to indicate the energy level. Let the atoms are described by a tuple of features Xl = (xl1, . . . ,x l n), where n denote the number of atoms and l denote the layer. Then given the positions R, the node embeddings are updated by the convolution with all surrounding atoms:
xl+1i = ( X l ∗W l ) i = natoms∑ j=0 xlj ◦W l (rj − ri) , (17)
where ”o” represents the element-wise multiplication. It is straightforward that the above function enables to include translational and rotational invariance by computing pairwise distances instead of using relative positions. After L convolutional layers, we perform a sum-pooling operator over the node embeddings to calculate the global embedding for the whole molecular structure. Then the global embedding is fed into a feedforward network to compute the scalar of the energy level.

heading: D TWO-STAGE DYNAMIC SYSTEM FOR SAMPLING
text: Algorithm 1 Sampling Procedure of the Proposed Method Input: molecular graph G, CGCF model with parameter θ, ETM with parameter φ, the number of optimization steps for p(R|d,G)M and its step size r, the number of MCMC steps for Eθ,φ N and its step size Output: molecular conformation R
1: Sample d(t0) ∼ N (0, I) 2: d = Fθ(d(t0),G) 3: for m = 1, ...,M do 4: Rm = Rm−1 + r∇R log p(R|d,G) 5: end for 6: for n = 1, ..., N do 7: Rn = Rn−1 − 2∇REθ,φ(R|G) + √ ω, ω ∼ N (0, I), 8: end for
E IMPLEMENTATION DETAILS
Our model is implemented in PyTorch (Paszke et al., 2017). The MPNN in CGCF is implemented with 3 layers, and the embedding dimension is set as 128. And the SchNet in ETM is implemented with 6 layers with the embedding dimension set as 128. We train our CGCF with a batch size of 128 and a learning rate of 0.001 until convergence. After obtaining the CGCF, we train the ETM with a batch size of 384 and a learning rate of 0.001 until convergence. For all experimental settings, we use Adam (Kingma & Ba, 2014) to optimize our model.

heading: F DETAILED DERIVATIONS OF ENERGY-BASED MODEL
text: Here we present the detailed derivations of the training objective function of Energy-based Tilting Model (ETM) in Eq. 9:
Lnce(R,G;φ) =− Epdata [ log
pθ,φ(R|G) pθ,φ(R|G) + pθ(R|G)
] − Epθ [ log
pθ(R|G) pθ,φ(R|G) + pθ(R|G) ] =− Epdata [ log
pθ(R|G) exp(−Eφ(R,G)) pθ(R|G) exp(−Eφ(R,G)) + pθ(R|G) ] − Epθ [ log
pθ(R|G) pθ(R|G) exp(−Eφ(R,G)) + pθ(R|G) ] =− Epdata [ log 1 1 + exp(Eφ(R,G)) ] − Epθ [ log
1 1 + exp(−Eφ(R,G)) ] .
(18)

heading: G MORE GENERATED SAMPLES
text: We present more visualizations of generated 3D structures in Fig. 3, which are generated from our model (CGCF + ETM) learned on both GEOM-QM9 and GEOM-Drugs datasets. The visualizations demonstrate that our proposed framework holds the high capacity to model the chemical structures in the 3D coordinates.

heading: H MORE RESULTS OF COVERAGE SCORE
text: We give more results of the coverage (COV) score with different threshold δ in Fig. 4. As shown in the figure, our proposed method can consistently outperform the previous state-of-the-art baselines CVGAE and GraphDG, which demonstrate the effectiveness of our model.
I IMPLEMENTATION FOR MMFF
In this section, we give a more precise description of the MMFF Force Field implementation in the RDKit toolkit (Riniker & Landrum, 2015).
In MMFF, the energy expression is constituted by seven terms: bond stretching, angle bending, stretch-bend, out-of-plane bending, torsional, van der Waals and electrostatic. The detailed functional form of individual terms can be found in the original literature (Halgren, 1996a). To build the force field for a given molecular system, the first step is to assign correct types to each atom. At the second step, atom-centered partial charges are computed according to the MMFF charge model (Halgren, 1996b). Then, all bonded and non-bonded interactions in the molecular system under study, depending on its structure and connectivity, are loaded into the energy expression. Optionally, external restraining terms can be added to the MMFF energy expression, with the purpose of constraining selected internal coordinates during geometry optimizations. Once all bonded and non-bonded interactions, plus optional restraints, have been loaded into the MMFF energy expression, potential gradients of the system under study can be computed to minimize the energy.

heading: J MORE EVALUATIONS FOR CONFORMATION GENERATION
text: Junk Rate. The COV and MAT score in Section 4.2 do not appear to explicitly measure the generated false samples. Here we additionally define Junk rate measurement. Intuitively, JUNK measures the fraction of generated conformations that are far away from all the conformations in the reference set. For each conformation in the generated set, it will be marked as a false sample if its RMSD to all the conformations of reference set are above a given threshold δ:
JUNK(Sg(G),Sr(G)) = 1 |Sg| ∣∣∣{R ∈ Sg∣∣RMSD(R,R′) > δ, ∀R′ ∈ Sr}∣∣∣, (19)
Typically, a lower JUNK rate means better generated quality. The results are shown in Tab. 4. As shown in the table, our CGCF model can already outperform the existing state-of-the-art baselines with an obvious margin. The results are further improved when combined with ETM to explicitly incorporate the long-range correlations.

heading: K DISTANCE DISTRIBUTION VISUALIZATION
text: In Fig. 5, we plot the marginal distributions p(duv|G) for all pairwise distances between C and O atoms of a molecular graph in the ISO17 test set. As shown in the figure, though primarily designed
for 3D structure generation, our method can make much better estimation of the distances than GraphDG, which is the state-of-the-art model for molecular geometry prediction. As a representative element of the pairwise property between atoms, the inter-atomic distances demonstrate the capacity of our model to capture the inter-atomic interactions.

Review summaries:
The authors of this manuscript proposed a generative dynamics system for the modelling and generation of 3D conformations of molecules. Specifically, there are three components: (1) conditional graph continuous flow (CGCF) to transform random noise to distances,  (2)a closed-form distribution p(R|d, G), and (3) an energy-based tilting model (ETM) to capture long-range interactions and correct the position matrix distribution. The proposed framework was compared with two deep learning methods for conformation generations -- CVGAE & GraphDG, as well as the computational chemistry tool RDKit on GEOM-QM9, GEOM-Drugs, and ISO17 data sets. Comparisons in terms of COV and MAT scores show that the proposed method (particularly the one enhanced with ETM) can outperform baselines. Further comparisons of distances densities show that CGCF (but without ETM) worked best over baselines. 

This paper presents an approach to generate diverse small molecule conformations given its graph by combining a conditional flow-based model with an energy-based model. Sampling is performed in two separate stages: 1) a normalizing flow produces a distribution over interatomic distances (which is then postprocessed into cartesian coordinates), 2) sampled coordinates are refined by Langevin dynamics with gradient signal produced from an energy-based model. The models are trained separately.