Direct Molecular Conformation Generation
Authors: Jinhua Zhu
Yingce Xia
Chang Liu
Lijun Wu
Shufang Xie
Wengang Zhou
Tao Qin
Houqiang Li
Tie-Yan Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Molecular Conformation Generation

Abstract: Molecular conformation generation, which is to generate 3 dimensional coordinates of all the atoms in a molecule, is an important task for bioinformatics and pharmacology. Most existing machine learning based methods first predict interatomic distances and then generate conformations based on them. This two-stage approach has a potential limitation that the predicted distances may conflict with each other, e.g., violating the triangle inequality. In this work, we propose a method that directly outputs the coordinates of atoms, so that there is no violation of constraints. The conformation generator of our method stacks multiple blocks, and each block outputs a conformation which is then refined by the following block.  We adopt the variational auto-encoder (VAE) framework and use a latent variable to generate diverse conformations. To handle the roto-translation equivariance, we adopt a loss that is invariant to rotation and translation of molecule coordinates, by computing the minimal achievable distance after any rotation and translation. Our method outperforms strong baselines on four public datasets, which shows the effectiveness of our method and the great potential of the direct approach. The code is released at \url{https://github.com/DirectMolecularConfGen/DMCG}. 

Under review as a conference paper at ICLR 2022
DIRECT MOLECULAR CONFORMATION GENERATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Molecular conformation generation, which is to generate 3 dimensional coordi- nates of all the atoms in molecule, is important task for bioinformatics and pharmacology. Most existing machine learning based methods first predict in- teratomic distances and then generate conformations based on them. This two- stage approach has a potential limitation that the predicted distances may conflict with each other, e.g., violating the triangle inequality. In this work, we pro- pose a method that directly outputs the coordinates of atoms, so that there is no violation of constraints. The conformation generator of our method stacks multiple blocks, and each block outputs a conformation which is then refined by the following block. We adopt the variational auto-encoder (VAE) frame- work and use latent variable to generate diverse conformations. To handle the aato-rranslttion equivariance, we adopt a loss that is invariant to rotation and translation of molecule coordinates, by computing the minimal achievable dis- tance after any rotation and translation. Our method outperforms strong base- lines four public datasets, which shows the effectiveness of our method and the great potential of the direct approach. The code is released at httpps:
INTRODUCTION
Molecular conformation generation aims to generate the 3D coordinates of all the atoms of molecules, which then can be used in molecular property prediction (Axelrod & Gombz-Bonbarellj 2021), docking (Roy et al. 2015), structure-based virtual screening (Kontoyianni (017). etc. Molec- ular conformation can be physically observed using X-ray crystallography, but prohibitively costly for industry-scale tasks (Mansimov et al. 2019)) Ab initio methods. such as density functional theory (DFT) (Parr] 1980, Baseden & Tye 201 can accurately predict the molecular shapes, but take up to several hours per small molecule (Hu et al. 202T).To handle large-scale molecules, people turn to leverage the classical force fields, like UFF (Rappe et al. 1992) or MMFF (Halgren 1996). to estimate conformations, which is efficient but unacceptably inaccurate (Kanal et al. 2018
Recently, people start to explore machine learning methods to generate conformation. Simm & Hernandez-LLbato) (2020), (Shi et al. al. (2020) and et al. (202T) leveraged variational auto-encoder (VAE), flow- models and score-based methods for conformation generation, respectively.
With learned models, molecule conformations can be sampled independently (Simm & Hernández-
Lobato 2020 Shi et al., 2020) or using Langevin dynamics (Xu et al.) 2021a Shi et al.) 2021).
The common part of these methods is that they are all built upon the interatomic distances among
atoms (i.e., the distance between atom pairs). Specifically, Simm & Hernández-Lobato) (2020); Shh
al. 22020)] Xu et al. (2021a b) use various generative methods model the distribution of inter- atomic distances, and then reconstruct conformations based on distances. (Shi et al. (202)) leveraged score-matching network model the density gradient of interatomic distances. Moreover, Win- ter et al.] (2021) use variational auto-encoder to first predict bond length, bond angle and dihedral angle and then reconstruct the coordinates based on the intermediate results. major reason of us- ing distance- based methods is that the interatomic distances are invariant to rotation and translation of the conformation. However, a potential drawback that the predicted distances might conflict with each other. For example, they did not explicitly consider the triangle inequality for distances among any three atoms. The underlying the number of atoms in molecule) while practically, these methods often generate distances with the offreedo of roughly N(N - 1)/2, which usually lead to violations of the triangle in- equality. For example, we found that in GraphDG (Simm& & Hermández-Looate 2020), a prevailing
freedom of these distances is only 3.V - 6 (N is

---

Under review as a conference paper at ICLR 2022
representative of distance-based method, 8.65% of molecular graphs in the GEOM-Drugs test set will produce distance matrix that violates the triangle inequality.
In this work, explore the possibility to directly generate the coordinates of all atoms. This is a straightforward and more natural choice and avoids violations of the triangle incquality, and has demonstrated remarkable performance on protein structure prediction by the AlphaFold 2 (Jumper et al. 2021). We design a model that generates atom coordinates directly. The generator in our model stacks multiple blocks, and each block outputs a conformation which is then refined by the following block. A block consists of several modules that can encode the previous conformation as well as the features of bonds, atoms and global information. At the end of each block, we add a normalization layer that centers the coordinates at the origin. Since a molecule may have multiple conformations, we use the variational auto-encoder (VAE) framework which allows diverse generation. To realize roto-translation equivariance, i.e., rotating and translating a set of atom coordinates do not change the conformation, we adopt loss that invariant to rotation and translation of atom coordinates.
We conduct experiments four benchmark settings, which are GEOM-QM9 and GEOM-Drugs with small-scale setting (Shi et al.) (222)) and large-scale setting (Axeirod & 2021]. Compared with previous methods, ours achieves state-of-the-art results on GEOM- (both ssall-scale and large-scale settings), and outperforms almost all previous basclines on GEOM- QM9, demonstrating the effectiveness of our method.
Our contributions are summarized as follows:
(1) We explore new direction of conformation generation, that directly generates the coordinates of a molecular conformation without generating interatomic distances. We empirically show that directly generating conformation achieves state-of-the-ar results on several tasks.
(2) We leverage a fine-grained loss function for training, that is invariant to the roto-translation of the conformation.
(3) We propose a new model that iteratively refines the conformations. Our model is inspired by mul- tiple advanced architectures like GATv2 (Brody al. (02T), graph network (GN) block (Battaglia et al. 2018), that can effectively model molecules.
PRELIMINARIES
In this section, we introduce the notations used in this work, give a formal definition of the molec- ular conformation generation problem, and briefly introduce how align two conformations under rotation and translation.
Notations: Let G (V, F) denote molecular graph, where V and F are collections of atoms and bonds, respectively. Specifically, V {E1, V2, ... } with the i-th atom Us Let egy denote the bond between atom i; and ui.. Let (t) denote the neighbors of atom i, i.e., N(2) = {uz Cij E E). We use to represent the conformation of G, where R E R x3 The i-th row of / is the coordinate of atom 24.
Let p(-) denote poto-translation operation, i.c. an affine transformation p(ft) = Rt + b where the orientation perioooiireeeeiinggggg orthogonal transformation SO(3) 3xx3 represents rotation and the vector b E IR! 1x represents translation. Let MLP (...) denote the multi-layer perception network, where the inputs are concatenated together and then processed by linear mapping, Batch Normal- ization (Toffe & Szegedy| 2015) and ReLU activation sequentially.
Problem definition: Given a graph G (V, our task is to learn mapping, that can output the coordinates of all atoms in V. i.e., R E R Vix3
Matching loss: Let Rj and R2 denote two conformations, both of which are N x 3 matrices (N is the number of rows R1 and Rz). The matching loss between R₁ and R3 is defined as follows:
(R,, R2) min (R)) R2ll
AA molecule corresponds to different conformations different energy level. To model such cases, the problem is generate hased G and random noise
2

---

Under review as a conference paper at ICLR 2022
where || I denotes the Frobenius norm, i.e., |||| = - 1.jl?. The matching loss is invariant
to the rotation and translation of either of the two input conformations. It is obviously nonnegative, and is zero if R is obtained by a roto-translational operation of R2.
Karney (2077) proposed to use quaternions to solve Eqn. D. A quaternion q is extension of complex numbers, q = You + qi + qal + gak, where go: 91, q2, Q3 are scalars and u, J. are orientation vectors. With quaternions, any rotation operation is specified by 3 x 3 matrix, where each element in the matrix is the basic algebra of go to qs and some constants. The solution to Eqn.[[ ( is the minimal eigenvalue of 4 4 matrix obtained by the basic algebra Rı and R2, the corresponding optimal rotation and translation p" can be obtained from the eigenvectors. Note that the best '" is related both R1 and Rg- To stabilize training, set the gradients of p" w.r.t. the model parameter as zero. More details can be found in Karney (2007).
3 OUR METHOD
We introduce our proposed method in this section.
3.1 CONFORMATION GENERATION FRAMEWORK
Our goal can be formalized as learning to generate samples from the conditional distribution p( ))). We follow the variational auto-encoder (VAE) scheme (Kingma & Welling 2014; Rezende et al., 2014 Sohn et al., 2013) as it enables flexible model architecture (vs. flow-based models), efficient (i.e., i..d) generation (vs. score-matching methods), and stable training and diverse sample genera- tion (vs. adversarial methods). Given a molecule graph G, the conditional variant of VAE generates R using likelihood model p(R|z, G) where 2 is latent code drawn from prior p(2) = (0 I). The corresponding p( R|G) = p(=)p((D R G') da is however intractable to evaluate and maximize to fit data. VAE handles this by introducing (conditional) inference model q(=|R, G) which leads to the following identity:
log p( () R. [logp(R|z, G)] (x + DKI. (g(2) () G)),
(2)
where DKL denotes the Kullback-Leilerr (KL) divergence, and p(=|R,
p(=)p((|, the
true posterior. Due to the nonnegativity of the third term, the first two terms form a lower bound of log pog((())), When q(2 R, G) is properly chosen, the bound is tractable optimize. Moreover, as the left-hand-side is independent of q(= G), tightening the bound (i.e., minimizing the third term) can be achieved by maximizing the bound w.r.t (2| G). Therefore, as a loss function be minimized, the objective is:
G)] + D
(3)
Now we develop our specification for the conformation generation task. We construct the likelihood model with regard to a decoded reference conformation f((2, G):
p[R]z, G) x exp {f - (1L(2, G), 1))
(4)
where y 0 is variance parameter that we choose fix (see Sec. 3.4 (1) for explanation). Note that the usage of las guarantees all roto-translation equivalent conformations have the same probability. For the inference model, we choose
<< G) -N ER,G),
(5)
where the conditional mean and variance are outputs from some encoder. It enables tractable loss optimization via reparameterization: ~ q(2|R, G) is equivalent to z = PR,G + ER,GE where ~ (O, I). With the above specification, the loss function becomes:
min E, (i((RR + MR,GE G), R) + 3D ER. I)),
(6)
where the minimization is applied over all model parameters. Note that in Eqn. 66, an additional hyperparameter 3 is introduced in the spirit of 3- VAE (Higgins et al.] (2016) to handle the strength of the prior regularization. The second term in Eqn. ( has closed-form expression for optimization.
3

---

Under review as a conference paper at ICLR 2022
p(R|z,G)
ty
Q2D
R 0 H U
Pdec
3D
N (uRR, 2n,6)
Z
R
G()|R,G)
p(x) (0,
Figure I: The workflow of our method. Green and orange lines represent pl R|2,, G) and q(2 G) respectively.
3.2 WORKFLOW
According to E)n.(3). we need a module PPD to model g(2 R), module Pa to model p(R|2, 2. G) and a module you model the conditional input G. In this subsection, we introduce the general workflow of how they work together (see Figure I Details of the yp.. modules will be covered in the next subsection.
The training workflow is shown as follows:
(1) The encoder (DD takes the molecular graph G its input, and outputs d-dimensional graph representations H E xd for all atoms, H (0) E xd for all bonds, global graph feature
(+() Rdd and initial conformation R( RIV x3, Formally, (H (0) H ,U((),, R())) (GD
(2) We use another encoder yPaD to extract features of the conformation R for constructing the con- ditional inference model q( R, G). According to the above specification, this encoder only needs to output the mean and variance of the Gaussian, formally, (MR,G, ΣR,G) = Pa G).
(3) We randomly sample a variable z from the Gaussian distribution N (uRR,, ER,G), and then feed (0) (0) U f((0) into the decoder Pace obtain the conformation it That is, A((2 Paec ( (G) =) = (H H R((0), (T(0) z). Note that sampling z ~ NNun, En,a) is equivalent to sampling e N0, I) and then setting MR,G + DR,GE.
(4) After obtaining R(2,, G) and and N (un.o En,G), we use Eqn. training. Remind that ft is related to spD, (P3D, SPece and PR.G. ΣR,G are related to yan.
6 as the objective function for
For the inference workflow, there are three steps in total: (1) Given a molecular graph G, P2D to encode G and obtain (()), H()) H!)) (7();; (2) sample a random variable 2 from Gaussian (0, I); (3) feed R(()), ( F7 (7(0), into (dde and obtain the eventual conformation R(2, Note that PaD is not used in inference phase.
3.3 DETAILED MODEL ARCHITECTURE
The encoders goDD PaD and the decoder &dee share the same architecture. They are all stacks of T. identical blocks. We introduce the 1-th block in the decoder (Paec, and leave the introduction of P2D and PaD in Appendix A In contrast to previous work, we introduce some novel designs that handle the task at finer level. (I) The conformation is iteratively refined by each decoding block. Specifically, we use normalization technique that moves the center of each decoded conformation the origin. (2) We use more advanced model components, like attention models (Brody et al.) 2021) and the GN block (Battaglia et al., 2018), that can more effectively represent the molecules.
The detailed architecture is shown in Figure The inputs include the following information output by the (l - 1) block: the conformation !!!-1), atom features H (![-1) E IR V|x3, edge features (((-1) and the global feature U((-1), Note (()) H(!)) ( and R(0) are the outputs of yzp- (((-1) is first mapped by ML. and get (() = ( The atom features are then fused with the coordinate features by H
H + (
where NNHRRO,
We use a variant of the GN block (Battaglia et al., 2018) as the backbone of the model due to its superior performance in molecular modeling. In each block, update the bond features, atom fea-
4

---

Under review as a conference paper at ICLR 2022
R((-))
encode conl
update bond
update atom
update global
U
Hy (1-1)
He-D U
(1-)
(
H
conf
R(O
g
pred
Network architecture of the 1-th block. Yellow dashed boxes are inputs, orange boxes denote the operations, and green boxes are outputs.
Figure 2:
tures and global features sequentially. Each block will output its own prediction on the coordinates of atoms. For ease of reference, we use h (single subscript) to denote the feature of atom i output by the I-th block, and use 1/ (two subscripts) to denote the bond features between atom and output by the 1-th block. The workflow of the I-th block is shown as follows:
(1) Update bond features: For each bond feature 12 (1-1) in H!ll1), it is updated by
h hig (!-1) MLP (t-1) 20-0 nn (1-1) (1-1) (((11)).
mij (((-)),
(2) Update alom features: The atom features are updated using attentive way. For any i [[]]]
Σ ayW.concat(hh.. hh JEN())
x cxp(a T LeakyReLU(W
4/ (1-1) + (h (111) ,,ly)))
(8)
h ( = ((-1)) + MLP h. (1-1) (0) (00-1)
In Eqn.(8) Wg W. and Wi are the parameters to be learned and concat(-,)) is the concatena- tion of two vectors. For atom 8, we first use GATv2 (Brody et al. 2021) to aggregate the features from its connected bonds and obtain h; and then update e based on h h (l-1) and ((-1))
(3) Update global features: The global feature is updated as follows:
T ((- MLP
(1) hig
((-1))
(9)
V
E
(4) Output the prediction: After obtaining the new features of the 1-th block, for any atom i [[]],
it predicts the conformations (1)) as follows:
R (N. (1). (n)
1
FC
R - ((()) (1) + R7 ((1-1)
(10)
V
that, after making initial prediction Rd we calculate its center and normalize their coordinates by moving the center to the origin. We add this kind of normalization to ensure that input coordinates of each block in reasonable numeric ranges.
An important step Eqn. 10
We use (L) output by the last block in Pace the final prediction of the conformation.
3.4 DISCUSSIONS
In this section, we discuss the relation between method and two previous work.
Comparison with CVGAE: Mansimov et al. (2019) developed an early attempt to generate conforma- tion by coordinates, but its performance not desired distance-based methods developed afterwards (Shi et al.) 2020 Simm & Hemández--Lobtoo 2020). Our method, pursuing the same spirit, makes several finer designs and implementations.
(1) Under the VAE formulation, Mansimov et al. (2019) also leaned the variance parameter y of the likelihood model. i.e., the Eqn.(4). However, decent VAE analysis (Dai & Wipf 2019
5

---

Under review as a conference paper at ICLR 2022
Thm. 3) revealed that the VAE objective prefers vanishing y. This is desired in the ideal case when the encoder and decoder are optimized perfectly (Dai & Wiph 2019 Thm. 4), but in practice it would distract the optimizer towards "lazy way" to focus on Y and stagnate the optimization of the encoder and decoder. Therefore, we choose to fix y. When y is fixed, the loss function is equivalent to using 7 1 and a parameter as the formulation in Eqn.(6). We hence choose a small value of 3 corresponding to fixing to small value in the original formulation (i.c. without 3) as advocated (Dai & Wipf 2019).
(2) Our model consists of more advanced modules, including GATv2 (Brody et al. (021) and GN block (Battaglia et al., 2018) to better model the input. In comparison, Mansimov et al. (2019) mainly leveraged GRU (Bahdanau et al. (2015) and its variant on graphs, which is outperformed by the modules used in our model.
(3) We iteratively refine the output of each block, while (Mansioov et al. 2019) only outputs the conformation in the last layer.
Comparison with ConfGF: Shi et al. (221) used score matching to model the gradients w.r.t inter- atomic distances, and then recover the coordinates based on the gradients. That is, (Shi et al. 2021) still tries to model the interatomic distances, while we completely abandon modeling the distances. We propose a new direction for molecular conformation generation.
4 EXPERIMENT
4.1 SETTINGS
Following prior works (Xu et al.] [021a]] Shi et al. 2021). use the GEOM-QM9 and GEOM-Drugs datasets (Axelrod & Gomez Bombarelli [021] for conformation generation. We verify our method on both small-scal setting and large- scale setting. First, we use the same datasets provided by (hi et al. (2021) for fair comparison with prior works. The training, validation and test sets of the two datasets consist of 200K, 2.5K and 22408 (for GEOM-QM9)11432 (for GEOM- Drugs) molecule-come conforme-oonooommtiionn pairs respectively. After that, we sample larger datasets from the original GEOM to validate the scalability of our method. We use all data in GEOM-QM9 and 2.2M pairs for GEOM-Drugs. The numbers of training, validation and test sets for the larger GEOM-QNE 1-M9 setting are 1.37M, 165K and 174K, and those for larger GEOM-Drugs 2M, 100K and 100K.
Datasets:
Model configuration: All of f2D. &ou and Sdee have the same number of blocks. The dimension d of the features is 256. Inspired by the feed-forward layer in Transformer (Vaswani et al. 2017),
MLP also consists of two sub-layers, where the first one maps the input features from dimension 256 to hidden states, followed by Batch Normalization and ReLU activation. Then the hidden states is mapped to 256 again using linear mapping. More parameters are summarized in Appendix B
Evaluation: Assume in the test set, the molecule x has N₂ conformations. Following Shi et al. 2020 2021). for each molecule x in the test set, we generate 2N, conformations. Let (9) and ( denote all generated and groundtruth conformations respectively. We use coverage score (COV) and matching score (MAT) to evaluate the generation quality. Mathematically,
1
(O),
H 3, lss (R, R') < < ER' E SSyll;
(11)
S
MAT (G),
min EM (R, R'), RES,
(12)
where EM is defined in Eqn. T good method should have high COV score and a low MAT score. Following (Shi et al. 2021 the ò's are set as 0.5 and 1.25 for QM9 and Drugs, respectively.
Baselines: (1) RDKit, which is a widely used toolkit and generates the conformation based on the
force fields; (2) CVGAE (Mansimov al. 2019), which is an early attempt to generate raw coor- dinates; (3) GraphDG (Simm& & -rimanndz--Lbbttee 2020). representative distance-based method with VAE; (4) CGCF (Xu et al. [2021a), which another distance- based method leveraging continu- ous normalizing flow; (5) ConIVAE (Xu et al., (2211), an end-to-end framework for molecular con- formation generation, which still uses the pairwise distances among atoms as intermediate variables;

---

Under review as a conference paper at ICLR 2022
(6) ConfGF (Shi et al. 2021). which uses score matching to generate the gradients w.rt. distances and then recover the conformation. Note that ConfGF leverages Langevin dynamics, which requires much more additional decoding time.
4.2 RESULTS
The results on the small-scale and large-scale datasets are Table and Table respectively.
Dataset
QM9
Drugs
COV(%)1
MAT (A)
COV(%)T
MAT (A)L
Methods
Mean
Median
Mean
Median
Mean
Median
Mean
Median
RDkit
83.26
90.78
0.3447
0.2935
60.91
65.70
1.2026
1.1252
CVGAE
0.09
0.00
1.6713
1.6088
0.00
0.00
3.0702
2.9937
GraphDG
73.33
84.21
0.4245
0.3973
8.27
0.00
1.9722
1.9845
CGCF
78.05
82.48
0.4219
0.3900
53.96
57.06
1.2487
1.2247
ConfVAE
80.42
85.31
0.4066
0.3891
53.14
53.98
1,2392
1,2447
ConfGF
88.49
94.13
0.2673
0.2685
62.15
70.93
1.1629
1.1596
Ours
81.44
96.61
0.2879
0.2428
77.78
86.09
1.0657
1.0563
Table 1: Experimental results on small-scale datasets.
Datasct
QM9
Drugs
COv(%)t
MAT (A)J
COV(%))
MAT (Á)L
Methods
Mean
Median
Mean
Median
Mean
Median
Mean
Median
RDkit
81.61
85.71
0.2643
0.2472
69.42
77.45
1.0880
1,0333
CVGAE
0.00
1.4687
1.3758
0.00
0.00
2.6501
2.5969
GraphDG
13.48
5.71
0.9511
0.9180
1.95
0.00
2.6133
2.6132
CGCF
81.48
86.95
0.3598
0.3684
57.47
62.09
1.2205
1.2003
ConfVAE
80.18
85.87
0.3684
0.3776
57.63
63.75
1.2125
1.1986
ConfGF
89.21
95.12
0.2809
0.2837
70.92
85.71
1.0940
1,0917
Ours
86.58
100.00
0.2457
0.1756
90.68
100.00
0.8934
0.8703
Table 2: Experimental results on large-scale datasets.
We have the following observations:
(I) On the four settings Table and Table 2 our method achieves state-of-the-art results on most of them, i.e., the two large-scale settings and small dataset setting GEOM-drug. The only exception is that our method not so good as ConfGF in terms of the COV mean and MAT mean small-scale QM9, and the mean COV score on large-scale QM9. The median COV(%) being 100% means that for more than half of the reference molecules, these exist generated molecules that close to them within predefined threshold. These results already showed the effectiveness and scalability of our method.
(2) Our method achieves more improvement on molecules with more heavy atoms. Take the results in Table [ an example. On average, QM9 and Drugs have 8.8 and 24.9 heavy atoms respectively. In terms of MAT median, QM9, method improves ConfGF by 0.026 points, while Drugs, the improvement is 0.3. To verify our conjecture, on Drugs, we categorize the molecules based on their numbers of heavy atoms. The number of heavy atoms in the i-th group lie in [10 (i]+
We compare our method against ConfGF and GraphDG. The results are in Table
Metric
COV(%))
MAT(Á).
i=1
i
2
==
average
i=
i 2
==3
average
ConfGi
99.95
66.28
15.34
62.54
0.7764
1.1510
1.5345
1.1637
GraphDG
15.11
1.78
0.0
3.12
2.0578
2,5863
2.9849
2.5847
Ours
98.13
80.18
51.72
77.78
0.8502
1.0552
1.2737
1.0657
Table COV and MAT mean scores w.r.t. numbers heavy atoms on small-scale GEOM-Drugs. The i indicates that the heavy atom number lies in range [10r + 1, 10(i + i)], (1,2,3).

---

Under review as a conference paper at ICLR 2022
Molecular
Reference
Ours
GraphDG
Figure Visualization of different conformations.
We observe that among the three algorithms, ConfGF achieves the best results the molecules whose numbers of heavy atoms lie in [11, 20), and our method is slightly worse than it. However, when the numbers of heavy atoms are larger than 21, our method significantly outperforms ConfGF and GraphDG. This is consistent with our conjecture, and demonstrates the potential of directly gen- erating coordinates for molecules with more heavy atoms. More discussions are Appendix C.5
(3) Our method is much more sample efficient than ConfGF. In (Shi et al. 202T generating conformation requires 5000 sequential forward steps. With our method, we only need to sample one variable from (00 I) and then generate a corresponding conformation by forwarding once. For fair comparison, following the official implementation of ConfGF, we split the test sets of small- scale GEOM-QM9 and GEOM-Drugs into 200 batches. ConfGF requires 8511.60 and 1 11830.42 seconds to decode QM9 and Drugs test sets, while our method only requires 32.6% and 54.89 seconds respectively. That is, our method speeds up the decoding more than 200 times.
In Figure 3 we visualize the conformation of different methods. We randomly select four molecules from the small-scale GEOM-drug dataset, generate several conformations, and visualize the best- aligned ones with the groundtruth. We can see that our method can generate high-quality conforma- tions than previous methods, which are the most similar the groundtruth.
4.3 PROPERTY PREDICTION
We conduct experiments the property prediction task, which is predict molecular properties based on an ensemble of generated conformations (Axelrod & Gomez- BombaeBB (202]). We first randomly choose 30 molecules from QM9 and Drugs sets, and then sample 50 conformations for each molecule using RDkit, ConfGF and our method as initial coordinates. Next, we use the package Psi4 (Smith al. 2020) calculate the energy, HOMO and LUMO for each generated conformation and groundiruth conformation. After that, we calculate ensemble properties including average energy F,, lowest energy Emin. average HOMO-LUMO gap Ae, minimum gap Aemin and maximum gap Aemax for the conformations. Finally, we use the mean absolute error to measure the gap of the above properties between the groundtruth and generated conformations. Although a better choice is to use Boltzmann distribution to re- re-weight each conformation, due to the lack of such the distribution the dataset, we are not able to use it now and use the average value instead.
QM9 Ae
Drugs Zz
Methods
E
Emin
Acpii
A
E
Emin
AEnia
Acmax
Rdkit
0.0902
0.0534
0.2001
0.2215
0.1837
3.4808
0.1669
0.4082
2,8640
0.2720
GraphDG
0.1108
0.0789
0.2309
0,3886
0.2431
8.9561
0.4485
0.9217
3.2755
0.4519
ConfGi
0.0719
0.0295
0.1533
0.2302
0.1640
2.9843
0.1763
0.2188
2.6481
0.2422
0.0656
0.0258
0.1358
0.2156
0.1638
0.1796
0.1468
0.2073
0.6740
0.2243
Table 4: Mean absolute error of predicted ensemble properties. Unit: eV..
8

---

Under review as a conference paper at ICLR 2022
COV(%)
MAT (A)
Methods
Mean
Median
Mean
Median
Ours
77.78
86.09
1.0657
1.0563
Ours w/o attention
72.18
82.76
1.1257
1.1188
Ours w/o conformation normalization
61.82
64.76
1.2045
1.1974
Ours w/ FF
85.00
92.05
0.8959
0.8992
Table 5: Ablation study small-scale GEOM-Drugs.
The results are shown in T On QM9, our method achieves slightly better results than ConfGF, while on Drugs, the results are significantly better than ConfGF and GraphDG. This shows that our method can provide good conformations for property prediction tasks.
4.4 ABLATION STUDY
To further verify our algorithm, we conduct the following ablation study: (1) We remove the attentive node aggregation; instead, it is replaced by simple MLP network. That is, Eqn. is replaced by
11-1) MLP(( (11) ((-
Σ f1 1)
(13)
(((1)
JEN(S
(2) We remove the normalization step in (qn. i.e., the m! is not used; (3) We further use the force field RDKit refine the output of our method. The results are summarized Table We can see that: (1) Without attentively aggregating the atom features, the performances drops: The mean COV drops 5.6 points and MAT score increases 0.06 points. (2) Without the conformation normalization, the performance is greatly effected, which shows the importance of this step; (3) Our method can still be refined by the RDKit, which shows that our method can be combined with the classic methods for more improvements.
Finally, we compute the COV and MAT scores of R!(0) against the groundtruth, which is the output conformation of the 1-th block in the decoder. R(0) is the output of (22D The results are shown in Figure 4 We can see that iteratively refining the conformations can improve the performances, which shows the effectiveness of our design.
1.12
0.36
0.80
(MATNA
(OV(%)
1.10
0.32
0.75
1.08
GEOM Drugs
0.28
GEOM
0.70
2
Block
Block
Figure 4: MAT and COV scores of 7(0 against the grondtruth.
CONCLUSIONS AND FUTURE WORK
In this work, we propose a new method, that directly generates the coordinates of conformations. For this purpose, we leverage loss function that is invariant to roto- tit--translation and design new model with many advanced modules (i.e., GATv2, GN block) that can iteratively refine the confor- mations. Experimental results on both small-scale and large-scale GEOM-QM9 and GEOM-Drugs demonstrate the effectiveness of our method.
For future work, there are many interesting directions. First, we will combine with more generative methods like flow-based model, Langevin dynamics and so on. Second, during case analyzing, we found that there are still some cases that current methods cannot successfully handle (e.g., rotatable rings) and we will improve them in the future (see Appendix C33) Third, current methods are mainly non-autoregressive,, where all coordinates autoregressive setting so as to further improve the accuracy. Fourth, the current loss function is not invariant to permutations of symmetric atoms, and we will improve it in the future.
generated simultaneously. We will study the

---

Under review as a conference paper at ICLR 2022
REFERENCES
Simon Axelrod and Rafael manez-Bombarelli.. Geom: Energy-amnotated molecular conformations for property prediction and molecular generation, 2021.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. ICLR, 2015.
Kyle A. Baseden and Jesse W. Tye. Introduction to density functional theory: Calculations by hand on the helium atom. Journal of Chemical Education, 91(12):2116-211) 2014. doi: 10.1021/ ed5004788.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzales Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1806.v1806..1.. 2018.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? preprint arXiv:2 105. /4491, 2021.
arXiv
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference Learning Representations, 2019.
Thomas A Halgren. Merck molecular force field. i. basis, form, scope, parameterization, and per-
formance of mmff94. Journal of computational chemistry, 17(5-6):490-555 519, 1996.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE Learning basic visual concepts with a strained variational framework. 2016.
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec, Ogb-Isc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2 103.09430. 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448-456, Lille, France, 07-09 Jul 2015. PMLR.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy. Michal Zielinski, Martin Steinegger, Michalina Pacholska. Tamas Berghammer, Se- bastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596(7883) 589, Aug 2021. ISSN 1476-4687. doi: 10.
URL
/t:: org 10. 1038/ 1 021 03819-2
aana Y. Kanal, John A. Keith, and Geoffrey R. Hutchison. sobering assessment of small- molecule force field methods for low energy conformer predictions. International Journal of Quantum Chemistry, 118(5)e25512, 2018. doi: https://doi.org// 100p///..0..00//....52..5 URL
www.wwiee...... com/ www.//oo//aaacccoocccooccc.coeco /oi////////b//o///// 1002 /ua.////auu... 25512
Charles FF Karney. Quaternions in molecular modeling. Journal of Molecular Graphics and Mod- elling, 2555):595-604 2007.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR 2014), Banff, Canada, 2014. ICLR Committee.
Maria Kontoyianni. Docking and Virtual Screening in Drug Discovery, pp. 255-266. Springer New York, New York, NY. 2017. ISBN 140---1--9999-77-111 doi: 10.1007/978 09-499--721.. 18. URL orr 33t
10

---

Under review as a conference paper at ICLR 2022
Iyy Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. CoRR, (bss/1008.0998, 2016. URL /ttt::://axxi..oo///bbbbb
03983
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- net formm?id
Learning Representations, 2019. URL
Bkg6RicqY7
Elman Mansimov, Omar Mahmood, Seokho Kang, and Kyunghyun Cho. Molecular geometry pre- diction using a deep generative graph neural network. Scientific Reports, 91):20031, Dec 2019. ISSN
doi:
URL httph:://cc..c.ooo 0tt.://000..00000..001000.
1115998-199550
Robert G. Parr. Density functional theory of atoms and molecules. Kenichi Fukui and Bernard Pullman (eds.), Horizons of Quantum Chemistry, pp. 5-15, Dordrecht, 1980. Springer Nether- lands, ISBN 978-94-009-9007-2.
A. K. Rappe. C. J. Casewit, K. S. Colwell, W. A. Goddard, and W. M. Skiff. Uff, a full periodic table force field for molecular mechanics and molecular dynamics simulations. Journal of the American Chemical Society, 114(25):10024-10000
1992. doi: 10. 10.1021/ja000511044)
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models, In International Conference on Machine Learn- ing. pp. 1278-1286, 2014.
Kunal Roy. Supratik Kar, and Rudra Narayan Das. Chapter 10 other related techniques. In Kunal Roy, Supratik Kar, and Rudra Narayan Das (eds.), Understanding the Basics of QSAR for Applications in Pharmaceutical Sciences and Risk Assessment, pp. 355-425. Academic Press, Boston, 2015. ISBN 978-0-12-801505.41 B978-0-
doi: http://doi.org/10..000 /ttp://wwwwwww// wwwp://cceneei.eeetccoossce.eeee
URL
Stticce/pa1/39700120000000
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: flow-based autoregressive model for molecular graph generation. In International Confer-
ence on Learning Representations, 2020. URL htttps::/openrevveew net. foat/forrmmiio
SlesMKHYPI
Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular confor- mation generation. In International Conference on Machine Learning, 2021.
Gregor N. C. Simm and José Miguel Hernnndez-Lobato. A generative model for molecular distance geometry. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Con- ference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp- 8949 8958. PMLR, 13-18 Jul 2020. URL httt:://pooceeeeee
ml wwilr.pres//vl..
s simm20a html
Daniel GA Smith, Lori A Burns, Andrew C Simmonett, Robert M Parrish, Matthew C Schieber, Raimondas Galvelis, Peter Kraus, Holger Kruse, Roberto Di Remigio, Asem Alenaizan, et al. Psi4 1.4: Open-source software for high-throughput quantum chemistry. The Journal of chemical physics, 152(18) 8): 184108, 2020.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, volume 28, pp. 3483-3491, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is you need. I. Guyon, U. V. Luxburg, S. Bengio, Н. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (ods.), Advances in Neural Information Processing Systems. volume 30. Curran Asso- ciates, Inc., 2017. URL
htw.aa/aiaaaa.........ooeeaaeaee Paper. -dff
Robin Winter, Frank Noé, and Djork-Amé Clevert. Auto-encoding molecular conformations. arXiv preprint 1rXiv:2101.0/6 2021.
11

---

Under review as a conference paper at ICLR 2022
Minkai Xu, Shitong Luo, Yoshua Bengio, Jian Peng, and Jian Tang. Learning neural generative dynamics for molecular conformation generation. In International Conference on Learning Rep- resentations, 2021a. [tL/tttss///eeerriiii net.
wttp:/wwwwwweee/eouuneieeee Eorum Srrpuumiid-ppA.llggeeeeee
Minkai Xu, Wujie Wang, Shitong Luo, Chence Shi, Yoshua Bengio, Rafael Hromez-Bombarelit and Jian Tang. An end-to-end framework for molecular conformation generation via bilevel program- ming. arXiv preprint arXiv:2105.077246, 2021b.
A DETAILS OF foD AND fPDD
The model architectures of P2n and Pan are similar to Piecs with the following differences.
Comparing $2DD with Pdee, the differences are the initial conformation R! and initial features (i.e., the H H and ((0)). P2D takes random conformation sampled from uniform distribution in [-1, as input. The imitial atom and edge features are the embeddings of the atoms and edges respectively. fPD will also output a prediction the conformation. Note that the random variable sampled from Gaussian N Gun,G, Σn,G) is not used in PPDD
Comparing faD with Pdez, the differences are the initial conformation R((), initial features (i.e., the HVO H and (1()) too. (anD takes the groundtruth conformation as input. The initial atom and edge features are the embeddings of the atoms and edges respectively. Another difference is that the fourth step Faee. i.e., Eqn. (1)). is not used. More details are available in our code.
B MORE EXPERIMENT DETAILS
We use A optimizer (Loshchilov & Hutter 2019) with initial learning rate Too 2 10-1 and weight decay 0.01. In the first 4000 iterations, the learning rate is linearly increased from cosine learning rate scheduler (Loshchilov & Hutter 2016), where
to 2 10 After that, the learning rate at the t-th iteration is 7o(1 + cos(x f 4)) where T is the half of the period (i.c., the iteration numbers of 10 epochs in our setting). Similarly,
also the cosine scheduler to dynamically set the B at range [0.0001, 0.008). The batch size is fixed as 128. All models are trained for 100 epochs. For the two small-scale settings, the experiments are conducted on a single V100 GPU. For the two large-scale settings, we use two V100 GPUs for experiments. The detailed pyper-parameters are described in Table
Small-Scale
Large-Scale
Layer number
3
6
Dropout
0.1
0.1
Learning rate
2e-4
2e-4
Batch size
128
128
Epoch
100
100
Vae 3 Min
0.0001
0.001
Vae 9 Max
[0.001, 0.002, 0.004, 0.008. 0.01)
(0.005, 0.01, 0.02, 0.04.0.05)
Latent size
256
256
Hidden dimension
512
1024
GPU number
Xx NVIDIA V100
2x NVIDIA V100
Table 6: Hyper-parameters for our experiments
C MORE EXPERIMENTAL RESULTS
C.I COMBINATION WITH DISTANCE-BASEE AND ANGLE-BASED LOSS FUNCTIONS
In addition to the matching loss defined in Eqn. which is related coordinates only, may be curious about whether using distance-based loss and angle-based can further improve the per- formance, since the latter two equivariant the transformation of coordinates. For ease of
12

---

Under review as a conference paper at ICLR 2022
reference, let R₃ denote the groundtruth coordinate of atom os and R. denote the predicted coordi- nate of atom 84, Remind in Section 2 we use E' to denote the collection of all bonds. We also define E2 ())((())) E E, ki) E #
Following (Winter et al.
2021 we use the following two functions:
Luugle
Σ
|| cosine(R, Rs, R R₂) - -oosine R R. Rs R.ll
(14)
E2l
(L.j.k)EE.
distance Ry, R₂) - distance RS R.))
bond
(15)
E
(Lf))E
where cosine(a, b) = H and distance(a, b) = ||a bll, and b are two vectors. That is, apply additional constraints to bond length and bond angles. Please note that with the above two auxiliary loss functions, our method still generates coordinates directly and does not need to generate intermediate distances and angles.
We set the weight of the above two loss functions as 0.1, and add them to Equ. We conduct experiments small-scale QM9 and Drugs. The results reported in Table which are in the row starting with "Ours + Aux". We can see after using those two auxiliary loss functions which are invariant the ooto-rranslationll operations on the coordinates, the results can be further improved. We will further explore this direction in the future.
Dataset
QM9
Drugs
COV(%)}
MAT (A)L
COV(%)
MAT (A)
Methods
Mean
Median
Mean
Median
Mean
Median
Mean
Median
RDkit
83.26
90.78
0.3447
0,2935
60.91
65.70
1.2026
1.1252
CVGAE
0,09
0.00
1.6713
1.6088
0.00
0.00
3.0702
2.9937
GraphDG
73.33
84.21
0.4245
0.3973
8.27
0.00
1.9722
1.9845
CGCF
78.05
82.48
0.4219
0.3900
53.96
57.06
1.2487
1.2247
ConfVAR
80.42
85.31
0.4066
0.3891
53.14
53.98
1.2392
1,2447
ConfGF
88.49
94.13
0,2673
0,2685
62.15
70.93
1.1629
1.1596
Ours
81.44
96.61
0.2879
0.2428
77.78
86.09
1.0657
1.0563
Ours + Aux
91.97
98.66
0.2388
0.2231
92.45
98.70
0.8983
0.9016
Table 7: Combination with iittance-based and angle-hased loss functions.
C.2 PROPERTY PREDICTION
Following Shi et (2021). besides energy, we also calculate HOMO and LUMO of all the con- formation using Psi4. Note that all the generated conformations are refined by MMFF Then we will calculate ensemble property average energy E, lowest energy Emins average HOMO-LUMO gap Ac, minimum gap Acmin and maximum gap Acman for each method and use mean absolute error (MAE) and median absolute error with ground-truth conformations to measure accuracy. The results are showed in Table 8 and Table 9 As the results shown, our method achieves the best accuracy in general.
QM9 Ac
Drugs Ac
Methods
E
Emioo
Acmin
Ammas
E
Emmn
Ammin
A Acmua
Rdkit
0.0507
0.0276
0.0853
0.1724
0.1384
0.1720
0.1255
0.1872
0.1790
0.2188
GraphDG
0.0859
0.0567
0.1723
0.2569
0.1977
5,6440
0.3520
0.6502
2,9129
0.2677
ConfGF
0.0458
0.0229
0.1198
0.1798
0.1531
2.7344
0.1379
0.3171
2.8166
0.1820
Ours
0.0418
0.0213
0.1163
0.1681
0.1450
0.1573
0.1129
0.1724
0.5469
0.1780
Table 8: Mean absolute error of predicted ensemble properties. Unit: eVV.
13

---

Under review as a conference paper at ICLR 2022
QM9
Drugs Ac
Methods
E
Emin
A Crsin
Cmaz
E
Emin
Amin
Cmar
Rdkit
0.0325
0.0233
0.0814
0.0917
0.1103
0.1619
0.0879
0,1418
0.1432
0.1179
GraphDG
0.0624
0.0309
0.1430
0.1562
0.1704
4.1807
0.2190
0,4412
3,3542
0.2127
ConIGF
0.0348
0.0209
0.0991
0.1497
0.1534
2.7917
0.0725
0,2700
2,9545
0.1972
Ours
0.0318
0.0160
0.1174
0.1336
0.1501
0.0695
0.0618
0.1431
0.1425
0.1043
Table Median absolute error of predicted ensemble properties. Unit:
C.3 FAILURE CASES
We notice some failure cases in model prediction. Two examples are shown in Figure 5 We can see that there exists rotatable ring at the end a molecule, where the ring is symmetric to the bond connecting itself to the rest of the molecule. Our method fails to generate the coordinates of such rings, but simply puts them using the additional loss functions conformations of those rings (see the row Ours + Aux). We will keep exploring along this direction.
line. This is because the model is trapped into local optimal. By introduced Appendix C..
successfully recover the
Molecular
Graph
Reference
Ours
Durs Aux
Figure 5: Failure case of method. "Our Aux" that add auxiliary loss method, and these two losses improve our method effectively.
C.4 STUDY OF MODEL PARAMETERS
In this section, compare the performances of method and ConfGR. By default, model has 13.29M parameters, and the ConfGF model has 0.81M parameters. We explore the following two settings: (1) reduce our model size 0.98M by reducing hidden dimension sizes and layer number; (2) we increase the model size of ConfGF to 12.28M by increasing hidden dimension sizes, and re-run the experiments. The results are shown in Table I0
Dataset
QM9
Drugs
COV(%))
MAT (A)J
COV(%)
MAT (A).
Methods
Mean
Median
Mean
Median
Mean
Median
Mean
Median
ContGF (0.81M)
88.49
94.13
0.2673
0.2685
62.15
70.93
1.1629.
1.1596
Ours (0.98M)
80.47
95.83
0,2859
0.2508
79.24
90.91
1.0777
1.0665
ConfGF (12. (12.28M)
86.86
93.49
0.3377
0.3450
55.36
58.20
1.2186
1.2134
Ours (13.29M)
81.44
96.61
0.2879
0.2428
77.78
86.09
1.0657
1.0563
Table 10: Comparison of our method and ConfGF with different model sizes
We can see that: (i) when we reduce our model size to 0.98M, our method still outperforms Con- fGF on GEOM-Drigs; GEOM-QM9, method outperforms ConfGF evaluated by the median
14

---

Under review as a conference paper at ICLR 2022
values of COV and MAT. (ii) The performances of model with 13.29M parameters and 0.98M parameters are similar. (iii) When we increase the model size of ConfGF, the performance becomes worse, which shows that ConfGF cannot benefit from more parameters. We observe that larger ConfGF (12.28M) suffers from larger training loss than a smaller ConfGF (0.98M), which shows its limitation.
C.5 MORE DISCUSSIONS ON THE CONFORMATION WITH MORE HEAVY ATOMS
In Table we observe that our method works better than distance-based methods molecules with more heavy atoms. Our conjecture is that for these distance-based works, they usually extend the molccular graph with 1,2,3-order neighbors, which is sufficient determine the 3D structure in principle. For GEOM-QM9 dataset, considering the number of atoms less than 10, this extended graph is nearly complete graph and provide enough signals to reconstruct the 3D structure. Therefore, these distance- based performances are good on GEOM FEOM-QM9 dataset. For GEOM-Drugh dataset, the numbers of atoms much more than those in GEOM- -M9.. Although in theory, the distances third-order extended graph can reconstruct the 3D structure, practically the signals still not enough. Our method does not rely the interatomic distances, and can achieve good results on large molecules.
15


Reviews:
[{"review_id": "qTsEulSoocU", "reviewer": "Reviewer_P4XU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOverall, this work appears to be an advance compared to the\nLangevin-based method it mainly compares against.  It is not clear to\nme if this is simply due to a different parameter setting (e.g. a\nlarger model than a year ago works better for larger molecules), or if\nthis improvement is due to a genuine algorithmic advance, and if so,\nwhat part of the algorithm helps the larger molecules the most and why\ncouldn't the authors beat the previous model on the smaller-sized\nmolecules.  I appreciated the authors effort to document the change of\nthe results across the methods in Table 3, but it would help if they\ncould also modify their new model (or ConfGf, or both) to a\ndrastically different size.\n\nI did not understand what was the contribution for the fine-grained\nloss function used in training. Unless I misunderstand something\nimportant, the particular matching loss is not invariant to\npermutations of symmetric atoms---think of the different ways to\nenumerate the atoms in a methyl group, or replace the hydrogen in a\nmethane by methyls, or take any symmetric groups in a molecule that\ncan generate a combinatorial number of equivalent coordinates that\nrepresent the same molecule graph. So I don't understand the statement\nof the authors that this loss is zero if and only if R1 and R2\nrepresent the same molecule structure---what would the loss be if I\nrandomly permute the symmetric hydrogen identities but not change the\ncoordinates?\n\nRegarding Secion 4.3, I understand that these types of tables have\nbeen published previously, but I wanted to state that I strongly\ndisagree with the terminology of calling the average energy of the\ngenerated conformers a \"property prediction\".  I don't think that\nthere exists a single experiment that can measure this particular\nquantity, although arguably one could use a Quantum Mechanical code,\nas these authors and others have done, and calculate a number for it.\nFrom a physics perspective, the somewhat more meaningful property is\nthe Boltzmann-weighted average of the energies of the molecules---is\nthat perhaps the quantity that this benchmark is supposed to evaluate?\nFurthermore, what is the meaning of switching to an MMFF force field\nfor optimization and then use Psi4---why not directly evaluate the\nconformation with Psi4?  Finally, what are the units in the energies\nof Table 4?  If these are in Hartree (which I thought were the default\nunits for Psi4, though I'm no expert in it), then the reported errors\nare all huge to be meaningful one way or another (to the order of more\nthan 13 kcal/mol for the smallest number listed in that table); these\ntypes of enormous errors have little meaning for any conformational\nenergy comparison, as a trivial small bond-length deviation could add\nthis level of energy, whereas a wrong dihedral angle couldn't (unless\nparts of the molecule almost clashed onto itself).  Perhaps the\nauthors would want to either clarify section 4.3 further (it took a\nwhile to parse what the authors actually calculate there, and from a\nquick check of the repo, I couldn't locate the relevant code for it)\nor simply drop it altogether since it probably distracts from the rest\nof this work.\n\nI also didn't understand the first two of the three future directions:\nI am not clear what the authors mean when they suggest combining this\nmethod with flow-based or Langevin dynamics models---can they clarify\nthis point?  Importantly, what are the cases that fail in the current\nmodel---can the authors document some examples in an appendix?\n\n", "summary_of_the_paper": "\nThe paper presents a direct method for generating a molecular\nconformation conditional on the molecular graph.  Inspired by the\nsuccess of AlphaFold2, the authors demonstrated that one could\ndirectly write out 3D coordinates for a small molecule rather than use\nthe previous ways of generating intermediate interatomic distance\nmatrices or iterate over forces. The method displays a decent\nperformance on the standard benchmark set of QM9.  It performs better\nthan past baseline methods for larger molecules.\n", "summary_of_the_review": "\nI think that this paper is an advance on previous work and probably represents\nthe state of the art in generating molecular conformations conditional\non the graph.  The paper could use a review of the language to clarify\ncertain aspects of it, but I don't think that it is strong enough for ICLR.  \nImportantly, it should attempt to clarify if the improvement is mainly \ndue to model size changes or due to algorithm changes.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lGmmH9xamkb", "reviewer": "Reviewer_Y6xJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is well-organized and clearly written. All content is easy to understand for me.\nThe experimental part is comprehensive. The author nicely compares the proposed method with competitive baselines on both small-scale and large-scale datasets. The results demonstrate that the model is better or comparable with state-of-the-art results with much less computational cost.\nWeaknesses:\nThe main concern of this paper is the originality. The main contribution of this paper is the decoder part, which is very similar to the “structure module” in Alphafold that iteratively updates the folding and node/edge features. However, I appreciate the efforts to extend the progress of protein folding to molecular modeling, so I also agree the adaption is non-trivial.\nA minor problem: the problem definition in Sec.2 seems not accurate enough. In your statement, it seems each graph only corresponds to a single conformation, but this is not true. Please restate it.\nI’m also interested in the design of the decoder block. I understand the training objective is invariant, however, I’m curious whether updating the coordinates with equivariant networks will further help the performance. Several recent works such as ConfGF have shown that equivariance is an important inductive bias for 3D generative models.\n", "summary_of_the_paper": "The paper proposed a new VAE-based generative model for generating molecular conformations from graphs. The whole formulation and training objective are similar to existing VAE-based models like CVGAE, and the main contrition lies in the architecture of the decoder. The architecture borrows the idea from the recent advance of AlphaFold: the proposed decoder is composed of several identical updating blocks, where they iteratively update the node/edge embeddings as well as the coordinates in each block. And the blocks are implemented with advanced graph neural networks. \n", "summary_of_the_review": "The paper proposed a VAE model for molecular conformation generation, where the decoder borrows the idea of Alphafold to iteratively refine the generated structure. Personally, the paper seems an extension of the recent progress of protein folding to the small molecule setting, but I appreciate the authors’ efforts to make it work. So currently I vote for a weak acceptance.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kecKXS2vDhL", "reviewer": "Reviewer_rZ2g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** The work suffers from the lack of originality. The described model shares the same idea as ConfVAE and the unnamed model, proposed in [1] and [2] respectively - the autoencoder-based framework for conformation generation, the iterative generation procedure, and RMSD objective were proposed earlier. The main novelty of the model is neural network architectures of encoder and decoder, that utilize advanced graph convolutions. Both of [1] and [2] works are missed in baselines, the model compared with.\n\n**Clarity and Quality:** The authors clearly describe their motivation and provide good discussion and ablation studies section, still paper contains vague contributions part and missed related work section. The paper is written in a hard-to-read way. The paper contains plenty of inline equations, inconvenient formatting of enumerated statements that make it hard to navigate over text and understand the method. The paper focuses too much attention on technical details of the model, that are minor and can be moved to the Supplementary materials.\n\n**Significance:** The novelty of the paper is minor. The experiment results show the increase of metric values between the proposed model and ConfGF baseline, still, it's hard to determine the significance of the performance gap due to the lack of standard deviation. Also, the significance of the results is doubtful since the important baselines [1] and [2] are missed in Experiment section.\n\n**Drawbacks / questions**:\n\n1. I would recommend using different Greek letters to denote the encoders/decoder parts instead of subscript text to increase the readability of formulas.\n2. The ablation studies cover only architecture details. Still [2] provide loss function based on distance differences that is also roto-translation invariant but do not require additional subproblem solving.\n3. The small-scale dataset results are redundant still large-scale dataset covers it. I would recommend training CVGAE and CGCF on the large-scale dataset and including the results of these models.\n4. The choice of conformation colors in Figure 3 is misleading. I would recommend to add an additional figure with aligned conformation cloud to show the diversity of generated conformations.\n\n[1] An End-to-End Framework for Molecular Conformation Generation via Bilevel Programming, Xu et al.\n\n[2] Auto-Encoding Molecular Conformations, Winter et al.", "summary_of_the_paper": "The authors propose the generative model that constructs a conformation (set of 3d coordinates) of the molecular graph with a variational autoencoder (VAE) framework. Unlike several recent works on conformation generation based on generating of distance matrix and further conformation recovering by solving a time-consuming distance geometry problem, the proposed approach directly generates 3d coordinates with an iterative process inside the decoder part of the model.", "summary_of_the_review": "The idea of the paper is not novel. The experiment part misses several important baselines. The text should be revised to improve readability and clarity. The ablation study section should provide experiments to justify loss function choice.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "3dY6BS4KnhF", "reviewer": "Reviewer_T1Ea", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The results are strong and the presented methodology is clear.\n* The ablations shown in Table 5 are very nice to see and I think are a useful part of the paper.\n\nWeaknesses:\n* Some of the discussion of results could be expanded, or at least included in an appendix. For example, \"This is consistent with our conjecture, and also demonstrates the potential of directly generating coordinates for complex molecules with more heavy atoms.\" This would be nice to have been further expanded upon.\n* The presented results are limited in scope, and largely rely on comparisons with ConfGF, though then some additional comparisons would be useful. For example, for the property prediction showing results comparing to Tables 3 and 4 in the ConfGF paper would be useful. \n\nComments:\n* In Fig. 3, what do the colors represent? Are two views shown?\n*  A table with exact hyper parameters/etc in the appendix could be useful (in addition to the details in section 4.1). It's great that code is provided.\n* Some methods, like G-SchNet (https://github.com/atomistic-machine-learning/G-SchNet) from my understanding do not produce confirmations which violate the triangle inequality. Is this a misunderstanding on my end, or is this approach of successively predicting distances an alternate approach that produces valid molecules?  I realise it is a different problem being approached, but molecules are still being generated. \n* In Table 2, what is your interpretation of the COV(%) being 100%? \n* For the property prediction, on QM9 various properties can be predicted. Could these all be shown?  \n* The text is at times a bit difficult to follow, but it does not detract from the ability to follow the work as a whole. \n* I don't quite follow the \"ours with /FF\" in Table 5, and why this is not the \"default\" flavour?", "summary_of_the_paper": "The authors present a method that is able to accurately generate molecule confirmations, achieving competitive results across four datasets. \nUnlike previous methods that often rely on directly predicting various inter-atomic distances (through a distance matrix) (which in some cases results in physically impossible molecules), the presented work predicts the location of all atoms. The loss function is invariant to rotation, an important property of many molecules. ", "summary_of_the_review": "The work presents strong results on molecular confirmation prediction. A novel method is developed that mitigates concerns of many previous methods by ensuring adherence to geometric rules. The results shown are very competitive, though I would like to see slightly broader comparisons and more exposition on various points that are raised in discussion. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a new VAE-based generative model for generating molecular conformations from graphs. Reading the paper, the reviews and the rebuttal, it looks like this project is a work in progress and not yet ready for publication. Some reviewers indicate that the paper lacks significant technical novelty. 

This is a worthy application. I encourage the authors to keep working and resubmit to another venue when they feel the work is ready and they have addressed all points raised by the reviewers.
Decision: Reject