Direct Molecular Conformation Generation
Authors: Jinhua Zhu
Yingce Xia
Chang Liu
Lijun Wu
Shufang Xie
Yusong Wang
Tong Wang
Tao Qin
Wengang Zhou
Houqiang Li
Haiguang Liu
Tie-Yan Liu
Venue: Accepted by TMLR ()

Abstract: Molecular conformation generation aims to generate three-dimensional coordinates of all the atoms in a molecule and is an important task in bioinformatics and pharmacology. Previous methods usually first predict the interatomic distances, the gradients of interatomic distances or the local structures (e.g., torsion angles) of a molecule, and then reconstruct its 3D conformation. How to directly generate the conformation without the above intermediate values is not fully explored. In this work, we propose a method that directly predicts the coordinates of atoms: (1) the loss function is invariant to roto-translation of coordinates and permutation of symmetric atoms; (2) the newly proposed model adaptively aggregates the bond and atom information and iteratively refines the coordinates of the generated conformation. Our method achieves the best results on GEOM-QM9 and GEOM-Drugs datasets. Further analysis shows that our generated conformations have closer properties (e.g., HOMO-LUMO gap) with the groundtruth conformations. In addition, our method improves molecular docking by providing better initial conformations. All the results demonstrate the effectiveness of our method and the great potential of the direct approach. The code is released at  \url{https://github.com/DirectMolecularConfGen/DMCG}.

Published in Transactions on Machine Learning Research (10/2022)
Direct Molecular Conformation Generation"
Jinhua Zhu!
teslazhu@mail ustc.edd.com yingce. xio miirosooo com changliu changliu@mierosoft Gmperoooff lijuau&mmcroo @mccroooft. com shouf huufongzie Druc. edu. on wangyusong 200 2000 0000000t00 @su.. sjuu. edu. watong @microsoft. com taogin @ @microsooft. com zhoug Bustc. edu. on ligg@ustc. edu. Sectiiaann@@iiccosse com Iyliu@microsoff.ccmm com
Yingce Xia
Chang Liu
com
Lijun Wu²
Shufang Xie
Yusong Wang*
Tong Wang
Tao Qin
Wengang Zhou
Houqiang Li
Haiguang Liu2
Tie- Liu?
University of Science and Technology of China
Microsoft Research A14Science
Renmin University of China
Xi'an Jiaotong University
Reviewed on OpenReview: https: // openreview. net/ forum? id- \CPOH6
Abstract
Molecular conformation generation aims to generate three-dimensional coordinates all the atoms a molccule and is an important task bioinformatics and pharmacology. Pro- vious methods usually first predict the interatomic distances, the gradients of interatomic distances the local structures (e-g., torsion angles) of molecule, and then reconstruct its 3D conformation. How to directly generate the conformation without the above intermedi- ate values is not fully explored. In this work, we propose method that directly predicts the coordinates of atoms: (1) the loss function is invariant to roto-translation of coordinates and permutation symmetric atoms; (2) the newly proposed model adaptively aggrogates the bond and atom information and iteratively refines the coordinates of the generated conformation. Our method achieves the best results on GEOM-QM9 and GEOM-Drug datascts. Further analysis shows that our generated conformations have closer properties (e.g., HOMO-LUMO gap) with the groundtruth conformations. In addition, our method improves molecular docking by providing better initial conformations. All the results demon- strate the effectiveness of our method and the great potential of the direct approach. The code is released at httpp::/gtt http:///ww..caaacaaaaaaaaoooooaa htt.a//efa.o..eooooooooooooooooa
1 Introduction
Molecular conformation generation aims to generate 3D atomic coordinates of a molecule, which then can be used in molecular property prediction (Axelrod & Gomez- -oanazz-ommbaellii 2021), docking (Roy et al. 2015). structure-based virtual screening (Kontoyianni 2017), etc. While molecular conformation is experimentally obtainable, such via X-ray crystallography, it is prohibitively costly for industry-scale tasks (Mansimov
el al. 2019). Ab initio methods, e.g., based on density functional theory (DFT) (Parr 1980: Baseden & Tye 2014), can accurately predict molecular structures, but take several hours per small molecule (Hu et al. 2021 To handle large molccules, people turn to leverage classical force fields, like UFF (Rappe et al. 1992)
or MMFF (Halgren 1996) and its extension (Cleves Jain 2017). to optimize conformations, which is efficient but at the cost of low accuracy (Knnll al. 2018).
*This work was done when Jinhua Zhu and Yusong Wang were interns Microsoft Research AT4Sclence. Correspondence Yingee Xia and Chang Liu.

---

Published in Transactions on Machine Learning Research (10/2022)
9
5
O
0
3
11
17
6
2
16
10
12
18
13
14
Figure 1: An example of symmetric substructure of a molecule.
Recently, machine learning methods have attracted much attention for conformation generation due to their accuracy and efficiency. Most of previous methods first predict some intermediate values, like interatomic distances (Simm & Mrrndande--Lobaaaee 2020 Shi et al. 2020 Xu et al. 2021a the gradients w.r.t. in- teratomic distances (Shi et al.] 2021 Luo et al. 202Tb), or the torsion angles (Ganea et al.) 2021 ; and then reconstruct the conformation based on them. While those methods improve molecular conformation gener- ation, the intermediate values they used should satisfy additional hard constraints, which are unfortunately violated in many cases. For example, GraphDG (Simm & Hernández-Lobate 2020) predicts the interatomic distances and then reconstructs the conformation based on them. The real distances (e.g., considering three atoms) should satisfy the triangle inequality, but the distances predicted by GraphDG violate the inequality out of 8.65% cases according to our study. For another example, ConIGF (Shi et al. 2021 predicts the gradi- ent interatomic distances, and the rank of a squared distance matrix at most five. Such constraint makes gradients ill-defined, because other distances cannot all be held constant while taking an infinitesimal change C for more details). Directly generating the coordinates without those intermediate values is a more straightforward strategy but is not fully explored. AlphaFold 2 (Jumper) et al. 2021 is such a kind of direct approach and has achieved remarkable performances on protein structure prediction. The success of AlphaFold 2 inspires us to explore the method of directly generating coordinates for molecular conformation.
to specific distance dij (see Appendix
A challenge of this approach is to maintain roto-translation invariance and permutation invariance. Specif- ically, (1) rotating and translating the coordinates of all atoms as a group do not change the conformation of molecule, which should be taken into consideration for the direct approach; (2) Permutation invariance should be considered for symmetry-related atoms. For example, as shown in Figure I duc to the symmetry of the pyrimidine part along the C-S bond (alom 11 and 12), aloms 13, 14 and aloms 17, 16 are equivalent. Therefore, swapping the coordinates of 13 with 17 and 14 with 16 yields the same conformation. According to our statistics on a subset of 40K molccules from GEOM-Drugs (Axelrod & Gomez- Bombbrellij 2021) on average, each molecule has 5.9 atom mappings which could result in the same conformation (more specifically, the average number of [S] Equ. is 5.9). The number is non-negligible for loss function design.
To maintain roto-translation and permutation invariance, in our method, design loss function as the minimal distance between two sets of coordinates after any roto-translation and permutation of symmatric atoms. Based on the new loss function, we design model that iteratively refines atom coordinates. The model stacks multiple blocks, and each block outputs a conformation which is then refined by the following block. A block consists of several modules that encode the previous conformation as well as the representa- tions of bonds, atoms and global information of molecules. At the end of each block, we add a normalization layer that conters the coordinates at the origin. Since molccule may have multiple conformations, inspired by variational auto-encoder (VAE), we introduce a random variable and regularization term on 2, which allows diverse generation.
We conduct experiments on four benchmarks: GEOM-QM9 and GEOM-Drugs with the small-scale set- ting (Shi el al. 2021) and large-scale setting (Axelrod & Gomez- Bombarll 2021). The small-scale GEOM- QM9 and GEOM-Drugs have 200K molecule-conformation pairs for training, and the large-scale GEOM- QM9 and GEOM-Drugs have 1.37M and 2.0M training pairs. Our method achieves stato-of-tho-art results on all of them, demonstrating the effectiveness of our method. Specifically, on small-scale GEOM-QM9, our method improves the recall-based mean coverage score and mean matching score by 4.7% and 0.3%. On
2

---

Published in Transactions on Machine Learning Research (10/2022)
small-scale GEOM-Drugs, the improvements are 7.4% and 16.3%. On the large-scale settings, the improve- ments are more significant: 7.3% and 47.1% for GEOM-QM9, and 25.3% and 36.0% for GFOM-Drugs. To further verify the generation quality, we use Psi4 (Smith et al. 2020) to calculate the properties of generated conformations and groundtruth conformations (e.g.. HOMO-LUMO gap). Our conformations have closer properties to the groundtruth compared with other methods. We also find that our generated conformations can help improve molecular docking by providing better initial conformations.
To summary, (1) we design dedicated loss function, that can maintains both permutation invariance symmetric atoms and roto-transse transsarnnnt invariance conformations; (2) we design new model that iteratively refines the conformation. (3) our method, named Direct Molecular Conformation Generation (DMCG), outperforms strong baselines and achieves state-of-the-al results on all benchmarks we tested.
Problem Definition: Let G = (V, denote a molecular graph, where V and E are collections of atoms and bonds, respectively. Spocifically, V = {ui, u2,,.. , with the i-th atom V,. Let Ctj denote the bond between alom v; and Vy. For ease of reference, we simply use E V and (i,j) E E to denote the i-th atom in V and the bond Eij in E. Let (i) denote the neighbors of atom i, i.e., (() = {i (i, E E}. We use R to represent the conformation of G, where R E Vxx,, The i-th row of R (denoted as R,) is the coordinate of atom e.. Given graph G = (V, E), our task is to learn mapping, that can output the coordinates R of all atoms in V, i.e., R E |R
2 Framework
In this section, we first introduce the loss function. After that, we present the overall training and inference workflow of our method. Finally, we introduce our proposod model.
2.1 Loss function
Roto-translationall and permutation invariance of conformations: Let R E RV and ît glviss denote the groundtruth conformation and the generated conformation. The roto-translation and permutation invariant loss defined as follows:
@RTP R, ft)= min || 1 ((
(1)
PES
In Eqn. (i) p denotes a voto-translational operation, which means to rolate and translate conformation
rigidly; (ii) S denotes the collection of the permutation operations on symmetric atoms. For example, in
Figure 1 contains two clements d1 and 02, where dn is an identical mapping, i.c. o1 = i for any
e {1,2, .............................. .......18))) 18), and 02 is the mapping on symmetric atoms of the pyrimidine: o2((3) = 17.
13, (34)) = 16, o216) = 14 and (1) = i for the remaining atom i's. (iii) || is defined as Σ., | In all, Eqn. defines a loss between R and ft as the minimal achievable distance under any rolo-translation operation and any permutation operation of symmetric atoms, hence is invariant to these operations. Equ.
can be solved via quaternions (Karney 2007 Hamilton 1840) and graph isomorphism (Meli & Biggin 2020
To solve Eqn. 1 the optimization can be decomposed into two sub problems: (S1) lरT = min, lle( ft) R
(S2) c9 minlre ||o(( R.
Karney (007)) propose to use quaternions (Hamilton 1840) to solve (S1). A quaternion @ is an extension of complex numbers, g = gou + qıi q2j + gak, where qu, Q1. 92, 93 are real scalars and u, i, j, k are orientation vectors. With quaternions, any rotation operation is specified by a 3 x 3 matrix, where each element in the matrix is the summation/muul commmationmmutipplcaationn /multpplicationn of go to 93. The solution to (S1) is the minimal eigenvalue of 4 x 4 matrix obtained by algebraic operations back-propagation through p (see Appendix B.3 for the ablation study).
R and R. To stabilize training, we stop gradient
To solve (S2), we need to find all elements in S, and then enumerate them to get the minimal value. S can be mathematically described as follows: (1) Vi E V, atom i and atom ai) have the same label, which is defined as the union of the atom type itself and also the types of all the bonds connected to i (2) There
For example, in Figure the label atom 11 is "S-2Sing"", and the label of atom 17 "N22rrmtaii"".
3

---

Published in Transactions on Machine Learning Research (10/2022)
exists bond between atoms i and j if and only if there exists bond between atoms oii) and o(j) in the same molecular graph. Therefore, we convert finding S into a graph isomorphism problem on molecular graphs. Inspired by Meli & Biggin (020), we use the graph_ tool toolki to find all permutations in S. By combining the above two strategies, we are able to solve Equ. (I) We provide several examples in the online supplementary material to show how our method works.
Hoperoft& & Wong 1974) proposed an algorithm whose complexity of testing planar graphs for isomorphism is O((E)) where E is the number of edges in graph. A planar graph can be regarded as type of graph that edges cross each other (see Wiki for quick introduction). For the widely used GEOM-QM9 and GEOM-Drugs datasets (Shi et al.) 2021 Xu et al.) 2022) of conformation generation, all the molecules are planar graphs. We also randomly sample 30M compounds from PubChem, and only 4.5k of them are not planar graphs (0.015%). This shows that although our method needs to test graph isomorphism, the time complexity could still be controlled. In addition, the |S['s of 99.8% molecules in GEOM-Drugs are smaller than 100 and efficient to enumerate them all in GPU. limitation is that, when stepping from small molecules to proteins with long chain, [8] will significantly increase, resulting in large computation cost of obtaining {RTP. We will improve it in the future.
One-to-many mapping of conformations: A molecule might correspond to multiple conformations. Thus, we introduce a random variable z to our model for diverse con conformation generation. Given a molecular graph G, different z could result in different conformations (denoted as R(2, G)). Inspired by the variational auto-encoder (VAE) (Kingma & Welling 2014 Rezende et al.) 2014 Sohn et al. 2015), we introduce a (conditional) inference model (2(R, G) to describe the posterior distribution of z, reform the reconstruction loss in probabilistic style Ext=|R,0) (ARTP(R, R(2, G))]. and append regularization term in the form of the Kullback-Leibler (KL) divergence w.r.t. prior distribution p(=), i.e. DKL. (g(z R, G)Il(()). In this way, the aggregated (i.e. averaged marginalizedd) posterior ʃ q(2RR, G) (R) dR is driven towards the prior p(z), which in turn allows generating a new conformation from Paata by passing through the decoder with a p(z) sample. It is easy to draw a random variable = from p(=) and encourages diversity.
By properly choosing qz G), the loss is tractable to optimize. We specify q(2]R, G) == ((ZlRR, ER.G) multivariate Gaussian with a diagonal covariance matrix, where the PR,G and ERGG are outputs from encoder. It enables tractable loss optimization via reparameterization (Kingma & Welling 2014):
(x is equivalent to 2 -RRGG + V D Vi, where e ~ (N 1), 200 and PR. are the i-th element (f.))) of 2 and MRG, and RR. denotes the i-th diagonal element. The KL divergence loss is specialized as DKL(W EI, I)), which has closed form solution.
Overall, the overall training objective function is defined as follows:
min E. (R, R(e,, G)) + (N(Hn,,
(2)
where 3 > 0 is hyperparamoter. The minimization in Equ. is taken over all the network parameters (including the conformation generator and auxiliary model q).
2.2 Training and inference flow
Now we show the training and inference workflow. The training process involves three modules, P2D, PaD and Fdee. The workflow is illustrated in Figure a). Specifically,
(1) The encoder PoD takes the molccular graph G as its input, and outputs several representations: H e g for all atoms, ( e ll for all bonds, a global graph feature (()), Rdd and initial conformation k(0) RIV1x3 Note d is the dimension of the representations. Formally, (HI(") H (,(),, k())) (G).
(2) The encoder @aD extracts features of the conformation R for constructing the conditional inference module R2 G). According to the above specification, @DD only needs to output the mean and covariance of the Gaussian, or formally, (Hn,a, ER,G) = PaD R, G).
htttps://graph-toom skeved.de
4

---

Published in Transactions on Machine Learning Research (10/2022)
G
94 R(0), Hy (0)
22
)
(0) U
R(z,6) G)
oll
q ddec
P3D
(u IR,G)
Z
R
((]|R,G)
(a) Training workflow.
R(z,G) G)
G
R(O), H. (1 U()))
2D
dec
Z
p(z) = N(O.1)
(b) Inference workflow.
Figure 2: The workflow of our method. Green and orange lines represent how to obtain R(2, G) and q( [2) G) respectively. Solid lines and dashed lines represent the model components and outputs respectively.
(3) We randomly sample a variable from the Gaussian distribution NuRRO, ER,C), and then feed 1 H U10), R()) into the decoder Pdee to obtain the conformation R(z,G). That is, R(z, G) = @dee(ppaa (P) 2) = Pdee (H H (,000)000,0 ,)))) 2).. Note that sampling 2 ~ N(HRa, ER,c) is equivalent to sampling (0, I) and then setting 200) PR.G + +VeRO (3.0) R.C.
(6)
(4) After obtaining A((z, G) and N(RR, kR,,)), we optimize Eqn. 2 for training. Recall that R(2, G) is related to P2D, PaDD Pdec: and PR,G, ER, are related to (an.
The inference workflow is shown in Figure bb,, where the well-trained fo2 and Pdee are leveraged: (1) Given a molecular graph G, we use 42D to encode G and obtain f((0), H H U()); (2) we sample a random variable z from Gaussian NN0, I); (3) we feed R H H( U(0), into Pdec and obtaín the eventual conformation R(z, G). Note that yaD is not used in inference phase.
2.3 Model architecture
The encoders (2D: PaD and the decoder iadee share the same architecture. They all stack identical blocks. We take the decoder Gdee as an example to introduce its (-th block, and leave the details of poD and FaD to Appendix A.1
Figure shows the architecture of the I-th block of Pdecc Roughly speaking, this block takes the outputs from its preceding block (including the conformation f((1-1), atom representations H(-1), edge representations H! (I-1) and the global representation ((-1)) of the whole molecule) and outputs refined conformation and representations of atoms, bonds, the whole graph. The process is repeated until the eventual output R(4) is obtained. For the input of the first block (i.e., I = 1), the H( H()) U and f(() are the outputs of p2D.
We use variant of the GN block (Battaglia et al. 2018 Addanki et al., 2021) as the backbone of our model due to its superior performance in molccular modeling. In each block, we first update bond representations, then atom representations, and finally the global molecule representation and the conformation. For ease of
reference, let (1) denote the representation of atom i output by the (-th block, and (!) the representation of the bond between atom and j. Also, let MLP denote a food-forward network.
Mathematically, the 1-th block takes following operations:
5

---

Published in Transactions on Machine Learning Research (10/2022)
R
Update
Update global
conformation
Hy
(1)
Update alom
Update bond
input: RQ-1), n )
u(--)
Figure Network architecture of the 1-th block.
(1) Update bond representations: We first incorporate the coordinate information into the representations by
T (1) h b((11) + MLP (1 + V
(1-1)
hg hij
(1) (t-1) + MLP (1-11) - - I1), v(ij) E E,
R
where 2 ~ N.URR,, After that, the bond representations are updated as follows: V(i.)) E E,
h = My (1-1) + MLP(() (1-1) n nn ,,
(1-1))
(4)
(2) Update atom representations: for any atom V,
h.00 Σ
a, a, co, ") where a, x exp(a T <(W (1 W.concat(h (1-1) [.l)))::
JEN(()
h1 1/t (1-1) MLP h
(111) (0,),, (().)))))
In Eqn. 5 a, Wq. Wo and Wa are the parameters to be learned, concat(-..) is the concatenation two vectors and Ç is the leaky ReLU activation. For atom u, we first use GATv2 (Brody et al. 2021) aggregate the representations from its connected bonds to obtain hú and then update os based on h h [1 and (1-1))
(4) Update global molecule representation:
U0 = (1(11) +MP
Rij
(6)
E
(5) Updote the conformation: ViEV,
R = ML.(
mn)
Σ
R
R (1) + ((1-1)
(7)
V
An important step in Eqn. is that, after making initial prediction R.D, we calculate its center and normalize their coordinates by moving the center to the origin. This normalization ensures that the coordinates generated by each black are in reasonable numeric ranges.
We use R(()) output by the last block in Pdee as the final prediction of the conformation.
3 Discussions with related work
CVGAE (Mansimov et al. 2019) an carly attempt directly generating conformation. Unfortumately, its performance is not good distance-hased methods developed afterwards, like (Shi et al. 2020 and

---

Published in Transactions on Machine Learning Research (10/2022)
(Simm& & Hernández-Lobato 2020). Our method, pursuing the same spirit, makes several finer designs: (1) We design a dedicated training objective that takes the invariance of both roto-translation and permutation on symmetric atoms into consideration. (2) We iteratively refine the output of each block, which is effective for conformation generation (see Figure 7 for ablation study). In comparison, CVGAE only outputs the conformation in the last. layer. (3) Our model integrates several advanced and more effective modules,
including GATv2 (Brody et al., 2021 and GN block (Battaglia et al. 2018). while CVGAE mainly leverages GRU (Bahdanau al.] 2015) and its variants on graphs, which are outperformed by the modules used in our model. GeoDiff (Xu et al. 2022) is concurrent work, which uses diffusion- based method for conformation generation and also directly predicts the coordinates without using intermediate distances. Compared with our method, GeoDiff does not consider the permutation invariance of symmetric atoms and is not as efficient as our method due to its sequential sampling.
ConfGF (Shi et al. 2021) and DGSM (Luo et al., 2021b) are two recent works that can also directly output the coordinates. They both model the gradient of log-density w.r.t interatomic distances, and then generate coordinates by running Langevin dynamics using the gradients. The gradient model is learned via score- matching. ConfGF considers the distances of I-hop, 2-hop and 3-hop neighbors, and DGSM also considers distances of two randomly sampled nodes to model non-bonded distances. In comparison, we completely get rid of modeling distances. More importantly, the permutation invariance of symmetric atoms are not considered in those works. Ganea et al. (2021) propose another method for conformation generation: they first build the local structure (LS) by predicting the coordinates non-terminal atoms, and then refine the LS by the predicted distances and dihedral angles. In comparison, our method does not require refinement based the predicted distances and angles. Furthermore, although Ganea al. (2021) use a permutation invariant loss, they only consider the terminal atoms. According our statistics on subset of 40K molecules from GEOM-Drugs, besides terminal atoms, on average, & molecule has 4.9 non-terminal symmetric atoms, accounting for 10.8% of all atoms. We consider all symmetric atoms.
Our method models the roto-translation and permutation invariance through the loss function, while pre- vious works model the molecules using equivariant networks (Hoogeboom et al. 2022 Xu el al., 2022). More specifically, these works use the diffusion model for conformation generation. Rotational invariance of the conformation distribution is implemented using an invariant latent prior and cquivariant model structure (reverse diffusion process) to map from the latent space to the conformation space. This effec- tively makes an invariant loss in the latent space. [Hoogeboom et al. (2022) also generate the composition variables. In comparison, our method removes the constraints on equivariant noural networks by introducing equivariance/ /invariance into loss function, which is different from previous works that rely on specific network designs to ensure equiv- ariance/invan /ivarinnvv...ee A recent work (Du et al. 2022) points that only using radial direction to represent the geometrie information (like the models used in Hoogeboom et al. (2022) and Xu et al. (2022) abandons high-order tensor information, thus bringing direction degeneration problem and is insufficient to express complex geometric qualities. Therefore, in our approach, we can adopt both equivariant network models and more general (non-cquivariant) networks, enabling the possibility of using more powerful non-oquivariant neural models.
of a molecule, by leveraging continuous representation of ardinal/categorinn
There are some other works on conformation generation, but they target at different problems. G- SchNet (Gebauer et al.] 2019 Hoogeboom et al. (0222) takes some proporties as input (not 2D graph) and output conformation with desired properties. Luo et al. (2021a) focus on generating conformation that can bind with specific binding pocket. We can combine our method with them in the future.
4 Experiments
4.1 Settings
Following prior works (Xu et al.) 2021a Shi et al. 2021). we use the GEOM-QM9 and GEOM- Drugs datasets (Axclrod & Gomez- Bombareli 2021) for conformation generation. We verify our method on both scall-scale setting and large-scale setting. For the scall-scale setting, we use the same datasets provided by Shi et al. (2021) for fair comparison with prior works. The training, validation and test sets of
Datasets:
7

---

Published in Transactions on Machine Learning Research (10/2022)
the two datasets consist of 200K, 2.5K and 22408 (for GEOM-QM9)/14824 (for GEOM-Drrus) molecule- conformation pairs respectively. After that, we work on the large-scale setting by sampling larger datasets from the original GEOM to validate the scalability of our method. We use all data in GEOM-QM9 and 2.2M molecule-conformmation pairs for GEOM-Drugs. The numbers of training, validation and test sets for the larger GEOM-QM9 setting are 1.37M, 165K and 174K, and those for larger GEOM-Drugs are 2M, 100K and 100K.
Model configuration: All of (Pan Pan and Pdee have 6 blocks. The dimension d of the features is 256. Inspired by the feed-forwatt layer in Transformer (Vaswani et al. 2017 MLP also consists of two sub-layers, where the first one maps the input features from dimension 256 to hidden states, followed by Batch Normalization and ReLU activation. Then the hidden states is mapped to 256 again using linear mapping. Considering that our method outputs a conformation y! at each block I, we also require that each A() should try to be similar to the groundtruth R. Therefore, the GRTP is Eqn. (2)
is implemented as
L-1 1
CRTP (f(()) ( R) + AAP Σ
ETTP Feep (R),,,,
(8)
where L is the number of blocks in the decoder, 10 is the output from pen, and À is determined according to validation performance. More details are summarized in Appendix A.2
Evaluation: Assuming in the test set, the molecule x has N₂ conformations. Following Shi et al. 2020 2021), for each molecule x in the test set, we generate 2N, conformations. Let S, and S, denote all generated and groundtruth conformations respectively. We use coverage score (COV) and matching score (MAT) to evaluate the generation quality. To measure the difference between R and R, we use the GetBestRMS in the RDKit package and denote the root-mean-square deviation as R.MD R). The recall-based coverage and matching scores are defined as follows:
S. [{ [ s, |RMSD(R, Rt) < o, E S, $/
CO,
(9)
MAT (,,
min RMSD(R, it).
RES,
RES.
A good method should have a high COV score and low MAT score. Following (Shi et al. 2021 Xu et al, 2022), the 8's are set as 0.5 and 1.25 for GEOM-QM9 and GEOM-Drugs, respectively. The COV-5 curves are left in Figure g of the appendix. There are also precision-based COV and MAT scores by switching the
s, and Sy in Equ. 9 We leave the precision- based results in Appendix B.1
Baselines: (1) RDKit, which is widely used toolkit and generates the conformation based the force fields; (2) CVGAE (Mansimov et al. 2019). which is an early allempl to generate raw coordinates; (3)
GraphDG (Simm & Hernández-Loobtoo 2020), a representative distance-based method with VAE; (4) CGCF (Xu et al., 2021a), which is another distance-based method leveraging continuous normalizing flow; (5) ConfVAE (Xu et al.) 2021b), an end-to-end framework for molecular conformation generation, which still uses the pairwise distances among atoms as intermediate variables; (6) ConfGF (Shi et al., 2021 and
DGSM (Luo al.] (20216), which uses score matching to generate the gradients w.r.t distances and then re-
cover the conformation; (7) GeoDiff (Xu et al. 2022), ; which uses diffusson model to generate conformations; (8) GeoMol (Ganea et al. 2021), which predicts local atomic 3D structures and torsion angles. Considering Ganca et al. (2021) use different data split from previous work, we reproduce their method following the
more commonly used data split. (Xu et al. 2021a Shi et al. 2021
4.2 Results
The recall-based results are shown in Table ! For small-scale datasets, we independently train our models with five different random seeds, and report the mean and standard derivations. We have the following observations:
(1) On the four settings in Table D our method achieves state-of-the-ar results on all of them. The median COV(%) being 100% means that for more than half of the groundtruth conformations, there exist generated
8

---

Published in Transactions on Machine Learning Research (10/2022)
Table 1: Recall-based coverage and matching scores. Bold fonts indicate the best results. The standard derivations of our method (five independent runs) on small-scale datasets are reported.
Small-scale QM9
Small-scale Drugs
(OV(%))
MAT
(A)4
COV(%)t
MAT
(A)4
Methods
Mean
Median
Mean
Median
Mean
Median
Mean
Median
RDKit
83.26
90.78
0.3447
0.2935
60.91
65.70
1.2026
1.1252
CVGAE
0.09
0.00
1.6713
1.6088
0.00
0.00
3.0702
2.9937
GraphDG
73.33
84.21
0.4245
0.3973
8.27
0.00
1.9722
1.9845
CGCF
78.05
82.48
0.4219
0.3900
53.96
57.06
1.2487
1.2247
ConIVAE
80.42
85.31
0.4066
0.3891
53.14
53.98
1.2392
1.2447
GeoMol
71.26
72.00
0.3731
0.3731
67.16
71.71
1.0875
1.0586
ContGF
88.49
94.13
0.2673
0.2685
62.15
70.93
1.1629
1.1596
DGSM
91.49
95.92
0.2139
0.2137
78.73
94.39
1.0154
0.9980
GeoDiff
90.54
94.61
0.2090
0.1988
89.13
97.88
0.8629
0.8529
DMCG
96.23
99.26
0.2083
0.2014
96.52
100.00
0.7220
0.7161
Std
+0.38
+0.37
+0.0052
=0.0040
#0.14
+0.00
#0.0027
#0.00
Large-scale Q,99
Large-scale Drugs
COV(%)1
MAT (A)4
COV(%)+
MAT (A)4
Methods
Mean
Median
Mean
Median
Mean
Median
Mean
Median
RDKit
81.61
85.71
0.2643
0.2472
69.42
77.45
1.0880
1.0333
CVGAE
0.00
0.00
1.4687
1.3758
0.00
0.00
2.6501
2.5969
GraphDG
13.48
5.71
0.9511
0.9180
1.95
0.00
2.6133
2.6132
CGCF
81.48
86.95
0.3598
0.3684
37.47
62.09
1.2205
1.2003
ConfVAE
80.18
85.87
0.3684
0.3776
57.63
63.75
1.2125
1.1986
ConfGF
89.21
95.12
0.2809
0.2837
70.92
85.71
1.0940
1.0917
GeoMol 91.05
95.55
0.2970
0.2993
69.74
83.56
1.1110
1.0864
DMCG
98.34
100.00
0.1486
0.1340
96.22
100.00
0.6967
0.6552
conformations that are close to them within predefined threshold. These results show the effectiveness and scalability of our method.
(2) For the molecules in GEOM-QM9 and GEOM-Drugs, our method achioves more improvement on molecules with more heavy atoms. Take the small-scale results Table [ as an example. On average, DEOM-QM9 and GEOM-Drugs have 8.8 and 24.9 heavy atoms respectively. In terms of MAT mean values, GEOM-QM9, our method improves ConfGF and GcoDiff by 22.7% and 1.2%, while GEOM-Drugs. the improvements are 37.9% and 16.3%. The results demonstrate the effectiveness of our method on large molecules.
More analysis is in Appendix B.5
(3) Our method much more sample-effccient than methods based on Langevin dynamics like ConfGF, since we can generate IID samples free of the auto-corrclation a. Markov chain. ConfGF requires 5000 sequential forward steps, while we only need sample once from N/O,, I) and forward through the model. For fair comparison, following the official implementation of ConfGF, GEOM-QM9 and GEOM-Drugs into 200 batches. ConfGF requires 8511.60 and 11830.42 seconds to decode QM9 and Drugs test sets, while our method only requires 32.68 and 54.89 seconds respectively. Our method speeds up the decoding more than 200 times. Our method is also much more efficient than the recent GeoMol algorithm, which takes 99.34s and 668.95s to decode the above two datasets.
split the test sets of small-scale
(4) As shown in Table 1 the standard derivations of our method are significantly smaller than the gain compared to the previously best results. This shows the effectiveness and robustness of method. Tn

---

Published in Transactions on Machine Learning Research (10/2022)
addition, considering that our method takes random conformation as imput, to test the confidence interval, run decoding with 10 different initial conformations. The mean COV and MAT scores on small-scale 0.12 and 0.2079 + 0.0010, and those two numbers on small-scale GEOM-Drugs are
GEOM-QM9 are 96.38 + 0.19 and 0.7239 = 0.0025. Our method is not sensitive to the choice of imitial conformations.
The number of rotatable bonds an important metric of how flexible a molecule We report coverage score w.r.t. the number of rotatable bonds in Figure 4 based on small-scale GEOM-Drugs. More rotatable bonds indicate harder generation. Our method outperforms previous baselines.
100
80
Recall COV(%)
60
40
20
ConfGi
ConfVAE
GeoMol
RdKit
CGCF
2
4
6
8
Figure Coverage scores w.r.t. number of rotatable bonds.
Moleculan Graph
Refermec
DMCG
GraphDG
ConGGF
GenVall
Figure 5: Visualization of different conformations.
10

---

Published in Transactions on Machine Learning Research (10/2022)
In Figure 5 we visualize the conformation of different methods. We randomly select three molecules from the small-scale CEOM-drug dataset, generate several conformations, and visualize the best-aligned ones with the groundtruth. We can see that our method can generate high-quality conformations than previous methods, which are the most similar to the groundtruth.
Computation cost analysis: We use PyTorch profiler to analyze the training time of the following components: (I) model forward time, which denotes the time of calculating the hidden representations from the imput layer to output layer; (2) transformation time, which denotes the time of calculating the optimal roto-transia -rontraanlation operation p²; (3) permutation time, which denotes the time of enumerating all possible permutations in S and find the optimal one a" E S; note that we can use torch. no goograd to reduce time and memory; (4) loss forward time, which is the total of calculating the loss after obtaining p* and o"; (5) loss backward time, which denotes the time of gradient backpropagation.
The time is summarized in Table 2 We can see that model forward and loss forward/backwer takes about 71.4% of the total computation time. The transformation and permutation takes 20.4% and 8.2% of the total time. Note that there are 7 transformation operations in the experiments (see (q For the full training pipeline where data loading, model forwarding, loss forwarding, gradient backpropagation, metric calculation and CPU/GPU communications are all considered, DMCG takes 20% more time than that without roto-translation and permutation.
Table 2: Computation time statistics of each part in 100 iterations.
Model forward
Transformation
Permutation
Loss forward
Loss backward
5.515 (52.8%) 2.136 (20.4%) 0.858 (8.2%) 0.052 (0.5%) 1.886 (18.1%)
We use graph isomorphism algorithms to find all S. Although the general graph isomorphism problem is NP-hard, the size of drug-like molecules is largely limited, otherwise the molecule's druggability is limited (one can refer Lipinski's rule of five). Therefore, our method does not need scalability to large scale. In our experiments, takes 4.9 seconds to process 10k molecules in GEOM-QM9, and 6.6 seconds to process 10k molecules in GEOM-Drugs. This is negligible compared to the training time, and only need to process the data for one time in data preparing stage.
4.3 Molecular docking
Molecular docking (Roy et al. 2015) a widely used technique in drug discovery, which aims to find the optimal binding conformation of a drug (i.e., the small molecule) in the pocket of a given target protein and the corresponding binding affinity. In most cases, the molecular docking algorithms treat proteins as rigid bodies and take one conformation of the small molecules as the initial structure inputs. The algorithms then search for the optimal conformation in the conformation space of the small molecules guided by the scoring function. However, due to the complexity of the conformation space, it is difficult for the algorithm to converge to global minimum. Therefore, the choice of the initial structure often lcads to different binding conformations and needs to be (aken seriously.
Previously, RDKit was often used to generate initial conformations of small molecules, which usually got reasonable but not optimal results after docking. To verify the effectiveness of our method, we compared the docked poses which take initial conformations generated by our method (DMCG), ConSGF, GeoMol, GeoDiff and RDKit as the initial conformations for docking respectively.
We use Smina Koes et al. 2013 for molecular docking and make evaluation on PDBbind refined set Liu et al. 2017) which is a comprehensive collection of experimentally measured binding affinity for all biomolec- ular complexes deposited in the Protein Data Bank We randomly select 100 protein-ligand pairs for evaluation. Appendix A.3 shows detailed optimization hyper-parameters.
httpr///wwraaae//oo////////steee httpss//www.rcsbborg/
11

---

Published in Transactions on Machine Learning Research (10/2022)
RDKit(-8.85)
RDKit(4.24)
GeaMol(-9.55)
GeoMal(3.955
0.5
0.4
ConfGF(-9.87)
ConIGF(3.91)
100114f(-10.14)
GeoDiff(3.85)
DMCG(-10.78)
0.4
DMCG(3.73)
Density
0.3
Density
0.3
0.2
0.2
0.1
0.0
-18
-14
-10
-6
-2
0.0
6
2
6
Docking score (kcal/mol).
(b) RMSD (A).
Figure 6: Histograms of docking scores and RMSD scores.
Two metrics were used lo evaluate the results of docking- One the docking score (roughly, the estimation of binding affinity), which measures how well molecule fits the binding site. smaller value indicates better binding affinity. The other is the root-mean-square deviation (RMSD, the smaller, the better) compared to the crystal complex structure. As shown in Figure 6(a) the distributions for the three methods have the similar shape but our method is much more left-shifted than the others. This shows that for the same small molecule, our method tends to help docking to find conformations with higher binding affinities. Furthermore, docking tends to find lower RMSD binding conformations using the conformation generated by our method as the initial con formatioon, suggesting that our method can help dookkig to find binding conformations that are closer the native crystal structures (Figure 6(b]]. We also summarize the mean values of the docking scores and RMSD of different algorithms in the legends of Figure 6 All these results show that our method provides more proper initial conformations for molecular docking and thus facilitates the real application in computer-aided drug discovery.
4.4 Property prediction
In addition to conformation generation task, also conduct experiments property prediction task, which is to predict molecular property based on ensemble of generated conformation (Axelrod & Gomez- Bombarelli 202I). We first randomly choose 30 molecules from GEOM-QM9 test sets, and then sample 50 conformations for each molccule using RDKit, ConfGF and our method. We use the quantum chemical calculation package Psi4 (Smith et al., 2020) to calculate the energy, HOMO and LUMO for each generated conformation and groundtruth conformation. Next, we calculate the ensemble properties of average energy E, lowest energy Emin, average HOMO-LUMO gap Ae, minimum gap Aemin and maximum gap Aemax based the conformational properties of each moleculd]] We use mean absolute error to measure the property differences bet ween the generated conformations and groundtruth conformations.
The results are shown in Table 3 Our method significantly outperforms GraphDG, ConfGF and the recent GeoMol, which shows the effectiveness of our method. We can observe that RDKit achieves the best results on Aemin and we will combine our method with RDKit in the future.
4.5 Ablation study
We conduct ablation study on the small-scale GEOM-Drugs dataset. The results are shown in Table
FFrm physics perspective, using the Boltemann-wwightted average of the energies of the molccules better choice, but the distribution missing from the dataset. Following (Sim Hernández- Lobato 2020 Shi use the average number here instead of the weighted version.
2021 Luo al. 2021b)
12

---

Published in Transactions on Machine Learning Research (10/2022)
Table 3: Mean absolute error of predicted ensemble properties. (Unit: eV).
Methods
E
Emin
Åс
Acmin
Amaax
RDKit 0.8875
0.6530
0.3484
0.5570
0.2399
GraphDG
45.1088
9.2868
3.8970
6.6997
1.7724
ConfGF
2.8349
0.2012
0.6903
4.9221
0.1820
GeoMol
4.5700
0.5096
0.5616
3.5083
0.2650
DMCG
0.4324
0.1364
0.2057
1.3229
0.1509
Table 4: Ablation study on small-scale GEOM-Drugs.
COv(%)t
MAT (A)
Methods
Mean
Median
Mean
Median
DMCG
96.52
100.00
0.7220
0.7161
No &p
77.78
86.09
1.0657
1.0563
attention
94.99
100.00
0.7611
0.7581
No normalization
92.77
98. 98.68
0.8002
0.7977
(1) We remove the permutation invariant loss and the roto-translation invariant loss only, i.c., the EITP
in Eqn. (2) is replaced with tRT defined in Section. 2.1 The results are denoted as "No 6p in Table
(2) We replace attentive node aggregation by a simple MLP nelwork. That is, Equ. is replaced by
1 = h (1-1) + MLP(N ua-1)
Σ h 1
IN(1
JEN(S)
The results are denoted as "No attention" in Table
(3) We remove the normalization step in Eqn. i.c., the ml) is not used. Denote the results "No normalization".
We can see that: (1) The permutation invariant loss is extremely important, without which the mean COV drops 18.91 while MAT increases 0.3434. We also visualize several cases in Appendix B.2 to compare the results with or without lp. (2) Without attentively aggregating the alom features, the mean COV drops 1.70 points and MAT score increases 0.0345 points. (3) Without the conformation normalization, the performance is also hurt. These results demonstrate the importance of the components in our method.
Finally, compute the COV and MAT scores of Á(0)) against the groundrutth, which is the output confor- mation of the [-th block in the decoder. p((0) is the output of 42D. The results are shown in Figure 7 We can see that iteratively refining the conformations can improve the performances, which shows the effectiveness of our design. This phenomenon consistency with the discovery in machine translation (Xia et al. 2017). image synthesis (Chen & Koltun 2017) and protein structure prediction (Jumper et al., 2021
We leave the discussion about additional constraints on loss functions, the comparison of model sizes and more discussions in Appendix B
5 Conclusions and future work
In this work, we propose purpose, we design dedicated loss function, which is invariant to roto-translation and permutation on symmetric atoms. We also design a new model with many advanced modules (i.e., GATv2, GN block) that can iteratively refine the conformations. Experimental results on both small-scale and large-scale GEOM- QM9 and GEOM-Drugs demonstrate the effectiveness our method.
method, that directly generates the coordinates of conformations. For this
For future work, first, we will incorporate chemical rules into deep learning models to improve generation quality. Second, current methods are mainly non-autoregressive, where all coordinates generated simul-
13

---

Published in Transactions on Machine Learning Research (10/2022)
0.98
0.80
0.26
(MAT)(A)
COV(%)
0.94
0.76
0.23
GEOM Drugs
GEOM QM9
0.90
0.72
0.20
4
0
2
4
6
Block
Block
(a) MAT score.
(b) COV score.
Figure The MAT and COV scores of R(0) output by different blocks.
taneously. We will study the autoregressive setting so as to further improve the accuracy. Third, Villar et al. (202I) point that equivariance/ suuivariaanee/ivarianeee /invariance can be universally approximated through polynomial functions. This is a good direction to explore in molecular conformation generation. Fourth, when the number of per- mutation invariant mappings molecule is extremely large, enumerating all of them not the best choice due to the exponentially increased computation cost. We will improve method along this direction. Fifth, we will deeply collaborate with chemists and biologists more case studies.
References
Ravichandra Addanki, Peter W. Battaglia, David Budden, Andreca Deac, Jonathan Godwin, Thomas Keek, Wai Lok Sibon Li, Alvaro Gannaez, Jacklynn Stott, Shantanu Thakoor, and Petar Velickovic. Large-scale graph representation learinn with very deep gnns and self-supervision. CoRR, abs/2107.094222, 2021. URL
09422
Simon Axelrod and Rafael Gomez-Bombarelll, Geom: Energy-annotated molecular conformations for prop- erty prediction and molecular generation, 2021.
Dzmitry Bahdanan, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.
Kyle A. Baseden and Jesse W. Tye. Introduction to density functional theory: Calculations by hand on the helium atom. Journal of Chemical Education, 91(12):2116 2123, 2014. doi: 10.1021/ed5004788.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Gonzallz, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adamn Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint orXiv:1806.011 01261, 2018.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXio preprint arXiv:2105.1444 2021.
Qifeng Chen and Vladlen Koltun. Photographic image syuthesis with cascaded refinement networks. 2017 IEEE International Conference on Computer Vision (ICCV) pp. 1520-1529, 1529, 2017.
Ann E. Cleves and Ajay N. Jain. Forcegen 3d structure and conformer generation: from small lead-like molecules to macrocyelic drugs. Journal of Computer-Aidee Molecular Design, 31(5):419-45) 439, May 2017. ISSN 1573-4951. doi:
Ivan Dokmanic, Reza Parhizkar, Juri Ranieri, and Martin Vetterli. Euclidean distance matrices: essential theory, algorithms, and applications. IEFE Signal Processing Magazine, 32(6):12-33 2015.
14

---

Published in Transactions on Machine Learning Research (10/2022)
Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tic- Yan Liu. SE(3) equivariant graph neural networks with complete local frames. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5583- 5608. PMLR, 17-23 Jul 2022. URL https: http://ppoeeeens....o mlr. http://nnn.......eess///////////
Octavian-Eugen Ganea, Lagnajit Pattanaik, Connor W. Coley, Regina Barzilay, Klavs Jensen, William Green, and Tommi S. Jaakkola. Geomol: Torsional geometric generation of molecular 3d conformer ensembles. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (ods.), Advances Neural Information Processing Systems, 2021. URL hntxps::/openrevieww httppppawweiieeeeeeoooorrrreeeee net/forumiid hnngtuNj] Jingstt
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation 3d point sets for the targeted discovery of molccules. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https://proceeddi www.eeddiinnnn..nissse.. www.ies..u.ppppppeeeee httc//appp/201///11222. .ddpp
Thomas A Halgren. Merck molecular force field. i. basis, form, scope, parameterization, and performance of mmf994. Journal of computational chemistry, 17(5-6))490-519, 1996.
W. R. Hamilton. On a new species of imaginary quantities, connected with the theory of quaternions.
Proceedings of the Royal Irish Academy (1836-1869), 2:424 434, 1840. ISSN 03027597. URL http:
7/www. jstor.stor.o org/sstalle2052017
Moritz Hoffmann and Frank Noé. Generating valid euclidean distance matrices. arXiv:1910.091 2019.
arxiv preprint
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3D. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (cds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8867 8887. PMLR, 17 23 Jul 2022. URL https://proceedingg mlr,
html
J. E. Hopcroft and J. K. Wong. Linear time algorithm for isomorphism of planar graphs (preliminary report). STOC '74, pp. 172-184, New York, NY, USA, 1974. Association for Computing Machinery. ISBN 9781450374231. doi: 10.1145/ 10.1145/800119, 800011980089000 URL https://dot: .ar.l.rrg/// 11111.11140 1115//1010999800000000
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-Isc: A large-scale challenge for machine lcarning on graphs. arXiv preprint arXiv:2103.09450 2021.
John Jumper, Richard Fvans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596(7773):553 589, Aug 2021. ISSN 1476-4687. doi: 10.1038/ 541556-001-03819-22 URL
aaaa Y. Kanal, John A. Keith, and Geoffrey R. Hutchison. A sobering assessment of small-molecule force field methods for low energy conformer predictions. International Journal of Quantum Chemistry, 118(5): c25512, 2018. doi: https://dde http://ooooor/./100///// qua.i...550/////2222... 25512. URL https:///onlmeeibrraa www.....ww................ comlyy.com//il. 0bb//10.102.. qua. 25512
Charles FF Karney. Quaternions in molecular modeling. Journal of Molecular Graphics and Modelling, 25 (5):595-600) 2007.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes In Proceedings of the International Conference on Learning Representations (ICLR 2014), Banff, Canada, 2014. ICLR Committee.
15

---

Published in Transactions on Machine Learning Research (10/2022)
D. R. Koes, M. P. Baumgartner, and C. J. Camacho. Lessons learned in empirical scoring with smina from the csar 2011 benchmarking exercise. J Chem Inf Model, 53(8):1893-900) 2013. ISSN 1549-960X (Electronic) 1549-9596 (Linking). doi: 10.1021/ci300620
Maria Kontoyianni, Docking and Virtual Screening in Drug Discovery, pp. 255-266. Springer New York, New York, NY, 2017. ISBN
doi:
18. URL htt.tt/s:://di..
Zhihai Liu, Minyi Su, Li Han, Jie Liu, Qifan Yang, Yan Li, and Renxiao Wang. Forging the basis for developing protein-ligand interaction scoring functions. Accounts of chemical reseorch, 50(2):302-309, 2017.
llya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. CoRR, abs/ 160880983, 2016. URL httttp://arxiv.rrg// a000a//.010000033303333 03983
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019. URL httppp//pppaareeeee.eeeeerrrrrrp nett//orrmmmdddB/eeR.CC"""
Shitong Luo, Jiaqi Guan, Jianzhu Ma, and Jian Peng. A 3d generative model for structure-based drug design. In Thirty-Fitph Fifth Conference on Neural Information Processing Systems, 2021a.
Shitong Luo, Chence Shi, Minkai Xu, and Jian Tang. Predicting molecular conformation via dynamic graph score matching. Advances in Neural Information Processing Systems, 34, 2021b.
Elman Mansimov, Omar Mahmood, Seokho Kang, and Kyunghyun Cho. Molecular geometry prediction using& a deep generative graph neural network. Scientific Reports, 91): Dec 2019. ISSN 2045-2322. doi: (100://41590-09--5335
URL htttps://doi.oor///0// httpa//a........./////111---0-10
Rocco Meli and Philip C. Biggin. spyrmsd: symmetry-correeted rmsd calculations in python. Journal of Cheminformatics, 12(1))49, Aug 2020. ISSN 1758-2946. doi: 10.1188/413322-0000004---4- URL https:
Robert G. Parr. Density functional theory of atoms and molecules. In Kenichi Fukui and Bernard Pullman (eds.), Horizons of Quantum Chemistry, pp. 5-15, Dordrecht, 1980. Springer Netherlands. ISBN 978-94- 009-9027-2.
A. K. Rappe, C. J. Casewit, K. S. Colwell, W. A. Goddard, and W. M. Skiff. Uff, full periodic table force field for molecular mechanics and molecular dynamics simulations. Journal of the American Chemical Society, 114(25):10024 10035, 1992. doi: 10.1021/j000551a440.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, pp. 1278-1286, 2014.
Kunal Roy, Supratik Kar, and Rudra Narayan Das, Chapter 10 - other related techniques. In Ku- nal Roy, Supratik Kar, and Rudra Narayan Das (ods.), Understanding the Basics of QSAR for Ap- plications in Pharmaceuticnl Sciences and Risk Assessment, pp. 357 425. Academic Press, Boston, 2015. ISBN 978-0-12-801505- doi: https:/ https://doi.orrg /to.i/o///oo////////oo 1016 /11///9978090 100-000--2---00 12-80150555 00150066.0001000-00 5.00011-7. URL https: 7/ www.ssciencedreec... htt.e//ew..................oaaaa /tt.///////.11100000000000000000
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a flow- based autoregressive model for molecular graph generation. In International Conference on Learning Representations, 2020. URL
Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation gencration. In International Conference on Machine Learning, 2021.
Gregor N. C. Simm and José Miguel Hernández-Lobate A generative model for molecular distance geometry. In Hal Daumé III and Aarti Singh (ods.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8949-8958. PMLR, 13-18 Jul 2020. URL hTRLhhtt:::/// http::/proccedinggg mlr.
16

---

Published in Transactions on Machine Learning Research (10/2022)
Daniel GA Smith, Lori A Burns, Andrew C Simmonett, Robert M Parrish, Matthew C Schieber, Raimondas Galvelis, Peter Kraus, Holger Kruse, Roberto Di Remigio, Asem Alenaizan, et al. Psi4 1.4: Open-source software for high-throughput quantum chemistry. The Journal of chemical physics, 152(18):184108, 2020.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep condi- tional generative models. In Advances in neural information processing systems, volume 28, pp. 3483 3491, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL 3ibee23s77ees1100001c4444eeee Paper, pdf
Soledad Villar, David W Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. Scalars are universal: Equivariant machine learning, structured like classical physics. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advonces in Neural Information Processing Systems, 2021. URL https:
Robin Winter, Frank Noé, and Ajnrk-Arné Clevert. Anto-encoding molecular conformations, arXiv preprint arXiv:2101.01618 2021.
Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tic- Liu. Deliberation networks: Sequence generation beyond one-pass decoding. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL. https: ctttps://proceednngg 46336499e22106666997777788888888 Paperrr pdf
Minkai Xu, Shitong Luo, Yoshua Bengio, Jian Peng, and Jian Tang. Learning neural generative dynamics for molecular conformation generation. In International Conference on Learning Representations, 2021a. URL
Minkai Xu, Wujie Wang, Shitong Luo, Chence Shi, Yoshua Bengio, Rafael Gomez-Bombarell,, and Jian Tang. An end-to-end framework for molecular conformation generation via bilevel programming, arXi preprint arXiv:2105.00 07246,, 2021b.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Repre- sentations, 2022. URL httt:://wppaaaeeeee..eeee//rerrs
A Procedure descriptions
A.1 Details of other model components
The model architoctures of (2n and Yan are similar Pdecs with the following differences.
Comparing $2D with iaec, the differences are the initial conformation k()) and initial features (i.e., the H H and (10)). ?2n takes a random conformation sampled from uniform distribution in [-1,1] as input. The initial atom and edge features are the embeddings of the atoms and edges respectively. $2D will also output a prediction of the conformation. Note that the random variable a sampled from Ganssian N(HRG, ER,G) is not used in P2D-
Comparing PDD with Pdec, the differences are the initial conformation ((0), initial features (i.c., the H H(0) and (700)) too. PaD takes the groundtruth conformation as input. The initial atom and edge features are the embeddings of the atoms and edges respectively. Another difference is that the fourth step of Paec: i.e, (q is not used.
17

---

Published in Transactions on Machine Learning Research (10/2022)
A.2 More details about training
We use AdamW optimizer (Loshchilov & Hutter 2019 with initial learning rale yo = 2 x 10-1 and weight decay 0.01. In the first 4000 iterations, the learning rate is linearly increased from 10 to 2 x 10 After that, we use cosine learning rate scheduler [Loshchilov & Huttor 2016), where the learning rate at the t-th iteration is mo + C f where is the half of the period (i.e., the iteration numbers of 10 epochs in our setting). Similarly, we also use the cosine scheduler to dynamically set the / at range [0.0001, {max] The batch size is fixed as 128. All models are trained for 100 epochs. For the two small-scale settings, the experiments are conducted for experiments. The À in Eqn. S) for large-scale QM9 is 0.1, and for the remaining set settings, À is set 0.2. The detailed hyper-parameters are described in Table 5 We grid search the best hyper-parameter with these papee-pprameters and the last eyper-parameters are selected according to validation performance, i.e., the setting corresponding to the best coverage score (COV) and matching score (MAT) on the hold-out validation set.
single V100 GPU. For the two large-scale settings,
use two V100 GPUs
Table 5: hepper-arramters for our experiments.
Small-Scale
Large-Scale
Layer number
Dropout
{0.1, 0.2)
{0.1, 0.2)
Learning rate
f1e-4, 2c-4, 5o-4)
{1c-4, 2c-4, 5c-4)
Batch size
128
128
Epoch
100
100
B Min
0.0001
0.001
B Max
{0.001, 0.002, 0.004, 0.008, 0.013
{0.005, 0.01, 0.02, 0.04,0.05)
Latent size
256
256
Hidden dimension
1024
1024
GPU number
1x NVIDIA V100
2x NVIDIA V100
A.3 More details about molecular docking
For RDKit, we generated one initial conformation as input and set mum__moee to 50 when performing docking For our method, ConfGF, GcoDiff and GcoMol, since the generated conformations are independent and diverse, we randomly selected five of them, performed five independent molecular docking calculations and set num hom__modes to 10 to ensure all three methods generate equal number of conformations. Fventually, each method got about 50 binding conformations. The conformation corresponding to the lowest binding affinity was selected as the final docked pose.
B More experimental results
B.1 Precision-basee results
The precision- coverage and matching scores are defined as follows:
&.) [Sg]
{ R
E S, RMSD(RR R) < 8, < e e }
(10)
MAT-P MA, (S,, S,)
Σ
min RMSD(R, Rt.. RES.
S
RES
The results are in Table 6 The results of GraphDG, CGCF, ConfVAE, ConfGF and GeoDiff are from et al. 2022 Our method is still the best one.
Xu
When using different random seeds, the conformations output by RDKit diverse enough. Therefore, we only chonse one here.
18

---

Published in Transactions on Machine Learning Research (10/2022)
Table 6: Precision-basee coverage and matching scores. Bold fonts indicate the best results.
Small-scale QM9
Small-scale Drugs
COV-P(%)T
MAT-P(A)L
COV-P(%)
MAT-P(A).
Methods
Mean
Median
Mean
Median
Mean
Median
Mean
Median
GraphDG
43.90
35.33
0.5809
0.5823
2.08
0.00
2.4340
2.4100
CGCF
36.49
33.57
0.6615
0.6427
21.68
13.72
1.8571
1.8066
ConfVAE
38.02
34.67
0.6215
0.6091
22.96
14.05
1.8287
1.8159
ConfGF
49.02
46.69
0.5111
0.4979
23.15
15.73
1.7304
1.7106
GeoDiff
52.79
50.29
0.4448
0.4267
61.47
64.55
1.1712
1.1232
GeoMol
84.98
89.90
0.3292
0.3269
75.54
94.13
1.0028
0.9082
DMCG
87.26
91.00
0.2872
0.2926
81.05
95.51
0.9210
0.8785
Dataset Methods
Large-scale QM9
Large-scale Drugs
Mean
Median
Mean
Median
Mean
Median
Mean
Median
COV-P(%)t
MAT-P(A)U
COV-P(%)
MAT-P(A)4
ConIGF
46.23
44.87
0.5171
0.5133
28.23
20.71
1.6317
1.6155
GeoMol
78.28
81.03
0.3790
0.3861
41.46
36.79
1.5120
1.5107
DMCG
90.86
95.36
0.2305
0.2258
74.57
81.80
0.9940
0.9454
B.2 Combination with distance-based and angle-based loss functions
One may be curious about whether using distance-based loss and angle-based further improve the performance, since the latter two are equivariant to the transformation of coordinates. For ease of reference, let R₂ denote the groundtruth coordinate of atom and R; denote the predicted coordinate of atom ua Recall in Section
we use to denote the collection of all bonds. We define Fiy as {((,),))((( E F, (1, e
E,k #
Inspired by (inter et al.
2021 and (Ganea al. 2021 we use the following two functions:
1
Langle EE2
Σ
cosino(RR R;, R& Ri) cosino(R, R, Rr RR)l
(11)
(
(distance( R2, R2) - distance R -2:))
hhnd
(12)
E
(
where cosinc(a, b) constraints to bond length and bond angles. Please note that with the above two auxiliary loss functions, our method still generates coordinates directly and does not need to generate intermediate distances and angles.
T and distanco(a, b) ||a bll, a and are two vectors. That is, we apply additional
We verify the following three loss functions:
L1 = Be CRT R(ung + ER,ce, G)) 3D ER,G)) I));
(13)
L2 En R. R(nno En,ce, G)) + 3D ( En, I)) A (fangglc (hond),
(14)
L3 = E. (R, R(uR,G + ER.GS G)) + (K, WR. + Ay + booa),
(15)
where \ 0.1. Note in Eqn. 13 and Eqn. 14
we the roto-translation loss only without considering
permutation invariant loss on symmetric atoms. We conduct experiments on GEOM-Drugs (small-scale setting). The results are reported in Table 7
We have the following observations:
(1) Comparing L1 with our method, we can see that using permutation invariant loss on symmetric atoms are important, without which the results significantly drop. (2) Comparing La with method, see that
19

---

Published in Transactions on Machine Learning Research (10/2022)
Table 7: Results of combining with constraints on bond lengths and bond angles.
COV(%)
MAT (A)4
Methods
Mean
Median
Mean
Median
DMCG
96.52
100.00
0.7220
0.7161
L1
77.78
86.09
1.0657
1.0563
L2
92.45
98.70
0.8983
0.9016
L3
96.01
100.00
0.7235
0.7199
when we do not use the permutation invariant loss, using more constraints on bond lengths and bond angles can help improve the performances. (3) When using both permutation invariant loss and coto-translation invariant loss, using bond and angle will not bring more significant improvement. These results demonstrate that for molecular conformation generation, it is important consider the permutation of symmetric atoms.
To illustrate the impact of the permatation invariant loss, show two examples in Figure g For these two examples, there exists a rotatable ring at the end of molecule, where the ring is symmetric to the bond connecting itself to the rest of the molecule. Without the permutation invariant loss (see the row No <p), our method fails to generate the coordinates of such rings, but simply puts them in a line. This is because the model trapped into local optimal. By using the permutation invariant loss, we can successfully recover the conformations of those rings (see the row "DMCG"). This shows the importance of using the permutation invariant loss lp as we proposed.
Molecular
Graph
Reference
No 4p
DMCG
Figure 8: The illustration of the impact of the permutation invariant loss. "No Ep means without the permutation invariant loss.
B.3 Gradient back- propaaropagaiion through roto-translational operation?
As introduced Section 2.1 the optimal roto-translation operation p" can obtained by calculating the eigenvalues and eigen vectors matrix. This is implemented by using the torch. linalllnaeeee eig. However, the official document lists warning of this function: "Gradients computed using the eigenvectors tensor will only be finite when A has distinct eigenvalues. Furthermore, if the distance between any two eigenvalues is close to zero, the gradient will be numerically unstable, it depends the eigenvalues A through the computation of mingys A;- (the words are from the official document). Therefore, we disable the gradients through p" for stability.
1
20

---

Published in Transactions on Machine Learning Research (10/2022)
We compare the performance between enabling and disabling the gradients. The results are in Table g Overall speaking, enabling the gradients slightly hurts the performance (especially the MAT for GFOM- Drugs) and increases computation time. Therefore, we recommend disabling the gradients through p².
Table 8: Results with/withool gradient paok-propagation through p²
Small-scale QM9
Small-scale Drugs
COV-P(%)f
MAT-P(()L
COV-P(%)
MAT-P (AT-P((A)
Methods
Mean
Median
Mean
Median
Mean
Median
Mean
Median
with gradient
96.14
99.43
0.2090
0.2062
96.08
100.00
0.7345
0.7247
without gradient
96.23
99.26
0.2083
0.2014
96.52
100.00
0.7220
0.7161
B.4 Study of model parameters
In this section, compare the performances of our method and ConfGF. The ConfGF model has 0.81M
parameters. We reduce the network parameter of our method to 0.98M. The results are shown in Table
Table 9: Comparison of our method and ConfGF with different model sizes
Dataset
GEOM-QM9
GEOM-Drugs
COV(%)t
MAT (A)4
COv(%))
MAT (A)J
Methods
Mean
Median
Mean
Median
Mean
Median
Mean
Median
ConfGF (0.81M)
88.49
94.13
0.2673
0.2685
62.15
70.93
1.1629.
1.1596
DMCG (0.98M)
94.28
98.20
0.2399
0.2361
89.40
97.06
0.8653
0.8670
DMCG (normal)
96.23
99.26
0.2083
0.2014
96.52
100.00
0.7220
0.7161
By reducing the network parameters of our method, the performance also drops, but still significantly better than ConIGF.
We also study the results of our method w.r.t. the representation dimension d (please kindly refer to the Section 2.2) and the dimension of the MLP layer (denoted as dMLP). Results are reported in Table 10 We that our model benefits from more parameters.
Table 10: Results small-scale FE0M-Drrug with different hidden dimensions.
COV(%)*
MAT (A)
Mean
Median
Mean
Median
(128, 512)
95.54
100.00
0.7641
0.7584
(128, 1024)
95.84
100.00
0.7514
0.7432
(256, 512)
95.88
100.00
0.7378
0.7387
(256, 1024)
96.52
100.00
0.7220
0.7161
B.5 More discussions on the conformation with more heavy atoms
In Table I observe that our method works better than distance-basede methods (include modeling the distances directly, or the gradients of distances) molecules with more heavy atoms. Our conjecture is that for these tistancc-basel works, they usually extend the molecular graph with 1,2,3-order neighbors, which is sulficient to determine the 3D structure in principle. For GEOM-QM9 dataset, considering the number of atoms is less than 10, this extended graph nearly a complete graph and can provide enough signals reconstruct the 3D structure. Therefore, these distancc-based performances are good on GEOM- QM9 dataset. For GEOM-Drugs dataset, the numbers of atoms are much more than those in GEOM- QM9. Although in theory, the distances in third-order extended graph can reconstruet the 3D structure,
21

---

Published in Transactions on Machine Learning Research (10/2022)
practically the signals are still not enough. Our method does not rely on the interatomic distances, and achieve good results on large molecules.
To verify our conjecture, on GEOM-Drugs, we categorize the molecules based on their numbers of heavy atoms. We choose one of the five independently run DMCG models for analysis. The number heavy atoms in the i-th group lie in [10i + 1, 10 + 1)]. We compare our method against ConfGF (the code of DGSM is available) and GraphDG. The results in Table II We have similar observation, that our method brings more improvements than previous method on larger molecules.
Table 11: COV and MAT mean scores w.r.t numbers heavy atoms on small-scale GEOM-Drugs. The indicates that the heavy atom number lies in range [10i + 1, 10(i + 1)), (1,
Metric
COV (%)
MAT(A)L
= 1
==2
i == 3
average
i ==
== 2
i=
average
ConfGF
99.95
66.28
15.34
62.54
0.7764
1.1510
1.5345
1.1637
GraphDG
15.11
1.78
0.0
3.12
2.0578
2.5863
2.9849
2.5847
DMCG
100.00
97.62
90.04
96.69
0.5305
0.7190
0.8794
0.7223
8.6 More results about property prediction
Table 12: Median absolute error of predicted ensemble properties. (Unit: eV).
Methods
E
Emin
Acmin
A Aémax
RDKit 0.8721
0.6119
0.3057
0.4414
0.1830
GraphDG
13.1707
1.9221
3.4136
7.6845
1.1663
ConfGF
1.5167
0.1972
0.6588
4.8920
0.1686
DMCG
0.4132
0.1100
0.1276
0.8486
0.1288
The median absolute error of the property prediction is shown in Table 12 We can see that our method still outperforms all deep learning based mothods, which demonstrate the effectivencss of our method.
1.0
1.0
0.8
0.8
COv
0.6
COv
0.6
0.4
ConfGF Recall
0.4
GeoMol Recall
DMCG Recall
0.2
0.2
ConfGF Precision
GeoMol Precision
DMCG Precision
0.0
0.2
0.4
0.6
0.8
1.0
0.5
1.0
1.5
2.0
(a) CEOM-QM9
(b) CEOM-Drags
Figure 9: The Coverage score w.r.t different threshold 6.
22

---

Published in Transactions on Machine Learning Research (10/2022)
B.7 Results with different training sizes
To investigate whether our method relies on large dataset, we subsample training data the large-scale GEOM-QM9 and GEON-Drugs to 10%, 25%, 50,, 75%. The validation and test sets remain unchanged. The results are in Table R
Results with different training sizes. The baseline mehtods are trained on the full dataset. For our DMCG, it is trained on 10%, 25%, 50,, 75% and the full datasct.
Table 13:
Datasct
GEOM-QM9
GEOM-Drugs
COV (%)t
MAT (A)L
COV (%)1
MAT (A)
Methods
Mean
Median
Mean
Median
Mean
Median
Mean
Median
ConfGF
89.21
95.12
0.2809
0.2837
70.92
85.71
1.0940
1.0917
GeoMol
91.05
95.55
0.2970
0.2993
69.74
83.56
1.1110
1.0864
DMCG (10%)
91.48
97.45
0.2748
0.2692
94.47
100.00
0.7934
0.7624
DMCG (25%)
97.65
100.00
0.1858
0.1676
95.17
100.00
0.7475
0.7092
DMCG (50%)
98.23
100.00
0.1606
0.1457
96.38
100.00
0.7057
0.6771
DMCG (75%)
98.31
100.00
0.1544
0.1384
96.14
100.00
0.6947
0.6562
DMCG (100%)
98.34
100.00
0.1486
0.1340
96.22
100.00
0.6967
0.6552
We can see that:
1. Generally, DMCG benefits from more training data.
2. With 10% training data, our method is better than previous baselines ConIGF and GeoMol.
B.8 Adding more blocks
In this section, we study whether adding more blocks are helpful. We increase the number of blocks from 12. The results are in Table I4 For GEOM-QMS QM9, we do not observer performance improvement by increasing the number of blocks. For GEOM-Drugs, increasing the number of blocks further improves the performance. Our conjecture that, the molecules in GEOM-Drugs more complex than those in CEOM-QM9, which benefit more from larger models.
Table 14: Results with different number of blocks.
Dataset
GEOM-QM9
GEOM-Drugs
cOV(%))
MAT
(A)4
COV(%)
MAT
(A)
# blocks
Mean
Median
Mean
Median
Mean
Median
Mean
Median
96.23
99.26
0.2083
0.2014
96.52
100.00
0.7220
0.7161
95.55
98.91
0.2217
0.2190
96.77
100.00
0.7122
0.7093
10
95.71
99.52
0.2215
0.2212
97.29
100.00
0.7089
0.7079
12
94.65
99.11
0.2280
0.2284
97.11
100.00
0.7092
0.6996
B.9 Iterative refinement v.s. recursive refinement
Currently, the parameters of the blocks the decoder (i.c., Pdec) are not shared. Another option implement recursive model, where the parameters of different decoder blocks are shared. The results are in Table 15 We can that using the recursive model hurts the performances.
23

---

Published in Transactions on Machine Learning Research (10/2022)
Table 15: Results with recursive decoder.
Dataset
GEOM-QM9
GEOM-Drugs
(%)
MAT (A)
COv(%)T
MAT (A)
# Method
Mean
Median
Mean
Median
Mean
Median
Mean
Median
DMCG
96.23
99.26
0.2083
0.2014
96.52
100.00
0.7220
0.7161
DMCG with recursive decoder
94.03
98.00
0.2726
0.2771
95.20
100.00
0.7883
0.7862
C Constraints on distances
Let G be a molecular graph with N atoms (N 2 3). Let dı denote the distance between atom i and atom j. Define D as the distance matrix, which an x N matrix, and dij locates in the i-th row and j-th column of D.
The triangle inequalities means that for any three different i, k, d,, dj 2 dz,k-
A valid distance matrix D is induced from the - degree-of- ofreeddomf (DOF) of N 3D-corrdinates excluding global translation and rotation, while the popular practice of independently generating distances to 2- or 3-hop neighbors (Xu et al. 2021a) often introduces more DOF.
Moreover, a distance matrix should have a rank at most after element-wise squared Dokmannic et al, 2015). In other words, the rank of matrix D {dl, Ja,j is at most 5. Such constraint is hard to guarantee even if the DOF is matched (Simm & Hernández- Lobato
2020 (e.g., AI has one DOF but is almost surely full-rank). It also makes gradients ill-defined (Shi et al. 2021 (other distances cannot all be held constant while taking infinitesimal change to diz). Careful treatments (Hoffmann & Noë 2019) often increase the order of computation complexity.
24


Reviews:
[{"review_id": "55Gdk256lg", "reviewer": "Reviewer_yvBx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. This method directly generates 3D coordinates, which can then immediately be used for docking, property prediction and virtual screening. \n2. The method features a loss function that is designed for permutation and rotation/translation invariance using a combination of literature from quaternions and graph isomorphism,  utilized in this novel context. Moreover the loss function itself involves optimizations, 1) one which is finding a rotation minimizes the loss, is solved analytically and, 2) another which finds the minimizing permutation and solved by  enumeration. This idea of using optimizations to compute the loss function (but not backpropagating through them) is interesting but not highlighted in the manuscript. \n3. The proposed method is effective for predicting conformations of large molecules,  especially those with heavier atoms. Decoding is much more efficient compared to  sequential methods of generation, and the conformations generated are free from autocorrelation.\n\nWeaknesses: \n\n1. The introduction tries to motivate direct molecular generation by listing methods which generate intermediate quantities such as  interatomic distances, gradients wrt interatomic distances, torsion angles first. However, it is not really clear what is wrong with the latter approach. Indeed many of these quantities, such as relative distances and angles are invariant to rotation/translation already. \n2. The paper assumes that permutation and roto-translation invariance is a requirement for training a neural network to generate molecules, but this is not actually clear. What if a more relaxed condition like equivariance was sought [see Hoogeboom et al. ICML 2022 \"Equivariant Diffusion for Molecule Generation\" or Villar et al. NeurIPS 2021 \"Scalars are universal: Equivariant machine learning, structured like classical physics\"].  In fact, even with equivariance, it is not clear if it creates a more desirable solution space. \n3. The paper basically ignores its main contribution and does not discuss it at all the implications of their proposed loss function. Is it effective to compute? What is the effect of not backpropagating through the optimizations? Graph isomorphism is NP-hard, so how does this effect the solutions?  Are these choices ablated. The loss function and its solution are not even explained in the main paper. \n4. The authors propose a stacked or iterative architecture. Why is this needed? Why not have this be recursive? \n5. Lack of comparisons to methods that directly generate 3D conformations including the above mentioned paper from ICML 2022 as well as Clevees & Jain https://link.springer.com/article/10.1007/s10822-017-0015-8.\n6. Comparison to non-ML method OMEGA is not performed, OMEGA is SOTA and outperforms RDKit\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for generating 3D coordinates of atoms of a molecule. Their construction rests on a loss function that is invariant to symmetric molecule perturbations and rotations and translations of the molecular conformation. This loss function is implemented via enumeration of perturbations and an eigenvalue minimization problem using quaternions as presented in Karney 2007. \n\nThe  molecular generation architecture  consists  of  three  components:  (1)  an  encoder,  j2D,  that  takes  a  molecular graph as input and produces representations of all atoms (HV), covalent bonds (HE),  global graph-level features (U), and initial conformation (R); (2) another encoder, j3D, that takes  ground truth conformation and molecular graph as input and generates a normal distribution for \nconditional inference; and (3) a decoder, jdec, that takes learned representations and samples  from the normal distribution to produce the final conformation. The encoders and decoder are  implemented using a stacked architecture, which iteratively updates HV, HE, U and R. The loss  formulation consists of a KL divergence term (to drive the marginalized posterior distribution of  the random variable towards the prior) and a reconstruction/matching term that enforces roto-translation and permutation invariance.  \n \nNumerical experiments are performed using small-scale and large-scale versions of GEOM-QM9  and GEOM-Drugs. Models are evaluated using mean coverage score (abbr. COV, higher is better)  and matching score (abbr. MAT, lower is better) based on RMSD between the predicted and  actual  conformations.  Additionally,  the  authors  compare  the  ensemble  averaged  quantum  properties  of  the  generated  conformations  and  ground  truth  conformations  (both  computed  using Psi4) via the mean absolute error. They also evaluate the docking performance (using 100 randomly sampled protein-ligand pairs) of the generated conformations. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Requested changes mirror the weaknesses:\n1. Discuss in more detail what prior work has been done in conformation generation including Hoogeboom et al. \n2. More clearly motivate the advantage of direct 3D generation. \n3. Motivate invariance vs equivariance and also add ablation studies that show the impact of ablating permutation and rototranslation invariance or relaxing to equivariance. \n4. Discuss the details and implementation of the loss function in the main paper---this is a key part of this work. Perform complexity analysis of this, and what the gradients miss when they \"skip\" the optimization. Is it possible to use another neural network to compute the loss function?. \n5. Compare stacked to recurrent architecture. \n6. Add comparisons to the above mentioned methods. \n\nEdits:\n• Some of the models used for comparing recall-based COV and MAT \nscores (Table 1) are missing from precision-based scores (Table 5). \n• The authors mention that the code for DGSM is not available. However, COV and MAT \nscores for DGSM are reported in Table 1. In fact, almost all values reported in Table 1 for \nsmall-scale datasets are identical to Table 1 of the DGSM paper, “Predicting Molecular \nConformation via Dynamic Graph Score Matching” by Luo et al. However, the values \nreported in Table 2 (MAE of property prediction) are very different from those reported \nin Table 2 of the DGSM paper, especially for GraphDG. Is this difference due to the \nrandom sampling of 30 molecules and 50 conformations?  \n• Although the COV score flattened out, the MAT score is still decreasing in Figure 7. Is \nthere room for further improvement by adding more blocks? \n• In Figure 4, the proposed model is labelled as DMCG. This is inconsistent with other \nfigures and tables, where they refer to their model as “Ours”. \n• In Table 5 and Table 8, the best result is not bolded. \n• In Appendix C.2, Eqns. 12, 13 and 14, colored LaTeX is not needed.  ", "broader_impact_concerns": "I don't see any ethical implications that merit adding such a statement. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "obloAnT2LT", "reviewer": "Reviewer_D4R5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The implementation of the method is made public.\n- The related work section includes many recent models for conformation prediction, including a work on prediction of active conformation in a binding pocket.\n- The proposed method is able to generate multiple conformations because of the latent vector $z$ that adds randomness to the model.\n- The roto-translational invariance is achieved by the construction of the loss function that uses a transformation that aligns ground-truth and predicted conformations. The calculations are based on the quaternion method.\n- The additional transformation that permutes symmetric atoms improves the results of the direct conformation prediction (significant contribution).\n- The experimental section shows state-of-the-art results on the task of prediction molecular conformations.\n- The ablation study justifies the architectural choices made.\n- Additional experiments show that atom positions predicted by the method can be used for molecular prediction or as initial conformations for molecular docking.\n- The paper is written in a clear way, with clear mathematical notation.\n- The figures included in the paper help to grasp the idea of the method.\n- The supplementary materials contain many more experiments and metrics.\n\n\nWeaknesses:\n- Neither confidence intervals nor standard deviations are provided for the results.\n- The paper does not include a detailed discussion about computational cost (e.g. complexity of calculating transformations included in the loss function) though it argues that the method requires less iterations than other methods that can be found in the literature.\n- There is no description of the hyperparameter tuning process. If the best set of hyperparameters was chosen based on the experimental testing results, then this might indicate a potential data leakage and unfair comparison with other methods.\n- Some of the claims are not properly supported or ambiguous (see the requested changes below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the Authors introduce a new method for predicting molecular conformations. The method uses an autoencoder type of a network that generates conformations by encoding molecular graphs and predicting all atom positions at the end of the network. The loss function contains two components. One is responsible for measuring the distance between the ground truth conformation and the predicted conformation (RMSD), and the other one is the Kullback-Leibler divergence applied to the latent vector $z$ that encodes 3D atom positions and allows the model to sample multiple conformations. To make the model invariant to rotations, translations, and the permutation of symmetric atoms, a composition of two transformations is applied to the predicted conformation, and the loss function is computed for the maximum agreement of conformational poses over these transformations. The results of the proposed model are demonstrated on both small- and large-scale GEOM-QM9 and GEOM-Drugs datasets. Additionally, the usability of the method for molecular property prediction and molecular docking (as initial conformation generation) is considered.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Please, explain how the hyperparameters shown in Section B.1 of the supplementary materials were selected.\n- The Authors say that “on average, a molecule has 5.9 symmetric substructures” in the GEOM-Drugs dataset. How were these substructures counted? Is this the number of the axes of symmetry? Can this number be converted into a percentage that is easier to interpret?\n- For the coverage metric, $\\delta$ was selected to be 0.5 and 1.25 for the GEOM-QM9 and GEOM-Drugs datasets, respectively. Why were these numbers selected and why are they different for each dataset? Could a plot showing COV under different deltas be added to the supplementary materials?\n- Why was only a selection of models used in some of the experiments? For example, ConfGF was used in the docking experiment, and not GeoMol or GeoDiff. \n- In the ablations study, Figure 7 shows metrics calculated after each layer. Was the loss function in this experiment constructed to optimize the conformation of each layer, or only the final conformation was compared to the ground-truth positions?\n- Would it not be more fair for the experiment in Section C.3 (Supplementary Material) to reduce the number of parameters in the proposed method instead of increasing the number of parameters for the other method? I believe the size of hidden dimensions was selected to be optimal by the authors of the other method (ConfGF).", "broader_impact_concerns": "I do not see any ethical concerns that should be raised here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gaUvUeUARp", "reviewer": "Reviewer_PiFm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- **Interesting and principled loss function**: which empirically leads to much better performance (section 4.5). I am surprised that other works have not used such a loss function before (although I am not super familiar with work in this area so I could be wrong).\n- **Strong empirical results on standard benchmarks**, also featuring numerical improvements over previous methods. The authors also compared to an impressive number of baselines!\n- **Additional insightful experiments** including an ablation study on the key features of their method, and a docking/HOMO-LUMO task. I thought these experiments were interesting and insightful, particularly the ablation study.\n- **Writing** is very clear and high-quality with few exceptions\n\n### Weaknesses\n\n- **Potential poor scaling of loss not discussed**: my understanding is that evaluating equation (1) requires enumerating all possible permutations, and for each permutation finding the best roto-translation analytically. However, for many larger molecules the number of possible permutations will be very high. For example, many amino acids (e.g. Tyrosine, Leucine, Valine; see https://en.wikipedia.org/wiki/Amino_acid) have at least 1 symmetric atom in their side chains. A peptide of $N$ amino acids will therefore have at least $2^N$ different permutations. This is a known issue (e.g. it is mentioned in the documentation for `GetBestRMS` in `rdkit`). Given that one claim made by the authors is that DMCG shows especially strong improvement for larger molecules (stated in section 4.2), I think that this potential limitation for large molecules should at the very least be mentioned, and ideally be thoughtfully discussed.\n- **Potential errors in implementing loss function**: from the description in section A.1, it appears that the set of symmetries is defined only using information about the connectivity of a node's immediate neighbours. This is incorrect; it is easy to come up with counterexamples of 2 atoms in a molecule with identical immediate neighbours but for which there is no symmetry. If this implementation detail is indeed incorrect, it could potentially invalidate many of the results in the paper... If symmetries were identified with `rdkit` (e.g. the functions behind `GetBestRMS`) then I would have more confidence.\n- **Some unclear experimental details**:\n    - Which baselines did the authors run themselves, and for which baselines did the authors copy numbers from the original paper?\n    - Were the number of parameters in the baseline methods scaled when moving from the small to large datasets? Since DMCG is scaled it seems fair that the number of parameters for all methods should be adjusted to e.g. minimize overfitting.\n- **Space limitations**: in several places the authors mention \"space limitations\" as the reason for not including something in the main text of the paper. While I appreciate the honesty of the authors, I think that this should be avoided, especially in TMLR where submissions can be of any length. This was especially surprising since the submission main text is currently just over 11 pages, which is less than the 12 page maximum required for fast review. I think that the entirety of appendix A could (and probably should) be moved to the main text since these are important details of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a method to generate 3D molecular structures called \"Direct Molecular Conformation Generation\" (DMCG). The method is a latent variable model which generates a conformer from the output of a trained graph neural network and a random latent variable. The model is trained using an ELBO-like objective where the latent variable is outputted by a learned 3D encoder which is regularized by a KL divergence term. The authors perform a variety of experiments on different conformer generation benchmarks and report promising results.\n\nThe main contributions of the work are:\n1. Formulating a loss which is invariant to permutations, rotations, and translations, so that the model is only penalized for generating an incorrect structure, not generating a correct structure in the wrong orientation/position/arbitrary breaking of permutation symmetry.\n1. A model architecture inspired by AlphaFold which generates a structure and iteratively refines it.\n1. Thorough empirical comparison of DMCG with other methods, both on standard benchmarks and on a more real-world docking and quantum chemical modelling task.\n\nOverall I think this is a fairly strong paper. There is a clear contribution, and the claims are well-supported by the experiments. I believe it will be of interest to the TMLR audience. I have some small concerns about correctness but I imagine that these will be resolved without issue.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "### Very important changes\n\n- Clarification of whether the algorithm to determine molecular symmetries described in section a.1 is incorrect, or whether I have misunderstood it (see \"Weaknesses\" above). If I have misunderstood it, then I suggest that the authors provide a clearer description.\n- Clarification of experimental details from the \"weaknesses\" section.\n\n### Nice changes\n\n- Remove \"space limitation\" stuff and move appendix A into the main text\n- You use the phrase \"expressiveness of our loss function\": what does this mean? I don't think that losses \"express\" anything... I would like to see this re-phrased to be clearer.\n- The equations for sampling from a standard normal distribution on page 3 are only correct if $\\Sigma$ is diagonal and if it is interpreted as standard deviation instead of variance (which is more common). This should be updated/clarified.\n- DMCG (and other similar methods) depend on having a large dataset of high-quality conformers, which is often unavailable in practice. It would be very good to comment on how the method's performance changes with dataset size (perhaps an additional experiment), or at the very least discuss this limitation explicitly in the paper.\n\n### Typos/other small things\n\n- \"we compares\" -> \"we compared\" on page 9\n- You never explicitly define the acronym DMCG; it just appears in your github link and in the legend of Figure 4. You should really change this.\n- In equation 2, it would be nice to see explicitly which variables the minimization is taken with. It was slightly unclear to me.", "broader_impact_concerns": "No concerns about ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to generate molecular conformation by directly predicting atom positions. Core to the method is a principled loss function that is invariant with respect to rotation/translations as well as permutations of atom indices that do not change the molecule. The invariance to index permutation is achieved by computing the loss over different precomputed permutations.

The core idea is simple and the method achieves strong empirical performance according to a thorough evaluation. Among the compared approaches, the proposed method is the only one that consistently improves upon Rdkit. The high accuracy of the predicted conformation is shown to translate into downstream improvements when used as input to the calculation of molecular properties or the binding energy to proteins.

After a thorough discussion, reviewers unanimously recommended accepting the paper. The principled construction of the loss function and strong empirical performance was seen as key strength. Limited comparison to previous work and lack of clarity about picking hyperparameters were some of the discussed weaknesses.

Authors have released code and instructions on how to reproduce the results. During the review process, it was discovered and addressed that instructions are not fully complete. Overall, please make sure it is straightforward to rerun key experiments from the paper.

All in all, it is my pleasure to recommend acceptance of the paper. Thank you for submitting to TMLR, and please make sure to address all comments raised by reviewers. In particular, remember to include a more detailed discussion of how hyperparameters were chosen, and please add an ablation study that helps understand how the performance depends on the exact choice of hyperparameters.
Decision: Accept (poster)