An Autoregressive Flow Model for 3D Molecular Geometry Generation from Scratch
Authors: Youzhi Luo
Shuiwang Ji
Venue: ICLR 2022 Poster (2022)
Keywords: 3D molecular geometry generation
flow models
SphereNet

Abstract: We consider the problem of generating 3D molecular geometries from scratch. While multiple methods have been developed for generating molecular graphs, generating 3D molecular geometries from scratch is largely under-explored. In this work, we propose G-SphereNet, a novel autoregressive flow model for generating 3D molecular geometries. G-SphereNet employs a flexible sequential generation scheme by placing atoms in 3D space step-by-step. Instead of generating 3D coordinates directly, we propose to determine 3D positions of atoms by generating distances, angles and torsion angles, thereby ensuring both invariance and equivariance properties. In addition, we propose to use spherical message passing and attention mechanism for conditional information extraction. Experimental results show that G-SphereNet outperforms previous methods on random molecular geometry generation and targeted molecule discovery tasks. Our code is publicly available as part of the DIG package (https://github.com/divelab/DIG).

Published as a conference paper at ICLR 2022
AN AUTOREGRESSIVE FLOW MODEL FOR 3D MOLEC- ULAR GEOMETRY GENERATION FROM SCRATCH
Youzhi Luo & Shuiwang Ji Department of Computer Science & Engineering Texas A&M University
College Station, TX 77843, USA {yzluo, jji ajuu.ummu.. amu. edu
ABSTRACT
We consider the problem of generating 3D molecular geometries from scratch. While multiple methods have been developed for generating molecular graphs, generating 3D molecular geometries from scratch largely ender-explored. In this work, we propose G-SphereNet, novel autoregressive flow model for gener- ating 3D molecular geometries. G-SphereNet employs a flexible sequential gen- eration scheme by placing atoms in 3D space step-by-step. Instead of generating 3D coordinates directly, we propose to determine 3D positions of atoms by gen- erating distances, angles and torsion angles, thereby ensuring both invariance and equivariance. addition, we propose to use spherical message passing and atten- tion mechanism for conditional information extraction. Experimental results show that G-SphereNet outperforms previous methods on random molecular geometry generation and targeted molecule discovery tasks. Our code is publicly available as part of the DIG package (https://gr) ptctps://githubb.cmm http:/huwbb.oom///eellb/b//bb//a
INTRODUCTION
Designing and synthesizing novel molecules with desirable properties is challenging task in drug discovery (Wang et al. 2022, Stokes et al. 2020) and chemical science. The size of search space of all chemical molecules estimated to be on the order of 1003 (Polishchuk al. 2013). thereby making exhaustive search infeasible. In recent years, advances in machine learning methods have greatly accelerated the progress in this filed. Many studies represent molecules as 2D molecular graphs and propose to automatically generate molecular graphs and optimize molecular properties with deep generative models, such as variational auto-encoders (Kingma & Welling, 2014).
However, complete information of molecules cannot be obtained from 2D molecular graphs because 3D structures, also known as 3D molecular geometries, are critical in determining many molecular properties. 3D molecular geometries represent the 3D coordinates of atoms and are important for the accurate prediction of quantum properties (Schütt et al.) (017). Hence, we argue that generating 2D) molecular graphs might not be the best way identify novel molecules with certain desirable quan- tum properties. Instead, developing a generative model that can generate 3D molecular geometries from scratch is promising solution to this problem. Currently, this area remains under-explored. Recently, a series of seminal studies (Xu et al.) 2021b al Shi et al. (021] have proposed to gener- ate 3D molecular geometries from given 2D molecular graphs. These methods themselves do not generate novel molecules once the 2D molecular graphs are given.
In this work, we propose G-SphereNet, a Generative model for 3D molecular geometry generation from scratch inspired by SphereNet (Liu et al.) (022). In G-SphereNet, 3D molecular geometries are generated by sequentially placing atoms 3D space. The 3D positions of atoms are implic- itly determined by generating distances, angles and torsion angles ensure both invariance and equivariance properties. Our work is inspired by SphereNet, which use distances, angles and tor- sion angles to compute predictive representations of molecules. In addition, G-SphereNet employs SphereNet and attention mechanism to extract conditional information. Experimental results show that G-SphereNet can outperform prior methods on the 3D molecular geometry generation task.

---

Published as a conference paper at ICLR 2022
2 BACKGROUND AND RELATED WORK
2.1 2D MOLECULAR GRAPH GENERATION
Recently, thanks to the advances of deep generative models, significant progress has been made in the problem of molecule design and generation. Some methods (Gómez-Bombarde) et al.] 2018 Kusner et al.) 2017 Dai et al. 2018 Liu et al., 2020) use sequence models to generate SMILES [Weininger] 1988 string representations of molecules. Other studies consider molecules as graphs, in which atoms and chemical bonds of the molecule are represented by nodes and edges, respectively. These studies either generate the node type and adjacency matrix of the graph (Simonovsky & Komodakis 2018 Ma et al.) 2018 De Cao & Kipi, 2018 Liu et al. (021c). or form the molecular graph by sequentially adding nodes and edges (You al. 2018 [Shi* al. 2020), or compose the molecule from a junction tree of molecular motifss (Jin et al. 2018).
However, these methods only generate the graph structures of molecules while crucial 3D molecular geometries are ignored. In other words, the 3D coordinates of atoms in the molecule are unknown. Hence, these generative models cannot distinguish spatial isomers, i.e., molecules with the same molecular graph but different 3D molecular geometries. In addition, 3D molecular geometries are required for the computation of some quantum properties of molecules. such HOMO-LUMO gap (Hu et al., [202T] Liu et al., 2201aa] Ying et al.) [021]] Addanki et al.) [20T]] Xu et al. (021c). Hence, these generation methods cannot be used when spatial isomers or quantum properties are needed.
2.2 3D MOLECULAR GEOMETRY GENERATION FROM 2D INFORMATION
Currently, some studies have proposed generate 3D molecular geometries from the 2D conditional information of target molecules. Some methods (Simm et al. 2020 2021) propose to generate 3D molecular geometries through minimizing the energy of atomic systems, in which the numbers and types of atoms are explicitly given in atom bags. Other methods (Mansimov et al.) 2019 Simm & Hernandez-Lobato) 2020 Gogineni et al.) 2020 Xu et al., 2021b al Shi et al. 2021 Ganea et al.) 2021) propose to randomly sample multiple 3D molecular geometries from the molecular graph of the target molecule with deep generative models. These conditional generation methods assume that given the target molecules in the form of molecular graphs, but we do not know their 3D geometries. However, this assumption does not hold in the targeted molecule discovery problem. In this problem, aim to discover novel molecules with good quantum properties, such as low HOMO-LUMO gaps. In other words, target molecules themselves are unknown and to be generated, and their 3D geometries are also needed because quantum properties cannot be accurately estimated only from molecular graphs. Hence, we argue that developing a method to generate 3D molecular geometries from scratch more useful this problem.
2.3
3D MOLECULAR GEOMETRY GENERATION FROM SCRATCHI
In this work, consider the problem of generating 3D molecular geometries from scratch. Let g = fr be a set of 3D molecular geometries, and the function S(G) E I computes a specific quantum property score of G. We consider the two generation tasks defined as below.
Learning random generation model pe(-) from G, so that the model can sample valid 3D molecular geometry G with a high probability po (G').
Learning a targeted molecule discovery model pe (-) so as to maximize (or minimize) the expected quantum property score Eg~pp [S(G)]
This problem is largely under explored and there are only few studies attempting to solve it. G- SchNet (Gebauer et al.) 2019) uses an autoregressive model based on SchNet (Schütt et al., 2017) to sequentially generate the new atom and place it at the local grid point of focal atom. On the other hand, EDMNet (Hoffmann & Noč 2019) and 3DMoINet (Nesterov et al. 2020 generate pairwise distances between atoms with generative adversarial networks (GAN) (Goodfellow et al.) 2014) and variational auto- auto-enccoerrs (VAE) (Kingma & Welling 2014), respectively. Besides, E-NFs Satorras et al.) 2021a) proposes geometry generation model by combining flow models (Rezende & Mohamed] 2015) with E(n) equivariant graph neural networks (Satorras et al., 2021b). It generates 3ID coordinates of all atoms in a one-shot fashion and defines the prior distribution in subspace of
2

---

Published as a conference paper at ICLR 2022
the latent space to ensure translation invariance. Different from these methods, our method employs a flexible sequential generation pipeline based on autoregressive flow models, which can capture the density of 3D molecular geometries more effectively.
2.4 FLOW MODELS
A flow model defines a parametric invertible mapping fo :2:22 E R2 + E R., where the data point and the latent variable both random variables. Given that is sampled from a known prior distribution pz and fo is invertible, we can compute the log-likelihood of as
logPx(z) log (fr'(z)) + log [detJ].
der where J 05. (x) the Jacobian matrix. To train the flow model on a given dataset X' {a; m
fi=1+
the log-likelihoods of data points are computed from Eqn. ( and maximized by gradient descent. Hence, tractable and cheap computation of det. is needed for efficient training. A common choice of fo in most flow models is the affine coupling mapping (Dinh et al. 2014 2016), in which case computing det./ is very easy because J is an upper triangular matrix.
Flow models have been used in variety of generation tasks (Rezende & Mohamed 2015 Tran
ct al. 2019 Köhler et al. 2020). Compared with VAE and GAN, they allow for exact likelihoor computation, and can model the density of data more accurately. Because of these advantages, many recent studies have used flow models in the molecule generation task. Some one-shot methods, including GraphNVP (Madhawa al., 2019), GRF (Honda et al. 2019), and MoFlow (Zang & Wang 2020). consider the node type and adjacency matrix as the generation targets, On the other hand, GraphAF (Shi et al., 2020) and GraphDF (Luo et al. 202I generate molecular graphs by generating nodes and edges sequentially through autoregressive flow models (Papamakarios et al.) 2017). These models have stronger capacity to model graph structures than one-shot methods, and achieve state-of-the-aa performance in the molecular graph generation task.
3 METHODS
While autoregressive flow models have been successfully applied to the molecular graph genera- tion task, it remains unclear whether they are sufficiently powerful to model more complicated 3D) molccular geometries. In this section, we present G-Sphereee,, a novel 3D molecular geometry generation method. It adopts flexible, effective, and efficient sequential generation pipeline based on autoregressive flow models, which can ensure the equivariance property of coordinates and the invariance property of likelihood simultaneously. In addition, expressive spherical message passing based graph neural networks (Liu et al., 2022) and multi-head attention networks (Vaswani et al., 2017) are used in G-SphereNet to extract 3D conditional information for accurate generation. To the best of our knowledge, G-SphereNet is the first likelihood-base autoregressive generative model for 3D molecular geometry generation.
3.1 SEQUENTIAL GENERATION
Let & be the number of atom types. We use a 3D point cloud G = (A, R) to represent the 3D geometry of molecule with n atoms, where A E {0, 1 is the atom type matrix and R E =x is the atom coordinate matrix. Each row in the matrix A is a one-hot vector, and Alj. u] = represents that the j-th atom has type u. The 3-dimensional row vector the j-th row of the matrix R represents the 3D Cartesian coordinate of the j-th atom.
We consider the generation of 3D molecular geometries as sequential decision process. We start from molecular geometry G with one carbon atom at the origin point, and generate the complete geometry by adding a new atom at each step. Specifically, at the i-th step, let the intermediate 3D molecular geometry generated from the previous atoms. The atom type a; E {0, 1 of the new atom is generated by the generative model g" based on the latent variable 2f. Afterwards, the generative model g" decides the 3D Cartesian coordinate rs E R3 of the new atom based on the latent variable z!. Both g" and g" are autoregressive functions of intermediately generated geometries. The overall sequential generation process can be described by the following equations:
steps be G, (A,, R2), which has i
a; = (2f; Ai, Ri), rt ===9" (#//s Ai, Ra), i2 1.
(2)
3

---

Published as a conference paper at ICLR 2022
G
33
Hydrogon alom
Stroggr ator
Carbon
atom
G2
Ga
G4
New stom
Figure 1: An illustration of the sequential generation process in G-SphereNett.
3.2 GENERATION WITH AUTOREGRESSIVE FLOW MODELS
We employ autoregressive flow models to generate the atom type a, of the new atom at each step. Since atom types discrete numbers, adopt the dequantization method (You al. 2018 Shi et al. 2020 to convert them into continuous numbers by adding real-valued noise as
ā = a; + u, ~~ (0, 1)*, i2 1,
(3)
where (00, 1) is the uniform distribution over the interval (0, To generate a,, we first sample the latent variable 2.th E & from the standard Gaussian distribution (0, 1), and then map 2, to å by the affine transformation
a = Of of 121,
(4)
where denotes the element-wise multiplication, the scale factor of and the shift factor ef both depend on the conditional information extracted from the intermediate geometry G; = (A,, R₂). Intuitively, å; should be invariant to any rigid transformation on R,, i... â; should not change if we rotate or translate R2 in 3D space. Hence, we use a symmetry-invariant model to compute of and lf from G, which are described in Sec. 3.4 in detail. After obtaining as Og can computed by taking the argmax of a; a, one- one-hot(ase arg max ā,).
However, we cannot generate the 3D coordinate r; in the same way as the generation of the atom type a, Directly calculating with the autoregressive flow model, as in Eqn. (4), neither satisfies the equivariance property of coordinates nor the invariance property of likehood. First, easy to find that if we rotate translate R3, then r; needs to be rotated or translated correspondingly. Formally, it means that for any orthogonal matrix Q E 3 and translation vector E R rı = g=""=]]" (=I;; Ar, Rz), have
Qr:+b=g g" A,, R²QT + +1b"),
where 1 denotes a vector of all ones of length If we directly compute r, by the autoregressive model similar to Eqn. 4 i.e. ri of C o + 4r then satisfy Eqn. (5). the correctness of s 9r+h ① 2 + 1. Qr+b = Qis O + translation vector b. However, it is very hard to design flow model satisfying this complicated condition. Second, the likelihood p(r; A, R2) should be invariant to rotations and translations as they do not change the 3D structure. In other words,
has to be ensured for any orthogonal matrix Q and
(Qr; ; + b/A,, R²Q + 15²) pril R:)
should be satisfied for any orthogonal matrix Q and translation vector follows from the change- of-variable theorem that (ril A, R,) p (=[' det 0z," Let the latent variable corresponding to
Qr: + be 2r+ The affine transformation in Eqn. (4 does not admit any relationship between def 0-Gr+h either p(z}) and p(2?r+b), or ör; and B(Qr:+6). so Eqn. (6) is not guaranteed to hold. Hence, we
cannot guarantee the invariance property.
Given these limitations, we instead propose to determine the 3D relative position the new atom by symmetry-invariane elements. Similar to G-SchNet (Gebauer al. 2019). we first choose focal atom among all atoms G,, which serves as the reference point for the location of the new atom. Then, the model generates the distance d,, the angle 6,, and the torsion angle you successively. Specifically, assuming that the focal atom is the f-th atom of Gū, the distance d, = I|r - -ryll2 from the focal atom to the new atom is first generated. Then, if i >> 2, the angle 0; E [0, πr] between the
4

---

Published as a conference paper at ICLR 2022
lines rr) and (ry,re) ( is generated, where is the atom closest to in G2. Finally, if 2 3, the torsion angle (ps E [-T, x] between the planes formed by positions (ry, Te, Te) and (ry. re) is generated, where is the atom closest to e but different from in Gj. Similar to a, dj, f, and (p;
are generated as
d. = 8 e +
i2 1, 0, - + i2 2,
47
94 = of
i2 3,
where =f, =Q of E IR are all latent variables sampled from standard Gaussian distributions, and the scale factors 5% s..
R and the shift factors t,,,t, t? E R are functions of G, Afterwards, the coordinate r; is computed from the relative positional elements d, 8t, (P; and the coordinates Te.TesPo We show that such a process of placing the new atom in 3D space can strictly satisfy the conditions in Eqn. (5) and (6) in Appendix A thereby satisfying both invariance and equivariance properties. The sequential generation process repeats until either the maximum number of atoms is reached, or no atom can be chosen as the focal atom by the atom-wise classifier. Our proposed se- quential generation method is related SphereNet (Liu et al. 2022) in that our generation targets are the 3D information used by SphereNet to extract features, so we name our method as G-sphereNet. An illustration of the overall generation process in G-SphereNet is given in Figure [
3.3 DISCUSSIONS
We argue that our proposed G-SphereNet method has many advantages over previous 3D molecular geometry generation methods. First, it is easier for G-SphereNet to generate valid geometries theo- retically because the exact 3D coordinate of each atom can always be obtained. However, EDMNet (Hoffmann & Noë| 2019) and 3DMolNet (Nesterov et al., 2020) generate the pairwise distances of atoms in the form of distance matrices. There no theoretical guarantee that the generated matrices are always valid Euclidean distance matrices, or correspond to coordinates in 3D space (Dokmanic let al. 2015). Second, the generation of 3D positions in G-SphereNet is more flexible than that in G-SchNet (Gebauer et al., 2019). In G-SchNet, the new atom has to be placed at one of the can- didate grid points circling around the focal atom, but it can be placed any relative position of the focal atom in G-SphereNet. Third, compared with E-NFs (Satorras et al. (021a),, G-SphereNet more efficient and effective. E-NFs maps latent variables to 3D coordinates of atoms by a flow model. Since coordinates are not translation invariant, E-NFs proposes to obtain latent variables by computationally expensive operations. Specifically, E-NFs first samples from the prior distribution defined in a subspace of the latent space, then maps the sampled variable to the latent variable by linear projection. In contrast, G-SphereNet obtain 3D positions by generating distances, angles and torsion angles, which are naturally translation invariant. Hence, G-Sphere can avoid the compli- cated operations of E-NFs. In addition, E-NFs generates the coordinates of all atoms in the geometry at time, while G- G-SphereNet obtains the coordinate of one atom at time. Though G-SphereNet may be slower, we argue that the sequential generation fashion helps the model to capture the de- pendency between atoms and the density of geometries more effectively. Experimental results also demonstrate that G-SphereNet can generate much more valid molecular geometries than E-NFs.
3.4 CONDITIONAL INFORMATION EXTRACTION
As we have mentioned in Sec. 3.2 generating the atom type and 3D position of the new atom quires capturing conditional information from the intermediate geometry at each step. Desirable conditional information should incorporate comprehensive 3D structural features of the geometry, and be invariant to any rigid transformation. To achieve this goal, we propose to capture condi- tional information using SphereNet (Liu al. (222)), an advanced 3D graph neural network model. directional cutoff graph. Denoting the fea-
SphereNet considers the input molecular geometry ture of k-th edge by eg and the feature of i-th node by n;, SphereNet initializes them with spherical basis functions and updates them to ea and ea with spherical message passing as
e = Φ² (ek: Uru: Usu: Ess, v/ = 6" (
(R.s.))
Here, Φ", o" are updating functions, pr, pr are aggregation functions, F' and 8y are the sending and receiving nodes of the k-th edge, R,, are the coordinates of neighboring nodes of 8k; E., and
5

---

Published as a conference paper at ICLR 2022
Es the features of edges incident node są and i, separately. SphereNet has powerful 3D structural feature extraction ability, and achieves good performance in multiple quantum property prediction tasks. Considering these advantages, extractor to capture conditional information from the intermediate molecular geometry.
the SphereNet model as the backbone feature
For the input molecular geometry Gj, let the node embeddings computed from the SphereNet be )/= To select the focal atom, use atom-wise multi-layer perceptron (MLP) taking the corresponding node embedding as input, and randomly choose the focal atom f from atoms whose classification scores are higher than 0.5. all the classification scores outputted from the atom-wise classifier are lower than 0.5, then no atom can be chosen as the focal atom and the sequential generation process terminates. Afterwards, the scale and shift factors Eqn. 44 and 7 computed. We ever tried computing them using solely the node embeddings of the reference points f. c, and e. However, our experiments show that it frequently causes incorrect placements of new atoms in 3D space. We think it is because node embeddings only contain local 3D information, which is insufficient for the accurate generation of the 3D positions of new atoms.
To tackle the above issue, we propose to augment node embeddings with global features extracted by a multi-head attention network (Vaswani et al.) (017). Formally, a multi-head attention network takes the query matrix Q. key matrix K, and value matrix V inputs, and extract global information from inputs by the multi-head attention mechanism:
(iK??)
Q: = QW. K, KWW, V vWY, ATT; = softmax
V, 1siso,
VD
MH-ATT K, V) = Con (ATTI, ATT.) W
where Con(-) denotes the concatenation operation, is the size of the second dimension of K, o is the number of attention heads, the matrices W.P. W.K, WY, and w are all trainable parameters. Let the node embedding matrix of G, be H₁ [hr,0, -.y 11,5-1"", o8 and 1 are computed
Con ()) MLP" (Con (hu,,, MH-ATT (h Hi H:)) 121,
(10)
where MH-ATT" is multi-head attention network and MLP is multi-layer perceptron. As for the scale and shift factors in Eqn. 7 we first multiply node embeddings with the atom type embedding vector include the atom type information:
h Embedding(a:)), Thi = h2 @ higs H, = -To,o ... h )"
121,05isi-1.
then compute them as
Con(s (2) MLPd
Con T MH-ATT (TT. Hi,
121,
Con(s Con(s?,t?) MLP
Con
Te. Tr... MH-ATT (NE H, H.
122,
(11)
Con(s fon(s,,17) (1f) MIPP Con
T Thi,e, MH-ATT (NT,, H, H. )) 123.
Here Embedding(-) is a lookup based embedding layer, MH-ATT" MH-ATT MH-ATT are all multi-head attention networks, and MIPd MIP", MLPP are all multi-layer perceptrons. The use of the multi-head attention networks helps extract more comprehensive 3D conditional information and is demonstrated to improve the generation performance lot in our ablation study.
3.5 TRAINING
To train the G-SphereNet model on dataset, we first need to split each individual molecular geom- etry in the dataset into a trajectory of atom addition steps. In other words, the generation order of atoms in the geometry and all corresponding focal atoms need to be determined. Since the generated atom is supposed to be placed in the local region of the focal atom during generation, we propose to obtain the training trajectory by applying Prim's algorithm the geometry. This procedure can ensure that the sampled focal atom is always the nearest neighbor of the new atom among all atoms in the intermediate geometry.

---

Published as a conference paper at ICLR 2022
Table 1: Comparison of different methods on the random molecular geometry generation task. The performance is evaluated by the chemical validity percentage and MMD distances of bond length distributions. Here f means higher value indicates better performance, while means the opposite.
MMD distances+
Method
Validity
C-C
C-N
C-O
H-C
H-N
H-O
Average
E-NFs 39.77%
0.775
0.209
1.218
1.218
1.748
2,478
1.274
G-SchNet 81.49%
0.183
0.078
0.320
1.236
1.396
2.399
0.935
G-SphereNet
88.18%
1.144
0.315
0.305
0.139
1.029
0.831
0.627
For a 3D molecular geometry G having n atoms (n > 3), we maximize its log-likelihood to train the G-SphereNet model. Specifically, obtain the generation targets, i.e., the atom type and 3D position of the atom to be generated at each step, then compute the log-likelihood of G as
Oā: Ü2f
Od 0z4
logp(G)
Pog (=() + log
log Pza (=²) + log
(12)
00,
0y; 0z
log pz. (2.) log
Σ
log Pz, (2f) log
2 where latent variables 2", -;: 2/ 2 are computed by reversing the mapping in Equ. (4) and (7).
and pz,, Pz, are all standard Gaussian distributions. Besides, the atom-wise classifier in
the G-SphereNet model, which is used for the focal atom selection, is trained by the binary cross entropy loss. The ground-truth label is if the atom not valence full filled, otherwise is O. We describe the detailed training and generation algorithm of G-SphereNet in Appendix B
4 EXPERIMENTS
In this section, we evaluate the proposed G-SphereNet method on the random molecular geometry generation task and the targeted molecule discovery task described in Sec. 2.3 We show that in these tasks, G-SphereNet can outperform previous 3D molecular geometry methods, including G- SchNet (Gebauer et al., 2019) and E-NFs (Satorras et al., 2021a). Note that we do not compare with recent methods (Xu al. 2021b Shi al., of generating 3D molecular geometries from 2D information because they cannot do targeted molecule discovery (Sec. (.2). In addition, we conduct extensive ablation studies evaluate the advantages of some designs G-SphereNet method.
4.1 RANDOM MOLECULAR GEOMETRY GENERATION
Data. For the random molecular geometry generation task, evaluate G-SphereNet the QM9 (Ramakrishnan et al.) 2014) dataset. The QM9 dataset provides over 130k molecules and their cor- responding 3DD molecular geometries computed by density functional theory (DFT). We randomly select 100k 3D molecular geometries as the training data and 10k 3D molecular geometries as the validation data. For fair comparison, the models of our G-SphereNet method and all other methods are trained with the same data split.
Setup. We the chemical validity percentage (Validity in short) evaluate the generation accu- racy of G-SphereNet. Specifically, all generated 3D molecular geometries are converted to molecular graphs by the method proposed in Kim & Kim (2015). and the Validity is defined the percentage of molecular graphs which do not violate chemical valency rules. In addition to the chemical va- lidity, we also evaluate the 3D structural accuracy the generated molecular geometries. We ever tried to follow G-SchNet (Gebauer et al.) 2019) to compute the aligned coordinate differences be- tween the generated geometries and their relaxed ones, but we find the relaxing process involves the expensive computation based on DFT, which takes hours for a single molecular geometry. Hence, we propose to evaluate by the Maximum Mean Discrepancy (MMD) (Gretton et al., (2012) distances of bond length distributions. Formally, for a certain type of bond, obtain its length distribution in the generated geometries and in the geometries of the dataset, separately, and compute the statistical discrepancy between the two length distributions with the MMD distance. We compute the MMD)

---

Published as a conference paper at ICLR 2022
Table 2:
Comparison of different methods on the targeted molecule discovery task. Here means that our objective is minimizing the property score, while 1 is maximizing the property score.
HOMO-LUMO gapt
Isotropic polarizability?
Method
Good percentage
Good
Mean
Optimal
Mean
Optimal
percentage
QM9 (Dataset)
6,833
0.669
3.20%
75.19
196.62
2.04%
G-SchNet
3,447
0.583
78.45%
76.94
204.04
30.18%
G-SphereNet
2.907
0.294
82.73%
88.18
349.98
35.49 %
distances on six most frequently appeared types of chemical bonds, including carbon-carbon single bonds (C-C), carbon-nitrogen single bonds (C-N), carbon-oxygen single bonds (C-O), hydrogen- carbon single bonds (H-C), mydrogen-nitrogee single bonds (H-N), and wydrogen-oxygee single bonds (H-O). All metrics are computed from 10,000 generated molecular geometries.
The implementation of the SphereNet (Liu et al. 2022) model used for condition information extrac- tion is based on the code of DIG (Liu et al. (b2 package. We use Adam (Kingma & Ba 2015) optimizer to train the G-SphereNet model for 100 epochs, with a batch size of 64 and a learning rate of 0.0001. See Appendix for model configuration and other training details. G-SphereNet is compared with G-SchNet (Gebauer et al. 2019) and E-NFs (Satorras et al. 202 in terms of Validity and MMD distances, and we run the code provided by authors to obtain the results of two baseline methods. We do not compare with EDMNet (Hoffmann & Noē 2019) or 3DMolNet (Nes-) terov et al., 2020) because the authors of 3DMolNet do not provide their implementation, and the EDMNet model cannot be trained on molecular geometries with variable numbers of atoms, which inhibits fair comparison with G-SphereNet.
Results. We present the performance of different methods in Table [ Our G-SphereNet method achieves the highest Validity of 88.18%, while E-NFs achieves much lower Validity of 39.77%. The good performance strongly demonstrates that the sequential fashion of G-SphereNet helps the model capture the dependency between atoms and learn the underlying chemical rules of molec- ular geometries more effectively. In addition, compared with G-SchNet, our G- G-SphereNet achieves lower MMD distances for 4 types of chemical bonds, which shows that our method can model the 3D structural distribution of molecular geometries more accurately. We visualize some molecular geometries generated by G-SphereNet in Figure 2 of Appendix D
4.2 TARGETED MOLECULE DISCOVERY
Setup. In the targeted molecule discovery task, we aim to maximize minimize the expected quan- tum property score. We conduct two targeted molecule discovery experiments, namely minimizing the HOMO-LUMO gap and maximize the isotropic polarizability. Following G-SchNet (Gebauer cet al. 2019). we fine-tune the G-SphereNet model that has been trained in Sec. [.1 on the biased datasets. Specifically, from the QM9 dataset, collect all molecular geometries whose HOMO- LUMO gaps are lower than eV eV and all molecular geometries whose isotropic polarizabilities are higher than 91 Bohr The G-SphereNet model is then fine-tuned on these two biased datasets so as to generate molecular geometries with low HOMO-LUMO gaps or high isotropic polarizabilities, respectively. Details about the model fine-tuning process are summarized in Appendix
In this task, we evaluate the performance by three statistic metrics over the quantum property scores of generated molecular geometries. Specifically, we generate 1000 molecular geometries with the trained model, and filter out the geometries that are not chemically valid. Afterwards. the PySCF (Sun al. 2018 2020) package is used to compute the quantum property scores of valid molecular geometries. The performance is then evaluated by three statistic metrics over these quantum property scores. Formally, we calculate the mean and the optimal value over all property scores, and the percentage of property scores falling into the good region (good percentage in short). The good region is defined as scores lower than 4.5 eV for the HOMO-LUMO gap and scores higher than 91 Bohr² for the isotropic polarizability, respectively. We find that E-NFs (Satorras et al. 2021a) fails to generate enough chemically valid molecular geometries and produce reliable results after fine-tuning, we only compare our G-SphereNet method with G-SchNet in this task.
8

---

Published as a conference paper at ICLR 2022
Table 3: Results of ablation studies evaluated by Validity, (a) Comparison between using local
or/and global features; (b) Comparison of using different 3D information; (c) Comparison of the
focal atom selection by the Sigmoid method (ours) and the Softmax method (Simm al. 2020p.
(a)
(b)
(c)
Local
Global
Validity
Distance
Angle
Torsion
Validity
Setting
Validity
79.93%
74.20%
Softmax
67.7
76.39%
82. 12%
Sigmoid
88.18%
88.18% %
88.18 %
Results. Results of targeted molecule discovery for two quantum properties are summarized in Table 2 For both properties, our G-SphereNet can outperform G-SchNet in all metrics, showing that G-SphereNet can generate more molecular geometries with good properties. Since the two methods use the same pretraining and fine-tuning pipeline, we argue that the better performance of G-SphereNet indicates its stronger ability to search molecular geometries with desirable properties. We illustrate some generated molecular geometries with good properties in Figure 3f Appendix D
4.3 ABLATION STUDIES
In previous sections, we have demonstrated the effectiveness of our G-SphereNet method on two 3D molecular geometry generation tasks. However, is unclear whether some designs in our method, such as the use of global features extracted by multi-head attention networks, can indeed lead to good performance. Hence, we conduct extensive ablation studies justifying the use of both local and global features, and the consideration of distance, angle, and torsion information in the 3D conditional information extraction of G-SphereNet. In addition, we study the effects of different focal atom selection methods. In each ablation study, different variants of G-SphereNet models are trained with the setting in Sec. 4.1 and evaluated by the Validity metric. Table show all the results of ablation studies.
Ablation on local and global feature. We compare with G-SphereNet variants which only use local features, ie., node embeddings extracted by SphereNet, and only use global features, i.e., outputs from multi-head attention networks) in Eqn. (10) and [II.. Results in Table 3(a] show that using only local global features both achieves worse performance.
Ablation on 3D information. To show the advantages of using comprehensive 3D information, we replace the SphereNet by SchNet (Schütt al. (017)) considering only distance information, and DimeNet++ (Klicpera et al. (020) considering only distances and angles, respectively. As presented in Table (bb) missing partial 3D information leads to performance degradation.
Ablation on focal atom selection. In G-SphereNet, an atom-wise MLP and sigmoid function per- forms the atom-wise binary classification finding atoms that are not valence full filled. Then the focal atom is randomly selected from those atoms. In contrast, Simm et al. 2020) proposes to di- rectly select the focal atom by an MLP and softmax function. We discuss the differences between two methods in detail in Appendix C To demonstrate the benefits of our method, we compare it with a G-SphereNet variant replacing our focal atom selection method by the one in Simm et al. (2020). We denote our method by Sigmoid and the method in Simm et al. (2020) by Sofimax. As shown in Table 3(c) our Sigmoid method can achieve much better performance than the Softmax method.
5 CONCLUSION
We propose G-SphereNet, novel autoregressive flow model for 3D molecular geometry generation from scratch. G-SphereNet employs sequential generation pipeline, in which the 3D positions of atoms are obtained through generating the relative distances, angles, and torsion angles. It is flexible and efficient, and ensure both equivariance and invariance properties. In addition, spherical mes- sage passing and attention mechanism are used to extract conditional information during sequential generation. We empirically demonstrate that, compared with previous methods, our G-SphereNet method models the distribution of 3D molecular geometries more accurately, and has stronger ca- pacity search molecules with good properties. In the future, will apply our G-SphereNet more complicated 3D structures, such as proteins and many-body particle systems.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We have included the source codes of our method in DIG (Liu et al. 2021b) library and provided experiment details in Appendix for reproducing the results.
ACKNOWLEDGMENTS
We thank Limei Wang for her help explaining implementation details of the SphereNet model and Xuan Zhang for his guidance on using PySCF package. This work was supported in part by National Science Foundation grant 11S-1908220.
REFERENCES
Ravichandra Addanki, Peter W Battaglia, David Budden, Andreea Deac, Jonathan Godwin, Thomas Keck, Wai Lok Sibon Li, Alvaro Sanchez-Gonzaled Jacklynn Stott, Shantanu Thakoor, et al. Large-scale graph representation learning with very deep GNNs and self-supervision. arXiv preprint arXiv:2107.09422 2021.
Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Le Song. coder for structured data. 7th International Conference on Learning Representations, 2018.
variational autoen-
Nicola De Cao and Thomas Kipf. MolGAN: implicit generative model for small molecular graphs. ICML 2018 workshop Theoretical Foundations and Applications of Deep Generative Models, 2018
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 4th International Conference on Learning Representations, 2016.
Ivan Dokmanic, Reza Parhizkar, Juri Ranieri, and Martin Vetterli. Euclidean distance matrices: essential theory, algorithms, and applications. IEEE Signal Processing Magazine, 32(6):12-30, 2015.
Octavian-Eure Eugen Ganea, Lagnajit Pattanaik, Connor W. Coley, Regina Barzilay, Klavs Jensen, William Green, and Tommi S. Jaakkola. GeoMol: Torsional geometric generation of molec- ular 3d conformer ensembles. In A. Beygelzimer, Y. Dauphin. P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL. https: //openreview. met/fform? d= Jt
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Aalché-Bucc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019.
Tarun Gogineni, Ziping Xu, Exequiel Punzalan, Runxuan Jiang, Joshua Kammeraad, Ambuj Tewari, and Paul Zimmerman. TorsionNet: a reinforcement learning approach sequential conformer search. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20152.0055.. Curran Associates, Inc., 2020.
Rafael
Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Länchez-Lengellngg Dennis Sheberla, Jorge Aguilera-Iparraguiree Timothy Hirzel, Ryan P Adams, and Alán Aspuru-Guzik, Automatic chemical design using data-driven contin- uous representation of molecules. ACS central science, 4(2):268-277 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27, pp. 2672-26880 Curran Associates, Inc., 2014.
10

---

Published as a conference paper at ICLR 2022
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, (3(1):723-773 2012.
Moritz Hoffmann and Frank Noé. Generating valid Euclidean distance matrices. arXiv preprini 1rXiv:191..0313,, 2019.
Shion Honda, Hirotaka Akita, Katsuhiko Ishiguro, Toshiki Nakanishi, and Kenta Oono. Graph residual flow for molecular graph generation. arXiv preprint arXiv: ar0iv:1909. 13521, 2019.
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovee. OGB-
LSC: A lcrle challenge for machine learning on graphs. In Thirty-fifih Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https: //openzeview. net/ d=g.1
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2323-2332, 2018.
Yeonjoon Kim and Woo Youn Kim. Universal structure conversion method for organic molecules: from atomic connectivity to three-dimensional geometry. Bulletin of the Korean Chemical Society, 36(7):166--177,, 2015.
Diederick P Kingma and Jimmy Ba. Adam: of the 3rd international conference on learning representations, 2015.
A method for stochastic optimization. In Proceddings
Diederik P Kingma and Max Welling. Auto-Encoding variational bayes. In 3nd International Conference on Learning Representations, 2014.
Johannes Klicpera, Shankari Giri, Johannes T Margraf, and Stephan Günnemann. Fast and uncertainty-awar directional message passing for non-equilibrium molecules. NeurlPS 2020 Machine Learning for Molecules Workshop, 2020.
Jonas Köhler, Leon Klein, and Frank Noc. Equivariant flows: Exact likelihood generative learning for symmetric densities. Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th ternational Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5361-5370. PMLR, 13-18 Jul 2020.
Matt J. Kusner, Brooks Paige, and José Miguel Hemánneez-Lbatce Grammar variational autoen- coder. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Con- ference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 1945-1954, 2017.
Meng Liu, Cong Fu, Xuan Zhang, Limei Wang, Yaochen Xie, Hao Yuan, Youzhi Luo, Zhao Xu, Shenglong Xu, and Shuiwang Ji. Fast quantum property prediction via deeper 2d and 3d graph networks. In NewrIPS 202/ Al for Science Workshop, 2021a. URL httl.tttts:/ httphhttt:///pppeee. //tp::eee///peeeveee. di FVNL.
Meng Liu, Youzhi Luo, Limei Wang, Yaochen Xic, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang, Yi Liu, Keqiang Yan, Haoran Liu, Cong Fu, Bora M Oztekin, Xuan Zhang, and Shuiwang Ji. DIG: A turnkey library for diving into graph deep learning research. Journal of Machine Learning Research, 22(240):1-9, 2021b. URL /ttp:///mmmrreeeeeeeeeeeee www.ll/.rg//apers//eoe 21 0343.html
Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. GraphEBM: Molecular graph generation with energy-based models. In Energy Based Models Workshop-ICLR 2021, 2021c.
Xianggen Liu, Qiang Liu, Sen Song, and Jian Peng. A chance-constrained generative framework for sequence optimization. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume I 119 of Proceedings of Machine Learning Research, pp. 6271-628 PMLR, 13-18 Jul 2020.

---

Published as a conference paper at ICLR 2022
Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. International Conference on Learning Representa- tions, 2022. URL hpwpr//eeeeeeeeeeeee.eeeee..eeee net wttt://orrmmiiieem t
Youzhi Luo, Keqiang Yan, and Shuiwang Ji. GraphDF: A discrete flow model for molecular graph generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp- 7192-7203. PMLR, 18-24 Jul 2021.
Tengfei Ma, Jie Chen, and Cao Xiao. Constrained generation of semantically valid graphs via regularizing variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bisa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 71 7113-7124. Curran Associates, Inc., 2018.
Kaushalya Madhawa, Katushiko Ishiguro, Kosuke Nakago, and Motoki Abe. GraphNVP: An in- vertible flow model for generating molecular graphs. arXiv preprint wrXiv:1905.11600 2019.
Elman Mansimov, Omar Mahmood, Seokho Kang, and Kyunghyun Cho. Molecular geometry pre- diction using a deep generative graph neural network. Scientific reports, 9(1):1-13, 2019.
Vitali Nesterov, Mario Wieser, and Volker Roth. 3DMolNet: a generative network for molecular structures. arXiv preprint arXiv:2010.066 06477, 2020.
George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 2338- 2347. Curran Associates, Inc., 2017.
Pavel G Polishchuk, Timur Madzhidov, and Alexandre Varnek. Estimation of the size of drug- like chemical space based on GDB-17 data. Journal of computer-aided molecular design, 27(8): 675-679, 2013.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp. and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learn- ing, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1138, Lille, France, 2015.
Victor Garcia Satorras, Emiel Hoogeboom, Fabian Bernd Fuchs, Ingmar Posner, and Max Welling. E(n) equivariant normalizing flows. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021a. URL https: //openreview. wwt.:nootmmddddSetoe..... NSWVA__oOOOWWAAAAA_-
Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural net- works. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp- $323-9332. PMLR, 18-24 Jul 2021b.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmicla, Alexandre Tkatchenko, and Klaus-Rober Müller. SchNet: A continuous filter convolutional neural net- work for modeling quantum interactions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Chence Shi*, Minkai Xu*, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. GraphAF: a flow-based autoregressive model for molecular graph generation. In 8th International Conference on Learning Representations, 2020.
Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular confor- mation generation. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 9558-955 9568. PMLR, 18-24 Jul 2021.
12

---

Published as a conference paper at ICLR 2022
Gregor Simm and Jose Miguel Hemnanez--Lbato. A generative model for molecular distance ge- ometry. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Con- ference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp- 8949-8958 PMLR, 13-18 Jul 2020.
Gregor Simm, Robert Pinsler, and Jose Miguel Hernandez-Lobato. Reinforcement learning for molecular design guided by quantum mechanics. In Hal Daumé III and Aarti Singh (eds.), Pro- ceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 8959-8999999 PMLR, 13-18 18 Jul 2020.
Gregor N. Simm, Robert Pinsler, Gábor Csányi, and José Miguel Hernández-Lobar Symmetry- aware actor-critic for 3d molecular design. In International Conference on Learning Representa- tions, 2021. URL wttttts:://ppeeeeeee
Martin Simonovsky and Nikos Komodakis. GraphVAE: Towards generation of small graphs using variational autoencoders. In Věra Kürková, Yannis Manolopoulos, Barbara Hammer, Lazaros Iliadis, and Ilias Maglogiannis (eds.), Artificial Neural Nenworks and Machine Learning ICANN 2018, pp. 412-422, Cham, 2018. Springer International Publishing.
Jonathan M Stokes, Kevin Yang. Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar Bloom-Ackernan, et al. A deep learning approach to antibiotic discovery. Cell, 188(4))688-702 2020.
Qiming Sun, Timothy C Berkelbach, Nick S Blunt, George H Booth, Sheng Guo, Zhendong Li, Junzi Liu, James McClain, Elvira Sayfutyarova, Sandeep Sharma, et PySCF: the python-based simulations of chemistry framework. Wiley Interdisciplinary Reviews: Computational Molecular Science, 8(1)xe1340, 2018
Qiming Sun, Xing Zhang, Samragni Banerjee, Peng Bao, Marc Barbry, Nick S Blunt, Nikolay A Bogdanov, George H Booth, Jia Chen, Zhi-Hao Cui, et al. Recent developments in the PySCF program package. The Journal of chemical physics, 153(2):024109, 2020.
Dustin Tran, Keyon Vafa, Kumar Agrawal, Laurent Dinh, and Ben Poole. Discrete flows: Invertible generative models of discrete data. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 14719-14728. Curran Associates, Inc., 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 5998-6008 Curran Associates, Inc., 2017.
Zhengyang Wang, Meng Liu, Youzhi Luo, Zhao Xu, Yaochen Xie, Limei Wang. Lei Cai, Qi Qi, Zhuoning Yuan, Tianbao Yang. and Shuiwang Ji. Advanced Graph and Sequence Neural Net- works for Molecular Property Prediction and Drug Discovery. Bioinformatiis, 02 2022. ISSN 1367-4803. doi: 00.1093//ionnorrmatis/aaaa 12. URL https: setpss://di... htttps://@ci.org//11 wrg//0..... 1093/ bioinformat ics ics/btacl b2ac1
David Weininger. SMILES, chemical language and information system. 1. introduction to method- ology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-33) 1988.
Minkai Xu, Shitong Luo, Yoshua Bengio, Jian Peng, and Jian Tang. Learning neural generative dynamics for molecular conformation generation. In International Conference on Learning Rep- resentations, 2021a. URL ltttps::/
net/ forum fott:/oormmiddpoommo qtheGk
Minkai Xu, Wujie Wang, Shitong Luo, Chence Shi, Yoshua Bengio, Rafael Gomez-Bombaree and Jian Tang. An end-to-end framework for molecular conformation generation via bilevel program- ming. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 11537- 11547. PMLR, 18 24 Jul 2021b.
13

---

Published as a conference paper at ICLR 2022
Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Andrew Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3D: A benchmark for predicting 3d geometries from molecular graphs. 2021c. URL
forumiidd
In5rEiGx0GiT.
Chengxuan Ying, Mingqi Yang, Zheng Shuxin, Ke Guolin, Luo Shengjie, Cai Tianle, Wu Chenglin, Wang Yuxin, Shen Yanming, and He Di. First place solution of KDD Cup 2021 & OGB large- scale challenge graph prediction track. arXiv preprint arXiv:2106.0827,, 2021.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Desa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 9410-6421. Curran Associates, Inc., 2018.
Chengxi Zang and Fei Wang. MoFlow: an invertible flow model for generating molecular graphs. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 317-626, 2020.
14

---

Published as a conference paper at ICLR 2022
A PROOF OF THE EQUIVARIANCE AND INVARIANCE
In G-SphereNet, at the i-th generation step, we generate the relative distance d,, 0, and pi by the model
d; 0, p. 24, A. Ri).
(13)
because we Without loss of generality, here we assume 2 3. Note that g is model, hence have extract features using commetry-invariann SphereNet (Liu al. 2022)
d, 0, pt = (z), 2f: A., R²Q" 16²)
(14)
for any orthogonal matrix Q E 33x3 and translation vector E IR3. The coordinate r; of the new atom is then computed from di, 0i, Pi and the coordinates ry, Te- Next, show that this generation process satisfies the equivariance and invariance properties.
Theorem 1. If T. = g (z] Ai, R.). then we have Q + = g" A,, R,Q + 16b) for any orthogonal matrix Q E 3x33 and translation vector b E IR3.
Proof. Here, we can get that 2 (2 2/. 2..) and
g g A; R₂) = h (g (= =f A,, Ra) irr,
where the function h(d. 0 9: r, re) is defined as
cos A(r - ry) + - ||re -
dsin O - re,of)
h(d, f, piry, Te, +
Te,ef 12
In this equation, Te.ef is the coordinate of the projection of on the line (r,, re), and Te.p is the coordinate of e after rotating the plane (ry, along the line (ry, by the torsion angle p.
For any orthogonal matrix Q
3x33 because ||Qr112 TT Qr =r 11r122 we have
h(d, 0, pi Qr, + b, Qr. + 6, Qre b)
Qry doos OaQ Qr1)
dsin O(Qrevp Qre
10Q, - Qr
1|Qres - Qra,t
cos 0 rg)
dsin Or Te,es)
=Qry +b+ Q
+
(15)
Irr ry
I|rew - reoss
dcos A(r. rf)
dsin ((re,p re.ef)
=Q r |Ire --e-7 11
+
I|Te,p - Te,erl13
Qhi(d, Fer +b.
Combining the conclusions from both Eqn. and 15 g" ( R²Q" + 1b²") holds for any orthogonal matrix Q E R3383 and translation vector
we easily get that Qr, +
be 133
Theorem For any orthogonal matrix Q p (Qr; + b|A, R2Q"* +16") (r.) R.).
RR333 and translation vector
R we have
Proof. We can easily find that the relative distance d, angle 0,, and torsion angle p; will not change if we transform r; and R, with the same Q and b. In addition, from Eqn. (14), we have
2 2f = g- (d,, G., sp:: A., R²Q" 1b").
(16)
Hence, the corresponding latent variables 21 2/ 2/ are invariant to the rotation and translation transformation. Since p(r:)A, R.) = pz, (2.(2 |,, R, R,)p2 (2f1A, R,)p2, (2 IA, R,). we can get that (Qr, + b)A,, R²Q" + 16²) = p (rilA, R2) is right for any orthogonal matrix Q E g3x3 and translation vector E 3
15

---

Published as a conference paper at ICLR 2022
B GENERATION AND TRAINING ALGORITHM
Algorithm Generation Algorithm of G-SphereNet
1: Input: G-SphereNet model, latent distribution pz., pZa+ PZy pz, maximum number of atoms
n
2:
3: Initialize molecular geometry G1 with one carbon atom, whose coordinate is ro [0,0,00 0]
4: for 1 to n do
5: 2," P%.
6: Generate a; from z
7: Get the candidate focal atom set by the atom-wise classifer.
8:
if the candidate focal atom set is empty then
9: Output G;
10: else
11:
Random select the focal atom from the candidate focal atom set
12:
end if
p7a+ 2 p2, (if 2 2). 2 pz, (if 3) zạ
Generate d, 0, (if 2 2), fr (if 2 3) from 2/ = =
15: Get r; from d, 8, po; and rf.
16: Add a new node with type a, and coordinate r; to G, and set the updated geometry as G
17:
end for
18: Output G,,
Algorithm Training Algorithm of G-SphereNet
1: Input: Molecular geometry dataset M, G-SphereNet model with trainable parameter w, latent distribution pz., PZ PZa pz,, learning rate a, batch size B
2:
3: repeat
4: Sample a batch of B molecular graphs G from M
5:
L=0
6:
for G G do
7:
Set n as the number of atoms in G
8:
Order the atoms in G
9:
for
10:
Get ag d,, 0 (if i 2 2), Pi (if 2 3)
11:
_d zf (if 2 2), 2 (if 2 3) Get 27, 2j
I. log (2
log (2/)) L log (2) if 2
14:
L -- L. log (2 if
Add the binary cross entropy loss for the focal atom selection to L
16:
end for
17:
end for
18:
L
19:
2
L
20:
until w is converged
21:
Output G-SphereNet model with parameter w
16

---

Published as a conference paper at ICLR 2022
C EXPERIMENT DETAILS
Model configuration. For conditional information extraction, the input 3D molecular geometry is first processed into a cutoff graph. Specifically, any two nodes whose distances are lower than 5.0 Á are connected. The node features are initialized to the one-hot vectors of atom types and the edge features are initialized by spherical basis functions as in Liu et (2022). We use the SphereNet (Liu et al.) (2022) model with 4 layers to extract features from the input geometry, where the input embedding size set to 64 and output embedding size set to 256. Afterwards, global features are extracted by the multi-head attention network with 4 attention heads. In addition, we employ 6 flow layers. Such model configuration is used for all experiments.
Training and generation details. In the random molecular geometry generation task, the G- SphereNet model is trained with Adam optimizer for 100 epochs, where the learning rate 0.0001 and the batch size is 64. We report the results corresponding to the epoch with the best validation loss. In the target molecule discovery task, the model is fine-tuned with a learning rate 0.0001, batch size of 32. The number of training epochs is 40 for the HOMO-LUMO gap and 80 for the isotropic polarizability. During generation, we use temperature parameters in the prior Gaus- sian distributions. Specifically, we change the standard deviation of the Gaussian distribution the temperature parameter. We use 0.5 for sampling zi 2 1), 0.3 for sampling zi 2 1), 0.4 for sampling 2 (f 2 2). and 1.0 for sampling 2 (i 3).
Focal node selection. In G-SphereNet, the atom- wise MLP takes the feature of each atom input, and outputs a binary classification score. We consider the atoms whose scores are higher than 0.5 being classified candidate focal atoms, and select the exact focal atom from candidate focal atoms. In our experiments, we do not observe any unstable training for this classifier. This atom- wise classifier is trained classify whether an atom not valence full filled so that it can be candidate for next focal atom selection or not. We believe it is not a hard binary classification task so there no unstable training around the threshold and the training is not sensitive to the threshold choice. Differently, in Simm et al. (2020), the MLP and softmax function directly outputs the probability of being selected to the focal atom for each atom. This MLP is trained by using the sampled focal atom as targets. From the results in Table 3(c) we can clearly find that our method results in much higher chemical validity than the method in Simm al. (022)).
17

---

Published as a conference paper at ICLR 2022
D MORE EXPERIMENT RESULTS
Figure 2: An illustration the sample molecular geometries generated by the G-SphereNet model trained in the random molecular geometry generation task.
(a) Generated by the model trained to minimize HOMO-IUMO gap.
(b) Generated by the model trained to maximize isotropic polarizability.
Figure 3: An illustration of the sample molecular geometries generated by the G-SphereNet model trained in the targeted molecule discovery task.
18


Reviews:
[{"review_id": "cI4OOnn_-id", "reviewer": "Reviewer_JRzq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of the paper is in (i) showing that generating a 3D molecular structure from scratch is feasible, and (ii) in demonstrating that a sequential approach is particularly effective for this task. Both aspects are novel to my knowledge, and worthy of communication to the relevant community.\n\nThe paper is fluent, clear, and easy to follow. The positioning with respect to the literature is also rather clear, and the analogies and differences with existing approaches are well described. The experimental section is well executed, and includes an extensive ablation study confirming the validity of the proposed pipeline in its entirety. Comparisons with very recent methods addressing the same task are included. Code is also included.\n\nParticularly significant are the quantitative results, which demonstrate an important gap in accuracy compared with other approaches (>10% in most cases).\n\nOne aspect that I feel might benefit from additional discussion is on the rationale behind a sequential model being more effective for generation. The paper attempts to give some motivations (e.g. that it helps to capture the relation between atom and density more effectively), but additional insight would be useful. If the authors can share more of their insight during the discussion period, it would help my understanding of the approach.\n\nPerhaps more importantly, something that does not really exude from the paper is the potential impact that the method could have on practical applications. The conclusions mention that a promising future direction would be the generation of more complex molecules (such as proteins), but at the current stage, what is the main benefit of this method for downstream tasks in structural biology? If this is to be taken as a first step toward practical applications is also fine, but it would be nice to see this clarified in the manuscript.\n\nAs a minor comment, it is not clear to me how Figures 2 and 3 (in the supplementary material) should be interpreted. Is there a way to read these figures, that suggests that the generative model is doing a good job at synthesizing new molecules?", "summary_of_the_paper": "The paper introduces a new generative model for 3D molecular geometry, taking the form of a graph embedded in 3D where each node is an atom, and each edge is a bond. Differently from existing methods, the proposed pipeline attempts to generate the 3D molecule from scratch, instead of starting, e.g., from a 2D molecular graph as other methods do. The pipeline is sequential and is based on autoregressive flow models, which have been previously used for the generation of 2D molecular graphs, but not for the present task. The paper convincingly shows that the proposed model works better than competing approaches for the same task.", "summary_of_the_review": "This is a well written paper, with a novel contribution that is likely to leave an impact in the emerging field of generative models for molecular structures. The main weakness, in my opinion, lies in the complexity of the generated molecules, which at this point is not clear whether it can lead to relevant practical applications or not. But overall, I am still leaning positive in the light of the gap in accuracy with respect to previous methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UQX659jF9pn", "reviewer": "Reviewer_1x5L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\n1. The paper proposes a good way to handle the invariance issue during generating 3D positions of the atoms in the molecule by outputting related position change (d, theta, phi) instead of outputting absolute coordinate r. They have given thorough reasoning in the paper why they should not use r.\n\n2. The paper has shown good results compared with previous methods and has done several ablation studies to show that different components in their method are useful.\n\nCon:\n1. The contribution seems limited. Most of their pipeline is based on previous work including (Liu et al. 2021d) and (Vaswani et al. 2017).\n\n2. The writing of the paper is a little bit messy. Some related algorithm like (Vaswani et al. 2017) is shown with proper math formulations, while some others like the message passing in (Liu et al. 2021d) are missing. It's better to rearrange the paper and make it more self-contained.", "summary_of_the_paper": "The paper proposes a new way to directly generate 3D molecular geometries from scratch. They output the atoms one by one, including the type of the atom and the related position from one chosen focal atom. They use a SphereNet (Liu et al. 2021d) - a spherical message-passing network and an additional multi-head attention network (Vaswani et al. 2017) to process the existed information in the partial generation results. They have shown in 2 different tasks that they are better than previous methods.", "summary_of_the_review": "The paper has improvements compared to previous work based on the system design and new algorithms, however, some critical parts seem incremental.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Sg8tL8SU9c1", "reviewer": "Reviewer_UA8J", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In G-SphereNet, the authors propose to generate the new atoms using a triplet of (d, theta, phi), which can be seen as spherical coordinates in a local system defined by the focal atom. Such a design seems to simplify the generating process. But such representation has a known issue of Gimbal lock, i.e. when theta = 0, phi no longer affects the atom generation, which I assume happens a lot in the molecule generation process. Despite pairwise distances used in EDMNet/3DMolNet might now indicate a valid 3D point, it doesn't necessarily matter to the generating process as long as (1) there always be a deterministic solution; (2) the distances are SE(3) equivariant/invariant. Compared to G-SchNeet, it makes sense to only generate new atoms near the focal atom. The constraint might serve as a regularizer to generate consistent/meaningful molecules and better capture the dependency between atoms. Most of the benefits listed in section 3.3 remain unproven. In my opinion, regarding the representation of the relative atom position, the author should devise a way to validate the necessity and the benefits of this contribution.\n\nOne thing I am confused about is that in Table 3(b), the performance with SchNet backbone is lower than G-SchNet. Does it indicate that the other contributions are actually lowering the performance of G-SchNet?\n\nThe organization of the paper structure could be improved. For example, (1) shorten the paragraph that describes the issue of SE(3) invariant in the traditional autoregression flow model; (2) there are two paragraphs describing the focal atom selection (Sec 3.2 and Sec 3.4), which brings confusion about whether the two steps use different focal atoms.\n\nI am not sure if MMD can substitute the original RMSD metric. As the RMSD between the generated model and the relaxed model evaluates the accuracy of the generated model, while MMD between the generated model and the dataset ignores the molecule structures.\n\nThere are a few details not clear to me in the paper:\n(1) Sec 2.3, G-SchNet also uses a flexible sequential generation pipeline based on autoregressive flow models?\n(2) Sec 3.3, what if c and e are mutually closest points?\n\nSome typos:\nGaussian distribution N(0, I) --> N(0, 1)\nSec 3.5 last sentence: genration --> generation", "summary_of_the_paper": "The paper addresses the problem of generating 3D molecular geometries.\nThe proposed G-SphereNet takes SphereNet as the backbone to generate 3D molecular geometries from scratch.\nThe generation is done in a step-by-step fashion, with each step generating a (distance, angle, torsion) triplet, conditioning on features extracted by an attention mechanism.", "summary_of_the_review": "Overall, the paper builds largely based on the setting of G-SchNet. There are a few designs to improve the performance. However, the benefits of the designs are not thoroughly/fairly evaluated.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HhEW37lVdqe", "reviewer": "Reviewer_euwK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOverall the paper does a good job of extending autoregressive flow models to support full 3D molecular geometry. The extension depends on a few relevant insights in how to best parameterize the problem and shows good overall results and improvements over prior methods. As this is not a commonly explored area and outside my expertise, it is hard to judge how significant the improvmeents in Table 2 and related results are, but from my limited understanding they seem sufficiently significant.\n\n\"These conditional generation methods assume that we are given the target molecules in the form of molecular graphs, but we do not know their 3D geometries. However, this assumption does not hold in the targeted molecule discovery problem.\" -- it seemed to me when I read this that it should still be possible to still do targeted molecule discovery, by first generating the molecular graph then generate 3D geometry from the graph  and applying the loss to the generated 3D geometry?\n\nSeveral times the option of \"is randomly chosen from the atoms whose classification scores are higher than 0.5\" is taken, but it's not really described how this operates when there are no classification scores higher than 0.5; and furthermore it seems like this type of classifier might make network training unstable around the threshold, so it would be good to discuss how sensitive training is to metaparameter choice.\n\nI found the ablations useful to demonstrate the different components, and I think molecular design is potentially an important area to focus on more.\n", "summary_of_the_paper": "The paper focuses on a generative model for 3D molecules that generates full 3D molecular gemoetry, instead of generating a molecular graph and then generating the 3D molecular geometry conditional on the graph. The method using an autoregressive flow model, which has been applied to molecular graph generation but not full geometry. This enables some new applications such as molecular design to target a geometry-dependent objective function.\n ", "summary_of_the_review": "\nThe paper extends autoregressive flow models for 3D molecular generation to also generative 3D geometry. This helps with some challenging molecular discovery tasks, and a set of ablations and comparisons demonsrtate the effectiveness of the method.  The main concern is that it is somewhat incremental over the most similar prior work which does not generate 3D molecular geometry.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work introduces an autoregressive flow model that generates molecular geometries by placing one atom at the time. 
In order to preserve the E(3) invariance of the density, successive atom locations are sampled relative to already placed atoms (in a coordinate system described by distance, angle and torsion).
The paper is overall well-written and experimental results are compelling.
Decision: Accept (Poster)