Generating valid Euclidean distance matrices
Authors: Moritz Hoffmann
Frank Noe
Venue:  (2020)
Keywords: euclidean distance matrices
wgan
point clouds
molecular structures

Abstract: Generating point clouds, e.g., molecular structures, in arbitrary rotations, translations, and enumerations remains a challenging task. Meanwhile, neural networks
utilizing symmetry invariant layers have been shown to be able to optimize their
training objective in a data-efficient way. In this spirit, we present an architecture
which allows to produce valid Euclidean distance matrices, which by construction are already invariant under rotation and translation of the described object.
Motivated by the goal to generate molecular structures in Cartesian space, we use
this architecture to construct a Wasserstein GAN utilizing a permutation invariant critic network. This makes it possible to generate molecular structures in a
one-shot fashion by producing Euclidean distance matrices which have a three-
dimensional embedding.

Under review as a conference paper at ICLR 2020
GENERATING VALID EUCLIDEAN DISTANCE MATRICES
Anonymous authors
Paper under double-blind review
ABSTRACT
Generating point clouds, e.g., molecular structures, in arbitrary rotations, transla- tions, and enumerations remains challenging task. Meanwhile, neural networks utilizing symmetry invariant layers have been shown to be able to optimize their training objective data-efficient way. In this spirit, we present an architecture which allows to produce valid Euclidean distance matrices, which by construc- tion are already invariant under rotation and translation of the described object. Motivated by the goal to generate molecular structures in Cartesian space, we use this architecture to construct a Wasserstein GAN utilizing a permutation invari- ant critic network. This makes it possible to generate molecular structures in a one-shot fashion by producing Euclidean distance matrices which have three- dimensional embedding.
INTRODUCTION
Recently there has been great interest in deep learning based on graph structures (Defferrard et al.
2016 Kipf & Welling 2016 Gilmer al. (017)) and point clouds (Qi et al. 2017 Li et al. 2018b prominent application example is that of molecules, for which both inference based on the chemical compound, i.c., the molecular graph structure (Kearnes al. [0166] Janct & Kulik 2017; Winter et al.] 2019a), and based on the geometry, i.c. the positions of atoms in 3D
Yang et al. 2019)..
space (Behler & Parrinello
2007 Rupp et al. 2012 Schütt al. 2017a Smith al. 2017 are
active areas of research.
particularly interesting branch of machine learning for molecules is the reverse problem of gen- erating molecular structures, as it opens the door for designing molecules, e.g., obtain new materi- als (Sanchez-Lengeling
Aspuru-Guzik, 2018 Barnes et al., 2018, Elton et al., 2018 et al.) 2018a). design or discover pharmacological molecules such as inhibitors or antibodies (Popova) (ct al. 2018: Griffen et al., 2018), optimize biotechnological processes (Guimaraes et al. 2017 While this area of research has exploded in the past few years, the vast body of work has been done on the generation of new molecular compounds, i.e. the search for new molecular graphs, based on string encodings of that graph structure or other representations (Gómez Bombarelli al.) 2018 Winter et al., 2019b). On the other hand, exploring the geometry space of the individual chemical compound is equally important, cquilibrium properties, such as binding affinity, solubility etc. Sampling different geometric struc- tures is, however, still largely left to molecular dynamics (MD) simulation that suffers from the rare event sampling problem, although recently machine learning has been used to speed up MD simu- lation (Ribeiro et al., 2018; Bonati et al., 2019 Zhang et al. 2019 Plattner et al., 2017 Doerr & Fabritiis| 2014) or to perform sampling of the equilibrium distribution directly, without MD (Noé ct al. 2019). All of these techniques only sample one single chemical compound geometry space.
the molecular geometries and their probabilities determine all
Here we explore-ta our best knowledge for the first time in depth-the the simultaneous generation of chemical compounds and geometries. The only related work we are aware of (Gebauer et al.) 2018 2019) demonstrates the generation of chemical compounds, placing atom by atom with an autoregressive model. It was shown that the model can recover compounds contained the QM9 database of small molecules (Ruddigkeit et al. 2012 Ramakrishnan al. 2014) when trained subset, but different configurations of the same molecule were not analyzed.
While autoregressive models seem work well in the case of small (< 9 heavy atoms) molecules like the ones in the QM9 database, they can be tricky for larger structures as the probability to completely form complex structures, such rings, decays with the number of involved steps.

---

Under review as a conference paper at ICLR 2020
To avoid this limitation, in our method investigate in shot models for point clouds which have no absolute orientation, i... the point cloud structure is considered to be the same independent of its rotation, translation, and of the permutation of points.
natural representation, which is independent of rotation and translation is the Euclidean distance matrix, which is the matrix of all squared pairwise Euclidean distances. Furthermore, Euclidean distance matrices are useful determinants of valid molecular structures.
While symmetric and non-negative matrix with zero diagonal easily be parameterized by, e.g., a neural network, it is not immediately clear that this matrix indeed belongs to set of n points in Euclidean space and even then, that this space has the right dimension.
Here we develop a new method to parameterize and generate valid Euclidean distance matrices without placing coordinates directly, hereby taking away lot of the ambiguity.
We furthermore propose a Wasserstein GAN architecture for learning distributions of pointclouds, e.g., molecular structures invariant rotation, translation, and permutation. To this end the data distribution as well as the generator distribution are represented in terms of Euclidean distance ma- trices.
In summary, our main contributions are as follows:
We introduce a new method of training neural networks so that their output are Euclidean distance matrices with predefined embedding dimension.
We propose a GAN architecture, which can learn distributions of Euclidean distance ma- trices, while treating the structures described by the distance matrices as set, i.e., invariant under their permutations.
We apply the proposed architecture to a set of C7O2H isomers contained in the QM9 database and show that it can recover parts of the training set well generalize out of it.
GENERATING EUCLIDEAN DISTANCE MATRICES
We describe a way to generate Euclidean distance matrices D E EDM" C g without placing coordinates in Cartesian space. This means in particular that the parameterized output is invariant to translation and rotation.
A matrix D is in EDM" by definition if there exist points 2i,..., nn E Fd such that Diy ||,; 13 for all i,j I,..., Such matrix D defines structure in Euclidean space R up to a combination of translation, rotation, and mirroring. The smallest integer d > 0 for which a set of points in Rd exists that reproduces the matrix D is called the embedding dimension.
The general idea of the generation process is to produce hollow (i.e., zeros on the diagonal) sym- metric matrix D and then weakly enforce D E EDM through a term in the loss. It can be shown that
1 - positive semi-definite,
D E EIDM" + 1
T and 1 (I,..., 1)T E 1 (Schoenberg, 1935 Gower 1982). However where - 1- 111
trying to use this relationship directly in the context of deep learning by parameterizing the matrix D poses problem, as the set of EDMs forms a cone (Dattorro (2010)) and not a vector space, which the underlying assumption of the standard optimizers in common deep learning frameworks. One can either turn to optimization techniques on Riemannian manifolds (Zhang et al.] (2016) or find reparameterization in which the network's output behaves like vector space and that can be transformed into an EDM.
Here, leverage a connection between EDMs and positive semi-definite matrices Alfakih et al. 1999]: Krislock & Wolkowicz (012)) in order to parameterize the problem in a space that behaves like vector space. In particular, for D EDD. by definition there exist points æ D,, Bd generating D. The EDM D has corresponding Gram matrix M E B by the relationship
Mi = (y:. ,,)2 = (Dis + Da - Dτ)
(2)
2

---

Under review as a conference paper at ICLR 2020
with U 2/ 21, ho 1,.................... and vice versa
Do Ma + M,; 2M
(3)
The matrix M furthermore has a specific structure
M (
0
(4)
L
with E Pa-1xn-I and is symmetric and positive semi-definite. therefore admits an eigen- value decomposition M = USUT (UVS)(UVS)) = YYT which, assuming that $ = diag(A1, An) with À1 2 X2 2 ... 2 A reveals composition of coordinates Y in the first d rows where d is the embedding dimension and the number of non-zero eigenvalues of M associated D.
Therefore, the embedding dimension d of D is given by the rank of M or equivalently the number of positive eigenvalues. In principle it would be sufficient parameterize symmetric positive semi-definite matrix L E R= p=-1xn-1, as it then automatically is also Gram matrix for some set of vectors. However, also the set of symmetric positive semi-definite matrices behaves like a cone, which precludes the use of standard optimization techniques.
Instead, we propose to parameterize an arbitrary symmetric matrix E R=-1xn-1, as the set of symmetric matrices behaves like vector space. This symmetric matrix can be transformed into symmetric positive semi-definie matrix
A1
1((1))
L L= 9
U
UT
U
UT
(5)
Ap-1
g(An
by any non-negative function g(.) and then used to reconstruct D via and (4)
This approach is shown in Algorithm| D for the context of neural networks and the particular choice of = sp, the softplus activation function. A symmetric matrix L is parameterized and transformed into a Gram matrix M and a matrix D. For M there a loss in place that drives it towards a specific rank and for D we introduce penalty negative eigenvalues of ( It should be noted that g(-) can also be applied for the largest eigenvalues and explicitly set to 0 otherwise, in which case the rank of M is automatically s d. In that case it is not necessary apply Irank-
Algorithm Algorithm to train generative neural network (in general non-uniformly) sample Euclidean distance matrices based on the neural network G, where N₂ is the dimension of the input vector, m the batch size, and n the number of points place relative to one another.
1: Sample A00 1) i.e. sample from simple prior distribution,
2: Transform X G(z) E ( (1)xm-1))))))))))------- via neural network G,
3: for iari=1 to m do
4: Symmetrize L.
(X; + XT)
5: Make positive definitf t sp (L) with 5
6: Assemble M (T) with 44
7:
Assemble D = ((M) with
8:
Compute eigenvalues pI...., Mn of - DJD, see !
9:
Ledm -= ReLU(-uu)
(5)
10: Compute eigenvalues A1,...,
......,,,, An M such that a 2 A 2 ...
11:
(6)
Σx=d+14 X
end for
13:
L+ MI 1 -
100 + 72 DTh T
L Tank
14:
Optimize weights of G with respect to V..
3 EUCLIDEAN DISTANCE MATRIX WGAN
We consider the class generative adversarial networks Goodffllow et al. (2014)) (GANs) and in partieular Wasserstein GANs (WGANs), i.e., the ones that minimize the Wasserstein-1 distance
3

---

Under review as a conference paper at ICLR 2020
in contrast to the original formulation, where the former be related to minimizing the Jensen- Shannon divergence Arjovsky G(-), which transforms prior distribution- in our into a target distribution IPg which should match the data's underlying distribution I as closely as possible. The other network is so-called critic network C(+) E IR, which assigns scalar values to individual samples from either distribution. High scalar values express that the sample is believed belong P,, low scalar values indicate Pg. The overall optimization objective reads
A et (201).. WGANs consist of two networks, one generator network
vector of white noise & _N 1)*
min max Ep, [C(x)] Ea-,, , [C(x)] CED
(6)
where 'D is the set of all Lipschitz continuous functions with a Lipschitz constant L < 1. We enforce the Lipschitz constant using a gradient penalty (WGAN-GP) introduced in Gulrajani et al. (2017). One can observe that the maximum in Eq. (6 is attained when as large as possible values are assigned to samples from P, and as small as possible values to samples from Pgg. Meanwhile the minimum over the generator network G tries to minimize that difference, which turns out to be exactly the Wasserstein-1 1 distance according to the Kantorovich- ciattrovicc-Rubiisteeen duality (Villani (2008). ). Since the Wasserstein-1 distance is a proper metric of distributions, the generated distribution P. is exactly the data distribution P, if and only the maximum Eq. (6) zero. The networks and C' are trained in an alternating fashion.
We choose for the critic network the message- paaaggggg neural network SchNet (Schütt al. 2 2018) CschNet (-), which was originally designed compute energies of molecules.
It operates on the pairwise distances (v (DDi)) D E EDDM" in structure and the atom types T". If there is no atom type information present, these can be just constant vectors that initially carry information. These atom types are then embedded into a state vector and transformed with variable sharing across all atoms. Furthermore there are layers in which continuous convolutions are performed based on the relative distances between the atoms. In a physical sense this corresponds to learning energy contributions of, e.g., bonds and angles. Finally all states are mapped to a scalar and then pooled in
Due to the pooling and the use only relative distances but never absolute coordinates, the output invariant under translation, rotation, and permutation.
The generator network G employs the construction of Section to produce approximately EDMs with fixed embedding dimension. Therefore this architecture is able to learn distributions of Eu- clidean distance matrices.
4 APPLICATION AND RESULTS
The WGAN-GP introduced Sec. is applied subset of the QM9 dataset consisting of 6095 isomers with the chemical formula C70,H10- To this end the distribution not only consists of the Euclidean distance matrices describing the molecular structure but also of the atom types. The generator produces an additional type vector in multi-task fashion which is checked against constant type reference with a cross-entropy loss. particular this means that there is another linear transformation between the output of the neural network parameterizing the symmetric matrix L (see, e.g., 5 and a vector E | which is due to the use of softmax non-linearity probability distribution over the types per atom. This probability distribution compared against a one-hot encoded type vector tre representing the type composition in the considered isomer with cross entropy term (t, t.er). ). As this example the chemical composition never changes, the type vector tyee is always constant.

---

Under review as a conference paper at ICLR 2020
1,20
dstances
0.50
0.35
02
0.20
1.20
0.75
0.50
0.25
0.0
0:0
0.4
dettnnne tampled
driance simpied
0.2
64 1
24
34
68 76
04
14
24 4.
56
Figure 1: Distribution of pairwise distances between different kinds of atom type after training a Euclidean distance matrix WGAN-GP (Sec. 3 on the C7O2H10 isomer subset of QM9.
Furthermore the prior of a minimal distance between atoms is applied, i.e., we have a loss penalizing distances that too small. Altogether optimize the losses
Leritic E(DA))P, [C(D,t)] E(D,)), [C(D,t)
(original WGAN loss)
(7)
+ ALGP
(gradient penalty of WGAN-GP)
(8)
+ Eannn [C(D,t)
(drift term Karras et al. (0177)
(9)
Lgen EiD.ypp, [C(D,t]
(original WGAN loss)
(10)
B(D,)=? [H(t, (ret)
(cross entropy for types)
(11)
k B.D.Y
Σ Dig r)2 itj
(harmonic repulsion)
(12)
+ Lodm
(for EDMs, see Alg.
(13)
with C(-) being a SchNet critic, oe reference type order, À = 10, Edrint = 10-3, 1003,,0 10, and r being the minimal pairwise distance achieved in the considered QM9 subset. The drift term leads to critic values around for real samples, as otherwise only the relative difference between values for real and fake samples matters. Although in principle the cross entropy loss (1I) is not required found in our experiments that it qualitatively helps convergence. The generator network G(.) uses a combination of deconvolution and dense layers.
The function g(-) ensuring positive semi-definiteness 5 was chosen to be the softplus activation g = sp for the largest three eigenvalues and we explicilly set all other eigenvalues to zero. This leads to a Gram matrix with exactly the right rank and the constraint does not need to be weakly enforced anymore in the generator's loss.
Prior to training the data was split into 50% training and 50% test data. After training on the training data set we evaluate the distribution of pairwise distances between different types of atoms, see Fig. T The overall shape of the distributions is picked up and only the distance between pairs of oxygen atoms are not completely correctly distributed.
After generation we perform computationally cheap validity test by inferring bonds and bond orders with Open Babel Boyle et al.] (20IT). On the inferred bonding graph we check for con- nectivity and valency, i.e., if for each atom the number of inferred bonds add up to its respective valency. This leaves us with roughly 7.5% of the generated samples.
For the valid samples we infer canonical SMILES representations which are a fingerprint of the molecule's topology in order to determine how many different molecule types can be produced us- ing the trained generator. Fig. Zishows the cumulative number of unique SMILES fingerprints when producing roughly 4000 valid samples. It can be observed that the network is able to generalize out of the training set and is able to generate not only topologies which can be found in the test set but also entirely new ones. Nevertheless, while the GAN implementation of our approach gen- erates substantial diversity in configuration space, the diversity in chemical space is limited. This
5

---

Under review as a conference paper at ICLR 2020
30
20
found In training set (8)
15
new molecules (31)
10
5-
500
1000
1500
2000
25000
3000
3500
4000
Figure 2: Number of unique molecular structures in terms of their topology for roughly 4000 valid generated samples and whether they could be found in the training set (blue), the test set (orange), or had a new topology altogether (green).
can be improved by better hyperparameter selection or choosing different generative structures, e.g., (Gebauer al. 2018, 2019).
While SMILES can be used to get an idea about the different bonding structures that were generated, contains no information about different possible conformations these bonding structures. To analyze the number of unique conformations that were generated, we compared each generated structure against all structures the considered QM9 subset. Since the architecture designed in such way that it is permutation invariant, i.c., applying the critic onto matrix D (Da,)).j= and D, (D (()()).ot))EE Yi j= for some permutation yields the same result, one first has find the best possible assignment of atoms.
To this end, apply the Hungarian algorithm Kuhn 1955) onto matrix C' R for EDMs D1, D2 and type vectors tı, t2 E Rn with
1 Ei,1 - 1 (D)aa)) if (t)); (12) O.j 00
(14)
otherwise.
Intuitively this means that the cost of assigning atom in the first structure to atom j the second structure depends whether their atom types match, in which case compare the mean distance from the i-th atom to all other atoms in its structure to the same quantity for the j-th atom in the second structure. If the atom types do not match, we assign a very high number so that this particular mapping is not considered. After we have found an assignment between the atoms, we superpose the structures using functionality from the software package MDTraj (McGibbon al. (2015)) and evaluate the maximal atomic distance between all heavy atoms (i.e., carbons and oxygens) after alignment. The cutoff which we consider structure be distinct conformation is maximal atomic distance between heavy atoms of more than deutoff = 0.6 i.c., more than half carbon- carbon bond length.
The results of this analysis are depicted in Fig. 3 One can observe that although the reported number of unique molecular structures via SMILES is rather low, under our similarity criterion lot of different valid conformations are discovered; in particular also some new conformations of structures that were already contained in the QM9 database.
Finally, we also check for the approximate total energies of the generated molecules compared to the database's. s. To this end, we use the semi-empirical method provided by the software package MOPAC Stewart| (1990) relax all structures in the considered QM9 subset well as all valid generated structures, see Fig. 4 can be observed that after relaxation all energies are contained within the same range of roughly - -1586 - 11581 eV. This means that architecture is capable to propose structures which after relaxation have energies comparable to the ones in the database.
In Figure 5 we show examples of generated molecules the top row (A-D) and the closest re- spective matches in the QM9 database in the bottom row (D')."). The closeness of match was

---

Under review as a conference paper at ICLR 2020
contomationn
new conformations for known structures (total 14)
contormation
10
107
35.00
Figure 3: Unique generated conformations up to a maximal heavy atom distance cutoff deutoff = 0.6 Å after assignment and superposition. We distinguish the categories of known conformations in the considered subset of the QM9 database (blue), new conformations for contained molecular structures (orange), and distinct conformations for molecular structures that not contained.
QM9
discovered conformations conformations new molecules
10a
10-1
10-2
-1585
-1584
-1583
-1582
-1811
energy
Figure 4: Total energies of structures that were relaxed with the semi-empirin method implemented by MOPAC, particular for molecules contained the considered QM9 subset (blue), structures that correspond to new conformations for contained molecules (orange), and unique conformations that belong to new molecules.

---

Under review as a conference paper at ICLR 2020
A
B
c
A'
B'
D
Figure 5: Sampled structures with the Euclidean distance matrix WGAN. Top row A to D are gener- ated samples, bottom row A' to D' are closest matches from QM9 database. Generated molecules A and B could be matched with A' and B' up to maximum atom distance of 0.6 Generated molccules and D are new molecular structures with their closest matches C' and D', respectively.
determined by the maximal atomic distance after assignment of atom identities and superposition. Configurations A and B could be matched with maximal atomic distance of less than deutoff.
5 CONCLUSION AND DISCUSSION
We have developed way parameterize the output of neural network that it produces valid Fuclidean distance matrices with a predefined embedding dimension without placing coordinates in Cartesian space directly. This enables of the described object. Given a network that is able to produce valid Euclidean distance matrices introduce a Wasserstein GAN that can learn one-shot generate distributions of point clouds irrespective of their orientation, translation, or permutation. The permutation invariance is achieved by incorporating the message passing neural network SchNet as critic.
be naturally invariant under rotation and translation
We applied the introduced WGAN to the C7O2H10 isomer subset of the QM9 molecules database and could generalize of pairwise distances in this set of molecules.
the training set as well achieve good representation of the distribution
In future work we want to improve on the performance of the network on the isomer subset as well as extend it to molecules of varying size and chemical composition. We expect the ideas of this work to be applicable for, e.g., generating, transforming, coarse graining, or upsampling point clouds.
To improve the quality of the generated molecular structures a follow up of this work would be including a force field so that more energetically reasonable configurations are produced and includ- ing penalty terms which favor configurations that do not contain invalid valencies, i.c., produce valid molecular structures. Furthermore optimizing a conditional distribution for particular molecule- ie., conditioning the molecular graph and exploring its conformations is natural extension of this work.
REFERENCES
Abdo Y Alfakih, Amir Khandani, and Henry Wolkowicz. Solving euclidean distance matrix com- pletion problems via semidefinite programming. Computational optimization and applications, 12(1-3):13-00, 1999.
8

---

Under review as a conference paper at ICLR 2020
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks.
In Interaational conference on machine learning, pp. 214-224222 214--23,20 2017.
Brian C Barnes, Daniel C Elton, Zois Boukouvalas, DeCarlos E Taylor, William D Mattson, Mark D Fuge, and Peter W Chung. Machine learning of energetic material properties. arXiv preprint arXiv:/807.06156. 2018.
J. Behler and M. Parrinello. Generalized neural-network representation of high-dimensional potential-energy surfaces. Phys. Rev. Lett., 98:146401, 2007.
L. Bonati, Y.- Y-Y.Zhang, and M. Parrinello. Neural networks-based variationally enhanced sampling. Proc. Natl. Acad. Sci. USA, DOI: 10. 11.117//paxx1/90779777333 2019.
Jon Dattorro. Convex optimization & Euclidean distance geometry. Lulu.com, 2010.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems, pp. 3844-3852, 2016.
S. Doerr and G. De Fabritiis. On-the-fly fly learning and sampling of ligand binding by thuugt molecular simulations. J. Chem. Theory Comput., 10:2064-2069, 2014.
Daniel C Elton, Zois Boukouvalas, Mark Butrico, Mark D Fuge, and Peter W Chung. Applying machine learning techniques to predict the properties of energetic materials. Scientific reports, 8 (1);9059, 2018.
Niklas WA Gebauer, Michael Gastegger, and Kristof T Schütt. Generating equilibrium molecules with deep neural networks. arXiv preprint arXiv:1810.11344 2018.
Niklas WA Gebauer, Michael Gastegger, and Kristof T Schütt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. arXiv preprint arXiv: 19.Xi:::900000957, 2019.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. Proceedings of the 34th International Conference on Machine Learming- Learning-Volume Volume 70, pp. 1263-1272. JMLR. org, 2017.
R. Gómez--oobarrlli, J. N. Wei, D. Duvenaud, J. M. Hemánndez-Lobato, B. Sánchez-Lengeling. D. Sheberla, J. Aguilera-pparaguirre, T. D Hirzel, R. P Adams, and A. Aspuru-Guzik, Automatic chemical design using data-driven continuous representation of molecules. ACS Cent. Sci., 4: 268-276, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2666 2680, 2014.
John Clifford Gower. Euclidean distance geometry. Math. Sci, 7(1):1-14, 1982.
Edward J Griffen, Alexander Dossetter, Andrew G Leach, and Shane Montague. Can we acceler- ate medicinal chemistry by augmenting the chemist with big data and artificial intelligence? Drug discovery today, 2018.
Gabriel Lima Guimaraes, Benjamin Sanchez- Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Aspuru-Guzik, Objective-reinforeed generative adversarial networks (organ) for se- quence generation models. arXiv preprint
10843, 2017.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017.
Jon Paul Janet and Heather J Kulik. Predicting electronic structure properties of transition metal complexes with neural networks. Chemical science, 8(7):5137-5152, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint 170. 2017.

---

Under review as a conference paper at ICLR 2020
Steven Kearnes, Kevin McCloskey, Mare Berndl, Vijay Pande, and Patrick Riley. Molecular graph convolutions: moving beyond fingerprints. Journal of computer- molecular design, 30(8): 595-608, 2016.
Thomas N Kipf and Max Welling. Sumi-super classification with graph convolutional net- works. arXiv preprint arXiv: 1609.02999 2016.
Nathan Krislock and Henry Wolkowicz. Euclidean distance matrices and applications. In Handbook on semidefinite, conic and polynomial optimization, pp. 879-914. Springer, 2012.
H. W. Kuhn. The hungarian method for the assignment problem. Nav. Res. Logist. Quart., 2:83-97, 1955.
Haichen Li, Christopher R Collins, Thomas G Ribelli, Krzysztof Matyjaszewski, Geoffrey Gordon, Tomasz Kowalewski, and David Yaron. Tuning the molecular weight distribution from atom transfer radical polymerization using deep reinforcement learning. Molecular Systems Design & Engineering, 3(3):496-5000 2018a.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointean: Con- volution on x-transformed points. In Advances in Neural Information Processing Systems, pp- 820-830, 2018b.
R. McGibbon, Beauchamp, M. Harrigan, C. Klein, J. M. Swails, C. Hernández, C. R. Schwantes, L. P. Wang, T. J. Lane, and V. S. Pande. Mdtraj: A modern open library for the analysis of molecular dynamics trajectories. Biophys J., 109: 109:1528-1532, 2015.
Frank Noć, Simon Olsson, Jonas Köhler, and Hao Wu. Boltzmann generators- sampling equilibrium states of many-body systems with deep learning. arXiv:1812.00179,, 2019.
Noel M O'Boyle, Michael Banck, Craig A James, Chris Morley, Tim Vandermeersch, and Geof- frey R Hutchison. Open babel: An open chemical toolbox. Journal of cheminformatics, 3(1):33, 2011.
N. Plattner, S. Doerr, G. De Fabritiis, and F. Noé. Protein pereein association and binding mechanism resolved atomic detail. Nat. Chem., 9:1005-1011, 2017.
Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 4(7))eaap7885, 2018.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings tfe the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652-660, 2017.
Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
João Marcelo Lamim Ribeiro, Pablo Bravo, Yihang Wang, and Pratyush Tiwary. Reweighted au- toencoded variational bayes for enhanced sampling (rave). J. Chem. Phys., 149:072301, 2018.
2018,
Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of chemical information and modeling, 52(1 21228644-775 2012.
M. Rupp, A. Tkatchenko, K.-R. Müller, and O. A. Von Lilienfeld. Fast and accurate modeling of molecular atomization energies with machine learning. Phys. Rev. Lett., 108:058301, 2012.
Benjamin Sanchez- and Alán Aspuru-Guzik. Inverse molecular design using machine learning: Generative models for matter engineering. Science, 361(6400):360-366 2018.
Isaac Schoenberg. Remarks maurice frechet's article"sur la definition axiomatique d' l'uee classe d'espace distances vectoriellement applicable sur l'espace de hilbert. Annals of Mathematics, pp.
724-732, 1935.
10

---

R. Winter, F. Montanari, A. Steffen, H. Briem, F. Noé, and D. A. Clevert. Ef- ficient multi-objective molecular optimization in continuous latent space. 2//he////...202444/////ii... Under review as a conference paper at ICLR 2020
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: continuous- filter convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems, pp- 991-1001, 2017a.
Kristof T Schlitt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko. Quantum-chemice insights from deep tensor neural networks. Nature communicationns, 8:13890, 2017b.
Kristof T Schüit, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241722,2 2018.
J. S. Smith, O. Isayev, and A. E. Roitberg. Ani-1: extensible neural network potential with dft accuracy at force field computational cost. Chem, Sci., 8:3192-3203, 2017.
James JP Stewart. Mopac: a semiempirical molecular orbital program. Journal of computer-aided molecular design, 4(1):1-103, 1990.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
R. Winter, F. Montanari, F. Noé, and D.-A. Clevert. Learning continuous and driven molecular descriptors by translating equivalent chemical representations. Chem. Sci., 10:1692-1701, 2012 2019a.
2019b.
Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariha- ran. Pointflow: 3d point cloud generation with continuous normalizing flows. arXiv preprint arXiv:1906. 12320, 2019.
Hongyi Zhang, Sashank J Reddi, and Suvrit Sra. Riemannian svrg: Fast stochastic optimization riemannian manifolds. In Advances in Neural Information Processing Systems, pp. 4592-4600, 2016.
Zhang, Y. Yang, and F. Noé. Targeted adversarial learning optimized sampling. ChemRxiv. DOI: 00226444cheemrii777337777 2019.


Reviews:
[{"review_id": "rkeVnp86YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors of this paper present architecture to produce valid Euclidean distance matrices. A Wasserstein GAN is then constructed by utilizing a permutation invariant critic network based on the architecture. Generating molecular structures in a one-shot fashion is conducted using the produced distance matrices in 3-d embedding.\n\nIn Section 2, the constraint on L makes M symmetric and positive semi-definite. This seems to be equivalent to treating M as a kernel matrix, and D is the pairwise distance between the kernel function induced by M. So, learning a valid Euclidian distance matrix is same as learning a kernel function.\n\nAuthors need to provide the evidence that Equation (5) holds for all function g, especially for the used softplus activation function. As I know, it holds if function g is a polynomial matrix function.\n\nIn Algorithm 1, what is the meaning of step 14? There is no definition or discussion about G and the \\nabla L. The construction function is the key contribution of this paper, which is incorporated the existing SchNet, so it is better to show the advantage of the proposed construction method comparing with SchNet.\n\nAlthough this paper is an application-oriented paper, the comparisons with baseline methods are preferred, such as some simple and straightforward baselines. Due to the lack of comparisons, it is hard to understand the statement given by authors that “ the current performance… is not optimal and can likely be improved by a better hyperparameter selection”.\n\nIn Section 4, authors calculate the similarity of generated molecules by the closest respective matches, which is determined by the maximal atomic distance after assignment of atom identities and superposition. Is this the standard way to compute the similarity between two graph structures?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skx9wBpHFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is extremely interesting, solid and very well written. The idea is simple but nonetheless developed in a smart and effective fashion. The underlying theory is solid, even if some choices should have been discussed more deeply (e.g. the chosen loss function). Introduction and references are adequate, and the paper is readable by a quite broad audience, despite the detailed technical sections. The main issue related to the manuscript is very narrow target of the experimental part, limited to the isomers of a given compound - it would have been interesting to check its potentialities in generating more different structures and distance matrices, and thus to compare its effectiveness versus alternative generative approaches.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkeMvX5TFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\t\nThis paper addresses the important problem of molecular structures generation, and more generally of efficient point-cloud distributions learning in d-dimensional space.\nThe paper is written clearly, the pseudo-code is presented in a clear way, striking a good balance between explicit-ness and concision.\n\nAfter my first reading I was disappointed in the experiments, because I read the paper quite quickly at first.  After a second more careful read I understood most of it and was much more enthusiastic. Disclaimer: I am inexperienced in this particular field (GANs and GANs for molecular generation) so I lack literature knowledge, and I may be over-evaluating the quality of performances (compared to recent results).\nBut currently I believe that the authors are a bit too humble about their results (a rather uncommon phenomenon). There is a part of the method that I would like to have clarified (about bond/atom types), but other than this clarification, I strongly recommend the paper to be accepted.\n\nThe paper deals with a couple of distinct, related problems.  One is that of sampling valid (Euclidean) distances matrices (EDM). This is done with algorithm 1.\nThe paper uses these EDM to train a generator G directly in EDM-space, against a Critic network C (the architecture of which is taken from existing literature).\nSome of the output configurations have to be discarded due to incorrect bond types assignment (this is the part that is still obscure to me).\nThe remaining outputs are chemically sensible in terms of bond types/valence/local chemistry/etc, but also, and quite impressively, have reasonable ground state energies !  In this sense, the paper produces samples that may deserve to be added into the QM9 data set (After some data augmentation using DFT or other physics-validated methods of course).\n\n\nI have mainly two requests:\n\n1. The bond type or atom assignment is not clearly explained. I ask further detailed questions below, but overall I think you should attempt to make a pedagogical explanation of how the atom types are assigned/learned, and how some of them are later discarded.\n\n'The  generator produces an additional type vector in a multi-task fashion which is checked against a  constant type reference with a cross-entropy loss.'\nCould you explain this better ? At least in the appendix. Otherwise this very dense sentence remains mysterious. Eq.11 is currently the single occurrence of the H, t, and tref terms.\n\nAbout the validity test using Open Babel. Again, what happens with bond types is not very clear. Are they set in stone at generation time, and then most of these affected types are 'wrong' and are discarded by Open Babel ?\nWhy don't you include this validation at training time (if it's very complicated to do, explain why) ?\nIn any case, please clarify this paragraph, as for now it is cryptic.\n\n2. Part of the paper goal is to achieve learning of point clouds distributions and not about chemistry.  However, discovery of new structures (and their conformations) is in itself a big topic.  I think it would be good to mention a couple of follow ups to your work in the conclusion. For instance, it would seem rather natural to me to include some equivalent of OpenBAbel and Energy-estimates within the learning loop, so that the generator directly generates valid (open-babel wise) and reasonable (energy level-wise) structures.\nBesides the conclusion, you should stress out better the significance of your results with a couple of comments here and there. (I have more precise suggestions below).\n\n\nSome other comments to improve the paper (in clarity or other):\n\nIn algorithm 1, you should precise that G() is your NN-based generator network. At this point of the paper G and z and N_z have never been mentioned. This is a problem when reading the paper for the first time.\nLine 5, you could explicitly precise, 'with (5)'.  Also I am puzzled by the use of the softplus here, but later in page 5 you say g() is softplus for the first three eigenvalues, and set to 0 for all others. Why not already anticipate and say this in section 2 ?\nLine 8, I would have written 'eigenvalues .... of D'  (what is the role of Eq. 1 here?)\n\nAt some point (i.e. around end of section 2), a comment would be welcomed, about whether you may sample the space of EDMs uniformly at random using algorithm 1 (I think you do not, and it is ok, but the question naturally arises and it's not addressed).\n\n´´which transforms a prior distribution into a target distribution´´\nyou could specify that this prior is the Gaussian N(0,1)^Nz in this case, to better connect with algorithm 1.\n\nAbout equation 6: it would be nice to have some intuitive explanation of what the output values of C(x) mean. They are scalars that represent the opinion of C on the molecule x, so they are the probability that the observed molecule is 'a true molecule x' ?  You should recall that for the inexperienced reader.\n\nStill about Eq.6 : could you quickly provide the motivation for demanding C to be L-Lipschit with  $L \\leq 1$ ?\n\nThe drift term Eq.9 seems to be some sort of regularization, maybe it could be mentioned once in the text for completeness?\n\nUsing Mopac Stewart and figure 4: you could insist more on this result. Up to that point I was very dubious about the usefulness of the whole work, because I would have expected many generated samples to be highly unrealistic, i.e. having huge energies, and so being extremely unstable. Instead here you show it is not the case, and even all your energies lie within the range of observed energies !  This is a very strong result, that is not obvious to expect, and is highly valuable (even after keeping only 7.5% of structures, this is still a strong result.)\nI think this test and the corresponding result should be emphasized more.\n\nYou observe new topological types and complain there are not many. I think ~20 new topologies is not a small number (for so few atoms), and you may be rather proud of it.  What is too bad is that you do not have a tool for differentiating between the very similar conformations (that correspond to thermal fluctuations around a given structure) and the conformations that encode a new structure (which does not necessarily means new topology). \nIf you had this tool, you could enrich figure 3 with a curve showing the new structures (not counting conformational variants).\n\nrefs 2017a and 2017b are the same.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a parametrisation of Euclidean distance matrices amenable to be used within a differentiable generative model. The resulting model is used in a WGAN architecture and demonstrated empirically in the generation of molecular structures. 

Reviewers were positive about the motivation from a specific application area (generation of molecular structures). However, they raised some concerns about the actual significance of the approach. The AC shares these concerns; the methodology essentially amounts to constraining the output of a neural network to be symmetric and positive semidefinite, which is in turn equivalent to producing a non-negative diagonal matrix (corresponding to the eigenvalues). As a result, the AC recommends rejection, and encourages the authors to include simple baselines in the next iteration. 
Decision: Reject