$G^3$: Representation Learning and Generation for Geometric Graphs
Authors: Han Huang
Stefan C Schonsheck
Rongjie Lai
Jie Chen
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Learning
Generative Models
Graph Neural Networks

Abstract: A geometric graph is a graph equipped with geometric information (i.e., node coordinates). A notable example is molecular graphs, where the combinatorial bonding is supplement with atomic coordinates that determine the three-dimensional structure. This work proposes a generative model for geometric graphs, capitalizing on the complementary information of structure and geometry to learn the underlying distribution. The proposed model, Geometric Graph Generator (G$^3$), orchestrates graph neural networks and point cloud models in a nontrivial manner under an autoencoding framework. Additionally, we augment this framework with a normalizing flow so that one can effectively sample from the otherwise intractable latent space. G$^3$ can be used in computer-aided drug discovery, where seeking novel and optimal molecular structures is critical. As a representation learning approach, the interaction of the graph structure and the geometric point cloud also improve significantly the performance of downstream tasks, such as molecular property prediction. We conduct a comprehensive set of experiments to demonstrate that G$^3$ learns more accurately the distribution of given molecules and helps identify novel molecules with better properties of interest.

Under review as a conference paper at ICLR 2022
G REPRESENTATION LEARNING AND GENERATION FOR GEOMETRIC GRAPHS
Anonymous authors Paper under double-blind review
ABSTRACT
A geometric graph a graph equipped with geometric information (i.e., node coor- dinates). A notable example is molecular graphs, where the combinatorial bonding is supplement with atomic coordinates that determine the three-dimensional struc- ture. This work proposes generative model for geometric graphs, capitalizing on the complementary information of structure and geometry to learn the underlying distribution. The proposed model, Geometric Graph Generator (G²), orchestrates graph neural networks and point cloud models in a nontrivial manner under an autoencoding framework. Additionally, izing flow so that one can effectively sample from the otherwise intractable latent space. G can be used in computer-aided drug discovery, where seeking novel and optimal molecular structures is critical. As a representation learning approach, the interaction of the graph structure and the geometric point cloud also improve significantly the performance of downstream tasks, such as molecular property prediction. We conduct comprehensive set of experiments demonstrate that Go learns more accurately the distribution of given molecules and helps identify novel molecules with better properties of interest.
augment this framework with normal-
INTRODUCTION
Geometric deep learning refers to the development of deep learning techniques for data from non- Fuclidean domains, such graphs and manifolds (Bronstein et al. (017). In recent years, graph neural networks (GNN) emerged be a promising family of architectures that models the relational inductive bias ubiquitous to graph structured data (Battaglia et al.] 2018 Zhou et al. 2018) With the rise of deep generative modeling, various generative models have been adapted for graphs and proven to be effective in learning the underlying distribution implicitly defined by set of given
graphs (Goodfellow et al., 2014 Kingma & Welling 2013 Rezende & Mohamed 2015 van den
Oord et al. 2016 De Cao & Kipf 2018 You al. 2018b Jin et 2018: Shi et al. 2020
In this work, we consider a special type of caaphs--geomericc graphs- which are equipped with geometric information in addition to the combinatorial structure (Pach) 2013). In practice, this additional information corresponds to low-dimensional geometric structures and typically appears as node coordinates in g or 3 The coordinates find broad uses molecules (Van Aalten et al. 1996). meshes (Alliez et al. 2005), and graph drawing (Frishman & Tal (2008). This work aims to develop generative model that captures both the combinatorial and the geometric characteristies exhibited in given collection of geometric graphs.
One driving application of geometric graphs is molecule generation. In pharmacology, drug discovery is the process of identifying new candidate medications in the form of molecules. The process is known to difficult, costly, and time-consumen consumnnn (Paul et al. 2010), because of the discrete nature of the search space and its vast size (estimated to contain at least T0 molecules) (Polishchuk et al. 2013). Recently, deep learning techniques have been developed to represent molecules as continuous vectors and apply continuous optimization (Jin et al. 2018) or reinforcement learning (You et al.) 2018ab to conduct an efficient search. almost all existing work based on the molecular graph approach, the graph is treated as combinatorial object and the dinen-iimesiooall geometric informa- tion serves as node features only. However, the atomic coordinates encode vital energy information of the molecule and they can be instrumental for the inference of structure and state. Therefore,

---

Under review as a conference paper at ICLR 2022
proper modeling of the geometric information is paramount for more accurate representation of the molecule in downstream tasks.
In this work, we develop autoencoder for geometric graphs, orchestrating GNN and point cloud model in a nontrivial manner. Besides the straightforward use of a GNN (Duvenaud et al. 2015 Kearnes et al. 2016 Gilmer al.) 2017) to encode the combinatorial structure, we treat the nodes with spatial coordinates as low-dimensional point clouds and use a point cloud model to encode the geometry. Different from decoders merely based on graph structures in conventional graph autoencoders, on the other hand, we decode the graph structure by first reconstructing the geometry. Specifically, we use a folding- -based technique (Yngg et al. [018 Pang et al. 2021) to map a template of points to the correct geometry, from which the combinatorial structure is inferred. This way, both sources of information are organically fused and processed. Furthermore, to fulfill the generative capability, we augment the autoencoder with a normalizing flow (Kobyzev et al., [219 Dinh et al.) 2016 2014), so that one able to effectively sample from the otherwise intractable latent space.
The proposed geometric graph generator, G addresses geometry that is rarely considered in past graph generative models. A naive alternative to incorporate node coordinates is to treat them as part of features when consumed by a GNN. This approach, however, biases the geometry by the graph structure owing to local neighborhood aggregation- the global geometry may be compromised. For geometric graphs, the structure itself is combinatorial and discrete, while the geometry is continuous. These two sources of information are often complementary and benefit from collaborative processing. Our strategy is exploit both graph techniques and point cloud techniques maximally retain information from the two sources.
To evaluate the effectiveness of G., we focus on molecule applications and use popular datasets (QM9 and ChEMBL) that contain atomic coordinates. We demonstrate that G outperforms a number of representative graph generative models generating novel and valid molecules and it also sigmificantly outperforms either GNN or a point model alone for property prediction. Appealingly, G decodes molecules substantially faster than do many popular models. Furthermore, with the use of Bayesian optimization in concert with a trained model, we can identify better molecules under metries of interest than do existing methods.
2 RELATED WORK
Generative models refer to a class of machine learning models that can learn the underlying distribu- tion, implicitly defined by given set of data, and to sample from it. Noteworthy generative models in the deep learning era include variational autoencoders (VAE) (Kingma & Welling) 2013), generative adversarial networks (GAN) (Goodfellow et al., 2014) and normalizing flows (NF) (Kobyzev et al., 2019: Dinh et al. 2016 2014). Among them, NF bears unique advantage in estimating the density (likelihood). For training. NF directly optimizes the likelihood, whereas VAE maximizes a lower bound of it (called the ELBO Kingma & Welling (2013) and GAN minimizes the discrepancy
between the input and the transformed noise distribution.
Under these generative models, graph can be generated in a sequential or one-shot manner: the former samples nodes and edges incrementally while the latter samples a graph in its entirety. Combining GAN and reinforcement learning, MolGAN (De Cao & Kipf 2018) and GCPN (You et al.) 2018a) generate molecules in a one-shot and sequential fashion, respectively. MolecularRNN (Popova al. (019)) is an autoregressive model based on recurrent neural networks; it performs validity checks during the sequential generation process and rejects invalid structures. JA- (Jin et al. 2018) a VAE-based tree model that constructs the molecular graph sequentially from sub-components.
Popova et al. (2119) point out that JA-VAE may suffer ambiguity in the conversion of the tree structure to a molecular graph, affecting property optimization. GraphNVP (Madhawa et al.) 2019) and GraphAF (Shi et al., 2020) are recently proposed basba models that parameterize the mapping from a latent vector to a graph by using a flow. The former employs one-shot sampling while the latter is sequential.
Molecules admit representations other than graphs. An alternative is the simplified molecular input line-entry system (SMILES) (Weininger] 1988). string notation that universally describes molecular structures. Grammar VAE (GVAE) (Kusner et al., (017) and et al. 2018) are VAE-based models that process SMILES strings and apply grammatical rules to
VAE (SD-VAL ---V (Dai
2

---

Under review as a conference paper at ICLR 2022
sequentially reconstruct the strings. Another family of models, which are motivated by quantum chemistry principles, models molecules as point clouds and learns a physically robust representation to prediet, e.g., molecular energies and equilibrium conformations, [Schütt et al. (2018) propose SchNet, a network with continuous- convolution layers that model interactions between atoms and learn representation that is invariant translation and rotation. Using the same filters, Gebauer ct al. (2019) devise Senerative-SchNe (G-SchNet), which accurately models the target molecule distribution and generates novel 3D conformations that are relatively stable.
In addition to the graph structure, our work treats nodes as a point cloud, which is a collection of points in R Point clouds are prominent subject in computer vision and computer graphics; they are obtained through, e.g., LIDAR scanning of object surfaces (Yang et al.] 2018). Deep learning with point clouds is faced with challenges in defining the convolution operator (Bruna al. 2013 Bronstein et al., 2017, Schonsheeck et al. 2018:; Jin et al.] 2019). Volumetric CNN (Wu et al. 2015 Qi et | 2016 Maturana& Scherer 2015) applies convolution filters on voxels obtained from discretization of the dirreednimniionall space. Multiview CNN (Su et al., 2015 Qi al. 2016) reduces point clouds to collection of 2D images and applies gridded convolutions on them. PointNet (Qi et al. 2017a b) uses point-wise MLPs to featurize individual points, followed by symmetric pooling that generates a permutation permutaaiinninnvarian global description.
3 GEOMETRIC GRAPH GENERATOR
In this section, we present G The model is autoencoder that contains interacting modules to process and reconstruct the geometry (i.c., node coordinates), the structure (i.e., edges and edge types), and additional features (c.g.. node features). Such a sophisticated autoencoder allows forming a more accurate representation of the geometric graph and learning better the input distribution.
proper processing of the geometric and combinatorial information is crucial to the success of G and downstream tasks. For example, an authentic representation of the node coordinates serves as the basis for decoding both the node and the edge types: the training hinges on an accurate registration between the reconstructed point cloud and the ground truth. The straightforward adaptation of GNNs for encoding the geometric information (as node features) results poor coordinate reconstruction and worse generation quality overall. Moreover, features with single decoder. However, mixing two modes of information together introduces unnecessary bias through, e.g., the difference in scales. Hence, separate but dependent decoders for features and geometry render a more accurate reconstruction.
can naively reconstruct coordinates and node
3.1 NOTATION
Let n be the number of nodes, dy be the number of node features, de be the number of edge types, and de be the dimension of the node coordinates (typically two or three). We denote a geometric graph as G (A, X), where A E {0, 1]xxmxde is the adjacency tensor and X [C, F' (xx((d+dd)) is the node matrix, where each note vector x; [ea, fa] is concatenation of coordinates e; E Igdd and features fa RRixd, For molecules, the node features are one-hot encoding of the atom type, and the edge types correspond to bond types.
3.2 ENCODER
The purpose of the encoder is to use latent vector to represent both the geometry and the structure of G. It contains two modules: a point encoder that maps the point cloud to a geometry descriptor Zpi and graph encoder that takes A and F input and maps them to a structure descriptor 2g. Both descriptors concatenated to form the latent representation
[Zp, 2g] E Rd=
usual graph encoder processes the structure information A and node features F by using a GNN. When the coordinate information C' is available, one could have treated C' as part of the node features and feed them into the GNN, indiscriminately often complementary to the structure. Hence, a drawback of such approach is that one heavily relies on the local neighborhoods from graph structures to parameterize the geometry, resulting in strong bias toward the structure. A notable feature of our design is that we separate the geometry
F. However, the geometry information of a graph is
3

---

Under review as a conference paper at ICLR 2022
from the structure when encoding a geometric graph; we will show ablation study supporting this design subsequently.
This idea offers means process all node information, including features F and coordinates C, independently of the combinatorial structure, as an additional parameterization over the pure use of a GNN. In what follows, we describe the two encoder modules and defer details of the specific network architecture in the appendix.
Point Encoder. In this module, we treat each row of C point and use a point cloud architecture to obtain the geometry descriptor 2p. First, a multilayer perceptron (MLP) is applied point- to the point cloud to amplify the feature dimensions. The point-wise features are then passed through series of multi-headed attention blocks (Vaswani et al. 2017) to further featurize the points. These attention-bre -baeed blocks are more effective than simple point-wise MLPs used in PointNet (Qi et al. 2017a) as they are able to leverage global information, i.c. features on all other points, to produce rich and expressive point-wisc features. At last, by an additional MLP to yield the geometry descriptor Zp.
a global max-pooling over the points followed
Graph Encoder. This module takes the graph adjacency tensor and the node feature matrix as inputs and outputs a structure descriptor 2g. Virtually all GNNs serve the purpose: some are even specialized to molecules (e.g.. Duvenaud et al. (2015); Kearnes et al. (2016); Gilmer et al., (2017)). Our empirical investigation finds that GINE, Graph Isomorphism Network with Edge features (Xu jet al. 2018), obtains good and stable performance for our purpose.
3.3 DECODER
Decoding a graph is substantially harder than encoding one. common idea to employ a sequence model and decode nodes and edges one by one Popova et al. (2019). Training such model increasingly more challenging for long sequences (i.e., large graphs). On the other hand, approaches decoding the adjacency tensor also face scalability challenges owing to the O(n²) degrees of freedom Samanta et al. (2020). In this work, we exploit the unique opportunity offered by a geometric graph decode first the geometry, then the point-wise features, and lastly the structure.
Specifically, we propose a three-step decoding process that decodes a graph G' = (A', from 2, where X" = [C], The first step is to decode C', the coordinates for the point cloud, which involves using the latent representation to morph template of points to the correct geometry. By isolating the decoding of coordinates, we can leverage specialized architectures to achieve accurate reconstructions of the geometry, which is crucial for the subsequent decoding of F' and A'. Secondly, the reconstructed geometry C' is recycled along with the latent code z to predict the point-wise features F' Finally, X' = [C", FF] is used to infer the graph structure A' through a link predictor. This procedure sidesteps the scalability challenge of both the sequential generation approach and the adjacency decoding approach.
In what follows, we describe each component of decoder modules and defer details of the specific network architecture in the appendix.
Coordinate Decoder The idea of decoding point cloud from latent vector is successively fold a template of points until it forms the desired geometry. Our coordinate decoder is inspired by TearingNet (Pang al. 202T), which is refinement over FoldingNet (Yang al. 2018 Tao 2020 a popular archetypal architecture for point cloud reconstruction. Note that the template design is dediiaationdeeendeen an effective template is guided by the geometry be modeled. For example, spherical template with roughly equidistant points is more suitable for the three-dimensional structure of molecules. This design is different from the original TearingNet architecture that uses regular 2D grid to model surfaces.
To decode the point-wise features F`, we apply the folding operations with the decoded coordinates C" from the aforementioned coordinate decoder as the starting template. The latent code 2 is replicated and concatenated to C". which is then forwarded through point-wise MLPs to complete one set of folding. We repeat two sets of folding to arrive at the decoded features. The architecture of the feature decoder is illustrated in Figure]
Feature Decoder
4

---

Under review as a conference paper at ICLR 2022
Repeal n.d.
(A. Axl
n x dg
Concat
x (d,
MLP
11 X d, F Intermediate Point Festures
Concati
(d, de
MLP
X de
Coneat
x (d. de)
/ Decoded Coordinates
Skip Connection
Skip Connection
F² F'
X' = [ F'
Decoded Point Features
Decoded
1st Folding
2nd Fobding
Point with Features
Figure 1: The G feature Decoder.
Link Predictor (Graph Decoder) The link predictor determines the existence of an edge (and its type so) for every pair of decoded nodes. typical link predictor takes the inner product between two nodes to indicate the strength of connection Trouillon et al. (2016). However, such link predictor is oblivious to the geometry because large inner product 3D does not necessarily indicate closeness. Rather, the displacement of two points, their respective neighborhoods, and their neighbors' features may all play role in determining the existence and type of an edge. For example, in molecules, the bond between pair of atoms depends their distance well as the neighboring atoms. Therefore, we augment the features of a decoded node z; from [ct, fi] to &; as
i
||| Cn2 : ... |c; Coe-ab fúr fRi fmi-il.
where #1...... 7 are the node indices of the sorted k-nearest neighbors of xy J Rdy and oj E Rd are decoded features and coordinates, respectively, and the absolute sign indicates element- wise absolute values. We then use these augmented features as well as the coordinate-wiss displacement |a - Gyl define the link predictor. Formally,
[(i.j) = MLP((E, lq -5ll) E Rd LinkPredictor((..) ((i,)) ((),d) Rd.
We highlight that our link predictor works remarkably well as standalone module for molecules, which can be characterized as procedure for inferring inter-atomic bonds based on molecular conformation and atom types. To the best of our knowledge, there are no reliable public software available for this task, especially for large molecules with noise in coordinates or atom types. Therefore, our proposed link predictor has its own value other than being part of G3. We refer appendix G for illustrating the outstanding performance of the standalone link predictor.
3.4 LATENT SPACE SAMPLING: NORMALIZING FLOW
To reliably sample from GD s latent space, we make use of a normalizing flow model (Kobyzev et al. 2019), which consists of sequence of invertible transformations that map between easily sampled base distribution (e.g., standard normal) and an unknown distribution (e.g.. the latent distribution of our autoencoder). The densities of the two distributions related by the Jacobian of the transformations, so that log-likelihood of the unknown distribution can be computed and optimized for training. In practice, we find that RealNVP (Dinh al. (016) to be simple yet effective model for our needs. After the training is finished, samples from the base distribution are forwarded through the normalizing flow and subsequently decoded into the generated molecules.
3.5 OVERALL MODEL AND TRAINING LOSs
The overall architecture of G is outlined in Figure which summarizes all the components described in the preceding subsections. To train this model, we use loss consisting multiple terms with a tunable weighting scheme: L d Lc + d2 LF Xa LE + X4 LxF + As LR-
Node coordinate loss Le. In this loss, we use only the coordinate part contained in the node matrix X. In our experiments, we use the Sliced Wasserstein-2 distance (SW2) as our loss for robust point cloud reconstructions (Rabin et al. [001I. Note that in the calculation of Lc, every point c E C identified with the nearest match c' C' and vice versa. This correspondence allows us feasibly define the following losses LF and LE with differentiability. We remark that another commonly used loss for point-cloud matching, Chamfer distance Yang et al. 2018), can also be used here.
5

---

Under review as a conference paper at ICLR 2022
Le
S
Ln LoossEntropyLoss(...
Ln VegencyRog(X X
X
Point Encoder
7p
LNF NLL(ZAE)
[C.P)
X
NF
ANY
N(
NF
ZAE
Coordhate
Feature
X
Link Preilictor
A
A
Graph Encoder
2g
Ry. Rcl
ZAE
Autoencoder Latent Space
Normalizing Flow Base Distribution
Lg Figure 2: Overall architecture of G
CrossEntrop (A,
Node feature loss Ly. Since node correspondence is established, the difference between and F be meaningfully quantified. When the otherwise, mean absolute error (MAE) or other losses may be appropriate.
one-hot vectors, cross-entropy loss is suitable;
Edge loss Lg. The edge loss quantifies whether edge types correctly inferred. Hence, cross- entropy loss between and A' is natural, where node correspondence needed.
Normalizing flow loss LNF- The normalizing flow model is equipped with standard negative log-likelihood loss (Dinh et al. 2016).
Additional loss LR- This term enforces structural and other chemical constraints innate to the specific application. Some constraints may be easily violated even if the decoding of nodes and edges reasonably accurate (as long the violation of the constraints helps improve semantic validity. For example, for molecules, one incorrect decoding of carbon atom to oxygen enough to create an invalid molecule, because the valence of oxygen is much smaller. In this case, we follow the work by Ma et al. (2018) and include a soft valence regularization term LR encourage higher validity of the generated molecules.
exists, however small). Hence, regularization that penalizes
EXPERIMENTS
4.1 DATASETS AND BASELINE MODELS
We perform experiments QM9 [Ruddigkeit et al. 2012 Ramakrishnan et al.) 2014) and ChEMBL (Mendez et al.) [2099] Davies al. 2015). The QM9 dataset contains 134k molecules with atom types and up to heavy atoms per molecule. The ChEMBL dataset contains 1.5 million molecules, and the largest one has hundreds of atoms. For feasible experimentation, we take subset of it that contains 250k molecules with 19 atom types and up to 39 heavy atoms molecule. We name this new dataset ChEMBL250K. Details of dataset proprocessing are provided the appendix.
benchmark datasets that come with atomic coordinate information:
We compare G with several representative architectures that feature variety of generative ap- proaches. SDVAE (Dai et al.] 2018) is a syntax-directed VAE model that processes SMILES representation of molecules and generates syntactically correct ones. MolGAN (De Cao & Kipf 2018) combines GAN and reinforcement learning to generate molecules with optimized properties. IT-VAE (Jin et al. 2018) is VAE-based model that assembles chemical substructures. boasting high validity of the generated molecules. GraphNVI VP (Madhawa et al. 2019)) is a flow-based model with one-shot generation procedure. GraphAF (Shi al. 2022)) is another flow-based autoregressive model that advertises high flexibility in density modeling. G-SchNet (Gebauer et al. 2019) is quantum-chemistry-i -inspiee model that learns accurate conformations and sequentially generates molecules close to the trained distribution.
4.2 GENERATION QUALITY
Metrics and processing. To evaluate the generative capability of different models, we measure the validity, uniqueness, and novelty (V, U, N) their generated molecules. Validity is the percentage of

---

Under review as a conference paper at ICLR 2022
molecules that pass RDKit's sanitization check. Uniqueness is the portion of valid molecules that distinct. Novelty is the portion of valid molecules that do not appear in the training set.
In addition, include metrics computed between the test and the generated molccules via MOSES (Polykovskiy et al., 2020): Frechet ChemNet Distance (FCD), which measures the distance between the two sets according to ChemNet's penultimate layer's activation; Fragment similarity (FRAG) measures the similarity in fragments present in both sets of molecules; Internal Diversity (IntDiv2) assesses the diversity within the generated molecules; LogP. QED, and Weight the ID Wasserstein distances between the two sets on different chemical properties: LogP measures the lipophilicity of a compound (Bhal 2007); QED is the quantitative estimate of druglike-ner (Bickerton al.) 2012); Weight is the sum of atomic weights in molecule. All scores are computed on 10k generated molecules.
In our experiments, G sometimes generates disconnected molecules, for which we treat the largest connected component as the final generated molecule (breaking ties arbitrarily) for evaluation. Additionally, we observe the normalizing flow cannot capture the autoencoder's latent distribution perfectly when trained on ChEMBL250K. Therefore, we interpolate between samples from the normalizing flow and the nearest encoded training data before decoding.
MODEL
Vt
Uf
Nt
FCD4
FRAG
INTDIV21
LoGP
QED4
WEIGHT
SDVAE
33.0
73.6
93.4
8.52
0.57
0.870
2.50
0.062
206.82
MOLGAN
74.5
23.1
52.5
3.69
0.66
0,369
0.63
0.028
2.17
JT-VAE
100
70.2
43.3
0.94
0.94
0.894
0.14
0.016
19.50
GRAPHNVP
86.6
83.5
56.5
3.83
0.86
0.886
1.06
0.027
1.04
GRAPHAF
100
69.8
86.1
7.86
0.22
0.826
0.74
0.057
62.83
G-SCHNET
79.2
94.1
63.9
1.99
0.90
0.892
0.38
0.007
5.06
G
92.7
97.0
72.2
1.23
0.94
0.900
0.25
0.006
1.60
Table 1: Molecular scores the QM9 dataset.
MODEL
VT
U1
Nt
FCD|
FRAGT
INTDIV21
LoGP]
QED1
WEIGHT
SDVAE
32.8
81.9
94.0
28.09
0.47
0.889
5.25
0.213
331.57
MOLGAN
0.02
48.52
0.270
1.04
0.403
284.11
JT-VAE
100
97.8
99.9
7.39
0.86
0.854
0.84
0.189
72.45
GRAPHNVP
31.6
95.8
99.8
39.62
0.13
0.771
1.70
0.271
69.94
GRAPHAF
100
72.0
91.9
26.95
0.46
0.895
1.39
0.117
182.12
G
78.2
99.4
94.1
7.28
0.95
0.893
0.59
0.067
64.10
Table 2: Molecular scores on the ChEMBL250K dataset.
Results and discussions. Results reported in Tables and invite a few observations. Overall, see that G has strong performances across all metrics in both QM9 and ChEMBL250K. Notably, G low FCD score indicates accurate modeling of the training distribution, while high novelty rate means G has not simply memorized the training set. This important because a model can always reproduce the training to get perfect scores on c.g. FCD. In contrast, although JT- JT-VAE has better FCD and LogP scores on QM9, its low novelty is characteristic of overfitting. In addition, GD validity scores are consistently the highest among the models with hard valency checks (the others have 100% validity). and G high uniqueness and molecular diversity suggest it is able to avoid mode collapse, which is evident MolGAN. Furthermore, G robust performance in LogP, QED, and Weight implies that G generated molecules are not only faithful to the training set in terms of (sub)structure, but also with respect to various chemical properties.
Efficiency. G performance on large molecular datasets is especially impressive considering its one-shot generation approach and far superior to other models in the same camp (GraphNVP. MolGAN). Comparing to the best sequential models, e.g. JT-VAE, G can generate molecules as good in quality in a much faster fashion, thanks to its one-shot nature. When benchmarked on a machine with 17- 17-8700K CPU, GeForee 1080 RTX GPU, and 16GB RAM. GraphAF and JT- VAE took 4804.36s and 2642.97s to generate 10k molecules when trained on ChEMBL250K, respectively. In contrast, G took 47. 77.9,, which is approximately 55 times faster than VT-VAE and 100 times faster
7

---

Under review as a conference paper at ICLR 2022
than GraphAF. As discussed, the inefficieney of sequential generation models is mainly due the challenge of parallelization.
Latent space interpolation. Lastly, present examples of G generated molecules well as their local latent space in Figure 3 The surrounding molecules are decoded along two random orthogonal axes starting from the center molecule highlighted by the blue box. The transition between molecules are gradual and chemically sound, which illustrates G great capability to smoothly parameterize discrete molecules in its latent space. More qualitative examples be found in the appendix.
oe ofo oe
og of
de
fra of
Ji o
2 00 go
300
300
8o
8
8.a
Figure 3: Latent space interpolation for QM9 (left) and ChEMBL250k (right).
4.3 ABLATION STUDY AND PROPERTY PREDICTION
To process the complementary geometry and structure information, we use pair of point encoder and graph encoder for G A natural question asks if two encoders are really necessary. Here, we conduct an ablation study and compare G with two simpler models. The first one uses only the point encoder and removes the graph encoder. Because of a lack of graph structure encoding, the edge decoding quality entirely depends on node information. The second simplified model remove the point encoder but retain the graph encoder. Since the graph encoder takes coordinates as part of the input, the geometry information still encoded, albeit through GNN module that was not designed to handle geometry in the first place.
ENCODER
V1
Ut
Nt
FCD|
FRAGT
INTDIV21
LOGP4
QED4
WEIGHT|
POINT
93.0
96.5
74.6
1.67
0.90
0.897
0.22
0.006
2.84
GRAPH
92.6
95.2
72.4
1.57
0.92
0.897
0.34
0.008
2.74
BOTH
92.7
97.0
72.2
1.23
0.94
0.900
0.25
0.006
1.60
Table 3: Ablation molecular scores on the QM9 datasct.
We train the models QM9 until convergence using the same hyperparameters and latent dimensions. From Table 3. G encoder achieves the best results in two-thirds of the metrics, and the remaining third is split between the other approaches. We further highlight the combined encoder's convincing lead in FCD, which is the best single indicator of the quality generated molecules in the absence of overfitting. As result, find " combined approach to be clearly favored.
To further indicate that the combination of geometry and graph information used in G learn a meaningful latent representation for the molccules, show downstream tasks for predicting chemical properties. In the subsequent experiment, we show that a G encoder pretrained in a unsupervised manner can be used to predict chemical properties of QM9 molecules with an additional MI.P fit
8

---

Under review as a conference paper at ICLR 2022
MODEL
H-- 4
INT. ENERGY]
FREE ENERGY
HEAT CAPACITY
PRETRAINED GRAPH ENCODER
0.099
0.040
0.040
0.050
PRETRAINED G ENCODER
0.091
0.026
0,026
0.049
GRAPH ENCODER
0.033
0.014
0,020
0,013
G ENCODER
0.027
0.006
0.005
0.006
Table Relative MAE on test set QM9 property prediction.
desired properties. Alternatively, one can also train G encoder in supervised manner to predict the desired properties, which yield better results. The pretrained models are first trained on the QM9 dataset in an unsupervised manner for 80 epochs. The weights on the encoders are then frozen, and an additional MI.P trained on the encoded vectors until convergence. The supervised models are trained on the QM9 dataset until convergence.
We predict four properties that come with the QM9 dataset: Homo-Lumo energy (H-L), internal energy, free energy, and heat capacity. In all cases, GDs graph-point encoder significantly outperforms the standalone graph encoder used in section 3.2 This further substantiates GD ability to leverage the geometric and combinatorial features in geometric graph more effectively by separating the graph and the point encoder.
4.4 PROPERTY OPTIMIZATION
One use of generative models with latent spaces is to search for better molecules under metrics of interest. Following the approach outlined by Kusner al. (2017), apply Bayesian optimization (BO) on the G model pretrained from Table and to optimize two metrics: penalized logP and QED. Penalized logP logP less the synthetic accessibility score and the number of long cycles Bhal 2007), and QED measures how likely a compound is to be drug (Bickerton et al. 2012).
CHEMBL250K
PENALIZED LOGP
QED
PENALIZED LOOP
QED
MODEL
1ST
2ND
3RD
1ST
2ND
3RD
1ST
2ND
3RD
1ST
2ND
3RD
SDVAE
4.96
4.85
4.12
0.528
0.515
0.509
5.04
2.09
1.95
0.843
0.809
0.806
GRAPHAF
2.50
2.24
2.24
0.593
0.588
0.584
3.09
2.53
2.49
0.830
0.821
0.820
4.99
0.623
0.620
4.70
0.934
G
5.97
5.64
5.37
0.640
0.621
0.614
5.46
4.94
4.82
0.941
0.929
0.927
Table S: Best scores found for QM9 ChEMBL250K.
The results for QM9 and ChEMBL2250K the best molecule under both metrics. We remark that BO works better for low dimensional search spaces (Frazier 2018). The latent spaces of SDVAE and JT-VAE are of similar nature to that of 3 but GraphAF must use a latent space of the same dimension as the input space. Hence, BO may not be the best method for GraphAF: Indeed, (Shi al. (2020) employs reinforcement learning chase for optimal metric values, which gives more promising results on ZINC250k. However, for fairness of comparison, opt to use BO for all methods. More details are provided in the appendix.
reported in Table One that G consistently finds
CONCLUSION
We have presented generative model G for geometric graphs that are equipped with additional geometry information. Geometric graphs appear many practical applications, a notable one being molecules, whose three-dimensional structure is rarely captured by existing graph generative models. Our model is an autoencoder that employs both GNN architectures and point cloud architectures to handle the complementary structure and geometry information. Moreover, normalizing flow is inserted into the latent space so that one can effectively sample new graphs. Experiment results show that G is able to more accurately learn the distribution of given graphs and better predict molecular properties. It helps discover novel molecules with better properties than do many representative graph generative models.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Pierre Alliez, David Cohen-Steiner, Mariette Yvinec, and Mathieu Desbrun, Variational tetrahedral meshing. In ACM SIGGRAPH 2005 Papers, pp. 617-625, 2005.
Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2008.
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. Preprint
2018.
Sanjivanjit K Bhal. Logp making sense of the value. Advanced Chemistry Development, Toronto, ON, Canada, pp. 1-4, 2007.
G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
Nicolas Bonnotte. Unidimensional and evolution methods for optimal transportation. PhD thesis, Paris 11, 2013.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv: 2013.
Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Le Song. Syntax-directed variational autoencoder for structured data, arXiv preprint arXiv:1802.087380 2018.
Mark Davies, Michat Nowotka, George Papadatos, Nathan Dedman, Anna Gaulton, Francis Atkinson, Louisa Bellis, and John P Overington. Chembl web services: streamlining access to drug discovery data and utilities. Nucleic acids research, 43(W1):W612-W220 2015.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs arXiv preprint arXiv: 1805. 1973, 2018
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 14Xivv/410..88 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: w0w.06000............ 2016.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. Advances in neural information processing systems, 28:2224 2232, 2015.
Peter Frazier. A tutorial on bayesian optimization. arXiv preprint 1rXiv:1807.00 2018
Yaniv Frishman and Ayellet Tal. Online dynamic graph drawing. IEEE Transactions on Visualization and Computer Graphics, 14(4):727-740, 2008.
Niklas WA Gebauer, Michael Gastegger, and T Schütt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. arXiv preprint arXiv:1906. 000577990600077, 2019.
Justin Gilmer, Samuel Schoenholz, Patrick F Rilcy, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. arXiv preprint 1704vv1000.000
2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27:2672-2680, 2014.
10

---

Under review as a conference paper at ICLR 2022
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456.. PMLR. 2015.
Pengfei Jin, Tianhao Lai, Rongje Lai, and Bin Dong. Npte-net: Narrow-band parallel transport convolutional neural network point clouds. arXiv preprint arXiv:1905.12218. 2019.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. arXiv preprint arXiv: arxivv//002 043444364, 2018.
Steven Kearnes, Kevin McCloskey, Mare Bernl,, Vijay Pande, and Patrick Riley. Molecular graph convolutions: moving beyond fingerprints, Journal of computer-aided molecular design, 30(8): 595-608, 2016.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv: 1312.61 14, 2013.
arXiv preprini
Ivan Kobyzev, Simon Prince, and Marcus A Brubaker. Normalizing flows: Introduction and ideas. arXiv preprint arXiv: 090.99998897777 2019.
Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoencoder. arXiv preprint arXiv: 703.0/ 2017.
Tengfei Ma, Jie Chen, and Cao Xiao. Constrained generation of semantically valid graphs via regularizing variational autoencoders. In Advances in Neural Information Processing Systems, pp. 7113-7124, 2018.
Kaushalya Madhawa, Katushiko Ishiguro, Kosuke Nakago, and Motoki Abe. Graphnvp: An invertible flow model for generating molecular graphs. arXiv preprint 1rXiv:1905.11600, 2019.
Daniel Maturana and Sebastian Scherer. Voxnet: 3d convolutional neural network for real-time object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 922-928. IEEE, 2015.
David Mendez, Anna Gaulton, Patrícia Bento, Jon Chambers, Marleen De Veij. Eloy Félix, María Paula Magariños, Juan F Mosquera, Prudence Mutowo, Michal Nowotka, et al. Chembl: towards direct deposition of bioassay data. Nucleic acids research, 47(D1):D930 D940, 2019.
János Pach. The beginnings of geometric graph theory. In Erdős Centennial, pp. 465-484. Springer, 2013.
Jiahao Pang, Duanshun Li, and Dong Tian. Tearingnet: Point cloud autoencoder to learn topology- friendly representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7453-7462, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
Steven M Paul, Daniel Mytelka, Christopher T Dunwiddie, Charles C Persinger, Bernard Munos, Stacy R Lindborg, and Aaron L Schacht. How to improve r&d productivity: the pharmaceutical industry's grand challenge. Nature reviews Drug discovery, 933):033-244 2010.
Pavel G Polishchuk, Timur Madzhidov, and Alexandre Varnek. Estimation of the size of drug-like chemical space based on gdb-1 17 data. Journal of computer-aided molecular design, 27(8):675 679, 2013.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Molecular sets (moses): benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:1931, 2020. 2020.

---

Under review as a conference paper at ICLR 2022
Mariya Popova, Mykhailo Shvets, Junier Oliva, and Olexandr Isayev. Molecularmn: Generating realistic molecular graphs with optimized properties. arXiv preprint arXiv:190S.13372 2019.
Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Volumetric and multi-view cnns for object classification on 3d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 652-660, 2017 7a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets metric space. Advances in neural information processing systems, 30: 5099-5108, 2017b.
Julien Rabin, Gabriel Peyré, Julie Delon, and Mare Bernot. Wasserstein barycenter and its application to texture mixing- In International Conference Scale Space and Variational Methods in Computer Vision, pp.
Springer, 2011.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthis Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):177,001 2014.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of chemical information and modeling, 52(1 28:448--28777 2012.
Bidisha Samanta, Abir De, Gourhari Jana, Vicenç Gómez, Pratim Chattaraj, Niloy Ganguly, and Manuel Gomez-Rodriguezz. Nevae: deep generative model for molecular graphs. Journal of Machine Learning Research, (1144:1-33, 2020.
Stefan C Schonsheck, Bin Dong, and Rongjie Lai. Parallel transport convolution: A new tool for convolutional neural networks on manifolds. arXiv preprint
2018.
Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Müller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241722, 2018.
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a flow-based autoregressive model for molecular graph generation. arXiv preprint arXiv:2001.099 09332. 2020.
Teague Sterling and John J Irwin. Zine 15-ligand discovery for everyone. Journal of chemical information and modeling, 55(11))2324-2337, 2015.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Millen Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pp. 945-953, 2015.
An Tao. http://aww...aoeooooo////////rne
Unsupervised point cloud reconstruction for classific feature learning. 2020.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference on Machine Learning, pp- 2071-2080. PMLR, 2016.
Daan MF Van Aalten, R Bywater, John BC Findlay, Manfred Hendlich, Rob WW Hooft, and Gert Vriend. Prodrg, program for generating molecular topologies and unique molecular descriptors from coordinates of small molecules. Journal of computer-aidee aided molecular design, 10(3):255-22 262, 1996.
12

---

Under review as a conference paper at ICLR 2022
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukeuoglu. WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Autention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint urXiv:/51..0633 2015,
Hanchen Wang. Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt Kusner. Unsupervised point cloud pre-training via occlusion completion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9782-97922 2021.
David Weininger. Smiles, a chemical language and information system. 1. introduction methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.0000 2018.
Yaoqing Yang, Chen Feng. Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 206-215, 2018
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In Advances in neural information processing systems, pp.
8a.
Jiaxuan You, Rex Ying, Xiang Ren, William L Hamilton, and Jure Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. arXiv preprint wrXiv:1802.08775 2018b.
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang. Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. arXiv preprint arXiv:1812.08444, 2018
13

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL DETAILS
Datasets We preprocess each molecule by centering it based on the 3D coordinates and aligning its orientation with the principal axes via PCA. The molecules in the datasets have different number of atoms. Hence, we pad each graph with "phantom nodes" to reach the maximum molecule size the dataset. The phantom nodes are indicated by one of the node features, and we also treat nonexistence of edges as an edge type.
Importantly, the phantom nodes need artificial coordinates. We cannot naively allocate all the phantom nodes to the origin, because overlapping coordinates cause non-unique correspondences in the point-cloud registration, making the learning of atom and edge types impossible. In practice, use an extra coordinate dimension and assign the phantom coordinates for the i-th phantom node as (0, 0, or (0, 0, if the original point cloud is in 2D 3D, respectively.
In addition, we make a distinction between each atom type and its aromatic counterpart whenever possible. For example, carbon atom may be part of aromatic ring, in which case we mark its atom type as aromatic carbon to distinguish it from an ordinary carbon atom. This facilitates the reconstruction of ring structures in the link predictor, as can predict aromatic bonds easily by checking if the atom types are aromatic.
Note that another popularly used benchmark, ZINC250K (Sterling Irwin 2015), does not contain coordinate information and hence we omit it. However, the scales of ChEMBL250K and ZINC250K are rather close, and their results unsurprisingly share similar trends when cross-checked against the performance of baseline models.
Models For G we use the same architecture and identical number of layers and hidden dimensions on both QM9 and ChEMBL250K. without separate fine tuning. Such a practice helps corroborate robustness of the model under different data size and complexity. The latent dimension used for QM9 is (dp. dg) = [10, 30), and [35, 25] for ChEMBL250k. The additional graph latent dimensions is required to accurately decode the diverse atom types present in ChEMBI 220kk, which is more than twice than that QM9.
NAME
MODEL
GENERATION
REPRESENT.
SDVAE
VAE
SEQUENTIAL
SMILES
MOLGAN
GAN
ONE- ONO-SHOT
GRAPH
G-SCHNET
AR
SEQUENTIAL
POINT CLOUD
JT-VAE
VAE
SEQUENTIAL
GRAPH
GRAPHNVP
NF
ONE-SHOT
GRAPH
GRAPHAF
NF
SEQUENTIAL
GRAPH
G
NF
ONE-SHOT
GEOMETRIC GRAPH
Table Summary of baseline models.
We did not check G-SchNet's performance on ChEMBL250k for two reasons. First, G-SchNet excels at reconstructing and generating conformers, which are the 3D structures of molecules that are crucial for various computations in quantum chemistry. The coordinates of ChEMBL250K, however, come from molecular graphs and are all in 2D. Second, G-SchNet's sequential generation process utilizes all previously placed atoms to predict the position of the new atom at each step, which scales poorly with respect to the size of molecules. As many ChEMBL. 2250k molecules have more than 100 atoms after adding Hydrogens, the computational resource required to train a G-SchNet model on ChEMBL250K is likely to be prohibitively high.
All baseline results are obtained by running the implementation provided by the authors, with default hyperparameters. The baseline results generally match the reported ones from the original papers. One discrepancy occurs on GraphAF with QM9, which is caused by the disabling of a filter that ignores small molecules. Turning off the filter amounts to a more fair comparison because small molecules are still sensible results. The V, U, N metrics for MolGAN on QM9 are also somewhat off, which is likely symptom of GAN's training instability.
14

---

Under review as a conference paper at ICLR 2022
Code G is implemented with the Pytorch framework. All relevant code well as the trained models will be made available upon publication.
B ARCHITECTURE DETAILS
In this section, we specify the architecture used to produce the reported experiment results. Let (n, h, o, 1) denote a (point-wise) multi-layer perceptron that has input dimension n, hidden dimension ht output dimension o, and number of hidden layers I. Let MHA(n, h) be the multi- headed attention block with embedding dimension n and h heads. The implementation multi-
headed attention is borrowed from Paszke al. (2019).
Point Encoder As illustrated in Figure the G point encoder is MLP(n, 256, 768, 1) + MHA (256,3) MHA(226,, 3) - GlobalMaxPooling M76 256, dp, 1)
Pointwise
Multi-Hleaded Attention x
Global Max Pooling
X de
MLPP
x h
h
x h
MLP Zp
C
Figure 4: The G Point Encoder.
Graph Encoder Let GINE(h, 1) denote GINE kernel (Xu et al. 2018 with embedding dimen- sion and hidden layers in the MLP. Let Set2Set Set2Set global pooling layer (Vinyals et al.) 2015). We first use linear layers to embed the node and edge features to 256 dimensional vectors. The graph encoder is
G26E 3) - G2NE -> GINE(256, 3) - Sct2Sct - MLP 256, dg, 2)
Batchnorm (Toffe & Szegedy 2015) layer is applicd after every GINE convolution.
Coordinate Decoder This module is inspired by (Pang et al. 2021). Let be the template with the i-th row denoded as u, [zp, 2g] be the latent vector, and Y be the output with the i-th row denoded as s/. Define the FoldingNet F, where F(U, 2) = Y performs the following operation for each point i:
v MLP (u;, =)
v² MLP (yl
5/4 MLP (v?
Define the Net T, where T(U, X, 2) Y'. It performs the following operation for each point
i:
y; + MLP(u: Ii,
y/ MLP (u:, v. 2z, 2)
The coordinate decoder does the following operations:
C² - F(U,))
U - X(U,
C' + F(U),2)
Where C' with the i-th row as of are the decoded coordinates. The MLPs all have two hidden layers with hidden dimensions 1024.
15

---

Under review as a conference paper at ICLR 2022
Feature Decoder Let C" be the decoded coordinates with the i-th denoded d/., and [&p, øg] be the latent vector. Using the same FoldingNet notation as above, the feature decoder does the following:
F' F
Where F" with the i-th row as f are the decoded features. The MLPs all have two hidden layers with hidden dimensions 1024.
Link Predictor Let dz 2((do + de) k)) de. The MLP inside the link predictor is MLP(dx, 1024, 3). 3).
C HYPERPARAMETERS
Experiment on generation quality G For QM9, the latent vector == = 2g] has dimension 10 + 30 and the loss weighting is (A1, A2, λs, A4. Às) = (10, 10-3 10). For ChEMBL250K, the latent dimension is 35 25 and the loss weighting (10, 0.1, 0.1,9,10-4,55
GraphAF: In this model, there an option to filter out generated molecules with fewer than m atoms. Since other models filtering, are more likely to be novel and unique, the scores reported by the original authors are higher than those we obtain. Moreover, have tried best efforts tune hyperparameters for QM9 the authors did not release their setting.
for fair comparison. Because larger molecules
Other models: We use the default hyperparameters released by the authors for QM9. For ChEMBL250K, we borrow those used for ZINC250K. We find this to be reasonable approach because ZINC250K and ChEMBL250k have the same number of molecules, and the maximum number of atoms per molecule is also quite close: 38 and 39, respectively. As such, it is expected that the resulting performance on these two datasets similar, which indeed the case.
Experiment on property optimization process is used fit the explored data, and batch of new candidates suggested based on the of an acquisition function. For all models, we use training molecules as the initial set of points and perform five iterations of Bayesian optimization, wherein the acquisition function is El (expected improvement). In each iteration, 60 candidates are generated.
each iteration of the Bayesian optimization, a Gaussian
The implementation provided by Kusner al. (2017) via custom build of Theano. To conduct Bayesian optimization on the baseline models, we use the default hyperparameters released by the authors whenever available, and follow the practice above otherwise.
D Loss DETAILS
For completeness, in what follows give the detailed expressions for all the training losses. They are generally straightforward or have been defined in the literature.
D.1 COORDINATE Loss
The Wasserstein distance is intimately connected with the study of Optimal Transport. Let (),FF) be measurable space, and P(22) be the set of all probability measures on (Ω, F). Define two probability measures μ, E P2))), where P2(1) {u P(Ω)) Sa [=]12p(dz) oo). The Wasserstein-2 distance Wa W2(n,, p) is defined as:
W3 v)
inf yEC(U.)) Jax || y113 dy)
Where C(p,v) is the set of all couplings between v that satisfies: 7(A Ω) ((A), AA F. 7(2 x B) = (B),, (,,VV' VB E F, and F is the sigma algebra associated with Q. is known that (2(,vv) defines metric P(L) (Ambrosio al. 2008).
16

---

Under review as a conference paper at ICLR 2022
In general, W2 (1,, 2/) is difficult to compute, especially in high dimensions. However, it has simple form in one-dimension:
W2(,v = (t)
Where Fu: Fg denote the cumulative distribution functions associated with μ., v, respectively. The simple one-dimensional form of W2(u, v) inspires the usage of sliced Wasserstein distance, which is the expected value of the projected one-dimensional Wasserstein distance. Define the push-forward operation as: f-p(AA = u(ff=(A)) AА E F, and the projection map as (xx) (8,x) x) for normalized direction 0. Formally, the sliced Wasserstein-2 distance SW2 (H, v) in d-dimension is:
[
SW2(p,
Iga-1 ""((,()))
where d0 is the uniform measure on Sd-1 metric on P(2), and it induces the same topology as W2(,, when Q is compact.
Bonnotte 2013) has shown that SW2(n, 2) also defines a
In practice, we compute SW2(u, v)) by approximating the integral via Monte Carlo sampling direc- tions on the sphere. We then compute the projection of the point clouds onto the sampled directions, sort the one- dieeeninnionall projections, and average their differences. Through experimentation, we found that 200 is an appropriate number of directions to sample.
D.2 NODE FEATURE Loss LF AND EDGE Loss LE
Let C E Raxde be the original point cloud, where each row C₁ = Ct,; gives the coordinates of point. Similarly, let C' Ru be the reconstructed point cloud with of = C. Define the permutation matrix as:
- [1, if = arg ming l|e dlll
oij Ozz 1 otherwise.
(0,
Let F, F be the original and the reconstructed node feature matrices, respectively. The node feature loss is defined as:
1 Σ H(F.., W. n
LF
where F" = of is the reconstructed node feature matrix that is aligned with F; w is the weighting vector for each coordinate; and (Ix y, w) the weighted cross entropy between probability vectors and y:
H(r,y,w)= = Σ waz, log(y:).
Let A, A be the original and the reconstructed adjacency tensors, respectively. The edge loss is defined as:
LE n(n
Σ i=
Σ H(Aij.. A
where A' is the reconstructed adjacency tensor that is aligned with A, satisfying A oA. for 1 = 1,2,... de: and E is the weighting vector for the edge classes. Note the exclusion = in the inner summation; self-loops are not counted.
We use the following weighting for the cross entropies:
N 1,
E = (
2 D)
where is a probability vector that gives the empirical distribution of classes computed from data; and the operations @ and In are element-wise.
17

---

Under review as a conference paper at ICLR 2022
D.3 NORMALIZNG FLOW Loss LNF
Let = E IR be a random vector with density Pa (z) and let æ be a vector resulting from a sequence of K invertible transformations from
fx fK-1 ... f1(2).
(= 1) for i 1,... K, and zk By the change-of-variabe formula, the density of a can expressed as
Define
K
Ofz
log (x) = log (2() -
Σ
log
det
(1)
0z,-1
1=1
If z is standard normal, then
1
D
log (2))
|=1|?
log 2n.
In our work, we use RealNVP Dinh et al. (2016) to parameterize the mappings. Specifically, mapping (w) = v is defined as
Vi:d Ui:d
Ud+1:D ud+1:D @ eXP(MLP, (wI.d)) MLP((
where d = D. One easily calculates that the Jacobian of is
0f
1
0 diag(exp (en(exx(MMPP.... (MLP)
Ou
Therefore,
0s
d
det
= exp
Σ
MLP, (u1:a)
(3)
Ou
Substituting equation 2 and equation
3 into equation
obtain the negative log-likelilhood loss
K
D-d
D 1110011 log 2n.
(x)
Σ
MLP ((z-1))aaa;
2
D.4 ADDITIONAL. LOSS L.R.
We follow the approach outlined in Ma et al. (2018) to introduce a soft valence regularization to penalize the cxcess of valence capacity for each atom. thereby encouraging the decoder to output chemically valid molecules. In particular, define Ln = Σ (V Uii) for each molecule, where is the ReLU function and (i) and (ii) the valence and capacity of atom respectively. The soft (thus differentiable) version is:
U(1) Σ Fl.
V(I) ΣΣ ΣΣM(4)A
where F" and A' are the reconstructed and aligned node feature matrix and adjacency tensor, re- spectively, Here, u(r) the valence of node type r, determined from data; and (kk) is the capacity for edge type k. Specifically, l((k) = 1, 2, 3, 1.5 for single, double, triple, and aromatic bonds, respectively. Moreover, u(r) and h(k)) are both for phantom nodes and edges.
E G PROPERTIES
A typical pitfall for many graph-based models is the ambiguity under node permutations. Reordering the nodes does not change the graph itself, but the latent representation of the graph may change properly modeled. Moreover, single latent vector may not be able to decode an adjacency tensor with an arbitrary node permutation. Thanks to the geometry, our model G able to identify node correspondence and maintain desirable permutation and translation invariance.
18

---

Under review as a conference paper at ICLR 2022
Proposition E.1. The G encoder and link predictor are invariant to input node permutations well as translations on the coordinates.
Proof. The G encoder is composed of point encoder and graph encoder. The point encoder is invariant to node permutations because the MLP is weight sharing for all points and the global pooling is permutation invariant. The graph encoder is also invariant to node permutations because the graph convolution layers of MoNet are. The overall latent vector is concatenation of the outputs of these two encoders and thus also invariant to node permutations. Additionally, the point clouds preprocessed that each is centered at the origin, invariant to translations.
The link predictor takes pairwise nodes as input and is therefore invariant to node permutations. In addition, node coordinates are used always in the form of displacements. Hence, the link prediction always unchanged regardless of translation.
F QUALITATIVE EXAMPLES
In Figure 5 we show the generated examples for QM9 as well as results of latent space interpolation. They correspond to the results reported in Table The generated samples and interpolation results for ChEMBL250k are in Figure 6. which correspond to the results reported in Table 2
At
Figure 5: Generated molecules for QM9
19

---

Under review as a conference paper at ICLR 2022
pho fo 8io
nao fast ap mao qro awp
s'a ptoo o go
tor Eoo meat mn0 no
9r at of Do mR
p dio oQp oper Lout
gor my
AO
Figure 6: Generated molecules for ChEMBL250k
20

---

Under review as a conference paper at ICLR 2022
G EXPERIMENTS ON THE STANDALONE LINK PREDICTOR
In this section, we empirically demonstrate the effectiveness of our link predictor as a standalone model. The task to predict the edge types for all edges in the graph given the ground truth node types and coordinates for all nodes. We train the link predictor on QM9 and ChEMBL250K until convergence, then report the accuracy and the F-1 score on the test set. The classification reports are computed by using scikit-learn, where the "ZERO" type corresponds to the non-existence of bond. In addition, we provide visualized comparisons between the ground truth molecules and the reconstructed molecules with ground truth nodes and predicted edges.
EDGE TYPE
PRECISION
RECALL
F-1 SCORE
SUPPORT
ZERO
0.99999
0.99999
0.99999
778401
SINGLE
0.99855
0.99842
0.99848
198214
DOUBLE
0.98009
0.98150
0.98080
15246
TRIPLE
0.99972
0.99972
0.99972
7234
AROMATIC
0.99974
1,00000
0.99987
15592
AGGREGATE METRICS
VALUE
VALUE
VALUE
SUPPORT
MACRO AVG
0.99562
0.99593
0.99577
1014687
WEIGHTED MACRO AVG
0.99941
0.99940
0.99940
1014687
ACCURACY
0.99940
1014687
Table 7: Standalone G link predictor's performance on the test set of QM9
EDGE TYPE
PRECISION
RECALL
F-1 SCORE
SUPPORT
ZERO
0.99984
0.99982
0.99984
37573897
SINGLE
0.97573
0.98228
0.97899
714020
DOUBLE
0.91250
0.87825
0.89505
89694
TRIPLE
0.98789
0.97759
0.98271
3838
AROMATIC
0.99904
0.99917
0.99910
633722
AGGREGATE METRICS
VALUE
VALUE
VALUE
SUPPORT
MACRO AVG
0.97500
0.96742
0.97114
3901517
WEIGHTED MACRO AvG
0.99920
0.99921
0.99920
39015171
ACCURACY
0.99921
39015171
Table Standalone G link predictor's performance test of ChEMBL250K
We remark that while link prediction is pairwise on nodes, the inclusion of neighbor information allows faithfully reconstruct complex structures like aromatic rings. Overall, G link predictor attains near perfect aggregate F-1 score and accuracy on both QM9 and ChEMBL250K. It therefore serves as viable standalone model for bond prediction in molecules.
21

---

Under review as a conference paper at ICLR 2022
Figure 7: Left: 50 ground truth molecules from the QM9 test set. Right: the same molecules with edges reconstructed from the ground truth nodes by the link predictor. All 50 molecules in the left figure are reconstructed exactly.
22

---

Under review as a conference paper at ICLR 2022
r Drg dec
p0mp"
Drg 8lol apou wpppp gofoo
gordy -
nooto 8ool ang
500
30 m0s poak ard
505
mas pour oi
320
6
tix 500 dap
tash 2op ga
B
foot poad ba g 80p
13pp
goo
fot
: 600
Arg 2ov
poad
600 fo:
.ox
yoody Arg
fot
youay
~ taun ga
fot y poo
FF. tauno $66
guae dcea
gridy
pea gru grea
sordy ang
oxp
and
ord
Sx²
ox
ofp
guc
0x00x
ox
oupll
glac d1ox
Soiph
doa
ando
doo
ando
Figure 8: Left: 50 ground truth molecules from the ChEMBL250K test set. Right: the same molecules with edges reconstructed from the ground truth nodes by the link predictor. 46 out of 50 molecules in the left figure are reconstructed exactly.
23

---

Under review as a conference paper at ICLR 2022
H VISUALIZATION OF GEOMETRIC FEATURES
In this section, we provide qualitative examples for the learned geometric features the point encoder. Similar to Wang et al. (202I, I. we present the channel-wise activation before the global pooling step in the point encoder of G pretrained on ChEMBI 250kk. Intuitively, the difference the level of activation reveals the geometric primitives that the neurons focus on.
In general, we observe that S point encoder can identify aromatic rings molecules based purely on geometry; i.e. bond information is provided. For example, Figure shows that the 63rd neuron identifies rings to the right end of the molecules (see the atoms).
In addition, different neurons can partition molecule into meaningful sub-components. In Figure TO
we show neurons that are excited by the five-member ring on the left, the benzene ring in the middle, and the benzene ring the right, respectively. In Figure IT the top figure presents neuron that identifies two rings on the right as group, whereas the bottom figures showcase more discerning neurons that distinguish the two rings separately.
Figure 9: Consistency of neuron activation on six-member rings Top row (from left to right): the molccular graphs of CwCCCC/CCCcccCCCCCoocccccccooooC Celeone(N2CC CCCCCCCCCCAAAecccCCCCooooddddde activation on the associate molecules, singling out (in red) the rings the right.
CCCcc(CCCCCCNNCCC)))))CC-C-- Bottom row: the 63rd neuron's
and
24

---

Under review as a conference paper at ICLR 2022
Figure 10: Top left: The molecular graph for CCOOGC/0NCCCNCCCCCCCCCCeceeCeee
Top
right: the 172nd neuron's 's activation on this molecule as point cloud, identifying the five-member ring on the left. Bottom left: the 24 neuron's 's activation on this molecules, identifying the benzene ring in the middle. Bottom right: the 57 1st neuron's activation this molecule, singling out the benzene ring the right. Red indicates higher activation values.
25

---

Under review as a conference paper at ICLR 2022
M112
Figure 11: Top left: The molecular graph of CCCeDD/CCCCOOXXNCCCCCCeeeeeeeeea
Top right: the 56th neuron's 's activation on this molecule, identifying the two rings on the right as group. Bottom left: the 663rd neuron's 's activation on this molecule. singling out the six-member ring on the right. Bottom right: the 670th neuron's activation on this molecule, identifying the five-member ring the right. Red indicates higher activation values.
26


Reviews:
[{"review_id": "YDzeIZh14jh", "reviewer": "Reviewer_PjEF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n\n1. It is true that restoring a graph is challenging. Works [1][2] at the same period tackle it via matching the representations in the latent space instead of defining reconstruction loss on the data space. This paper proposes a different way via separating (or..cascading) 3d and feature decoding processes, whose effectiveness and efficiency are empirically validated.\n\n2. a good integration of previous works, especially connecting generative models in point clouds and molecules.\n\n\n**Weakness**\n\n1. Link predictor is quite interesting (and overselling), more empirical results and interpretations will be helpful to support the claim of \"We highlight that our link predictor works remarkably well as a standalone module for molecules, ...\"\n2. Unclear when interpreting tables 1 & 2. How to determine the angle directions? In comparisons with all the baselines or a certain subset of baselines (e.g., methods without using hard valency checks)?\n3. Table 3 seems not to well support the claim of \"It is clear that both ablated models are inferior to G 3 in nearly all molecular metrics, which attests to the efﬁcacy of G 3 ’s overall architecture.\", for instance, Validity and Novelty are decreasing when combining both encoders, and so on. More elaborations?\n\n**Comments**\n\n1. What do you mean by the term \"large inner product in 3D\" (page 5)?\n2. It would be helpful to bold the best results in the tables.\n\n**Open Questions**\n\nNot addressing the following comments won't have negative effects on my rating. \nBut it would be nice if the authors can make a few clarifications, which also helps with my understandings!\n1. explorations on the template design in the coordinate decoder\n2. interpretability on the learned geometrical information in the point cloud encoders for molecules (something like [3])\n3. a better (analytical/empirical) way to support the claim that geometry and structure are \"complementary\"? I don't find an inspiring explanation in [1] or in this draft.\n\n**Reference**\n\n1. Pre-training Molecular Graph Representation with 3D Geometry, arXiv 2021\n2. 3D Infomax improves GNNs for Molecular Property Prediction, arXiv 2021\n3. Unsupervised Point Cloud Pre-Training via Occlusion Completion, ICCV 2021", "summary_of_the_paper": "This paper is well written and focuses on an interesting problem, to design a better framework for a molecular generation while incorporating 3D geometry, using connections between 3D point clouds and GNNs.", "summary_of_the_review": "In general, I like the idea but given the fact a lot of concerns need to be addressed, I tend to give a borderline. \n\nHowever, I am willing to adjust my scores based on the reviewer's reply.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PEjxos7lE1w", "reviewer": "Reviewer_qSkw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written and well motivated, and easy to follow. The network is comprised of multiple parts which are clearly defined and explained.\nThe numerical experiments indeed show an improvement over other methods, however not in all metrics.\n\nI have a few concerns:\n1. If I understand correctly, the authors choose to first learn the geometry, via template folding (as in FoldingNet), and then, once the node features are learned, use a link prediction mechanism to predict the topology of the graph. To the best of my knowledge, the approach of template folding is bound to its original topology (e.g. genus 0), which is limited. It would be beneficial and enriching if the authors can elaborate on this point. Were other methods considered to learning the geometry, and perhaps the authors can state what is the influence of using such template, which in many cases may not fit to the actual topology of the underlying graph.\n\n2.How did the authors choose the link prediction mechanism? In what sense is it different/better than using something like GAT and its variants ?\n\n3.The authors present their results on 2 popular datasets. However, in order to reach to a more conclusive evidence of the importance and contribution of this work, I would suggest the authors to add more applications, such as learning the geometry and topology of meshes (e.g., ModelNet, ShapeNet). Also, the task of protein folding (e.g., AlphaFold) seems to be appropriate for the suggested method.\n\n", "summary_of_the_paper": "The authors propose to combine point cloud and graph encoders with coordinate and feature decoders in order to learn the geometry first, and then try to predict the adjacency matrix of the graph, to obtain a better overall graph representation of the considered applications.\n", "summary_of_the_review": "The authors propose an interesting combination of known methods to learn the geometry and topology of the  graph for molecular-based applications. The results reveal that some improvements over existing method is obtained. Some of the limitation could be further discussed, and more experiments would strengthen the contribution of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7kEhlC68VCV", "reviewer": "Reviewer_LcmV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this is a well-written paper with clear motivations and description of each model component. The problem of generating geometric graphs is important and the paper demonstrates that the proposed G^3 can leverage both graph structures and the point cloud geometry well. The experiments are extensive and the results are clear with domain-knowledge interpretation.\n\nOn the negative side, it looks like many components of G^3 are be studied before, hence reducing the novelty of the paper.", "summary_of_the_paper": "The paper proposes a generative models for learning over the space of geometric graphs -- those graphs whose nodes are associated with geometric coordinates (point clouds). The point cloud serves as basis for decoding graph structure. This makes the entire system efficient. An extensive suite of experiments on chemical benchmarks (QM9 and ChEBML) show that the proposed method is competitive against state-of-the-art rivals on a variety of measures and tasks.\n", "summary_of_the_review": "This is a nice paper on an important topic and with a successful proposal evaluated extensively. The novelty may be less than ideal due to the integration of existing ideas.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1Nd8jBWvPPH", "reviewer": "Reviewer_rFAq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The run time of G3 is impressive compared with strong baselines such as  GraphAF and JT-VAE due to the one-shot nature while obtaining as good in quality as these baseline methods. The designed method may be applied to the application of drug discovery, etc.\n\nWeaknesses: The technical contribution is unclear/limited. The comparison (from both the theoretical side and architecture design side) between G3 and other popular baseline methods is missing. It is unclear why G3 is better than others and why it is so efficient but not other methods. How this designed architecture is useful to guide other works? The other concern is the generalization ability of the proposed method since the proposed method was only tested on two datasets.", "summary_of_the_paper": "This paper studies the problem of geometric graph generation (mainly focusing on molecule graph generation). Specifically, the authors propose a new method,  namely Geometric Graph Generator (G3), which generates three-dimensional geometric graphs. Different from others, G3 can capture both the combinatorial and the geometric information of graphs. The proposed algorithm can successfully generate molecule graphs and runs faster than other baseline methods. The key parts of G3 are the point encoder and graph encoder, and the three-step decoding process, that is the coordinate decode, feature decoder, and finally graph decoder.  The experimental results show that G3 has better performance than other generators in terms of validity, uniqueness, and novelty.\n", "summary_of_the_review": "Overall, the problem studied is interesting. The authors aim at designing new NN architecture to generate geometric graphs. The experimental results look promising. Moreover, G3 is much faster than current strong baseline methods. \n\nHowever, the generalization ability of the proposed architecture to other datasets is unknown. The encoding part seems heavily based on other works (similarly decoder parts). Compared with other methods, the reviewer does not see the real novelty part of the technical section. (The problem of graph generation is out of my research area. I may miss something important.)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a generative model for geometric graphs.  The main contribution is to separate the representation and generation of geometry from that of graph structure and features.  Based on this idea the authors assembled a set of existing ideas and built an auto-encoder style generative model for geometric graphs.

This paper sits on the borderline, with reviewers split on both sides.  I appreciate the clarifications from the authors during the rebuttal and the interactions with the reviewers.  The main concern is the novelty of this approach, as the main contribution is the idea of separating geometry from graph structure, and most other components of the pipeline already exist in the literature.  Because of this I think the paper can probably devote a bit more to this ablation study.  In particular the paper currently lacks detail about whether the size of the models were controlled when doing the ablation, which could be a confounding factor that explains why the joint model with both geometry and graph structure works better.  Also the different architecture choices may also factor into the difference, it would be more convincing if for example the same combination of multi-head attention blocks and GINE networks are used for the ablated graph encoder (you can simply concatenate the features from both on all layers, or even at the end).

Based on this I would recommend rejection at this time but encourage the authors to improve the paper and send it to the next venue.
Decision: Reject