GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation
Authors: Chence Shi*
Minkai Xu*
Zhaocheng Zhu
Weinan Zhang
Ming Zhang
Jian Tang
Venue:  (2020)
Keywords: Molecular graph generation
deep generative models
normalizing flows
autoregressive models

Abstract: Molecular graph generation is a fundamental problem for drug discovery and has been attracting growing attention. The problem is challenging since it requires not only generating chemically valid molecular structures but also optimizing their chemical properties in the meantime. Inspired by the recent progress in deep generative models, in this paper we propose a flow-based autoregressive model for graph generation called GraphAF. GraphAF combines the advantages of both autoregressive and flow-based approaches and enjoys: (1) high model flexibility for data density estimation; (2) efficient parallel computation for training; (3) an iterative sampling process, which allows leveraging chemical domain knowledge for valency checking. Experimental results show that GraphAF is able to generate 68\% chemically valid molecules even without chemical knowledge rules and 100\% valid molecules with chemical rules. The training process of GraphAF is two times faster than the existing state-of-the-art approach GCPN. After fine-tuning the model for goal-directed property optimization with reinforcement learning, GraphAF achieves state-of-the-art performance on both chemical property optimization and constrained property optimization. 

Published as a conference paper at ICLR 2020
GRAPHAF: A FLOW-BASED AUTOREGRESSIVE MODEL FOR MOLECULAR GRAPH GENERATION
Chence Shi*! Minkai Xu Zhaocheng Zhu Weinan Zhang?, Ming Zhang', .Jan Tang
"Department of Computer Science. Peking University, China
"Shangaai Jiao Tong University, China
3 Mila Québec AI Institute, Canada
Université de Montréal, Canada
6 HHEC Montréal, Canada
CCIARR AI Research Chair
[chenceshi mzhang_cs) Saapscsssppppp @epku. eduu.ceddcco {mkxu, wnzhang)@apent sjtu. edu, on zhaocheng. whw@umont jian. tang@hec.ca
ABSTRACT
Molecular graph generation is fundamental problem for drug discovery and has been attracting growing attention. The problem is challenging since it re- quires not only generating chemically valid molecular structures but also optimiz- ing their chemical properties in the meantime. Inspired by the recent progress in deep generative models, in this paper we propose flow-based autoregressive model for graph generation called GraphAF. GraphAF combines the advantages both autoregressive and flow-based approaches and enjoys: (1) high model flexibilityy for data density estimation; (2) efficient parallel computation for train- ing; (3) an iterative sampling process, which allows leveraging chemical domain knowledge for valency checking. Experimental results show that GraphAA is able to generate 68% chemically valid molecules even without chemical knowledge rules and 100% valid molecules with chemical rules. The training process of GraphAF is two times faster than the existing state-of-the-art approach GCPN. After fine-tuning the model for goal-directed property optimization with reinforce- ment learning. GraphAF achieves state-of of of-the-art performance on both chemical property optimization and constrained property optimization
INTRODUCTION
Designing novel molecular structures with desired properties is a fundamental problem in a variety of applications such drug discovery and material science. The problem is very challenging, since the chemical space is discrete by nature, and the entire search space is huge, which is believed to be large 1033 (Polishchuk et al. 2013}. Machine learning techniques have a big opportunity in molecular design thanks to the large amount of data in these domains. Recently, there are increas- ing efforts in developing machine learning algorithms that can automatically generate chemically valid molecular structures and meanwhile optimize their properties.
Specifically, significant progress has been achieved by representing molecular structures graphs and generating graph structures with deep generative models, e.g., Variational Autoencoders (VAEs) (Kingma Welling 2013), Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) and Autoregressive Models Van Oord et al. 2016). For example, Jin et al. (2018) proposed Junction Tree VAE (JT-VA) VAE) for molecular structure encoding and decoding. De Cao & Kipf (2018) studied how to use GANs for molecular graph generation. You et al. (2018a) proposed an approach called Graph Convolutional Policy Network (GCPN), which formulated molecular graph genera- tion sequential decision process and dynamically generates the nodes and edges based on the
"Equal contribution, with order determined by flipping a coin. Work was done during internship at Mila.
Code available at

---

Published as a conference paper at ICLR 2020
Table 1: Previous state-of-the-at algorithms for molecular graph generation. The comparison of training
only conducted between autoregressive models.
Generative Model
Sampling Process
Training Process
Name
VAE
GAN
RNN
Flow
One-shot
Iterative
Sequential
Parallel
JT-VAE
RVAE
GCPN
v
MRNN
V
GvaphNVP
v
V
GraphAF
V
existing graph substructures. They used reinforcement learning to optimize the properties of gener- ated graph structures. Recently, another very related work called MolecularRNN (MRNN) (Popova et al. 2019) proposed to use an autoregressive model for molecular graph generation. The autore- gressive based approaches including both GCPN and MRNN have demonstrated very competitive performance in variety of tasks on molecular graph generation.
Recently, besides the aforementioned three types of generative models, normalizing flows have made significant progress and have been successfully applied to variety of tasks including density esti-
mation (Dinh et al. 2016 Papamakarios et al.) (2017), variational inference (Kingma et [016] Louizos & Welling 2017 Rezende & Mohamed 2015). and image generation (Kingma & Dhariwal
2018]. Flow-based approaches define invertible transformations between a latent base distribution
(e.g. Gaussian distribution) and real-world high nigh-dimensional data (e.g. images and speech). Such an invertible mapping allows the calculation of the exact data likelihood. Meanwhile, by using multiple layers of non-linear transformation between the hidden space and observation space, flows have high capacity to model the data density. Moreover, different architectures can be designed to pro- mote fast training (Papamakarios et al. (017) or fast sampling (Kingma et al. 2016) depending the requirement of different applications.
Inspired by existing work autoregressive models and recent progress of deep generative mod- els with normalizing flows, we propose a flow-based autoregressive model called GraphAF for molecular graph generation. GraphAF effectively combines the advantages of autoregressive and flow-based approaches. It has a high model capacity and hence is capable of modeling the density of real-world molecule data. The sampling process of GraphAF designed as an autoregressive model, which dynamically generates the nodes and edges based on existing sub-graph structures. Similar to existing models such GCPN and MRNN, such sequential generation process allows leveraging chemical domain knowledge and valency checking in each generation step, which guar- antees the validity of generated molecular structures. Meanwhile, different from GCPN and MRNN autoregressive model during training. GraphAF defines feedforward neural network from molecular graph structures to the base distribution and is therefore able to compute the exact data likelihood in parallel. As result, the training process of GraphAF is very efficient.
We conduct extensive experiments on the standard ZINC (Irwin et al. 2012) dataset. Results show that the training of GraphAF is significantly efficient, which is two times faster than the state-of-the- art model GCPN. The generated molecules are 100% valid by incorporating the chemical rules dur- ing generation. We are also surprised to find that checking during generation, the percentage of valid molecules generated by GraphAF can be still as high as 68%, which is significantly higher than existing state-of-the-an GCPN, This shows that GraphAF indeed has the high model capability to learn the data distribution of molecule structures.
without using the chemical rules for valency
We further fine-tune the generation process with reinforcement learning to optimize the chemical properties of generated molecules. Results show that GraphAF significantly outperforms previous state-of-the-art GCPN on both property optimization and constrained property optimization tasks.
RELATED WORK
variety of deep generative models have been proposed for molecular graph generation re- cently (Segler et al. 2017 Olivecrona et al.) 2017 Samanta al. 2018 Neil et al. 2018). The RVAE model (Ma et al. 2018) used a variational autoencoder for molecule generation, and pro- posed novel regularization framework to ensure semantic validity. [ii et al. (2018) proposed to
2

---

Published as a conference paper at ICLR 2020
represent a molecule as a junetion tree of chemical scaffolds and proposed the JT-VAE model for molecule generation. For the bas-b approaches, the optimization of chemical properties is usually done by searching in the latent space with Bayesian Optimization (Jin et al. 2018 De Cao & Kipf (2018) used Generative Adversarial Networks for molecule generation. The ffae-of-the-an models are built on autoregressive based approaches (You al. 201 [Popova et al., 2019). (You
et al., 2018a) formulated the problem as sequential decision process by dynamically adding new nodes and edges based on current sub-graph structures, and the generation policy network trained by reinforcement learning framework. Recently, Popova et al. (2019) proposed an autoregressive model called MolecularRNN to generate new nodes and edges based on the generated nodes and edge sequences. The iterative nature of autoregressive model allows effectively leveraging chemical rules for valency checking during generation and hence the proportion of valid molecules generated by these models is very high. However, due to the sequential generation nature, the training process is usually slow. Our GraphAF approach enjoys the advantage of iterative generation process like au- toregressive models (the mapping from latent space to observation space) and meanwhile calculates the exact likelihood corresponding to a feedforward neural network (the mapping from observation space to latent space), which can be implemented efficiently through parallel computation.
Two recent work work-Grape Normalizing Flows (GNF) (Liu et al. 2019) and GraphNVP (Madhawal eet al. 2019)aare also flow-based approaches for graph generation. However, our work is funda- mentally different from their work. GNF defines a normalizing flow from a base distribution to the hidden node representations of pretrained Graph Autoencoders. The generation scheme is done through two separate stages by first generating the node embeddings with the normalizing flow and then generate the graphs based on the generated node embeddings in the first stage. By contrast, in GraphAF, we define an autoregressive flow from a base distribution directly to the molecular graph structures, which can be trained end-to-end. GraphNVP also defines normalizing flow from a base distribution to the molecular graph structures. However, the generation process of GraphNVP is one-shot, which cannot effectively capture graph structures and also cannot guarantee the validity of generated molecules. In our GraphAF, we formulate the generation process sequential decision process and effectively capture the sub-graph structures via graph neural networks, based on which define policy function generate the nodes and edges. The sequential generation process also allows incorporating the chemical rules. As a result, the validity of the generated molecules can be guaranteed. We summarize existing approaches in Table
PRELIMINARIES
3.1 AUTOREGRESSIVE FLOW
A normalizing flow (obyzey et al. 2019) defines a parameterized invertible deterministic trans- formation from base distribution (the latent space, e.g., Gaussian distribution) real-world observational space Z (e.g. images and speech). Let f : E + Z be an invertible transformation where e (e) is the base distribution, then we can compute the density function of real-world data z, i.e. Pz (2), via the change-of-vartie variables formula:
Pz(2)= PE (fo (x)) (x)) det 05 0
02
Now considering two key processes of normalizing flows a generative model: (1) Calculating Data Likelihood: given a datapoint z, the exact density Pz(2) can be calculated by inverting the transformation f. c = fo (2); (2) Sampling: 2 can be sampled from the distribution pz (z) by first sample (e) and then perform the feedforward transformation
fo(e). To efficiently perform the above mentioned operations, So is required to be invertible with an easily computable Jacobian determinant. Autoregressive flows (AF), originally proposed in Paaaaaaaariss al. (2017), is a variant that satisfies these criteria, which holds triangular Jacobian matrix, and the determinant can be computed linearly. Formally, given 2 E R D (D is the dimension of observation data), the autoregressive conditional probabilities can be parameterized as Gaussian distributions:
p(2z|=1-a-1) 1) Mazalua (aa)?), where μa ga (2 1;0), as = go. (znd-115)))
(2)
where 9n and g., are unconstrained and positive scalar functions of 21:d11 respectively to compute the mean and deviation. In practice, these functions can be implemented as neural networks. The
3

---

Published as a conference paper at ICLR 2020
affine transformation of AF can be written
fa(ca) = zd = paa + Odd cdi So (zd) = <d Fd Pd
The Jacobian matrix in AF triangular, since 5e,
az is non-zero only for j S i. Therefore, the determi-
nant can be efficiently computed through I ag. Specifically, to perform density estimation, we can apply all individual scalar afñin transformations in parallel to compute the base density, each of which depends on previous variables 21:d-1; to sample we can first sample IR and compute 21 through the affine transformation, and then each subsequent 2d can be computed sequentially based on previously observed 21:d-1-
3.2 GRAPH REPRESENTATION LEARNING
Following existing work, also represent molecule a graph G (A, where A is the adjacency tensor and is the node feature matrix. Assuming there are n nodes in the graph, and are the number of different types of nodes and edges respectively, then
{0, 1] jx and X E [0,1] Aijk = 1 if there exists a bond with type k between jth and juh nodes.
Graph Convolutional Networks (GCN) (Duvenaud et al. 2015 Gilmer al. 2017 Kearnes al., 2016 Kipf & Welling 2016; Schütt et al., (017)) ) are a family of neural network architectures graphs. In this paper, we use a variant of Relational GCN (R- GCN) (Schlichtkrull et al. 2018) to learn the node representations (i.e. atoms) of graphs with cat- egorical edge types. Let i denote the embedding dimension. We compute the node embeddings H RR at the 1th layer of R-GCN by aggregating messages from different edge types:
for learning representations
H' Agg (ReLU & H e (I.....b)))
(4)
where Fg Ap..a denotes the 7th slice of edge-conditioned adjacency tensor, F, = F. + and is a trainable weight matrix for the 7th edge type. Agg(-) denotes an
D = E. E. k. w:!h) aggregation function such as mean pooling or summation. The initial hidden node representation H set as the original node feature matrix X. After T. message passing layers, we use the the final hidden representation H as the node representations. Meanwhile, the whole graph representations can be defined by aggregating the whole node representations using a readout function (Hamilton) al. 2017). e.g., summation.
4 PROPOSED METHOD
4.1 GRAPHAF FRAMEWORK
Similar to existing works like GCPN You et al. 2018a) and MolecularRNN (Popova et al. 2019 formalize the problem of molecular graph generation sequential decision process. Tet G (A, X)) denote a molecular graph structure. Starting from an empty graph G, in each step new node X, generated based on the current sub-graph structure Gū, i.e., p(X, Afterwards, the edges between this new node and existing nodes are sequentially generated according to the current graph structure, i.e., p(A,yICG [G,, X;, X,, 1,1 This process is repeated until all the nodes and edges are generated. An illustrative example is given in Fig. ((a)
GraphAF is aimed at defining an invertible transformation from a base distribution (e.g. multivariate Gaussian) to a molecular graph structure G = (A, X). Note that we add one additional type of edge
between two nodes, which corresponds to no edge between two nodes, i.e., A E {0, 1) 1 (b++1) Since both the node type X, and the edge type Aij are discrete, which do fit into flow-based model, standard approach is to use Dequantization technique (Dinh et al.) 2016 Kingma & Dhari- wal. 2018) to convert discrete data into continuous data by adding real-valued noise. We follow this approach to preprocess discrete graph G (A, X) into continuous data (2d xx)x;
X3 + u, u U 1)4; N
Aij + u, a 1)
(5)
4

---

Published as a conference paper at ICLR 2020
X
81
X2
82
E-1-1
3
E31-- I
Edi- I
A21
E21
Noise ( I)
Node: Atom
Parallel Training
E32-+ +
E42-+ !
Edge: Single bond
X1
E1
Edge: Double bond
2
Edge: No bond
X2
8/2
Affine Transformation for Node G enneration
E43-+1 |
A21
F21
Affine Transformation for Edge G eneration
1
Sampling / TTriiiig Order
Sequential Sampling
(a) Sampling Phases
(b) Framework
Figure 1: Overview of the proposed GraphAF model. (a) Illustration of the generative procedure. New nodes
edges marked red. Starting from an empty graph and iteratively sample random variables map them atom/bond features. numbered first three steps correspond to the maps in bottom figure Fig. [Tb] (b) Computation graph of GraphAF. The left side are the nodes and edges and the right are latent variables.
We present further discussions on dequantization techniques in Appendix the conditional distributions for the generation as:
A Formally, we define
p(2 G.)) =N (a)))²)
(6)
where 11," Jnx (Ge), aj = (Gi).
p(233 Gt, X,, 1,1-11) = (uij (a j where 1 9,, -G: X,, A1,15-1),, = -. (G;, X,, A.1--1),
1},
where 4 and - 4 are parameterized neural networks for defining the mean and standard deviation a Gaussian distribution. More specifically, given the current sub-graph structure G, we I--layer of Relational GCN (defined in Section 3.2) learn the node embeddings H.P RRa and the embedding entire sub-graph h2 E R. based on which define the mean and standard deviations of Gaussian distributions generate the nodes and edges respectively:
R-GCN: H. = R-GG h, sum(H+))
Node-MLPPs 9μ.x ,,* (Tie), Jax = -max (his)
(8)
Edge-MLPs: 9nA mn (his H. H( Ian = man (fy, H H!)))
where sum denotes the sum-pooling operation, and Hi,, R& denotes the embedding of the j-th node in the embeddings H.¹ m,,x, m,x are Multi-Layer Perceptrons (MLP) that predict the node types according the current sub-graph embedding. and m,,a, m MLPs that predict the types of edges according to the current sub-graph embedding and node embeddings.
To generate
node X; and its edges connected existing nodes, just sample random variables 6; and Gj from the base Gaussian distribution and convert it to discrete features. More specifically,
= 3 + p E ER..
2/ Eig o m jE i 1}, Ei E g
where @ is the element-wise multiplication. In practice, a real molecular graph is generated by taking the argmax of generated continuous vectors, i.e., X, = nd argmax(=;") and Ai = -bb+ where oe denotes dimensional one-hot vector with gth dimension equal to 1.
"aggma(=dd)""
Let = {e1, 62, 621 E3, E31 E. En C,....., Cn.n1))) where n is the number of atoms in the given molecule, GraphAF defines an invertible mapping between the base Gaussian distribution
5

---

Published as a conference paper at ICLR 2020
and the molecule structures ,, xx According to Eq. 9 the inverse process from (z x²) to e can be easily calculated as:
1
G (2)
x
Gy = (2) mnj) ...je j (1,2...... 13,
(10)
where and a denote element wise reciprocals of ag and at respectively. of
4.2 EFFICIENT PARALLEL TRAINING
In GraphAF, since E 2 is autoregressive, the Jacobian matrix of the inverse process f-1 2 + is a triangular matrix, and its determinant can be calculated very efficiently. Given a mini- batch of training data G, the exact density of each molecule under given order can be efficiently computed by the change-of-variabe of training data.
formula in Eq. Our objective is to maximize the likelihood
During training, we are able to perform parallel computation by defining feedforward neural network between the input molecule graph G and the output latent variable by using masking. The mask drops out some connections from inputs to ensure that R-GCN is only connected to the uub-graph G; when inferring the hidden variable of node i, i.e., Gis and connected to sub-graph G,, 1 when inferring the hidden variable of edge A.z. i.e., Cij- This is similar to the ap- proaches used in MADE (Germain et al., 2015) and MAF (Papamakarios et al. 2017). With the masking technique, GraphAF satisfies the autoregressive property, and at the same time p(G) can be efficiently calculated in just one forward pass by computing all the conditionals in parallel.
To further accelerate the training process. the nodes and edges of training graph are re-ordered according to the breadth-first search (BFS) order, which is widely adopted by existing approaches for graph generation You et al. 2018b; Popova et al., 2019). Due the nature of BFS, bonds can only be present between nodes within the same or consecutive BFS depths. Therefore, the maximum dependency distance between nodes is bounded by the largest number of nodes in a single BFS depth. In our data sets, any single BFS depth contains no more than 12 nodes, which means we only need model the edges between current atom and the latest generated 12 atoms.
Due to space limitation, we summarize the detailed training algorithm into Appendix
4.3 VALIDITY CONSTRAINED SAMPLING
In chemistry, there exist many chemical rules, which can help to generate valid molecules. Thanks to the sequential generation process, GraphAF can leverage these rules in each generation step. Specifically, we can explicitly apply valency constraint during sampling check whether current bonds have exceeded the allowed valency, which has been widely adopted in previous models (You) jtt al. 2018a; Popova et al. 2019). Let |Aigl denote the order of the chemical bond Aij- In each edge generation step of A,j. we check the following valency constraint for the 7th and jth atoms:
Σ |Asl S Valency(X,) and Σ |Aal S Valency(Xx).
(11)
If the newly added bond breaks the valency constraint, we just reject the bond Aij, sample new Gj in the latent space and generate another new bond type. The generation process will terminate one of the following conditions is satisfied: 1) the graph size reaches the max-size n, 2) no bond is generated between the newly generated atom and previous sub-graph. Finally, hydrogens are added to the atoms that have not filled up their valencies.
4.4 GOAL-DIRECTED MOLECULE GENERATION WITH REINFORCEMENT LEARNING
So far, we have introduced how to use GraphAF to model the data density of molecular graph structures and generate valid molecules. Nonetheless, for drug discovery, we also need to optimize the chemical properties of generated molecules. In this part, we introduce how to fine-tune our generation process with reinforcement learning optimize the properties of generated molecules.
State and Policy Network. The state is the current sub-graph, and the initial state is an empty graph. The policy network is the same as the autoregressive model defined in Section 4.1, which

---

Published as a conference paper at ICLR 2020
includes the process of generating a new atom based the current sub- graph and generating the edges between the new atom and existing atoms, i.e., p (X,]G,) and p [,y|G,, X,, A...--1) The policy network itself defines distribution Po of molecular graphs G. If there are no edges between the newly generated atom and current sub-graph, the generation process terminates. For the state transition dynamics, also incorporate the valency check constraint.
Reward design. Similar to GCPN et al. (2018)). we also incorporate both intermediate and final rewards for training the policy network. A small penalization will be introduced the intermediate reward if the edge predictions violate the valency check. The final rewards include both the score of peeeted-propetiess of generated molecules such as octanol- -wter partition coefficient (logP) or drug- min-liienesss (QED) (Bickerton et al., 2012)) and the chemical validity reward such as penalties for molecules with excessive steric strain and or functional groups that violate ZINC functional group filters (Irwin et al.) 2012). The final reward is distributed to all intermediate steps with a discounting factor to stabilize the training.
In practice, we adopt Proximal Policy Optimization (PPO) (Schulman et al.) 2017). an advanced pol- icy gradient algorithm to train GraphAF the above defined environment, Let Gij be the shorthand notation of sub-graph G, U X; U A.1:j-1. Formally,
I(0)= = FiG Ex min X.), (1000
(12)
+ Ej min (ro (l,
Pe (X.IG.) and ri)( = po(A:; G:y) are ratios of probabilities output by old and new policies, and (state, action) the estimated advantage function with moving average baseline to reduce the variance. More specifically, we treat generating a node and all its edges with existing nodes as one step and maintain a moving average baseline for each step. The clipped surrogate objective prevents the policy from being updated to collapse for some extreme rewards.
where r(()) = G.)
Pe (G4))
EXPERIMENTS
5.1 EXPERIMENT SETUP
Evaluation Tasks. Following existing works molecule generation (Jin et al. 2018 You et al., 20T8a Popova et al., 2019), we conduct experiments by comparing with the state-of-the-ae ap- proaches on three standard tasks. Density Modeling and Generation evaluates the model's 's capacity to learn the data distribution and generate realistic and diverse molccules. Property Optimization concentrates on generating novel molecules with optimized chemical properties. For this task, fine-tune our network pretrained from the density modeling task to maximize the desired proper- ties. Constrained Property Optimization first proposed in Jin al. (2018), which is aimed at modifying the given molecule to improve desired properties while satisfying a similarity constraint.
Data. We use the ZINC250K molecular dataset (Irwin
2012) for training. The dataset contains 250,000 000 drug-like molecules with maximum atom number of 38. It has 9 atom types and 3 edge types. We use the open-source chemical software RDkit (Landrum) (0166) to preprocess molecules. All molecules are presented in kekulized form with hydrogen removed.
Baselines. We compare GraphAF with the following state-of-the-arr approaches for molecule gener- ation. JT-VAE (Jin et al. 2018) is VAE-based model which generates molecules by first decoding tree structure of scaffolds and then assembling them into molecules. JT-VAE has been shown to outperform other previous VAE-based models (Kusner et al.) 2017] Gomez-Bombarelli et al.) 2018 Simonovsky & Komodakis 2018). GCPN is ofate-of-the-- approach which combines re- inforcement learning and graph representation learning methods to explore the vast chemical space. MolecularRNN (MRNN), another autoregressive model, uses RNN to generate molecules in a se- quential manner. We also compare model with GraphNVP (Madhawa al. 2019). recently proposed blowd model. Results of baselines are taken from original papers unless stated.
Implementation Details. GraphAF is implemented in PoTorc (Paszke et al. 2017). The R-GCN is implemented with 3 layers, and the embedding dimension is set as 128. The max graph size set as 48 empirically. For density modeling, train our model for 10 epochs with batch size of 32 and
7

---

Published as a conference paper at ICLR 2020
Table 2: Comparison of different models density modeling and generation. Reconstruction is only evaluated on latent variable models. Validity woo check is only evaluated on models with valency constraints. Result with i is obtained by running GCPN's open-source code. Results with $ are taken from Popova et al. (2019).
Method
Validity
Validity w/o check
Uniqueness
Novelty
Reconstruction
JT-VAE
100%
100%
100%
76.7%
GCPN 100%
20%
99.97%
100% !
MRNN 100%
65%
99.89%
100%
GraphNVP 42.60%
94.80%
100%
100%
GraphAF
100%
68%
99.10%
100%
100%
Table 3: Results of density modeling and generation on three different datasets.
Method
Validity
Validity w/o check
Uniqueness
Novelty
Reconstruction
ZINC250K
100%
68%
99.10%
100%
100%
QM9
100%
67%
94.51 %
88.83% 100%
100%
MOSES
100%
71%
99.99%
100%
learning rate of 0.001. For property optimization, we perform grid search on the hyperparameters and select the best setting according the chemical scoring performance. We use Adam (Kingma & Ba 2014) to optimize our model. Full training details can be found in Appendix
5.2 NUMERICAL RESULTS
Density Modeling and Generation. We evaluate the ability of the proposed method to model real molecules by utilizing the widely-used metrics: Validity is the percentage of valid molecules among all the generated graphs. Uniqueness is the percentage of unique molecules among all the generated molecules, Novelty is the percentage of generated molecules not appearing in training set. Reconstruction is the percentage of the molecules that can be reconstructed from latent vectors. We calculate the above metrics from 10,000 randomly generated molecules.
Table that GraphAF achieves competitive results on all four metrics. As flow-based model, GraphAF holds perfect reconstruction ability compared with VAE approaches. Our model also achieves a 100% validity rate since we can leverage the valency check during sequential generation. By contrast, the validity rate of another flow-based approach GraphNVP is only 42.60% due to its one-shot sampling process, An interesting result is that even without the valency check during generation, GraphAF can still achieve validity rate as high as 68%, while previous state-of-the- approach GCPN only achieves 20%. This indicates the strong flexibility of GraphAF to model the data density and capture the domain knowledge from unsupervised training the large chemical dataset. We also compare the efficiency of different methods on the same computation environment, machine with Tesla V100 GPU and 32 CPU cores. To achieve the results in Table JT-VAE and GCPN take around 24 and 8 hours, respectively, while GraphAF only takes 4 hours.
To show that GraphAF is not overfitted to the specific dataset ZINC250K, we also conduct experi- ments on two other molecule datasets, QM9 (Ramakrishnan et al. 2014) and MOSES (Polykovskiy al. 2018). QM9 contains 134k molecules with 9 heavy atoms, and MOSES is much larger and more challenging, which contains 1.9M molecules with up to 30 heavy atoms. Table 3 shows that GraphAF can always generate valid and novel molecules even on the more complicated dataset.
Furthermore, though GraphAF is originally designed for molecular graph generation, it is actually very general and can be used to model different types of graphs by simply modifying the node and edge generating functions Edge-MLPss and Node-MLPPs (Eq.8) Following the experimental setup of Graph Normalizing Flows (GNF) (Liu et al.) 2019). we test GraphAF on two generic graph datasets: COMMUNIT--MMALL. which is synthetic data set containing 100 2-community graphs, and (GO-SMALL, which is a set of graphs extracted from Citeseer dataset (Sen et al. 2008). In prac- tice, we one-hot indicator vectors node features for R-GCN. We borrow open source scripts from GraphRNN You et al.) 2018b) to generate datasets and evaluate different models. For evalua- tion, we report the Maximum Mean Discrepancy (MMD) (Gretton et al., 2012) between generated
8

---

Published as a conference paper at ICLR 2020
Table 4: Comparison between different graph generative models on general graphs with MMD metrics. We
follow evaluation scheme GNF (Liu al. 2019). Results baselines are also taken from GNF.
COMMUNITY-MAALL,
EGO-SMALL Cluster
Method
Degree
Cluster
Orbit
Degree
Orbit
Graph
0.35
0.98
0.54
0.13
0.17
0.05
DEEPGMG
0.22
0.95
0.4
0.04
0.10
0.02
GraphRNN
0.08
0.12
0.04
0.09
0.22
0.003
GNF
0.20
0.20
0.11
0.03
0.10
0.001
GraphAF
0.18
0.20
0.02
0.03
0.11
0.001
G112hRNN(1024)
0.03
0.01
0.01
0.04
0.05
0.06
GNF(1024)
0.12
0.15
0.02
0.01
0.03
0.0008
GraphAF(1022)
0.06
0.10
0.015
0.04
0.04
0.008
Table 5: Comparison of the top property scores of generated molecules.
Penalized logP
QED
Method
1st
2nd
3rd
Validity
1st
2nd
3rd
Validity
ZINC (Dataset)
4.52
4.30
4.23
100.0%
0.948
0.948
0.948
100.0%
JT-VAE (Jin al. 2018
5.30
4.93
4.49
100.0%
0.925
0.911
0.910
100.0%
GCPN
You el al.
2018a
7.98
7.85
7.80
100.0%
0.948
0.947
0.946
100.0%
MRNN
Popova al.
2019
8.63
6.08
4.73
100.0%
0.844
0.796
0.736
100.0%
GraphAF
12.23
11.29
11.05
100.0%
0.948
0.948
0.947
100.0%
12.23
11.29
0.948
0.948
-8.25
-0.11
11.05
10.83
0.947
0.946
-8.66
-1.10
(a) Penalized logp optimization
(h) QED optimization
(c) Constrained property optimization
Figure 2: Molecules generated in property optimization and constrained property optimization tasks. (a) Molecules with high penalized logP scores. (b) Molecules with high QED scores. (c) Two pairs of molecules constrained property optimization for penalized logP with similarity (top and 0.64(bottom).
and training graphs using some specific metrics on graphs proposed by You et al. (2018b). The results in Table Table/Jdemonsteee demonstrate that when applied to generic graphs, GraphAF can still consistently yield comparable or better results compared with GraphRNN and GNF, We give the visualization of generated generie graphs in Appendix D
Property Optimization. In this task, aim generating molecules with desired properties. Specifically, we choose penalized logP and QED as our target property. The former score is logP score penalized by ring size and synthetic accessibility, while the latter one measures the drug- likeness of the molecules. Note that both scores are calculated using empirical prediction models and we adopt the script used in You et al. (018a) make results comparable. To perform this task, we pretrain the GraphAF network for 300 epochs for likelihood modeling, and then apply the RL process described in section 4.4 to fine-tune the network towards desired chemical properties. De- tailed reward design and works, we report the top-3 scores found by each model.
setting can be found in Appendix C Following existing
'The scores reported here recalculated hased top molecules presented in the original paper (Popova 2019) using GCPN's script.

---

Published as a conference paper at ICLR 2020
Table 6: Comparison of results constrained property optimization.
JT-VAE Similarity
GCPN Similarity
GraphAF Similarity
8
Improweent
Success
Improwement
Success
Improveennt
Success
0.0
1.91
0.28 0.15
97.5%
4.20 1.28
0.32 + 0.12
100%
13.13 + 6.89
0.29 = 0.15
100%
0.2
1.68 1.85
0.33 = 0.13
97.1%
4.12 1.19
0.34 = 0.11
100%
11.90 + 6.86
0.33 2: 0.12
100%
0.4
0.84 1.45
0.51 0.10
83.6%
2.49 1.30
0.47 0.08
100%
8.21 6.51
0.49 0.09
99 88%
0.6
0:21::00
0.71
0.09 - 0.06
46.4%
0.79 0.63
0.68 = 0,08
100%
4.98 + 6.49
0.06 = 0.05
96.88%
As shown in Table [ GraphAF outperforms all baselines by a large margin for penalized logP score and achieves comparable results for QED. This phenomenon indicates that combined with RL process, GraphAF successfully captures the distribution of desired molecules. Note that we re-evaluate the properties of the top-3 molecules found by MolecularRNN, which turn out to be lower than the results reported in the original paper. Figure 2(a) and 2(b) show the molecules with the highest score discovered by our model. More realistic molecules generated by GraphAF with penalized logP score ranging from 5 to 10 are presented in Figure 6 in Appendix
One should note that, as defined in Sec Sec4.4 our RL process is close to the one used in previous work GCPN (You et al.J (2088)). Therefore, the good property optimization performance is believed to come from the flexibility of flow. Compared with the GAN model used in GCPN, which is known to suffer from the mode collapse problem, flow is flexible at modeling complex distribution and generating diverse data (as shown Table and and T). This allows GraphAF to explore a variety of molecule structures in the RL process for molecule properties optimization.
Constrained Property Optimization. The goal of the last task is to modify the given molecule to improve specified property with the constraint that the similarity between the original and modified molecule is above threshold 8. Following J al. (2018) and You et al. 22018a). we choose to optimize penalized logP for 800 molecules in ZINC250K with the lowest scores and adopt Tanimoto similarity with Morgan fingerprint (Rogers & Hahn
2010) as the similarity metric.
Similar to the property optimization task, we pretrain GraphAF via density modeling and then fine- tune the model with RL. During generation, we set the initial states as sub-graphs randomly sampled from 800 molecules to be optimized. For evaluation, we report the mean and standard deviation of the highest improvement and the corresponding similarity between the original and modified molecules in Table O Experiment results show that GraphAF significantly outperforms all previous approaches and almost always succeeds improving the target property. Figure [ visualizes two optimization examples, showing that our model is able to improve the penalized logP score by large margin while maintaining a high similarity between the original and modified molecule.
CONCLUSION
We proposed GraphAF, the first flow-based autoregressive model for generating realistic and diverse molecular graphs. GraphAF is capable to model the complex molccular distribution thanks to the flexibility of normalizing flow, as well as generate novel and 100% valid molecules in empirical experiments. Moreover, the training of GraphAF is very efficient. To optimize the properties of gen- crated molecules, we fine-tuned the generative process with reinforcement learning. Experimental results show that GraphAF outperforms all previous state-of-the-art baselines on the standard tasks. In the future, we plan to train our GraphAF model on larger datasets and also extend it to generate other types of graph structures (e.g., social networks).
ACKNOWLEDGEMENT
We would like to thank Min Lin, Meng Qu, Andreea Deac, Laurent Dinh, Louis Pasccl A. C. Xhonneux and Vikas Verma for the extremely helpful discussions and comments. This project is supported by the Natural Sciences and Engineering Research Council of Canada, the Canada CI- FAR AI Chair Program, and a collaboration grant between Microsoft Research and Mila. Ming Zhang is supported by National Key Research and Development Program of China with Grant No. as well as Beijing Municipal Commission of Science and Technology under Grant No. Z181100000 100008918005 Weinan Zhang is supported by National Natural Science Foundation of China (61702327, 61772333, 61632017, 81771937) and Shanghai Sailing Program (17YF1428200).
10AAAA001100/2011444000000
10

---

Published as a conference paper at ICLR 2020
REFERENCES
G Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90, 2012.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint arXiv: /805. /1973, 2018.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv
preprint arXiv: 08803, 2016.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances in neural information processing systems, pp. 2224-2232, 2015.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pp. 881-889 2015.
Justin Gilmer, Samuel Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning- Gearning-Volume 70, pp. 1272. JMLR. org, 2017.
Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez- Lengeling Dennis Sheberla, Jorge Ryan P Adams, and Alán Aspuru-Guzie Guzik. Automatic chemical design using data-driven contin- uous representation of molecules. ACS centrul science, 4(2):268-22 276, 2018.
Timothy Hirzel,
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test. Journal of Machine Learning Research, 13(MMr))723-773, 2012.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances Neural Information Processing Systems, pp. 1024-1034, 2017.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improv- ing flow-based generative models with variational dequantization and architecture design. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 2019. URL lttt:://proceee
John J Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. Zinc: a free tool to discover chemistry for biology. Journal of chemical information and modeling, 52 (7):1757-1768, 2012.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. arXiv preprint 1rXivv1802.04364 2018
Steven Kearnes, Kevin McCloskey, Mare Berndl, Vijay Pande, and Patrick Riley. Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design, 30(8): 595-608, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3nd Interna- tional Conference on Learning Representations, 2014.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ix1 convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743-4751 2016.
11

---

Published as a conference paper at ICLR 2020
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- 2016.
works. arXiv preprint
Ivan Kobyzev, Simon Prince, and Mareus A. Brubaker. Normalizing flows: Introduction and ideas.
arXiv preprint arXiv: 0908: 09200. 2019.
Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoen- coder. In Proceedings of the 34th International Conference on Machine Learning-Volune 70, pp. 1945-1954. JMLR. org, 2017.
Greg Landrum. Rdkit: Open-source cheminformatics software. 2016.
Jenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, and Kevin Swersky. Graph normalizing flows. arXiv preprint arXiv: 1905. 13177, 2019.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 2217-2227. JMLR. org, 2017.
Tengfei Ma, Jie Chen, and Cao Xiao. Constrained generation of semantically valid graphs via regularizing variational autoencoders. In Advances in Neural Information Processing Systems, pp. 7113-1124 2018.
Kaushalya Madhawa, Katushiko Ishiguro, Kosuke Nakago, and Motoki Abe, Graphnvp: An invert- ible flow model for generating molecular graphs. arXiv preprint arXiv:1905. 2019.
Daniel Neil, Marwin H. S. Segler, Laura Guasch, Mohamed Ahmed, Dean Plumbley, Matthew Sellwood, and Nathan Brown. Exploring deep recurrent models with reinforcement learning for molecule design. In 6th International Conference Learning Representations, Workshop Track Proceedings, 2018
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformaties, 9(1):48, 2017.
George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-WW W, 2017.
G. Polishchuk, T. I. Madzhidov, and A. Varnek. Estimation of the size drug-like chemical space based gdb- 17 data. Jourmal of Computer-Aided Molecular Design, 27(8):675-66) 6799, Aug 2013.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengelingg Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin, Sergey Nikolenko, Alan Aspuru-Guzik, and Alex Zhavoronkov. Molec- ular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. arXiv preprint 1rXiv:1811.12223. 2018.
Mariya Popova, Mykhailo Shvets, Junier Oliva, and Olexandr Isayev. Molecularnn: Generating realistic molecular graphs with optimized properties. arXiv preprint arXiv: /9055. 13372, 2019.
Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and O Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint 1505.005 2015.
David Rogers and Mathew Hahn. Extended fingerprints. Journal of chemical informa- fion and modeling, 50:742-754, 2010.
Bidisha Samanta, Abir De, Niloy Ganguly, and Manuel Jomnez-Roriiuezz Designing random graph models using variational autoencoders with applications to chemical design. arXiv preprint arXiv: 1802, 1rX2v:1802.05520 05288. 2018.
12

---

Published as a conference paper at ICLR 2020
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling, Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pp. 593-5593-9 Springer, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Kristof T Schüit, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko. Quantum-chemice insights from deep tensor neural networks. Nature communications, 8: 8:13890, 2017.
Marwin HS Segler, Thierry Kogej. Christian Tyrchan, and Mark P Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1): 120-131, 2017.
Prithviraj Sen, Galileo Mark Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi-Rad. Collective classification in network data. AI Magazine, 29(3):93-106, 2008. URL
http://www. cs iit. aeddu//mm/dff//eeeeeeeeeeeee aaeaaaaaaaaaaaaaaaaaaaaaaaa pdf
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference Artificial Neural Networks, pp. 412- 422. Springer, 2018.
L. Theis, A. van den Oord, and M. Bethge. A note on the evaluation of generative models. In International Conference Learning Representations, 2016. URL http://arxiv.org) abs/ 0tss/511.0080 01844 11Xiv:1511.01844
Aaron Van Oord. Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp. 1747-1756, 2016.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional pol- icy network for goal-directed molecular graph generation. In Advances in Neural Information Processing Systems, pp. 6410-6421, 6421, 2018a.
Jiaxuan You, Rex Ying, Xiang Ren, William L Hamilton, and Jure Leskovec. Graphran: Generating realistic graphs with deep auto-regressive models. arXiv preprint arXiv:1802.08773, 2018b.
13

---

Published as a conference paper at ICLR 2020
Appendix
A DISCCUSIONS ON DEQUANTIZATION TECHNIQUES
The dequantization techniques allow mapping the discrete data into the continuous one by adding a small noise to each dimension. By adding noise from O0 1), we can ensure that the range of different categories will not overlap. For example, after dequantization, the value of 1-entry in the one-hot vector lies in [1, 2) while the O-entry lies in [0, 1). Therefore, we can map the dequantized continuous data back to the discrete one-hot data by easily performing the argmax operation in the generation process. Theoretically, as shown in Theis al. (2016); Ho et al. (2019), training continuous density model on uniform dequantized data can be interpreted as maximizing a lower bound on the log- lig-liielhoodd for the original discrete data. Mathematically, this statement holds for both image data and binary/categorical data.
Furthermore, as suggested in Ho al. (2019), instead of adding random uniform noise to each discrete data for dequantization, a more advanced dequantization technique is to treat the noise as hidden variable and variational inference infer the optimum noise added to each discrete data, which we would like to explore in our future work.
B PARALLEL TRAINING ALGORITHM
Algorithm Parallel Training Algorithm of GraphAF
Input: 77 learning rate, M batch size, P' maximum dependency distance in BFS, Adam hyperparam eters B1, 32, use Prod(-) the product of elements across dimensions of tensor Initial: Parameters of GraphAF (R-GCN, Node-MLP and Edge-MLP)
1: while 0 is not converged do
2: for m = ..., .M do
3: Sample molecule mol from dataset and get the graph size N
4: Convert mol to G X) with BFS re-ordering
5: for 1...., N do
6:
= = =x,
u ~ 1)
7:
p² = (G.). a 9ax (G.)
8:
e, = (z - u
9:
2x log( log(Prod(pe (G))) log((Pod) 1
for j max -P...., i.... do
11:
2 = Aij uu ~ U10 1)b+1
12:
" Iμ (Gi, X;, A1,15-1) at = Ia (Gi, Xi, Ar1z-1)
13:
E2 (= - m
14:
C - log Prod (Gg) ))) - log (rood 1)
15:
end for end for
17:
CC
-E (C. + E/= Fi)
18:
end for
0 ADAM( Σm CG n, B1, 32)
20:
end while
C EXPERIMENT DETAILS
Network architecture. The network architecture is fixed among all three tasks. More specifically, the R-GCN is implemented with 3 layers and the embedding dimension set as 128. We use batch normalization before graph pooling to accelerate the convergence and choose sum-pooling as
14

---

Published as a conference paper at ICLR 2020
the readout function for graph representations. Both node MLPs and edge MLPs have two fully- connected layers equipped with tanh non- non-lineare con-iinarr.ty.
Density Modeling and Generation. To achieve the results in Table train GraphAF on ZINC250K with a batch size of 32 on 1 Tesla V100 GPU and 32 CPU cores for 10 epochs. We optimize our model with Adam with a fixed learning rate of 0.001.
Property Optimization. For both property optimization and constrained property optimization, we first pretrain a GraphAF network via the density modeling task for 300 epochs, and then fine- tune the network toward desired molecular distribution through RL process, Following are details about the reward design for property optimization. The reward of each step consists of step-wise validity rewards and the final rewards discounted by a fixed factor y. The step-wise validity penalty fixed as -1. The final reward of a molecule m includes both property-targeted reward and chemical validation reward. We adopt the same chemical validation rewards as GCPN. We define property- targeted reward follows:
r(m) = 1 (m)
[log pen (mol)
(13)
r(m) exp
Y set to 0.97 for QED optimization and 0.9 for penalized logP optimization respectively. We fine-tune the pretrained model for 200 iterations with fixed batch size of 64 using Adam optimizer. We also adopt a linear learning rate warm-up to stabilize the training. We perform the grid search to determine the optimal hyperparameters according the chemical scoring performance. The search space summarised in Table
Table Luned-parameters for policy gradient and their search space.
PARAM
Description
Search space
Ir
Learning rate
{0.001, 0.0005, 0.0001} {2, 4, 5} [3, 4, 5} (12, 24, 36)
t1
Coefficient for QED score
l/
Temperature for exponential function Number of warm up iterations
Constrained Property Optimization. We first introduce the way we sample sub-graphs from 800 ZINC molecules. Given molecule, we first randomly sample BFS order and then drop the last m nodes in BFS order as well as edges induced by these nodes, where m is randomly chosen from 1, 2, 4, 5) each time. Finally, we reconstruct the sub-graph from the remaining nodes in the BFS sequence. Note that the final sub-graph is connected due to the nature of BFS order. For reward design, we set as the improvement of the target score. We fine-tune the pretrained model for 200 iterations with batch size of 64. We also use Adam with a learning rate of 0.0001 to optimize the model. Finally, each molecule is optimized for 200 times by the tuned model.
D VISUALIZATION OF GENERATED GENERIC GRAPHS
We present visualizations of graphs from both the training set and generated graphs by GraphAF in Figure 3 and Figure 4 The visualizations demonstrate that GraphAF has strong ability to model different graph structures in the generic graph datasets.
E MORE MOLECULE SAMPLES
We present more molecule samples generated by GraphAF in the following pages. [iguree5present 50 molecules randomly sampled from multivariate Gaussian, which justify the ability of our model model is
to generate novel, realistic and unique molecules. From Figure 6 we can see that able to generate molecules with high and diverse penalized logP scores ranging from 5 to 10. For constrained property optimization of penalized logP score, as shown by Figure 7 our model can either reduce the ring size, remove the big ring or grow carbon chains from the onginal molecule, improving the penalized logP score by large margin.
15

---

Published as a conference paper at ICLR 2020
(a) Graphs from training set
(b) Graphs generated by GraphAF
Figure 3: Visualizations of training graphs and generated graphs of EGO-SMALL.
(a) Graphs from training sel
(b) Graphs generated by GraphAF
Figure 4: Visualizations of training graphs and generated graphs of COMMUNITY-SMALL.
16

---

Published as a conference paper at ICLR 2020
asko
mp
ant
OtG
Figure 50 molecules sampled from prior.
17

---

Published as a conference paper at ICLR 2020
10.75
10.21
9.79
9.18
8.29
8.08
7.75
7.64
CD
8
6.27
6.09
5.48
5.28
Figure Molecule samples with high penalized logp score generated by GraphAF.
0.61 19.06
0.64 11.86
0.63 9.01
0.71 2.10
0.50 9.47
0.53 9.38
0.46 7.84
0.50 7.13
gio
Figure 7: More results on constrained property optimization for penalized logP score. Numbers beside the anoow denote similarity and improvement of the given molecule pair respectively.
18


Reviews:
[{"review_id": "ryxiyLlF_S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a generative model architecture for molecular graph generation based on autoregressive flows. The main contribution of this paper is to combine existing techniques (auto-regressive BFS-ordered generation of graphs, normalizing flows, dequantization by Gaussian noise, fine-tuning based on reinforcement learning for molecular property optimization, and validity constrained sampling). Most of these techniques are well-established either for data generation with normalizing flows or for molecular graph generation and the novelty lies in the combination of these building blocks into a framework. Training can be carried out in parallel over the sequential generation process, as no hidden states with sequential dependency are assumed (unlike a regular RNN). Experimental validation is carried out on a standard ZINC molecule generation benchmark (graphs with up to 48 nodes) and the reported metrics are competitive with recent related work.\n\nOverall, the paper is very well written, nicely structured and addresses an important problem. The framework in its entirety is novel, but the building blocks of the proposed framework are established in prior work and the idea of using normalizing flows for graph generation has been proposed in earlier work (see [1] and [2]). Nonetheless, I find the paper relevant for an ICLR audience and the quality of execution and presentation of the paper is good.\n\nI have two major (technical) concerns with the flow-based formulation used in the paper with regards to order-invariance and the utilized de-quantization scheme.\n* Order-invariance: The paper states that the “exact density of each molecule can be efficiently computed by the change-of-variables formula”. This seems to be incorrect, as the exact density is a product over all order-specific densities for all possible permutations in which the molecular graph can be represented. The change-of-variables formula does not provide an efficient way to circumvent this order-invariance issue, at least not in the way it is presented in the paper. Even when using BFS-ordered representations, the subspace of possible permutations is still typically too large to allow for efficient evaluation of the exact density. I suspect that the authors assume a canonical ordering of the graph representations, which is a strong assumption, but does not seem to be mentioned in the paper. How is the canonical ordering chosen? How is local structural symmetry broken in a consistent manner?\n* De-quantization: The de-quantization scheme used in this paper seems to be ill-suited for categorical variables. What motivates the use of adding Gaussian noise to categorical (one-hot encoded) variables, other than that it seems to work OK in the reported experiments? Adding Gaussian noise in this way can move these variables outside of the probability simplex — is this a valid technique in the framework of normalizing flows? Adding Gaussian noise makes sense if the data represents quantized continuous data, e.g. bit-quantized image data, but I have concerns about the validity of using this method for categorical data (both edge type and node features are categorical in this application). Other comparable generative models for graph-structured data use a relaxed discrete distribution (concrete / Gumbel softmax), e.g. in MolGAN [De Cao & Kipf (2018)], to address this issue — would this also be applicable here?\n\nI think that these two issues will have to be addressed before this paper can be considered for publication, and I recommend a weak reject at this point.\n\n[1] Madhawa et al., GraphNVP: An invertible flow model for generating molecular graphs. (2019)\n[2] Liu et al., Graph Normalizing Flows. (2019) — not cited\n\n\nUPDATE:\n\nMy two main technical concerns have been addressed in the rebuttal and I think that the revised version of the paper can be accepted to ICLR (my comment w.r.t. novelty still holds and hence I recommend 'weak accept').", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gjCzZAFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new invertible-flow based graph generation model. \nThe main difference from the previous flow-based generation model (GraphNVP) is the choice of flow mappings (coupling flow --> autoregressive flow). \nThe authors formulate conditional probabilities of iterative node/edge generation. Iterative sampling scheme naturally allows incorporating validity checks in each sampling step, assuring 100% validity in graph generation. The paper also proposes an implementation of molecule lead optimization combined with a RL framework. The experimental results show superiority in terms of valid graph generation and property optimization. \n\n\nOverall, the paper is written well. I feel no difficulty in understanding the main idea and the equations in the paper.\n \nIntroduction of the normalizing flows (Sec 3.1) can be expanded to reach non-expert users. Advantages of using invertible flows (against other generative models such as GANs and VAEs) are not described rigorously in the current manuscript. I also suggest citing a nice review for invertible flows appeared recently: \n\nIvan Kobyzev, Simon Prince, and Marcus A Brubaker, ``Normalizing Flows: Introduction and Ideas'', arXiv: 1908.09257, 2019.\n\n\nExplanations of the Sec 4.4 (+ appendix B) is simply insufficient to reproduce the experiments. More descriptions or references are required. \n\n\nExperimental results seem promising. A better validity score than the previous flow model illustrates the efficacy of the autoregressive flow against the coupling flow. \nThe performance on the property optimization (Table 3) seems brilliant. However, there is no discussion why the combination of the autoregressive flow and the RL performs greatly, compared to baselines. Some discussions will help the community to further improve the optimization tasks in the future. \n\n+ Overall, a good paper. well written, easy to understand. \n+ A new variant of the invertible-flow based graph generation model.  The novelty lies in the iterative generation process, naturally combined with the autoregressive flow. \n+ Superior to the one-shot flow baseline (GraphNVP) even if additional validity checks are omitted (Table 2)\n+ Good performances in property optimizations (Table 3, 4) \n- The explanation for RL process is simply insufficient for reproduction.\n-- No discussions about reasons why GraphAF+RL performs great in property optimization. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxZqYuWcr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Post Rebuttal\n\nThe authors have partially and satisfactorily addressed my concerns. In line of this I am raising my score to Weak Accept.\n\nThis paper proposes a new molecular graph generative model (GraphAF) which fuses the best of two worlds of generative networks - reversible flow and autoregressive mode. Such integration enjoys a) faster training due to parallel computation b) molecular validity checker during inference supported by sequential sampling process and c) exact likelihood maximisation due to invertible encoder. In lieu of such advantages, the model trains two times faster than the existing state-of-the-art and generates 100% valid molecules when trained on ZINC dataset. Further, it also demonstrates that additionally if the chemical properties are optimised during training with reinforcement learning policy then GraphAF outperforms all the prior works.\n\nAlthough the paper presents an interesting fusion of different generative models, in its current form it leans towards rejection due to the following factors:\n1) The empirical validation of GraphAF is contained to single dataset - ZINC with a maximum of 38 atoms. From the table 2, it seems to me every prior method works pretty well on important metrics. There is very little room for improvement. I recommend including results on QM9 and CEPDB datasets. \n2) The model being data-agnostic, it makes sense to evaluate them on generic graph datasets - synthetic and real.\n3) The novelty of the model is limited. The flow-based graph generative model is introduced in Graph Normalizing Flow (GNF) (NeurIPS'19, NeurIPS'18 workshop). The reversible flow is extended to whole graph in GraphNVP. Unlike GNF, GraphNVP and GraphAF do away with decoder. The major difference being the sampling process - one-shot to sequential.\nI am willing to improve my rating given that some of this points are addressed.\n\nClarification:\n1. What are the inputs edge-mlp's operate on ? Given the generation step is sequential, it is not clear to me why all the node embeddings H_i^L is given as input in eq (8). I also noted that the dimension of H_i^L varies with size of sub-graphs. Also note mismatch in the notation 'f' used in algorithm 1 and 'g' from the main text. \n2. Please compare inference time. \n\nOther weakness:\n1. Due to invertible flow modeling, the latent space is usually restricted to small dimension. In current case it is 9 for node feature and 3 for edge features. This drawback alongside the sequential edge generation prevents GraphAF from scaling to complex and large graphs with many labels.\n2. Moreover, GraphAF utilizes only single layer of flow i.e., eq (9). This is clearly not sufficient to model complex graphs. And in its current form it is not clear how one can extend to multi-layer flow.\n3. The encoder modeling in GraphAF also shares similarity with Variational graph auto-encoder. Instead of constraining latent distribution using KL divergence, GraphAF maximizes graph likelihood to enforce base distribution.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that this paper is essentially a combination of existing ideas, making it a bit incremental, but is well-executed and a good contribution.  Specifically, to quote R1:

"This paper proposes a generative model architecture for molecular graph generation based on autoregressive flows. The main contribution of this paper is to combine existing techniques (auto-regressive BFS-ordered generation of graphs, normalizing flows, dequantization by Gaussian noise, fine-tuning based on reinforcement learning for molecular property optimization, and validity constrained sampling). Most of these techniques are well-established either for data generation with normalizing flows or for molecular graph generation and the novelty lies in the combination of these building blocks into a framework. ... Overall, the paper is very well written, nicely structured and addresses an important problem. The framework in its entirety is novel, but the building blocks of the proposed framework are established in prior work and the idea of using normalizing flows for graph generation has been proposed in earlier work. Nonetheless, I find the paper relevant for an ICLR audience and the quality of execution and presentation of the paper is good."
Decision: Accept (Poster)