Modular Flows: Differential Molecular Generation
Authors: Yogesh Verma
Samuel Kaski
Markus Heinonen
Vikas K Garg
Venue: NeurIPS 2022 Accept (2022)
Keywords: normalizing flow
molecule generation
graph neural networks
neural ode

Abstract: Generating new molecules is fundamental to advancing critical applications such as drug discovery and material synthesis. Flows can generate molecules effectively by inverting the encoding process, however, existing flow models either require artifactual dequantization or specific node/edge orderings, lack desiderata such as permutation invariance, or induce discrepancy between encoding and decoding steps that necessitates post hoc validity correction. Inspired by graph PDEs, we circumvent these issues with novel continuous normalizing E(3)-equivariant flows, based on a system of coupled node ODEs, that repeatedly reconcile locally toward globally aligned densities. Our models can be cast as message passing temporal networks, and result in superlative density estimation and  molecular generation. In particular, our generated samples achieve state of the art on both the standard QM9 and ZINC250K benchmarks.

Modular Flows: Differential Molecular Generation
Yogesh Verma, Samuel Kaski, Markus Heinonen
Aalto University
fyogesh. verma, samuel kaski, markus. markus.heinonene hwwnonen)eaalto........ .i
Vikas Garg
Yai Ltd and Aalto University vgargocsail mit.edu; vikas@yaiyai. fi
Abstract
Generating new molecules is fundamental to advancing critical applications such as drug discovery and material synthesis. Flows can generate molecules effectively by inverting the encoding process, however, existing flow models either require artifactual dequantization or specific node/edge orderings, lack desiderata such permutation invariance, or induce discrepancy between the encoding and the decoding steps that necessitates post hoc validity correction. We circumvent these issues with novel continuous normalizing -e3-vvaaiaa flows, based on system node ODEs coupled as a graph PDE, that repeatedly reconcile locally toward globally aligned densities. Our models can be cast message passing temporal networks, and result in superlative performance on the tasks of density estimation and molecular generation. In particular, generated samples achieve state of the art on both the standard QM9 and ZINC250K benchmarks.
Introduction
Generative models have rapidly become ubiquitous in machine learning with advances from image synthesis (Ramesh et al., 2022) to protein design (Ingraham et al.
soflmax
Assignment
2019). Molecular ar generation (Stokes al.] 2020) has
Time t
also received sigmificant attention owing to its promise for discovering new drugs and materials. Searching for
(a) Graph Structure
valid molecules in prohibitively large discrete spaces is, however, challenging: estimates for drug-like structures range between 10 and 10 but only tiny fraction on the order of 10 has been synthesized (Polishchuk) et al. 2013 Merz et al.) 2020). Thus, learning repre- sentations that exploit appropriate molecular inductive biases (e.g., spatial correlations) becomes crucial.
Donak
025 023
Earlier models focused on generating sequences based on the SMILES notation (Weininger) 1988) used in Chemistry to describe the molecular structures as strings. However, they were supplanted by genera- tive models that capture valuable spatial information such as bond strengths and dihedral angles, e.g., by embedding molecular graphs via some graph neural network (GNNs) (Scarselli al. 2009; Garg et al. 2020). Such models primarily include variants of Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Normalizing
Density & Vector field (b)
felt Ax,
Figure toy illustration ModFl ow action with a two-node graph. The two local flows 21 and co-evolve toward more complex joint density. both driven by the differential f.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Flows Dinh et al. 2014 2016). Besides known issues with their training, GANs (Goodfellow et al. 2014; Maziarka et al. 2020) suffer from the well-documented problem of mode collapse. thereby generating molecules that lack diversity. VAFs (Kingma and Welling 2013 Lim et al.) 2018 Jin jet al. 2018), on the other hand, are susceptible to a distributional shift between the training data and the generated samples. Moreover, optimizing for likelihood via a surrogate lower bound is likely insufficient to capture the complex dependencies inherent the molecules.
Flows are especially appealing since, in principle, they enable estimating (and sampling from) complex data distributions using sequence of invertible transformations on samples from a more tractable continuous distribution. Molecules are discrete, so many flow models (Madhawa et al. 2019) Honda et al. 2019) (Shi et al. 2020) add noise during encoding and later apply a dequantization procedure. However, dequantization begets distortion and issues related to convergence (Luo et al. 202T). Moroever, many methods segregate the generation of atoms from bonds, so the decoded structure is often not valid molecule and requires post hoc correction to ensure validity (Zang) and Wang 2020), effecting a discrepancy between the encoding and the decoded distributions. Permutation dependence is another undesirable artifact of these methods. Some alternatives have been explored to avoid dequantization, e.g., (Lippe and Gavves 2021 encodes molecules a continuous latent space via variational inference and jointly optimizes a flow model for generation. Discrete graph flows (Luo et al. 2021) also circumvent the many pitfalls of dequantization by resorting to discrete latent variables, and performing validity checks during the generative process. However, discrete flows follow an autoregressive procedure that requires a specific ordering of nodes and edges during training. In general, one shot methods can generate much faster than discrete flows.
We offer a different flow-based perspective tailored to molecules. Specifically, we suggest coupled continuous normalizing equiivvaaa flows that bestow generative capabilities from neural partial differential equation (PDE) models on graphs. Graph PDEs have been known to enable designing new embedding methods such as variants of GNNs CChamberlain al. 2021), extending GNNs to continuous layers as Neural ODEs (Poli et al. 2019), and accommodating spatial information (lakovlev et al., 2020). We instead seek to bring to the fore their efficacy and elegance tools to help generate complex objects, such as molecules, viewed as outcomes resulting from an interplay of co-adapting latent trajectories (i.e., underlying dynamics). Concretely, flow is associated with each node of the graph, and these flows are conjoined as a joint ODE system conditioned on neighboring nodes. While these flows originate independently progressively toward more complex joint distributions as they repeatedly interact with the neighboring flows. We view molecules samples generated from the globally aligned distributions obtained after many such local feedback iterations. We call the proposed method Modular Flows (ModFlows) to underscore that each node can be regarded as module that coordinates with other modules. Table summarizes the capabilities of ModFlow compared to some previous generative works.
samples from simple distributions, they adjust
Contributions. We propose to learn continuous-time flow based generative models, grounded on graph PDEs, for generating molecules without resorting to any validity correction. In particular,
we propose ModFlow, novel generative model based on coupled continuous normalizing E(3) -equivrarrr flows. ModFlow encapsulates essential inductive bias using PDEs, and defines multiple flows that interact locally toward a globally consistent joint density;
Table 1: A comparison of generative modeling approaches for molecules.
Method
One-shot
Modular
Invertible
Continuous-time
JT-VAE
X
Jin et
2018
MRNN
X
X
X
X
Popova et al.
2019
GraphAF
X
X
Shi et al.
2020
GraphDF
X
X
X
Luo et al.
2022
MoFlow
X
X
Zang and Wang
(2020)
GraphNVP
X
/
X
Madhawa et al.
(2019
ModFlow
/
/
V
this work
2

---

=(0) ~ ((
2 pr
20
F
softmax
argmax
i.(1) (1(6).)))))a (6), NoA)
Time
(a) Graph
Latent Flows
(6) Atom Probability
(d) Assignment
Figure 2: A demonstration of the modular flow generation. The initial Gaussian distributions N/ I) evolve into complex densities z(T') under f and subsequently translated into probabilities and labels.
we encode permutation, translation, rotation, and reflection equivariance with E(3) equivari- ant GNNs adapted to molecular generation, and leverage 3D geometric information;
ModFlow is end-to-end trainable, non-autoregrressive, and obviates the need for any external validity checks or correction;
empirically, ModFlow achieves state-of-the-art performance on both the standard QM9 (Ra- makrishnan et al. 2014) and ZINC250K (Irwin et al. 2012) benchmarks.
2
Related works
Generative models. Earlier attempts for molecule generation (Kusner al. 2017 Dai al. 2018 aimed at representing molecules as SMILES strings (Weininger) 1988) and developed sequence generation models. A challenge for these approaches is learn complicated grammar rules that can generate syntactically valid sequences of molecules. Recently, representing molecules as graphs has inspired new deep generative models for molecular generation (Segler al., 2018 Samanta et al. 2018. Neil et al. 2018), ranging from VAEs (Jin et al. 2018 Kajino 2019) to flows (Madhawa et al. 2019 Luo et al. 2021 Shi al. 2020). The core idea learn encode molecular graphs into latent space, and subsequently decode samples from this latent space to generate new molecules (Atwood and Towsley 2016, Xhonneux et al. 2020 You et al. 2018).
Graph partial differential equations. Graph PDEs is an emerging area that studies PDEs structured data encoded as graphs. For instance, one can define a PDE on graphs to track the evolution of signals defined over the graph nodes under some dynamics. Graph PDEs have enabled, among others, design of new graph neural networks; see, e.g., works such as GNODE (Poli et al.) 2019). NeuralPDE (lakovlev et al. 2020),, Neural operator (Li al.] 2020). GRAND (Chamberlain et al.) 2021), and PDE-GCN (Eliasof et al. 2021). Different from all these works, we focus on using PDEs for generative modeling of molecules (as graph-structured objects). Interestingly, ModFlow proposed
in this work may be viewed as a new equivariant temporal graph network (Rossi et al.) 2020 Souza
al. 2022).
Validity oracles. A key challenge of molecular generative models is to be able to generate valid
molecules, according various criteria for molecular validity or feasibility. is common practice to call on external chemical software as rejection oracles to reduce exclude invalid molecules, or
do validity checks as part autoregressive generation (Luo al. 2021 Shi et al., 2020 [Popova et al. 2019). An important open question has been whether generative models can learn to achieve high generative validity intrinsically, i.e., without being aided by oracles or resorting to additional checks. ModFlow takes major step forward toward that goal.

---

3
Modular Flows
We focus on unsupervised learning of an underlying graph density p(G) using a dataset of observed molecular graphs 'D {G, We learn a generative flow model po(G) specified by flow parameters and use it to sample novel high-probabillty molecules.
3.1 Molecular Representation
Graph representation. We represent each molecular graph G = (V, E) of size M as tuple of vertices V (V)...., UM) and edges E V V. Each vertex takes a value from an alphabet on atoms: A {C, H. N, P. while each edge E B {1, 2, abstracts some bond type (i.e., single, double, or triple). We assume that, conditioned on the edges, the graph likelihood factorizes as a product of categorical distributions over vertices given their latent representations:
p(G) : :0 (z)) II Cat(l; (z.))
where 2; = (zse; -.....) ) RIA is a set of atom scores for node such that zu E R pertains to type E A, and the softmax function
exp(z.k)
(z,): En exp(zx))
(2)
which turns the real-valued scores 2, into normalized probabilities. ModFlow also supports 3D molecular graphs that contain atomic coordinates and angles as additional information.
Tree representations. We can obtain alternative representation for molecules: we can decompose each molecule into a tree-like structure, by contracting certain vertices into a single node (denoted as cluster) such that the molecular graph becomes acyelic. Following Jinn al. (2018), we restrict these clusters to ring substructures present in the molecular data, in addition to the atom alphabet. Thus, we obtain an extended alphabet Auee = AU {C1, C2.... where each cluster label C,- corresponds to some ring substructure in the label vocabulary X. We then reduce the vocabulary to the 30 most commonly occurring substructures of Atree For further details, see Appendix A.2
3.2 Differential modular flows
Normalizing flows Kobyzev et al. (021) provide a general recipe for constructing flexible probability
distributions, used density estimation (Cramer et al. [021]] HHungg al., 20188) and generative
modeling (Zhen et al. 2020 Zang and Wang 2020). We propose to model the atom scores 2;(() as Continuous-time Normalizing Flow (CNF) (Grtthwohl et al. 2018) over time / R... We assume the initial scores at time ( 0 follow an uninformative Gaussian base distribution z, (0) N/ /) for each node Node scores evolve parallel over time according to the differential equation
(t) So [t (1),,,
(1) = J
2N, (1), X., XN.), i (l,.,...,
where N₁ [j: (1,5) E F. the set of neighbors of node i and Zvz (1) = (z, (t) j E M3 the scores the neighbors time t; x; and xv; denote, respectively, the positional (2D/3D) information of and its neighbours; and 0 denotes the parameters of the flow function f to be learned. Stacking together all node differentials, we obtain a modular system of coupled ODEs:
(1)
( (1 (1) (N, (1), X1, XNi)
²(t)
: (1 aNw (1), XM, Xxw)
(t)
z (T7) 2(0)) + /- T
*(t)dt.
(5)
This coupled system ODEs may viewed as a graph PDE (lakovlev et al. 2020 Chamberlain al. 2021), where the evolution of each node depends only its neighbors.

---

The joint flow induces a corresponding change in the individual densities in terms of divergence of f (Chen et al. 2018).
dog ((()((() tr @5o 2i((,, (N)) Xi, Xx,)))
(6)
dt
Oz:
starting from the base distribution po();(0)) = (0) I). The trace picks only the diagonal elements of the Jacobian of which interprets the input from neighbors, 2, 'control' for each node z; at each instant f. An ODE solver is used for such systems, and the gradients are computed via the adjoint sensitivity method (Kolmogorov al. 1962). This approach incurs low memory cost, and explicitly controls the numerical error. Notably, moving towards modular flows translates sparsity also the adjoints.
2h'
Proposition 1: Modular adjoints are sparser than regular adjoints. They can be computed as
dA,
di -(1) 0f(1, (1t), (t), x,, XN,)
2₂(1) 05 (t) 21,, X;, XA,)
Σ
On
dzj
JEN.U()
where the partial derivatives
0f
dz Lzz ky are sparse (see Appendix A.l for the derivation).
084 =
3.3 Equivariant local differential
Our goal is to have a differential function f that a PDE operator used in Eqqation 4 and that satisfies the natural equivariances and invariances of the molecules. Specifically, this function must be (i) translation equivariant: translating the input results in an equivalent translation of the output; (ii) rotational (and reflection) equivariant: rotating the input results in equivalent rotation of the output; and (iii) permutation equivariant: permuting the input results in the same permutation of the output. Therefore, we chose to use E(3)-Equivariant GNN (EGNN) (Satorras et al. 2021), which is translation, rotation and reflection equivariant (E(n)), and permutation equivariant with respect to an input set of points (see Appendix A.33 for details). EGNN takes input the node embeddings as well as the geometric information (polar coordinates (2D) and spherical polar coordinates (3D)). ). Interestingly, ModFlow can be viewed as message passing temporal graph network (Rossi et al. 2020 Souza et al. 2022 as shown next.
Proposition 2: Modular Flows be cast message passing Temporal Graph Networks (TGNs). The operations are listed in Table where ModFlow is subjected to a single layer of EGNN. (See Appendix A.4 for more details).
Table 2: ModFlow as a temporal graph network (TGN). Adopting notation for TGNs from Rossi et al. (020) v; is node-wise event on é; eij denotes an (asymmetric) interaction between i and j: s; is the memory of node i; and and t- denote time with L being the time of last interaction before 1, e.g., s.((T) is the memory of just before time (; and msg and agg are learnable functions (e.g.. MLP) to compute, respectively, the individual and the aggregate messages. For ModFlow, we use riy to denote the spatial distance x; Xy. and azy to denote the attributes of the edge between i and j. The functions de. Φz. and Φn are as defined in (Satorras et al. 2021).
Method
TGN layer
ModFlow
m1)(1) = msg (s: (t") sy (t ( At, (0) (1) agg ((m' (1) [ E N.)
mi,(!) = de (z,, (0,10
Edge
(t) ESENro miy
(t) r.. (t) &e (m.)(t)
(f()) C EjENio ())
Memory state
(t) mem
x, + 1) = (t) + (t)
Node
x1(1)= Σ Jex, (s. 8, c. v,()),
2,(1 + 1) = dh (2) (1)
5

---

3.4 Training objective
Normalizing flows are predominantly trained minimize KL Ipo), i.e., the KL-divergence between the unknown data distribution Plata and the flow-generated distribution pa- This objective is equivalent maximizing Epaat. [log pel (Papamakarios et 2021). However, note that the discrete graphs G and the continuous atom scores x(?) reside in different spaces. Thus, in order to apply flows, mapping between the observation space and the flow space is needed. Farlier approaches use dequantisation to turn graph G' into a distribution of latent states, and argmax to deterministically map latent states to graphs (Zang and Wang 2020).
z(0)
((T)
0
(z((T)
G
sofemax
argnux
flow
Figure 3: Plate diagram showing both the inference and generative components of ModFlow.
We instead reduce the learning problem to maximizing (apata(((T)) log (z())))) where turn the observed set of graphs {G,) into a set of scores {z,.} using
Zn (G,,: e)
- onehot(G,))
[A5 1M
where onehot(G,) is matrix of size (n) x |A,1 (i.e., rows equal to the number of nodes in G and columns equal to the number possible node labels) such that (, that is if the vertex is labeled with atom k, and ( otherwise; l is vector with q entries each set I; A, E Atree: and [o. is added to model the noise in estimating the posterior ((z(T) due to short-circuiting the inference process from G to 2(T) skipping the intermediate dependencies, thereby inducing an unconditional distribution Pdata that slightly different from the true data distribution Pdata. The plate diagram in Figure summarizes the overall procedure.
if U2 ag E As.
Effectively, exploit the (non-reversible) composition of the argmax and softmax operations to transition from the continuous flow space to the discrete graph space, but skip this composition altogether in the reverse direction. Importantly, this shrot-circuiting allows ModFlow to keep the forward and backward flows between z( and z(T)) completely aligned (i.e., reversible) unlike previous approaches. We maximize the following objective over N training graphs:
argmax Epacaal log po
(8)
0
log (a(T) zn)
((n)
M(n)
/ tr Öz,(t)
asal1, 2;(),, 2N, (N, Xi, dt (10)
Σ
log (2)
Σ/
which factorizes over the size (n) of the n training molecule. The encoding probability follows from Equation 6 where z(0) can be traced by traversing the flow backward in time starting from 0. practice we solve ODE integrals using numerical solver such as
Zn at time ł = until Runge- Kutta. We thus delegate this task to general solver of the form ODESolve( zz fo. T), where map fo is applied for T steps starting with z. An optimizer optim also required for updating A.
3.5 Molecular generation
Given a molecular structure, we can generate novel molecules by sampling an initial state z(0) ~ (0, /). and running the modular flow forward in time for steps and obtain x(T). This procedure maps a tractable base distribution Po to more complex distribution pr. We follow argmax to pick the most probable label assignment for each node (Zang and Wang 2020). We outline the procedures for training and generation in Algorithm
I and Algorithm respectively.

---

Algorithm Training ModFlow
Algorithm 2 Generating with ModFlow
Require: Dataset D, iterations Tiiters batch size
1: Sample z(0) ~ (0
B, number of batches ng
x(T) = ODESolve(z(O), Ja, T)
1: Initialise parameters of ModFlow (EGNN)
3: Assign labels by (rgmax((((((T))
2: for k ..... Riter do
3: for b = 1, ...... B do
4: Sample D = [G....., GB} from D
5: Define (T) == (T) G, E }
6:
Set z (T) to 2 (G:ic)
7:
Lb - B
Σ
log (2, (T)), using
D.
76 (0) ODESolve (T), (T)
8: end for
9: 0 < optim T 2b Lsj
end for
Experiments
We first demonstrate the ability of Modular Flows (ModFlow) learn highly discontinuous synthetic patterns on 2D grids. We also evaluated ModFlow models trained, variously, on (i) 2D coordinates, (ii) representation, and (iv) 3D coordinates
3D coordinates, (iii) 2D coordinates on the tasks of molecular generation and optimization. Our results show that ModFlow compares favorably other prominent flow and non-flow based molecular generative models, including GraphDF (Luo ct al.
representation
202T). GraphNVP (Madhawa et al., 2019), MRNN(Popova et al., 2019). and GraphAF (Shi al. 2020 Notably, ModFlow achieves sfate-of-the-art results without validity checks or post hoc correction. We also provide results of our ablation studies to underscore the relevance of geometric features and equivariance toward this superlative empirical performance.
4.1 Density Estimation
We generated synthetic data in the following way. We considered two variants of chessboard pattern, namely, (i) x grid where every node takes binary value, 1, and neighboring nodes have different val- ues; and (ii) 16 x 16 grid where nodes in each block of 4 x all take the same value (0 1). different from the ad- jacent blocks. We also experimented ModFlow with a 20 x 20 grid describing alter- nating stripes of Os and 1s.
Stripes
16x16 chess board
buard
Data
Figure 4 shows that ModFlow learn neural differential functions fe that reproduce the patterns almost per- fectly, indicating sufficient capacity to model complex patterns. That is, ModFlow is able to transform
Figure ModFlow can accurately learn to reproduce complex, discontinuous graph patterns.
the initial Gaussian distribution into different multi-modal and discontinuous distributions.
4.2 Molecule Generation
Data. We trained and evaluated all the models on ZINC250K (Irwin et al. 2012) and QM9 (Ray makrishnan et al., 2014 4) datasets. The ZINC250K set contains 250,000 drug-like molecules, each consisting of up to 38 atoms. The QM9 set contains 134,000 stable small organic molecules with atoms from the set {c, H, D, F). The molecules are processed to be the kekulized form with hydrogens removed by the RDkit software (Landrum et al. 2013

---

Table 3: Random generation on QM9 (top) and ZINC250K (bottom) without post hoc validity corrections. Results with are taken from Luo et al. (002I) Higher values are better for all columns.
Method
Validity %
Uniqueness
Novelty
Reconstruction %
GVAE
60.2
9.3
80,9
96.0
GraphNVP*
83.1
99.2
58.2
100
GRF
84.5
66
58.6
100
GraphAF
67
94.2
88.8
100
GraphDF
82.7
97.6
98.1
100
MoFlow
89.0
98.5
100
ModFlow (2D-EGNN)
96.2 1 1.7
99,5
100
100
ModFlow (3D-EGNN)
98.3 0.7
99.1
100
ModFlow (JT-2D-EGNN)
97.9 + 1.2
99.2
100
100
ModFlow (JT:3D-EGNN)
99.1 = 0.8
99.3
100
100
Method
Validity %
Uniqueness
Novelty
Reconstruction %
MRNN
65
99.89
100
n/a
GVAE
7.2
9
53.7
GCPN
20
99.97
100
n/a
GraphNVP
42,6
94.8
100
100
GRF
73.4
53.7
100
100
GraphAF
68
99.1
100
100
GraphDF
89
99.2
100
100
MoFlow
50.3
99.9
100
100
ModFlow (2D-EGNN)
94.8
99.4
100
ModFlow (3D-EGNN)
95.4 + 1.2
99.7
100
100
ModFlow (JT-2D-EGNN)
97.4 + 144
99.1
100
100
ModFlow (JT-3D-EGNN)
98.1 + 0.9
99.3
100
100
Setup. We adopt common quality metrics to evaluate molecular generation. Validity is the fraction of molecules that satisfy the respective chemical valency of each atom. Uniqueness refers to the fraction of generated molecules that is unique (i.e, not duplicate of some other generated molecule). Novelty is the fraction of generated molecules that is not present in the training data. Reconstruction the fraction of molecules that can be reconstructed from their encoding. Here, we strictly limit ourselves to comparing all methods on their validity scores without resorting to external correction. We trained each model with random weight initializations, and generated 50,000 molecular graphs for evaluation. We report the mean and the standard deviation scores across these multiple runs.
The models were implemented in Py (Paskke al. 2019). The EGNN method used only a single layer with embedding dimension of 32. We trained with the Adam optimizer (Kingma and Ba 2014) for 50-100 epochs (until the training loss became stable), with batch size 1000 and learning rate 0.001. ModFlow is significantly faster compared to autoregressive models such GraphAF and GraphDF. For more details, Appendix A.5
Implementation.
Results. Table 3 reports the performance on QM9 (top) and ZINC250K (bottom) respectively. ModFlow achieves state-of-the-art results across all metries. Notably, its reconstruction rate is 100% (similar to other flow models); in addition, however, the novelty (100%) and uniqueness scores (209%) are also very high. Moreover, ModFlow surpassed the other methods validity (95%-99%)
In Appendix A.6 we document additional evaluations with respect to the MOSES metrics that access the overall quality of generated molecules, well as the distributions of chemical properties. All these results substantiate the promise ModFlow as an effective tool for molecular generation.
4.3 Property- -targeed Molecular Optimization
The task of molecular optimization is to search for molecules that have better chemical properties. We choose the standard quantitative estimate of drug-likeness (QED) as our target chemical property. QED measures the potential of molecule be characterized as a drug. We used pre-trained

---

(a) QM9 Dataset
(b) ZINC250K Dataset
Figure 5: Samples of molecules generated by ModFlow. More examples are shown in Appendix A.7
ModFlow model I to encode molecules M into their embeddings = f(M), and applied linear regression obtain QED scores J' from these embeddings. We then interpolate in the latent space of each molecule along the direction of increasing QED via several gradient ascent steps, i.c. updates of the form 2' Z + À dy where denotes the length of the search step. The final embedding thus obtained decoded as a new molecule via the reverse mapping f-1
d2
QED:0332 0.33
QED: 0.428
QED: 0557
Figure 6: Example of chemical property optimization the ZINC250K dataset. Given the left-most molecule, interpolate in latent space along the direction which maximizes its QED property.
OH
QED: 0.355
QED:9.41 0.419
QED: 0.552
QED: 0664
Figure 7: Example of chemical property optimization the QM9 dataset. Given the left-most molecule, interpolate latent space along the direction which maximizes QED property.
Table 4: Performance in terms of the best QED scores (baselines are taken from Luo al. 2021
Method
1st
2nd
3rd
ZINC (dataset)
0.948
0.948
0.948
JTVAE
0.925
0.911
0.910
GCPN
0.948
0.947
0.945
MRNN
0,844
0.799
0.736
GraphAF
0,948
0.948
0.947
GraphDF
0,948
0.948
0,948
MoFlow
0.948
0.948
0.948
ModFlow
(2D-EGNN)
0.948
0.941
0.937
ModFlow (3D-EGNN)
0.948
0.937
0.931
ModFlow (T-2D-EGNN)
0.947
0.941
0.939
ModFlow (JT-3D-EGNN)
0.948
0.948
0.945

---

Figure 6 and Figure 7 show examples of the molecules decoded from the learned latent space using this procedure, starting with molecules having a low QED score. Note that the number of valid molecules decoded back varies on the query molecule. We report the discovered novel molecules sorted by their QED scores QED scores.
Table Clearly, ModFlow able to find novel molecules with high
4.4 Ablation Studies
We also performed ablation experiments to gain further insights about ModFlow, as we describe next.
-(3i-eauvya versus not equivariant. Molecules exhibit translational and rotational symme- tries, so we conducted an ablation study to quantify the effect of incorporating these symmetries in our model. We compare the results obtained using EGNN with non-equivariant graph convolutional network (GCN). For our purpose, we used a 3-layer GCN with layer sizes 64-32-32. The validity scores in Table provide strong evidence in favor of modeling the symmetries explicitly in the proposed Modular Flows.
Table 5: Random generation performance on ZINC250K and QM9 dataset with E(3)-EGNN vs GCN.
Dataset
Method
Validity
Uniqueness %
Novelty
ModFlow (3D-EGNN)
95.4 + 1.2
99.7
ZINC250K
ModFlow (GCN)
90.3 = 1.9
99.7
100
ModFlow (3D-EGNN)
98.3 + 0.7
99.1
100
QM9
ModFlow (GCN)
93.3 = 0.5
98.8
100
2D versus 3D. Finally, we study whether including information about the 3D coordinates improves the model. Note that the EGNN-coupled differential function obtains either the 2D 3D positions as polar coordinates, where the 3D positions have an extra degree of freedom. Table 6 shows that transitioning from 2D 3D improves the mean validity score.
Table Random generation on ZINC250K and QM9 dataset with 2D versus features.
Dataset
Method
Validity
Uniqueness %
Novelty
ModFlow (3D-EGNN)
95.4 + 1.2
99.7
100
ZINC250K
ModFlow (2D-EGNN)
94.8 = 1.0
99.4
100
ModFlow (3D-EGNN)
98.3 + 0.7
99.1
100
QM9
ModFlow (2D-EGNN)
96.2 + 1.7
99.5
100
5 Conclusion
We proposed ModFlow, new generative flow model where multiple flows interact locally according to a coupled ODE, resulting in accurate modeling of graph densities and high quality molecular generation without any validity checks correction. Interesting avenues open up, including the design of (a) more nuanced mappings between discrete and continuous spaces, and (b) extensions of modular flows to (semi-)sun suuervisueeeeeeed settings.
Acknowledgments
The calculations were performed using resources within the Aalto University Science-IT project. This work has been supported by the Academy of Finland under the HEALED project (grant 13342077).
10

---

References
James Atwood and Don Towsley. Diffusiion-oonvllutioall neural networks. Advances in neural information processing systems, 29, 2016.
G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty drugs. Nature chemistry, 4(2):90-98, 2012.
Andrew M. Bradley. PDE-constrained optimization and the adjoint method. Technical report, 2019.
Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. International Conference on Machine Learning, pages 1407-1418. PMLR. 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Fike Cramer, Alexander Mitsos, Raul Tempone, and Manuel Dahmen. Principal component density estimation for scenario generation using normalizing flows, 2021. URI http.tttt:/:riiiii wrw. wrrg/abs/2104.. 10410
Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Le Song. dirextd variational autoencoder for structured data, 2018. URL.htts://arxiv org 1tt///88000088888
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv: 2014.
Laurent Dinh, Jascha Sohl-Dick Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 2016.
Moshe Eliasof, Eldad Haber, and Eran Treister. Pde-gen:
Novel architectures for graph neural networks
motivated by partial differential equations. Advances in Neural Information Processing Systems, 34, 2021.
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminformatics, 1(1):1-11, 2009.
Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. Generalization and representational limits of graph neural networks. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 Proceedings of Machine Learning Research, pages 3419-3430. PMLR, 2020. URL htPphttpp/prroeeednngs mlr axres/vv19/geet www.gaa..g..aaarme html
Ian J. Goodfellow, Jean Pouget-Ahadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. httptt//t///aatt org/abs. 1406..80 2661
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, 1lya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint 17100.7810. 2018.
Shion Honda, Hirotaka Akita. Katsuhiko Ishiguro. Toshiki Nakanishi, and Kenta Oono. Graph residual flow for molecular graph generation. arXiv preprini arXiv:1909. 1909.1321, 2019.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows, 2018.
URL htt.tbttt//aaattte org/ www./////////////////.///.e/eee/ 1804. 00779
Valerii lakovley, Markus Heinonen, and Harri Lähdesmäki. Learning continuous-time pdes from sparse data with graph neural networks. arXiv preprint trXiv:2006.0895 2020.
John Ingraham. Vikas Garg. Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. In H. Wallach, H. Larochelle, A. Beygelzimer. d' d'Alché-Buc, Fox, and R. Garnett, editors, vances in Neural Information Processing Syxtems, volume 32, Curran Associates. Inc., 2019. Utt /// |proceedings neurips http://ppe..ooeeeeee.eaeeaeeaaaa
Paper pdff
John J Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. Zine: a free tool to discover chemistry for biology. Journal of chemical information and modeling. 52(7):1757-174 2012.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molccular graph generation. arXiv preprint arXiv: 4r244:1802.04364, 2018
Hiroshi Kajino. Molecular hypergraph grammar with its application to molecular optimization. In International Conference on Machine Learning, pages 3183 3191. PMLR, 2019.

---

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: arXiv:1412.6988 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2013. URL httptttttaaaai....... 0bb//312.6o 6114
Ivan Kobyzev, Simon J.D. Prince, and Marcus A. Brubaker. Normalizing flows: An introduction and review of
current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11): 1):39113393997 nov 2021. doi: 10. 1000ppam..200002999990000
Jttpps//s://doi..... hrtp::::///oi..r/11///// 1109X2Ftpami. 2020 2992934
Andrei Nikolaevich Kolmogorov. Ye F Mishchenko, and Lev Semenovich Pontryagin. A probability problem of optimal control. Technical report, JOINT PUBLICATIONS RESEARCH SERVICE ARLINGTON VA, 1962.
Matt Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoencoder, 2017.
URL http:/ttt////i...otot abs/ 1703.000000000 01925
Greg Landrum et al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling, 2013.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485 2020.
Jacchang Lim, Scongok Ryu, Jin Woo Kim, and Woo Youn Kim. Molecular generative model based on conditional variational autoencoder for de novo molecular design. Journal of cheminformatics, 10(1):1-9, 2018.
Phillip ipppe and Efstratios Gavves. Categorical normalizing flows via continuous transformations. In Inter- national Conference on Learning Representations, 2021. URI. http://ppp///ppamerrreee.//////p GLNZeVDuik
Youzhi Luo, Keqiang Yan, and Shuiwang Ji. Graphdf: A discrete flow model for molecular graph generation, 2021. URL
org/ abo/2102...222 01189
Kaushalya Madhawa. Katushiko Ishiguro. Kosuke Nakago, and Motoki Abe. Graphnyp: An invertible flow model for generating molecular graphs. arXiv preprint arXiv:1905.1 11600, 2019.
Lukasz Maziarka, Agnieszka Pocha, Jan Kaczmarczyk, Krzysztof Rataj. Tomasz Danel, and Michal Warchot. Mol-cyclegan: generative model for molecular optimization. Journal of Cheminformatics, 12(1):1-18, 2020.
Kenneth M. Merz, Gianni De Fabritiis, and Guo- Wei Wei. Generative models for molecular design. Journal of Chemical Information and Modeling, 60(12) 10:3555633-5666 2020. doi: (0021/acs.jcim.0000 https::/doi. org/
URL
10. 1021/acc..c... jeim. 0c01388
PMID: 33378853.
Daniel Neil, Marwin Segler, Laura Guasch, Mohamed Ahmed, Dean Plumbley, Matthew Sellwood, and Nathan Brown. Exploring deep recurrent models with reinforcement learning for molecule design. 2018
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 1-64, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killcen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Michael Poli, Stefano Massaroli, Junyoung Park, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. Graph 2019.
neural ordinary differential equations. arXiv preprint
Pavel G Polishchuk, Timur Madzhidov, and Alexandre Vamnek. Estimation of the size of drug-like chemical space based on gdb-17 17 data. Journal of computer-aided molecular design, 27(8) 6799, 2013.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin. Simon Johansson, Hongming Chen, Sergey Nikolenko. Alan Aspuru-Guzik, and Alex Zhavoronkov. Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. Frontiers in Pharmacology. 2020.
12

---

Mariya Popova, Mykhailo Shvets, Junier Oliva, and Olexandr Isayev. Molecularrnn: Generating realistic molecular graphs with optimized properties. 2019. URL htttps:///azzii org/ hwtg/bbs/1905..0 13372
Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Gunter Klambauer. Fréchet chemnet distance: metric for generative models for molecules in drug discovery. Journal of chemical information and modeling, 58(9):1736-11) 1741, 2018.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 2014.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022. httpps//////xxii org/ wwwp///4...0000 06125
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs, 2020. URL https:// org/ abs/ 20066.0633)
Bidisha Samanta, Abir De, Niloy Ganguly, and Manuel Gomez Rodriigzz. Designing random graph models using variational autoencoders with applications to chemical design. arXiv preprint arXiv: 2018.
Victor Garcia Satorras, Emiel Hoogeboom. and Max Welling. E(n) equivariant graph neural networks, 2021. URL (RLL/atps://arxil
Franco Scarselli, Marco Gori, Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20x 1)--880, 2009. doi: 10. 10.1109//NN,2008. 2005605,
Marwin HS Segler. Thierry Kogej. Christian Tyrchan, and Mark P Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1): 2018.
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a flow-based autoregressive model for molecular graph generation. arXiv preprint arXiv:2001. 09382. 2020.
Amauri Souza, Diego Mesquita, Samuel Kaski, and Vikas Garg, Provably expressive temporal graph networks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Jonathan M. Stokes, Kevin Yang. Kylc Swanson, Wengong Jin, Andres Cubillos-Ruiz Nina M. Donghia, Craig R. MacNair. Shawn French, Lindsey A. Carfrae, Zohar BlkemmAckermann, Victoria M. Tran, Anush Chiappino- Pepee, Ahmed H. Badran, Ian W. Andrews, Emma J. Chory, George M. Church, Eric D. Brown, Tommi Jaakkola, Regina Barzilay. and James Collins. A deep learning approach to antibiotic discovery. 00088888--222211 702.c13, 2020. ISSN 0092-86777 doi: https:://oii coll 2020.01.2 URL http////aa////////////00//000000
Cell, 180(4))688
https://www.se.. https://www..eeeccdeeee..cm com wwt.///c/ccccieee sttpssecccccccccceceeeeee
David Weininger. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36 1988.
Scott A Wildman and Gordon M Crippen. Prediction of physicochemical parameters by atomic contributions. Journal of chemical information and computer sciences, 39(5).868 873, 1999.
Louis-Pascal Xhonneux, Meng Qu, and Jian Tang. Continuous graph neural networks. In International Conference on Machine Learning, pages 10432-10444 10441. PMLR, 2020.
Jiaxuan You, Rex Ying. Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. In International conference on machine learning, pages 5708-$717. PMLR. 2018
Chengxi Zang and Fei Wang. Moflow: an invertible flow model for generating molecular graphs In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 617-626. 2020.
Xingjian Zhen, Rudrasis Chakraborty, Liu Yang, and Vikas Singh. Flow- generative models for learning 10013
manifold to manifold mappings, 2020. URL
13


Reviews:
[{"review_id": "nPEiLsMq4DA", "reviewer": "Reviewer_HLkc", "summary": "This work introduces ModFlow, a modular continuous normalizing flow model parametrized by an E(3) equivariant GNN. To demonstrate the efficacy of ModFlow, the model is trained on QM9 and ZINC250K to the end of generating valid, unique, and novel samples. ", "questions": "1. Line 156, 157: should these be rotational equivariant and permutation equivariant instead of invariant?\n2. Line 180-183: what does it mean for the forward and backward flows to be completely aligned?\n3. To the best of my knowledge, the ZINC250k dataset only contains 2D atom coordinates. Are the generated 3D structures shown in Figure 5 ZINC250K computed via additional libraries, or are they generated by ModFlow? Similarly, are the QM9 conformers generated by ModFlow, or are they computed based on the molecular graph?\n4. How does ModFlow compare to other models in terms of training/inference time? How does the training/inference time of ModFlow scale with molecule size (QM9 \\to ZINC250K)?\n5. I understand that it is appealing to have a model that captures chemical composition laws from data rather than relying on e.g. valency checks from external libraries. However, I am wondering if there is an example application in which a model's intrinsic generative validity matters?", "limitations": "While ModFlow can generate molecules with high validity by directly sampling from the latent space, there does not seem to be a straightforward way to incorporate valency checks into the sampling procedure of ModFlow, should one desire to do so.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The writing is clear for the most part, and the model is well-presented. \n2. The validity, uniqueness, and novelty results for QM9 and ZINC250K look great. The generated samples also seem reasonable.\n\nWeaknesses:\nMy main concerns with this work are related to its numerical experiments.\n1. More metrics should be included for a comprehensive assessment of ModFlow's generation quality. This is because validity, uniqueness, and novelty scores may not correlate with how realistic the samples are. To this end, one may use the Frechet ChemNet Distance (FCD), which measures the similarity between the set of generated molecules and the set of training molecules. One can easily compute the FCD as well as a variety of other metrics through MOSES: https://github.com/molecularsets/moses, and the inclusion of these additional metrics will make the paper much more convincing.\n2. More experiments are needed to fully demonstrate the effectiveness of ModFlow. For example, property optimization is conducted in e.g. JTVAE and GraphAF to find the best generated molecules with certain chemical scores, e.g. penalized LogP, QED. Such optimization showcases that the model is able to learn a meaningful latent space useful for downstream tasks. Other possibilities are latent space interpolation and property prediction.\n\nEdit: in light of the added experimental results during the rebuttal, I have now raised my score to 6 as I believe the paper makes a solid contribution in cheminformatics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mCRWts9MNfA", "reviewer": "Reviewer_obfC", "summary": "The authors of this paper propose to use a continuous-time normalizing flow and E(3)-Equivariant GNN as the differential function in the neural PDE for molecular generation. Experimental results on QM9 and ZINC show that the proposed model, ModFlow, obtained better validity and novelty in comparison with existing graph flow based models. ", "questions": "The authors should (1) use more informative metrics, (2) dissect a test set for each dataset, and (3) compare with non-flow-based state of the art molecular generation models.  ", "limitations": "Only technical limitations are mentioned as future work in one sentence in Section 5 Conclusion. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The consideration of using continuous-time normalizing flow as encoder makes sense. The authors find the their proposed model is equivalent to a variant of temporal graph network (TGN). \n\nWeakness: \n1. The evaluation metrics used in this paper are validity, uniqueness, novelty, and reconstruction, which are usually used for sanity check. To further look into the consistency of between the generated samples and the training samples, more informative metrics should be used. For example, the metrics in MOSES can be considered. \n2. It seems that an intact test set was not used for result evaluation. \n3. Furthermore, the computational comparison should not be limited to flow-based methods. Other state-of-the-art deep learning based molecular generation approaches should be compared as well. \n4. Minors: (1) Moroever -> Moreover. (2) Check the capitalizations in each reference item. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "9fWebeutI-", "reviewer": "Reviewer_c5ye", "summary": "The authors propose a new generative model for molecules based on normalizing flows. The network is based on a set of ODEs (one per node), that are coupled together to form a PDE, which lets the model graph densities accurately. The model is also E(3) equivariant, which is an important physical property for molecules.\n\nThe paper presents experiments on QM9 and Zinc250K datasets and evaluate results on validity, uniqueness, novelty, and reconstruction metrics. The ModFlow method performs well on these metrics.", "questions": "1. Please include a discussion of the runtime of your model for training and inference.\n2. How does this scale with the number of atoms? Can it effectively generate larger systems like materials or bio molecules?", "limitations": "Please discuss limitations in terms of computational efficiency, classes of atomic systems that can be generated by this method etc.\nFor example, this method cannot directly be used for materials where periodic boundary conditions need to be respected.\n\nAlso, potential negative societal impact (such as generating toxic materials, etc.) should be mentioned.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* As the ModFlow method is based on combining normalizing flows with neural ODEs, it can generate graphs in one-shot, and also provide density estimates (unlike GANs).\n* The model is E(3) equivariant, which is a desirable property for molecules and other atomic systems.\n\nWeaknesses\n* It is not entirely clear how useful the metrics used in the paper are. As such they are weak metrics -- for example, validity simply measures the fraction of molecules that do not violate chemical valency rule. Most chemical applications require generating based on more complex properties, such as generating non-toxic drug molecules, generating synthesizable materials, etc. Therefore, it is unclear how useful this method is in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1h885PzzTWb", "reviewer": "Reviewer_FLBL", "summary": "The presented method combines continuous E(3)-equivariant flows and PDEs on graphs to generate molecular structures.", "questions": "- Eq. (1) seems to encapsulate a central assumption of this probabilistic model. Why should it be that, given edge information, the nodes can be labeled independently?\n- Sometimes, I am unsure whether you are working with 2D or 3D graphs. On the one hand, you use concepts such as bond orders and chemical validity; on the other, you are working with 3D structures. Do you convert generated 3D structures back to 2D graphs?\n- Tree representation:\n  - Is it always possible to decompose a molecular graph into a tree-like structure?\n  - I understand that the model generates 3D molecular structures. How do you handle substructures, i.e., groups of atoms consisting of more than one atom?\n- How is the neighborhood of an atom $\\mathcal{N}_i$ defined? Do you recompute that throughout the generation process?\n- Section 3.3: are you confusing invariant and equivariant in this section?", "limitations": "The authors adequately addressed the limitations of their work.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- Figure 2 helped me get a high-level understanding of the model\n- Experiment 4.1 show-cased the capabilities of the model with a simple example\n- Sections 4.2 and 4.3 are easy to follow and fairly clear.\n- The results seem impressive: the model's scores on \"validity\", \"uniqueness\", \"novelty\", and \"reconstruction\" seems impressive.\n\n# Weaknesses\n\n- The paper is not exceptionally well-written. I had a hard time following Sections 3.2 and 3.3 -- the main contributions of this paper.\n- The reader is expected to be familiar with the EGNN model. If the reader doesn't happen to know the exact inputs and mode of operation of this model, the reader will get lost. The importance of this borrowed architecture in this approach warrants a short introduction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept