Continuous Graph Flow
Authors: Zhiwei Deng
Megha Nawhal
Lili Meng
Greg Mori
Venue:  (2020)
Keywords: graph flow
normalizing flow
continuous message passing
reversible graph neural networks

Abstract: In this paper, we propose Continuous Graph Flow, a generative continuous flow based method that aims to model complex distributions of graph-structured data.  Once learned, the model can be applied to an arbitrary graph, defining a probability density over the random variables represented by the graph. It is formulated as an ordinary differential equation system with shared and reusable functions that operate over the graphs.  This leads to a new type of neural graph message passing scheme that performs continuous message passing over time. This class of models offers several advantages: a flexible representation that can generalize to variable data dimensions; ability to model dependencies in complex data distributions; reversible and memory-efficient; and exact and efficient computation of the likelihood of the data. We demonstrate the effectiveness of our model on a diverse set of generation tasks across different domains: graph generation, image puzzle generation, and layout generation from scene graphs. Our proposed model achieves significantly better performance compared to  state-of-the-art models.

Under review as a conference paper at ICLR 2020
CONTINUOUS GRAPH FLOW
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we propose Continuous Graph Flow, a generative continuous flow based method that aims to model complex distributions of graph-structured data. Once learned, the model can be applied to an arbitrary graph, defining probabil- ity density over the random variables represented by the graph. It is formulated as an ordinary differential equation system with shared and reusable functions that operate over the graphs. This leads to a new type of neural graph message pass- ing scheme that performs continuous message passing over time. This class of models offers several advantages: flexible representation that can generalize variable data dimensions; ability to model dependencies in complex data distri- butions; reversible and memory-effeiient; and exact and efficient computation of the likelihood of the data. We demonstrate the effectiveness of our model on a diverse set of generation tasks across different domains: graph generation, image puzzle generation, and layout generation from scene graphs. Our proposed model achieves significantly better performance compared state-of-the-art models.
INTRODUCTION
Modeling and generating graph-structured data has important applications in various scientific fields such building knowledge graphs (Lin et al. 2015; Bordes al., 2011). inventing new molecular structures (Gilmer et al., 2017) and generating diverse images from scene graphs (Johnson et al., 2018. Being able train expressive graph generative models is an integral part of Al research.
Significant research effort has been devoted in this direction. Traditional graph generative meth- ods (Erdös Rényí 1959 Leskovee et al., 2010) Albertt & Barabási 2002 Airoldi al., 20088) are based on rigid structural assumptions and lack the capability to learn from observed data. Modern deep learning frameworks within the variational autoencoder (VAE) (Kingma & Welling 2014) for- malism offer promise of learning distributions from data. Specifially, for structured data, research efforts have focused on bestowing VAE based generative models with the ability to learn structured latent space models (Lin et al. 2018 He et al. 2018 Kipf & Welling 2016). Nevertheless, their capacity is still limited mainly because of the assumptions placed on the form of distributions. An- other class of graph generative models are based on autoregressive methods YYou al. [018 Kippf let al. 2018). These models construct graph nodes sequentially wherein each iteration involves gen- eration of edges connecting generated node in that iteration with the previously generated set of nodes. Such autoregressive models have been proven to be the most successful so far. However, due to the sequential nature of the generation process, the generation suffers from the inability to main- tain long-term dependencies in larger graphs. Therefore, existing methods for graph generation are yet realize the full potential of their generative power, particularly, the ability to model complex distributions with the flexibility to address variable data dimensions.
Alternatively, for modeling the relational structure in data, graph neural networks (GNNs) or mes- sage passing neural networks (MPNNs) (Scarselli et al., 2009 Gilmer et al. 2017]] Duvenaud et al.) 2015 Li et al. 2017 Kipf & Welling 2017]] Santoro et al. 2017]] ZZhng et al. 2018) have been shown to be effective in learning generalizable representations over variable input data dimensions. These models operate on the underlying principle of iterative neural message passing wherein the node representations are updated iteratively for fixed number of steps. Hereafter,
the term message passing to refer to this neural message passing in GNNs. We leverage this representational ability towards graph generation.
In this paper. we introduce new class of models Continuous Graph Flow (CGF): graph gener- ative model based on continuous normalizing flows (Chen et al. 2018 Grathwohl et al. 2019) that

---

Under review as a conference paper at ICLR 2020
lteratione
Eo
[1 Time (Contnazus)
(a) Discrete neural message passing
(b) Continuous graph flow
Figure 1: Illustration of evolution of message passing mechanisms from discrete updates (a) to our proposed continuous updates (b). Continuous Graph Flow leverages normalizing flows to transform simple distributions (c.g. Gaussian) at to to the target distributions at t1. The distribution of only one graph node is shown here for visualization, but, all the node distributions transform over time.
generalizes the message passing mechanism in GNNs to continuous time. Specifically, to model continuous time dynamics of the graph variables, adopt neural ordinary different equation (ODE) formulation. Our CGP model has both the flexibility to handle variable data dimensions (by using GNNs) and the ability to model arbitrarily complex data distributions due free-form model architectures enabled by the neural ODE formulation. Inherently, the ODE formulation also imbues the model with following properties: reversibility and exact likelihood computation.
Concurret work on Graph Normalizing Flows (GNF) (Liu et al. 2019) also proposes a reversible graph neural network using normalizing flows. However, their model requires fixed number of transformations. In contrast, while our proposed CGF is also reversible and memory efficient, the underlying flow model relies on continuous message passing scheme. Moreover, the message pass- ing in GNF involves partitioning of data dimensions into two halves and employs coupling layers to couple them back. This leads to several constraints on function forms and model architectures that have a significant impact performance (Kingma & Dhariwal 2018). In contrast, our CGF model has unconstrained (free-form) Jacobians, enabling to learn more expressive transforma- tions. Moreover, other similar work GraphNVP et al. (2199) is also based normalizing flows as compared to CGF that models continuous time dynamics.
We demonstrate the effectiveness of our CGF-based models three diverse tasks: graph generation, image puzzle generation, and layout generation based on scene graphs. Experimental results show that our proposed model achieves significantly better performance than state-of-the-ard models.
2
PRELIMINARIES
Graph neural networks. Relational networks such as Graph Neural Networks (GNNs) facilitate learning of non-linear interactions using neural networks. In every layer of GNN, the embedding h ( corresponding to a graph node accumulates information from its neighbors of the previous layer recursively as described below.
h (f (fro(n! (e-1) (1-1) "/ (eff-")) s(i)
(I)
where the function g is aggregator function, S(i) is the set of neighbour nodes of node i, and Jij is the message function from node to node i, æ, (f-1) and a, (6-1) represent the node features corresponding to node and at layer (E - 1) respectively. Our model uses a restricted form of GNNs where embeddings of the graph nodes updated in-place (a, +- h2), thus, we denote graph node as x and ignore h hereafter. These in in-paace updates allow using æ; in the flow-based models while maintaining the same dimensionality across subsequent transformations.
Normalizing flows and change of variables. Flow-based models enable construction of com- plex distributions from simple distributions (e.g. Gaussian) through sequence of invertible map- pings (Rezende & Mohamed 2015). For instance, a random variable is transformed from an initial state density 2n the final state žκ using chain of K invertible functions f& described
fx ... f2 / (z()).
The computation log- likeliiel of random variable uses change of variables rule formulated as:
K
log det 0fe(zt-1)
log (*k) = log))(z0))
Σ
(3)
Oz
2

---

Under review as a conference paper at ICLR 2020
where (z is the Jacobian of fx for k (1.)....KK)
Contimuous normalizing flows. Continuous normalizing flows (CNFs) (Chen et al. 2018 Grath- wohl et al. 2019) model the continuous- dynamics by pushing the limit number of transfor- mations, Given a random variable 2, the following ordinary differential equation (ODE) defines the change in the state of the variable.
Oz
ôt (t)
(4)
Chen et al. (018 extended the change of variables rule described Eq. 3 to continuous version. The dynamics of the log- ogglikklihood of random variable is then defined as the following ODE.
Öp(z(t)
Of
(5)
Ōt
(z(()
Following the above equation, the log likelihood of the variable time t starting from time to is
log p(=((t))) log p(x((t))) - -f.² Tr (z(t)
@f(z)
(6)
where the trace computation is more computationally efficient than computation of the Jacobian in Equation (4). Building CNFs, present continuous graph flow which effectively models continuous time dynamics over graph-structured data.
3 CONTINUOUS GRAPH FLOW
Given a set of random variables X containing n related variables, the goal to learn the joint distribution p( X) of the set of variables X. Each element of set X is æ; R where 1,2 ...... and m represents the number of dimensions of the variable. For continuous time dynamics of the of variables X, formulate an ordinary differential equation system as follows:
(t)
f')((t))
(()(t))
:
(t)
(X)((())
where à; = dat:/tt and (1) is the set of variables at time f. The random variable æ; at time to follows base distribution that can have simple forms, e.g. Gaussian distributions. The function f implicitly defines the interaction among the variables. Following this formulation, the transforma- tion of the individual graph variable defined as
(x(tt) (to) + I(' (((
This provides transformation of the value of the variable from time to to time 41
3.1 CONTINUOUS MESSAGE PASSING
The form in Eq. & represents a generic multi-variate update where interaction functions are defined over all the variables in the set X. However, the functions do not take into account the relational structure between the graph variables.
To address this, define neural message passing process that operates the update functions over variables according to the graph structure. This process begins from time to where each variable æ, (to) only contain local information. At time t, these variables are updated based on the information gathered from other neighboring variables. For such updates, the function I' Eq. is defined as:
graph by defining
r (x 9()]= (1), S(O)).
where f.,(-) is a reusable message function and used for passing information between variables x; and v.j S(1) is the set of neighboring variables that interacts with variable 2;, and g(.) ()) is function
3

---

Under review as a conference paper at ICLR 2020
that aggregates the information passed to a variable. The above formulation describes the case of pairwise message functions, though this can be generalized to higher-order interactions.
We formulate it continuous process which climinates the requirement of having predetermined number of steps of message passing. By further pushing the message passing process to update at infinitesimally smaller steps and continuing the updates for an arbitrarily large number of steps, the update associated with each variable can be represented using shared and reusable functions as the following ordinary differential equation (ODE) system.
1 (t)
(((ff) (,)),, s(1))
(()
(12 (1)b S(2))
(10)
(t)
9(t.fng (a, (t), a (t))li 8(n)))
where é; dx,/dt. Performing message passing to derive final states is equivalent to solving an initial value problem for an ODE system. Following the ODE formulation, the final states of variables can be computed as follows. This formulation can be solved with an ODE solver.
(t1) = (to) + f.". (ffrttets) E s(i) 5(0)) sE fl..... .n).
(11)
3.2 CONTINUOUS MESSAGE PASSING FOR DENSITY TRANSFORMATION
Continuous graph flow leverages the continuous message passing mechanism (described in Sec. 3.1 and formulates the message passing as implicit density transformations of the variables (illustrated Figure Given set variables with dependencies among them, the goal is to learn a model that captures the distribution from which the data were sampled. Assume the joint distribution p( p(X) at time to has a simple form such as independent Gaussian distribution for each variable x, (to). The continuous message passing process allows the transformation of the set of variables from X X(t) to X (11). Moreover, this process also converts the distributions over variables from simple base distributions to complex data distributions. Building the independent variable continuous time dynamics described in Eq. 5) we define the dynamics corresponding to related graph variables as:
Ö pl (((())) -Tr
OF
(12)
(
where represents set of reusable functions incorporating aggregated messages. Therefore, the joint distribution of set of variables X can be described
Tr OF Ox
log pl X (ti)) = log X(
(13)
Here we use two types of density transformations for message passing: (1) generic message trans- formations transformations with generic update functions where trace in Eq. [3 can be approxi- mated instead of computing by brute force method, and (2) multi-scal message transformations transformations with generic update functions at multiple scales of information.
Generic message transformations. The trace of Jacobian matrix in Eq. 13 modeled using generic neural network function. The likelihood is defined as:
log (((())) = log X (fo)) - Exte) [" aF
edt
(14)
(x
where F denotes neural network for message functions, and is noise vector and usually can be sampled from standard Gaussian or Rademacher distributions.
Multi-scale message transformations. As generalization of generic message transformations, we design model with multi-scale message passing to encode different levels of information in the variables. Similar to Dinh al. (2016), we construct multi-scale CGF model by stacking

---

Under review as a conference paper at ICLR 2020
several blocks wherein each flow block performs message passing based generic message trans- formations. After passing the imput through a block, we factor out a portion of the output and feed as input to the subsequent block. The likelihood is defined as:
logp( X (16) = log p( X (16-1)) - Pp(()) (")
OF
(o-1 8 (t-1)) Edtt
E
(15)
where b = 1,2....., architecture. Assume at time (to to t1). (t,) is factored out into two. We use one of these (denoted as X (tt)) as the imput to the (b + 1)th block. Let X (ts) be the input the next block, the density transformation is formulated as:
1) with B as the total number of blocks in the design of the multi-scale
pbai [ OF
log p( X (1++1)) log p( (1+)) Ppte)
edt.
(16)
(t,))))
4 EXPERIMENTS
To demonstrate the generality and effectiveness of our Continuous Graph Flow (CGF), evalu- ate our model on three diverse tasks: (1) graph generation, (2) image puzzle generation, and (3) layout generation based scene graphs. Graph generation requires the model learn complex distributions over the graph structure. Image puzzle generation requires the model to learn local and global correlations in the puzzle pieces. Layout generation has a diverse set of possible nodes and edges. These tasks have high complexity in the distributions of graph variables and diverse potential function types. Together these tasks pose a challenging evaluation for our proposed method.
4.1 GRAPH GENERATION
Datasets and Baselines. We evaluate our model on graph generation on two benchmark datasets EGO-SMALL and COMMUNITY-SIN SMALL YYou et al., 2018) against four strong state-of-the-an base- lines: VAE-based method (Simonovsky & Komodakis 2018). autoregressive graph generative model GraphRNN You et al. 2018) and DeepGMG (Li et al. 2018), and Graph normalizing flows
[Lu et al. 2019).
Evaluation. We conduct a quantitative evaluation of the generated graphs using Maximum Mean Discrepancy (MMD) measures proposed in GRAPURNN (You et al. 20188). The MMD evaluation in GRAPURNN was performed using a test set of N ground truth graphs, computing their distribution over the nodes, and then searching for set of N generated graphs from a larger set of samples generated from the model that best matches this distribution. As mentioned by Liu et al. (2019), this evaluation process would likely have high variance as the graphs are very small. Therefore, also performed an evaluation by generating 1024 graphs for each model and computing the MMD distance between this generated set of graphs and the ground truth test set. Baseline results are from Liu et al. (011).. Implementation details refer to Appendix A.
Results and Analysis. Table shows the results in terms of MMD. Our CGF outperforms the base- lines by a significant margin and also the concurrent work GNE, We believe our CGF outperforms GNF because employs free-flow functions forms unlike GNF that has some contraints necessitated by the coupling layers. Fig. 2 visualizes the graphs generated by our model. Our model can capture the characteristics of datasets and generate diverse graphs that are not seen during the training. For additional visualizations and comparisons, refer to the Appendix A.
4.2 IMAGE PUZZLE GENERATION
Task description. We design image puzzles for image datasets to test model's ability on fitting very complex node distributions in graphs. Given an image of size W x W, we design puzzle by dividing the original image into non-overlapping unique patches. A puzzle patch is of size w x w, in which represents the width of the puzzle. Each image is divided into
W/w puzzle patches both horizontally and vertically, and therefore we obtain P P= =px x p patches in total. Each patch cor- responds to node in the graph. To evaluate the performance of instead of training the model with all nodes, we sample ğ adjacent patches where P is uniformly .....,P) P) input the model during training and test. In our experiments, sampled from (1, (1,...,
model on dynamic graph sizes,
5

---

Under review as a conference paper at ICLR 2020
Table 1: Quantitative results on graph generation. MMD measures between test set and gener- ated graphs (lower is better). The second set shows GRAPHRNN evaluation with node distribution matching (averaged over different models with 3 trials). The third set shows evaluation for the test set for all 1024 generated graphs (averaged over 5 models). (max (VV max ((E)) )) is also shown.
Method
COMMUNITY-SMALL (20,83)
EGO-SMALL (18,69)
DEGREE
CLUSTERING
ORBIT
DEGREE
CLUSTERING
ORBIT
GRAPHIVÆE
0.35
0.98
0.54
0.13
0.17
0,05
DEEPGMG
0.22
0.95
0.4
0.04
0.10
0.02
GRAPHRNN
0.08
0.12
0.04
0.09
0.22
0.003
GNF + AE
0.20
0.20
0.11
0.03
0.10
0.001
CGF
0.10
0.30
0.08
0.02
0.11
0.001
GRAPHRNN (1024)
0.03
0.01
0.01
0.04
0.05
0.06
GNF + AE (1024)
0.12
0.15
0.02
0.01
0.03
0.0008
CGF (1024)
0.02
0.02
0.001
0.002
0.007
0,0002
COMMUNITY-SMALL
EGO-SMALL
Figure 2: Visualization of generated graphs from model. Our model capture the charac-
teristic of datasets and generate diverse graphs not appearing in the training set.
patch size 16, p E (2, 3.,3,4 and edge function for each direction (left, right, up, down) within a neighbourhood of node. Additional details are Appendix
Datasets and baselines. We design the image puzzle generation task for three datasets: MNIST (Le- Cun al. 1998). CIFARIO (Krizhevsky et al. 2009), and CelebA (Liu et al.) 2015).. CelebA dataset does not have validation set, thus, we split the original dataset into training set of 27. 7,0000 images Dhariwal 2018). We compare our model with six and test set of 3,000 images as (Kingma VAE VAE (He et al. 2018), (2) Graphite GGover al. state-of-the-arr VAE based models: (I) Structured
2019), (3) Variational message passing using structured inference networks (VMP-SIN) (Lin et al,
2018 (4) BiL.TTM VAE: bidirectional L.STM used to model the interaction between node latent
variables (obtained after serializing the graph) an autoregressive manner similar to Grregor et al.) (2015). (5) Variational graph autoencoder (GAE) (Kipf & Welling 20166), and (6) Neural relational
inference (NRI) (Kipf et al. 20188): we adapt this to model data for single time instance and model interactions between the nodes.
Results and analysis. We report the negative log likelihood (NLL) in bits/dimension (lower is better). The results in Tableeindica that CGF significantly outperforms the baselines. In addition to the quantitative results, we also conduct sampling based evaluation and perform two types of generation experiments: (1) Unconditional Generation: Given a puzzle size p. p² puzzle patches are generated using vector sampled from Gaussian distribution (refer Fig. aa);; and (2) Conditional Generation: Given Pi patches from an image puzzle having p² patches, we generate (p² P1) patches of the puzzle using model (see Fig. b)). We believe the task of conditional generation is easier than unconditional generation as there is more relevant information in the input during flow based transformations. For unconditional generation, samples from base distribution (e.g. Gaussian) are transformed into learnt data distribution using the CGF model. For conditional generation, map &. E X, where X, C X to the points in base distribution obtain 2,, and
remaining

---

Under review as a conference paper at ICLR 2020
<
<2
Unconditional Generation
Conditional Generation
Figure 3: Qualitative results for image puzzle generation. Samples generated using our model for 2x2 MNIST puzzles (above horizontal line) and 3x3 CelebA-HQ puzzles (below horizontal line) in (a) unconditional generation and (b) conditional generation settings. For setting (b), generated patches (highlighted in green boxes) are conditioned on the remaining patches (from ground truth).
subsequently concatenate the samples from Gaussian distribution to Fa to obtain z' that match the dimensions of desired graph and generate samples by transforming from z' x X using the trained graph flow.
Table 2: Quantitative results on image puzzle generation. Comparison of our CGF model with standard VAE and state-of-the-art VAE based models in bits/dimension (lower is better). These results are for unconditional generation using multi-scale version continuous graph flow.
Method
MNIST
CIFAR-10
CelebA-HQ
2x2
3x3
4x4
2x2
3x3
4x4
2x2
3x3
4x4
BiLSTM VAE
4.97
4.77
4.42
6.02
5.20
4.53
5.72
5.66
5,48
StructuredVAE HHe et al. 2018
4.89
4.65
3.82
6.03
5.02
4.70
5.66
5.43
5.27
Graphite Grover al. 2019)
4.90
4.64
4.02
6.06
5.09
4.61
5.71
5.50
5.32
VMP-SIN (Lin et al. 2018)
5.13
4.92
4.44
6.00
4.96
4.34
5.70
5.43
5.27
GAE Kipf & Welling 2016)
4.91
4.89
4.17
5.83
4.95
4.21
5.71
5.63
5.28
NRI Kipi et al. 2018
4.58
4.35
4.11
5.44
4.82
4.70
5.36
5.43
5.28
CGF
1.24
1.21
1.20
2.42
2.31
2.00
3.44
3.17
3.16
4.3 LAYOUT GENERATION FROM SCENE GRAPIIS
Task description and evaluation metries. Layout generation from scene graphs is crucial task in computer vision and bridges the gap between the symbolic graph-based scene description and the object layouts in the scene (Johsson et al. 2018 Zhao al. 2019 1yothi et al. (019). Scene graphs represent scenes as directed graphs, where nodes are objects and edges give relationships between objects. Object layouts described by the set of corresponding bounding box annotations (Johnson et al. 2018). Our model uses scene graph as inputs (nodes correspond to objects and edges represent relations). edge function is defined for each relationship type. The output contains set object bounding boxes described by {[2,, Vi, h, ) where 2; Yi are the top-left coordinates, and W/; h2 are the bounding box width and height respectively. We use negative log likelihood per node (lower is better) for evaluating models on scene layout generation.
Datasets and baselines. Two large-scale challenging datasets are used evaluate the proposed model: Visual Genome (Krishna et al. 2017) and COCO-Stufi (Caesar et al.) 2018) datasets. Vi- sual Genome contains 175 object and 45 relation types. The training, validation and test contain 62565, 5506 and 5088 images respectively. COCO-S Stuff dataset contains 24972 train, 1024 valida- tion, and 2048 test scene graphs. We use the same baselines as in Sec. 4.2
Results and analysis. We show quantitative results in Table 3 against several state-of-the-art base- lines. Our CGF model significantly outperforms these baselines in terms of negative log likelihood. Moreover, show some qualitative results in Fig. 4 Our model leam the correct relations defined in scene graphs for both conditional and unconditional generation, Furthermore, our model capable learn one-to-many mappings and generate diverse of layouts for the same scene graph.

---

Under review as a conference paper at ICLR 2020
minn
Feee
Condioonal layout
Sample
Sample
Semple
Sample
(a) Scona graph
(b) Layout samples
Figure 4: Visualization for layout generation Visual Genome. Our CGF model can generate diverse layouts for the same scene graph. Upper row: layout samples with unconditional generation. Lower row: Layout generation conditioned on known layout. Best viewed in color.
Table 3:
Quantitative results for layout generation for scene graph in negative log-likelihood.
These results are for unconditional generation using CGF with generic message transformations.
Method BiLSTM + VAE
Visual Genome
COCO-Stuff
1.20
-1.60
VeructuredVAB (He al. 2018
-1.05
-1.36
Graphite (rrver et al. 2019)
-1.17
-0.93
VMP-SIN (Lin et al. 2018
-0.61
-0.85
GAE Kipf & Welling
2016
-1.85
-1.92
NRI (Kipf ct al.
2018
-0.76
-0.91
CGF
-4.24
-6.21
4.4 ANALYSIS: GENERALIZATION TEST
To test the generalizability our model to variable graph sizes, we design three different evaluation settings and test image puzzle task: (1) odd even: training with graphs having odd graph sizes and lesting on graphs with even numbers of nodes, (2) less to more: training on graphs with smaller sizes and testing on graphs with larger sizes, and (3) more to less: training on graphs with larger sizes and testing on graphs with smaller. In the less to more setting, we test the model's ability use the functions learned from small graphs on more complicated ones, whereas the more less setting evaluates the model's 's ability to learn disentangled functions without explicitly seeing them during training. In our experiments, for the less to more setting, we use sizes less than G/2 for training and more than G/2 for testing where G is the size of the full graph. Similarly, for the less to more setting, we use sizes less than G/2 for training and more than G/2 2 for testing. Table reports the NLL for these settings. The NLL of these models are close to the performance the models trained on full dataset demonstrating that our model is able to generalize to unseen graph sizes.
Table 4: Generalization test in three different evaluation settings for image puzzle sizes 3x3 for three image datasets bits/dimension.
Settings
MNIST
CIFAR-10
CelebA-HQ
Odd to even
1.33
2.81
3.31
Less more
1.37
2.91
3.66
More to less
1.34
2.83
3.44
CONCLUSION
In this paper, we presented continuous graph flow, generative model that generalizes the neural message passing in graphs to continuous time. We formulated the model as an neural ordinary dif- ferential equation system with shared and reusable functions that operate over the graph structure. We conducted evaluation for diverse set of generation tasks across different domains: graph gener- ation, image puzzle generation, and layout generation for scene graph. Experimental results showed that continuous graph flow achieves significant performance improvement over various of state-of- the-art baselines. For future work, we will focus on generation tasks for large-scale graphs which is promising our model is reversible and memory-efficiae
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Edoardo M Airoldi, David M Blei, Stephen E Fienberg, and Eric P Xing. Mixed membership stochastic blockmodels. Journal of machine learning research (JMLR), 2008.
Réka Albert and Albert-László Barabási. Statistical mechanics of complex networks. Reviews of modern physics, 2002.
Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. Learning structured embed-
dings of knowledge bases. In AAAI Conference on Artificial Intelligencce, 2011.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differ- ential equations. In Advances in Neural Information Processing Systems (NeurIPS). 2018.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. Inter-
national Conference Learning Representations (ICLR), 2016.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzize and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances Neural Information Processing Systems (NIPS), 2015. 2015.
Paul Erdōs and Alfréd Rényi. On the evolution of random graphs. Publicationes Mathematicae (Debrecen), 1959.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George Dahl. Neu- ral message passing for quantum chemistry. In International Conference on Machine Learning (ICML), 2017.
Will Grathwohl, Ricky TQ Chen, Jesse Betterncurrt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. International Confer- ence on Learning Representations (ICLR), 2019.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. International Conference on Machine Learning (ICML), 2015.
Aditya Grover, Aaron Zweig, and Stefano Ermon. Graphite: Iterative generative modeling of graphs. International Conference on Machine Learning (ICML), 2019.
Jiawei He, Yu Gong, Joseph Marino, Greg Mori, and Andreas Lehrmann. Variational autoencoders with jointly optimized latent dependency structure. In International Conference on Learning Representations (ICLR), 2018
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), 2018.
Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochas- tic scene layout generation from a label set. IEEE International Conference on Computer Vision (ICCV), 2019.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR), 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions, In Advances of Neural Information Processing Systems (NeurIPS), 2018
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. International Conference on Machine Learning (ICML),, 2018.
Thomas N Kipf and Max Welling. Variational graph auto-encoders. Bayesian Deep Learning Work- shop, NIPS, 2016.

---

Under review as a conference paper at ICLR 2020
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. International Conference on Learning Representations (ICLR), 2017.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV), 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998.
Jure Leskovec, Deepayan Chakrabarti, Jon Kleinberg, Christos Faloutsos. and Zoubin Ghahramani. Kronecker graphs: An approach to modeling networks. Journal of Machine Learning Research (JMLR), 2010.
Ruiyu Li, Makarand Tapaswi, Renjie Liao, Jiaya Jia, Raquel Urtasun, and Sanja Fidler. Situation recognition with graph neural networks. IEEE International Conference on Computer Vision (ICCV),
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. In International Conference on Machine Learning (ICML), 2018.
Wu Lin, Nicolas Hubacher, and Mohammad Emtiyaz Khan. Variational message passing with struc- tured inference networks. In International Conference on Learning Representations (ICLR), 2018.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In AAAI conference artificial intelligence, 2015.
Jenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, and Kevin Swersky. Graph normalizing flows. In
Advances in Neural Information Processing Systems (NIPS), 2019.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In IEEE International Conference on Computer Vision (ICCV), 2015.
Kaushalya Madhawa, Katushiko Ishiguro, Kosuke Nakago, and Motoki Abe. Graphnvp: An invert- ible flow model for generating molecular graphs. arXiv preprint arXiv: 16000, 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning (ICML), 2015.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems (NIPS), 2017.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.
The graph neural network model. IEEE Transactions on Neural Networks, 2009.
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference Artificial Neural Networks (ICANN),
2018.
Jiaxuan You, Rex Ying, Xiang Ren, William L Hamilton, and Jure Leskovec. Graphrnn Generating realistic graphs with deep regresseee models. International Conference on Machine Learning (ICML). 2018
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. AAAI Conference on Artificial Intelligence, 2018
Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In JEEE Conference Computer Vision and Patern Recognition (CVPR), 2019.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
We provide supplementary materials to support the contents of the main paper. In this part, we describe implementation details of our model. We also provide additional qualitative results for the generation tasks: Graph generation, image puzzle generation and layout generation from scene graphs.
A.1 IMPLEMENTATION DETAILS
The ODE formulation for continuous graph flow (CGF) model was solved using ODE solver pro- vided by NeuralODE (Chen al. 2018). this section, we provide specific details of the con- figuration of our CGF model used in our experiments on two different generation tasks used for evaluation in the paper.
Graph Generation. For each graph, firstly generate its line graph with edges switched to nodes and nodes switched to edges. Then the graph generation problem now generating the current nodes values which represents the adjacency matrix in the original graph. Each node value is binary (0 or 1) and is dequantized to continuous values through variational dequantization, with global learnable Gaussian distribution as variational distribution. For architecture,
two blocks of continuous graph flow with two fully connected layers in Community-sman dataset, and one block of continuous graph flow with one fully connected layer in Citeseer-small dataset. The hidden dimensions are all 32.
Image puzzle generation. Each graph for this task comprise nodes corresponding the puzzle pieces. The pieces that share an edge in the puzzle grid are considered to be connected and an edge function is defined over those connections. our experiments. each node is transformed an embedding of size 64 using convolutional layer. The graph message passing is performed over these node embeddings. The image puzzle generation model designed using multi-scale continuous graph flow architecture. We use two levels of downscaling in our model each of which factors out the channel dimension of the random variable by 2. We have two blocks of continuous graph flow before each downscaling wth four convolutional message passing blocks in each of them. Each message passing block has a unary message passing function and binary passing functions based on the edge types all containing hidden dimensions of 64.
Layout generation for scene graphs. For scene graph layout generation, a graph comprises node corresponding to object bounding boxes described by {[x, y4, h4, > where 24, y represents the top-left coordinates, and w;, w;,h; represents the bounding box width and height respectively and defined based the relation types. In our experiments, the layout generation model uses two blocks of continuous graph flow units, with four linear graph message passing 64 hidden dimensions, and takes the
edge functions
blocks in each of them. The message passing function embedding of node label and edge label in unary message passing function and binary message passing function respectively. The embedding dimension is also 64 dimensions. For binary message passing function, we pass the messages both through the direction of edge and the reverse direction of edge increase the model capacity.
A.2 IMAGE PUZZLE GENERATION: ADDITIONAL QUALITATIVE RESULTS FOR CELEBA-HO -HQ
Fig. and Fig. 6 presents the image puzzles generated using unconditional generation and condi- tional generation respectively.
A.3 LAYOUT GENERATION FROM SCENE GRAPH: QUALITATIVE RESULTS
Fig. 7 and Fig. g show qualitative result on unconditional and conditional layout generation from scene graphs for COCO-stuff dataset respectively. Fig. g and Fig. 10 show qualitative result unconditional and conditional layout generation from scene graphs for Visual Genome dataset re- spectively. The generated results have diverse layouts corresponding single scene graph.

---

Under review as a conference paper at ICLR 2020
Figure 5: Qualitative results on CelebA-HQ for image puzzle generation. Samples generated using our model for 3x3 CelebA-HQ puzzles in unconditional generation setting. Best viewed in color.
Figure 6: Qualitative results on CelebA-HQ for image puzzle generation. Samples generated using our model for 3x3 CelebA-HQ puzzles in conditional generation setting. are highlighted in green. Best viewed in color.
Generated patches
inotorcycli
watl-otter
cettingortte
elothed ght
calingratie
zellig-other
light
below
below
right
mell-other
decchs
ceiling-one otther
flour-other
clothes
foor other
ahove
below
below
Roor-ot oeher
hor
light
flooroother
Baronal
persoci left of
floo other
plastic
insoe)
of
Plart rson
bo
bonehhi
beal
plastic
(pensoos)
banana
banarr
person
be vara
bananaa
banana
Tobr other
Boursother
banana
Tborather
night
will Laick
sky-other
Iree
ahowe
right
ahove
nie
person
@ky-oltber
nett gress
net
(bonne
Jaft
[an
person vi "ilree
persoyteebriikk
dog
brish
dog
Sample
Sample
Sample 3
Sample
(a) Scene Graph
(b) Layout samples
Figure 7: Examples of Unconditional generation of layouts from scene graphs for COCO-Stuff dataset. We sample 4 layouts. The generated results have different layouts, but sharing the same scene graph. Best viewed in color. Please zoom in to see the category of each object.
12

---

Under review as a conference paper at ICLR 2020
1eg
bu lifi
building other
bulkcing-other
Hodferm De Palaee
building
pol
Persen
pelitiinn
patfom
paverent
platromm
Motfeenn paveement
Atratte Slove
grraffe
Por wood
Wall Good
gf tafte
graffe
dirt
Scuncrete
We l-conccese
Grallwoodd
Buiding
co tediiing othir
de Sggrding Ai
bulidt fence
earii ncrete
person
tree
pance
aill concrote
Hatlel
povement
wall concrete
pevemiall concrete
lence
Sample
Sample 2
Samplo
Sample
(a) Scene Graph
(b) Conditional layout
(b) Layout samples
Figure 8: Conditional generation of layouts from scene graphs for COCO-stuff dataset. We sample 4 layouts. The generated results have different layouts except the conditional layout objects in (b), but sharing the same scene graph. Best viewed in color. Please zoom in to see the category of each object.
mnnn
Pair
meah
be
wall
bokktt
person
walkinng
Sitting
perzon
Sidewall
pinsin
light
light light
Willl
wall
Vmai
behind
nest
BNN
chalt
foor
Sall
bed
table
light
Goblee
chair
well
light
loht
wall
pillww
table
Light
Sample
Sample
Sample
Sample
(a) Scene Graph
(b) Layout samples
Figure 9: Unconditional generation of layouts from scene graphs for Visual Genome dataset. We sample 4 layouts for each scene graph. The generated results have different layouts, but sharing the same scene graph. Best viewed in color. Please zoom in to see the category of each object.
A.4 GRAPH GENERATION: ADDITIONAL QUALITATIVE RESULTS
Fig. II and Fig. 12 present the generated graphs for EGO-SMALL and COMMUNITY-SMALL respec- tively.
13

---

Under review as a conference paper at ICLR 2020
Soud
dboud
RAN
Soud
SCRSKE
SIS
TCP
grawn
grass
COWA
mill
Wil
Hike
Will
Iocand
[ears
board
char
Sample
Sample
Sample
Sample
Scene Graph
Conditional layout
Layout samples
Figure 10: Conditional generation of layouts from scene graphs for Visual Genome dataset. We sample 4 layouts for each scene graph. The generated results have different layouts except the conditional layout objects in (b), but sharing the same scene graph. Best viewed in color. Please zoom in to see the category of each object.
A.S ANALYSIS: NUMBER OF FUNCTION EVALUATION VS NUMBER OF NODES
We analyze the variation in number of function evaluations (NFE) required solve the ODE the number of nodes in the graph changes. Refer to Figure [3 The results interestingly show that the average number of function evaluation does not increase linearly with the increment in number of graph nodes, which would be the case if the variables were independent.
14

---

Under review as a conference paper at ICLR 2020
(a) CGF samples
(b) GRAPHRNN SAMPLES
Figure 11: Graph generation on EGO-SMALL. Samples generated using (a) our CGF and (b)
GRAPHRNN.
15

---

Under review as a conference paper at ICLR 2020
(a) CGF samples
(b) GRAPHIRNN SAMPLES
Figure 12: Graph generation COMMUNITY-SMALL. Samples generated using (a) our CGF and
(b) GRAPHRNN.
16

---

Under review as a conference paper at ICLR 2020
NFE vs. Graph size
40
30
NFE
10
10
21
36
55
78
153
171
Graph size
(a)EGO-SMALL
NFE vs. Graph size
30
20
NFE
50
75
100
125
150
175
Graph size
(bJCOMMUNTY-SMALL
Figure 13: Visualization illustrating the effect on number of nodes in the graph on the number of function evaluations (NFE) required by the ODE solver for (4)EGO-SMALE and (b)COMMUNITY- SMALL datasets used in our graph generation model.
17


Reviews:
[{"review_id": "SkgZi5y0Yr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method to do Continuous Normalizing Flow for graphs by defining the density evolution via ODEs over shared variables. Continous message passing scheme introduced can leverage the free-form model architectures which allow flexibility in handling variable data dimensions. The results of three different generation tasks indicate improved performance over the baseline methods. \n\nNovelty: The basic methodological contribution of the paper seems somewhat limited. The main idea of the paper follows directly from NeuralODE (Chen et al., 2018) and FFJord (Grathwohl et al., 2019). Nevertheless, I do agree that adapting the NeuralODEs and continuous flows for unstructured data is challenging and deserves its own analysis. However, I find the experiment section to be not very clear (See specific comments below).  \n\nRecommendation: Weak Reject\n\nConcerns: \n\n- The experiment on Graph Generation is not clear to me. What is the motivation of using the dual of the graph? What is the input to the full-connected layers? Is the input dimension N^2 (the number of edges)? Please clarify. It would be helpful if the implementation details in Appendix A.1 can be formalized in the main paper since this is the main contribution of the paper. Overall the experiment section in the main paper lacks sufficient details to comment on the efficacy of the method.\n       \n- NeuralODEs involve iterative numerical solvers which can be very slow. It seems that the proposed method involves simultaneously solving ODEs that linearly scale with the number of nodes (Equation 11)? I think there should be some analysis of how the proposed method scales with the graph size. I am not sure if NeuralODE would perform well when the number of nodes is large.\n\n- Compared to other baseline methods (see You et al. 2018), the datasets used for analysis are smaller versions of the original datasets. There are other datasets that can be used for experiments on graph generation:  See Grid, Protein ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byxc74-0Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new variant of invertible flow-based model for graph structured data. \nSpecifically, the authors proposed a continuous normalizing flow model for graph generation, first in the graph generation literature. \nThe authors claim that the free-form model architecture of neural ODE formulation of continuous flow is advantageous against standard discrete flow models. Experimental results show that the proposed model is superior to recent models in image puzzling and layout generation datasets. \n\nOverall, manuscript is well organized. \nThe combination of continuous flows and graph structured data is new in the literature (as far as I know). Proposed formulation seems natural and reasonable. I find no fatal flows in the formulation. In experiments, the proposed model achieved good scores against recent GNN works. \n\nConcerning the existing invertible flow-based models for graph structured data, Madhawa’s work is one of the first attempts in the literature. I think the paper below should be refereed appropriately:  \nMadhawa+, “Graph NVP”, arXiv: 1905.11600, 2019. \n\nThe way of incorporating relational structure into flows are very similar to the GraphNVP and Graph Normalizing Flow: using neighboring nodes’ hidden vectors as parameters (or input to parameter inference networks). \nIn addition, I found no special tricks or theoretical considerations to achieve the continuous flow for graphs. Based on these points, I think technical contribution of this paper is somewhat limited. \n\nI cannot find information about the specific chosen forms of f-hat and g in Eq.(10) within the manuscript. Are the choices of f-hat and g are crucial for performance? It is preferable if the authors can present any experimental validations concerning this issue. \n\nMy main concerns are in the experimental section. \n\nI’m not fully convinced in the necessity of the continuous normalizing flows for the experimental tasks. None of the experimental tasks have `````'' intrinsic continuous time dynamics over graph-structured data (Sec. 2)''. Then, what is the rationale to adopt Continuous graph flow for these tasks? \n\nOne reason to adopt continuous flows is that the ODE formulation allows users to choose free-form model architecture, yielding more complicated mappings to capture delicate variable distributions. \nI expect some assessments are made concerning this issue. My suggestion is (i) to test the discretized model of the proposed Continuous Graph Flow and see how the discretization deteriorate the performance, and (ii) to test several choices of f-hat and g (Eq.10) to show the necessity of ODE formulation, accommodating free-form model architecture.  \n\nIn the current manuscript, Graph Normalizing Flow (GNF) is the closest competitor. However, GNF is not tested in the puzzle and the scene graph experiments. Why is that?\n\nI’d like to hear opinions of the authors concerning these issues, and hope some discussions are included in the manuscript. \n\n\nSummary\n+ continuous normalizing flow is first applied to graph structure data\n+ manuscript is well organized\n+- natural and reasonable formulation. But at the same time, technological advancement is limited. \n- Less convinced to adopt continuous flow for graph-structured data without no intrinsic continuous dynamics. \n- Necessity or advantages of ``continuous’’ flows are not well assessed in the experiments. Please consider some additional assessments suggested in the review comment. \n- GNF, the closest competitor, is omitted in the 2d and the 3rd experiments. No explanations about this. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BkgReQ9aKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes continuous graph flow (CGF), a flow-based generative model for graphs.  Based on the idea to build continuous-time flows with neural ODEs, the node features are transformed into Gaussian random variables via message passing. The log-likelihood can be approximated stochastically.\n\nI find it hard to assess the novelty of this work because 1) the algorithm looks like a trivial application of continuous-time normalizing flow to graph data using message passing algorithm, and 2) the concurrent work graph normalizing flow (GNF, Liu et al., 2019).  I failed to find any algorithmic innovation more than a mere application of continuous-time flow to graph data. The very idea of a flow-based model for graphs using the message passing algorithm may be considered as a contribution, but this is also blurred because of the concurrent work GNF. \n\nThe experiments look interesting, but there are some minor concerns. For the unsupervised graph generation task, CGF is shown to perform better than GNF, but I think the comparison here may not be fair because the results for GNF seem to be obtained from the paper uses a different way to construct initial node features to be fed into message passing. Specifically, according to the GNF paper, they inject an i.i.d. Gaussian noise as initial node features, but this work uses different schemes based on the dual graph (btw I think the term dual graph may be misleading. There already exists a common term called dual-graph with different definition). So I think to compare the expressive power of flows, it would be fair to start from a common scheme to build initial node features. GNF should also be compared to CGF for the other experiments. Seems like the baselines presented in Table 2 and Table 3 are quite dumb baselines not well suited for the tasks considered here. \n\nIn page 2, the authors stated that \"GNF involves partitioning of data dimensions into two halves and employs coupling layers to couple them back. This leads to several constraints on function forms and model architectures...\". I think GNF can be improved using more advanced transformation other than affine coupling, such as recently proposed mixture logistic CDF [1] or neural spline flow [2].  Continuous-time flow may still have an advantage in memory usage, but at least in terms of expressive power, I think it is not clear whether CGF is particularly better.\n\n[1] Ho et al., Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design, 2019\n[2] Durkan et al., Neural Spline Flows, 2019", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Novelty of the proposed model is low. Experimental results are weak.
Decision: Reject