Title: GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders
Authors: Martin Simonovsky
Nikos Komodakis
Venue: ICLR 2018
Keywords: graph
generative model
autoencoder

Abstract: Deep learning on graphs has become a popular research topic with many applications. However, past work has concentrated on learning graph embedding tasks only, which is in contrast with advances in generative models for images and text. Is it possible to transfer this progress to the domain of graphs? We propose to sidestep hurdles associated with linearization of such discrete structures by having a decoder output a probabilistic fully-connected graph of a predefined maximum size directly at once. Our method is formulated as a variational autoencoder. We evaluate on the challenging task of conditional molecule generation. 

TL;DR: We demonstate an autoencoder for graphs.

Review discussion:
This work proposed an interesting graph generator using a variational autoencoder. The work should be interesting to researchers in the various areas.
However, the work can only work on small graphs. The search space of small graph generation is usually very small, is there any other traditional methods can work on this problem?
Moreover, the notations are a little confusing.
Thank you for your review.
Regarding traditional methods besides stochastic blockmodels [Snijders & Nowicki, 1997], we should have also mentioned the research on random graph models, such as [Erdös & Rényi, 1960] or [Barabási & Albert, 1999]. These models make fairly strong assumptions and cannot be used to model e.g. chemical compounds, though. You can consult e.g. "A Survey of Statistical Network Models" [Goldenberg et al, 2009] for a detailed review.
Regarding the little confusing notation, we have updated the paper in several places today; could you please provide more details so that we can further improve the manuscript?
The authors propose a variational auto encoder architecture to generate graphs.
Pros:
- the formulation of the problem as the modeling of a probabilistic graph is of interest
- some of the main issues with graph generation are acknowledged (e.g. the problem of invariance to node permutation) and a solution is proposed (the binary assignment  matrix)
- notions for measuring the quality of the output graphs are of interest: here the authors propose some ways to use domain knowledge to check simple properties of molecular graphs
Cons:
- the work is quite preliminary
- many crucial elements  in graph generation are not dealt with:
a) the adjacency matrix and the label tensors are not independent of each other, the notion of a graph is in itself a way to represent the 'relational links' between the various components
b) the boundaries between a feasible and an infeasible graph are sharp: one edge or one label can be sufficient for acting the transition independently of the graph size, this makes it a difficult task for a continuous model. The authors acknowledge this but do not offer ways to tackle the issue
c) conditioning on the label histogram should make the problem easy: one is giving away the number of nodes and the label identities after all;
however even in this setup the approach fails more often than not
d) the graph matching procedure proposed is a rough patch for a much deeper problem
- the evaluation should include a measure of the capacity of the architecture to :
a) reconstruct perfectly the input
b) denoise perturbations over node labels and additional/missing  edges
Thank you for your review. We address your critique in the following.
# a) adjacency matrix and the label tensors are not independent of each other
Our decoder uses a single stream of feature channels until its last layer, which should make the three predicted tensors rather dependent.
In fact, we tried to go a step further and derive the adjacency matrix from feature tensors by introducing a virtual "not-present" edge and node class.
However, this did not improve performance, likely due to a the fact that this required whole feature tensors to be correct, whereas our presented loss ignores unmatched parts of these tensors.
# c) conditioning on the label histogram should make the problem easy: one is giving away the number of nodes and the label identities after all;
however even in this setup the approach fails more often than not
Thank you for making this hypothesis. We performed an additional experiment by training in unconditioned setting on QM9 (see updated Table 1).
Indeed, conditional training is able to reach a lower loss, though this difference diminishes with increasing size of the embedding (likely due to the autoencoder having more freedom to capture such statistics by itself). The number of valid samples fluctuates over configurations and is roughly the same for both conditional and unconditional setting.
We managed to improve our results on QM9 (so that can it succeed slightly more often than not), compared them to previous work, and found that the ratio of valid samples can be similar to a grammar-based decoder [Kusner et al, 2017] on QM9 while offering much higher variance; see Tables 1 and 3 in the updated paper.
Unlike Kusner et al, we could achieve this without manual specification of a grammar or other rules, besides the help from maximum spanning tree.
# A measure of the capacity of the architecture to reconstruct perfectly the input
This is a very good point. To this end, we removed the regularization and trained our architecture as a standard autoencoder, where the only goal is to aim for perfect reconstruction. Unfortunately, it turned out the architecture is not powerful enough to perfectly reconstruct the input, unless the set of possible inputs is rather small (e.g. for a fixed set of 1000 training examples). We added this information to Appendix C. In this light, we did not pursue the scenario of denoising autoencoder, which you also suggested.
This paper studies the problem of learning to generate graphs using deep learning methods. The main challenges of generating graphs as opposed to text or images are said to be the following:
(a) Graphs are discrete structures, and incrementally constructing them would lead to non-differentiability (I don't agree with this; see below)
(b) It's not clear how to linearize the construction of graphs due to their symmetries. Based on this motivation, the paper decides to generate a graph in "one shot", directly  outputting node and edge existence probabilities, and node attribute vectors.
A graph is represented by a soft adjacency matrix A (entries are probability of existence of an edge), an edge attribute tensor E (entries are probability of each edge being one of d_e discrete types), and a node attribute matrix F, which has a node vector for each  potential node. A cross entropy loss is developed to measure the loss between generated A, E, and F and corresponding targets.
The main issue with training models in this formulation is the alignment of the generated graph to the ground truth graph. To handle this, the paper proposes to use a simple graph  matching algorithm (Max Pooling Matching) to align nodes and edges. A downside to the algorithm is that it has complexity O(k^4) for graphs with k nodes, but the authors argue that this is not a problem when generating small graphs. Once the best correspondence is found, it is treated as constant and gradients are propagated appropriately.
Experimentally, generative models of chemical graphs are trained on two datasets. Qualitative results and ELBO values are reported as the dimensionality of the embeddings is varied. No baseline results are presented.
A further small set of experiments evaluates the quality of the matching algorithm on a synthetic setup.
Strengths:
- Generating graphs is an interesting problem, and the proposed approach seems like an easy-to-implement, mostly reasonable way of approaching the problem.
- The exposition is clear (although a bit more detail on MPM matching would be appreciated)
However, there are some significant weaknesses.
First, the motivation for one-shot graph construction is not very strong:
- I don't understand why the non-differentiability argued in (a) above is an issue. If training uses a maximum likelihood objective, then we should be able to decompose the generation of a graph into a sequence of decisions and maximize the sum of the logprobs of the conditionals. People do this all the time with sequence data and non-differentiability is not an issue.
- I also don't agree that the one shot graph construction sidesteps the issue of how to linearize the construction of a graph. Even after doing so, the authors need to solve a matching problem to resolve the alignment issue. I see this as equivalent to choosing an order in which to linearize the order of nodes and edges in the graph.
Second, the experiments are quite weak. No baselines are presented to back up the claims motivating the formulation. I don't know how to interpret whether the results are good or bad. I would have at least liked to see a comparison to a method that generated SMILES format in an autoregressive manner (similar to previous work on chemical graph generation), and would ideally have liked to see an attempt at solving the alignment problem within an autoregressive formulation (e.g., by greedily constructing the alignment as the graph was generated). If one is willing to spend O(k^4) computation to solve the alignment problem, then there seem like many possibilities that could be easily applied to the autoregressive formulation. The authors might also be interested in a concurrent ICLR submission that approaches the problem from an autoregressive angle (https://openreview.net/pdf?id=Hy1d-ebAb).
Finally, I would have expected to see a discussion and comparison to "Learning Graphical State Transitions" (Johnson, 2017). Please also don't make statements like "To the best of our knowledge, we are the first to address graph generation using deep learning." This is very clearly not true. Even disregarding Johnson (2017), which the authors claim to be unaware of, I would consider approaches that generate SMILES format (like Gomez-Bombarelli et al) to be doing graph generation using deep learning.
Overall, the paper is about an interesting subject, but in my opinion the execution isn't strong enough to warrant publication at this point.
Thank you for your review. We address your critique in the following.
# Ordering vs Alignment
The choice between linearization and matching is certainly an interesting topic, these are indeed two sides of the same coin. The graph canonization problem (i.e. consistent node ordering) is at least as computationally hard as the graph isomorphism problem (i.e. matching), which is NP-hard for general graphs. Fortunately, there are practical algorithms available for both problems, such as Nauty [McKay & Piperno, 2014] for canonization and max-pooling matching [Cho et al, 2014] for approximate isomorphism, used in our paper.
Thus, both ways are feasible for small graphs, though not for free.
We decided for one-shot construction with matching to allow the decoder to find its own orderings, motivated by the empirical result of [Vinyals et al, ICLR'16] that the linearization order matters when learning on sets. It is a priori unclear that enforcing a specific canonical ordering of vertices with a strategy for incremental construction (e.g. adding vertices one by one and connecting them to existing nodes) would lead to the best results. In this sense, we indeed sidestep the issue of how to linearize the construction by postponing the correspondence problem to the loss for the final result. We do not avoid the computational penalty of alignment. Note that our matching approach can be seen as inexact search of output permutation in Equation 9 in [Vinyals et al, ICLR'16].
One could also consider incremental (likely autoregressive, as you suggested) construction with matching.
However, [Johnson, ICLR'17] noted in his construction of probabilistic graphs that a loss function for only the final result was insufficient and deep supervision over individual construction steps was necessary for good performance. Your idea of "greedily constructing the alignment as the graph was generated" certainly sounds quite promising in this context, thank you for it. It might nicely combine the idea of the concurrent submission (https://openreview.net/pdf?id=Hy1d-ebAb) and our paper.
Though we would consider it as a direction for future work at this momement, as it would lead to extensive modification of our current submission.
# Non-differentiability
We agree that non-differentiability is not a major obstacle if the generation of a graph is linearized, i.e. decomposed into a sequence of decisions in the ground truth. This may be given by the nature of some tasks, such as those addressed by [Johnson, ICLR'17], where graphs are built according to a sequence of statements. In general, however, the choice of such a decomposition is not clear, as we argue above. In this regard, it is interesting to learn from the mentioned concurrent submission (https://openreview.net/pdf?id=Hy1d-ebAb) that random orderings seem to work well.
Nevertheless, even ML training with teacher forcing is not the perfect solution due to exposure bias (possibly poor prediction performance if the RNN's conditioning context diverges from sequences seen during training, i.e. the inability to fix its mistakes) [Bengio et al, 2015].
# Baselines
We agree that the omission of baselines was clearly a weak point. In the updated paper, we compare with character-based decoder of [Gomez-Bombarelli et al, 2016] and grammar-based decoder of [Kusner et al, 2017]. We found that the ratio of valid samples can be similar to a grammar-based decoder [Kusner et al, 2017] on QM9 while offering much higher variance; see Tables 1 and 3 in the updated paper.
Unlike Kusner et al, we could achieve this without manual specification of a grammar or other rules, besides the help from maximum spanning tree.
# Other points
Thank you for the reference to [Johnson, ICLR'17], we have updated the paper in this regard and toned down our statement on being the first, in this light. We also included a short appendix on MPM matching.
Thanks for the response, adding baselines, and a better treatment of related work. I've raised my score by a point.
Thank you!
- We added comparison to two baselines ([Kusner et al, 2017] and [Gomez-Bombarelli et al, 2016]) on QM9 and ZINC, results of unconditioned models on QM9 (Table 1), and results with unregularized training (Appendix C).
- We introduce a model variant with a higher percentage of valid samples by making node probabilities a function of edge probabilities (Appendix B).
- We added a brief summary of max-pooling matching (Appendix A).
- We made multiple minor edits over the paper to enhance clarity, mention further observations, and refer to more related work such as [Johnson, 2017], [Vinyals et al, 2016], and [Stewart et al, 2016].

Meta-review:
role: PC
data: comment: The authors present GraphVAE, a method for fitting a generative deep model, a variational autoencoder, to small graphs.  Fitting deep learning models to graphs remains challenging (although there is relevant literature as brought up by the reviewers and anonymous comments) and this paper is a strong start.

In weighing the various reviews, AnonReviewer3 is weighed more highly than AnonReviewer1 and AnonReviewer2 since that review is far more thorough and the reviewer is more expert on this subject.  Unfortunately, the review from AnonReviewer1 is extremely short and of very low confidence.  As such, this paper sits just below the borderline for acceptance.  In general, the main criticisms of the paper are that some claims are too strong (e.g. non-differentiability of discrete structures), treatment of related work (missing references, etc.) and weak experiments and baselines.  The consensus among the reviews (even AnonReviewer2) is that the paper is preliminary.  The paper is close, however, and addressing these concerns will make the paper much stronger.

Pros:
- Proposes a method to build a generative deep model of graphs
- Addresses a timely and interesting topic in deep learning
- Exposition is clear

Cons:
- Treatment of related literature should be improved
- Experiments and baselines are somewhat weak
- "Preliminary"
- Only works on rather small graphs (i.e. O(k^4) for graphs with k nodes)

Decision: Reject