Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this paper, a GraphNVP framework for molecular graph generation is proposed. The main difference from the previously proposed models is the use of the invertible normalizing flow idea for the generative model, which doesn’t require a separate decoder for sampling. This architecture is implemented with coupling layers combined with a multi-layer perceptron. The model is evaluated and compared on QM9 and ZINC chemical molecular datasets.

This method combines a number of existing techniques to obtain a new model for the molecular graph generation problem. The paper is very well written.

I have several concerns with regards to this model and the proposed algorithm:
1. How many parameters does GraphNVP model have? The coupling layers should have at least O(LN^2R) and that must be multiplied by the number of MLP parameters of the adjacency tensor which I suppose is of order O(N^2R), is this correct? This number must be huge. How can one ensure that such a model does not overfit? Moreover, 100% reconstruction accuracy is rather an indicator that it actually does overfit, isn’t it?
2. I’m concerned whether the use of dequantization for this particular application is valid. Indeed, images are usually modeled as vectors taking values in [0, 255] and adding a uniform on [0,1] variable actually corresponds to noise. However, the graph adjacency tensor takes either 0 or 1 values and adding a similar uniform variable (potentially scaled by 0.9) is actually more than simply adding noise. Say one value is 0 and added 0.8, while the other is 1 and added 0.1; the transformed variables are now much closer to each other. Therefore, the likelihood of the transformed variables is going to be significantly different from the original one. Although one can indeed recover the original graph from the dequantized one, I doubt that there is a correspondence between two likelihoods extrema. This also contradicts one of the motivations to this paper that this approach uses precise log-likelihood. Could you please comment on this?
3. I am confused by this sentence: “Our objective is maximizing the log likelihood (Eq. 1) of z over minibatches of training data.” Does this mean that log(p_z(z)) is the objective? If yes, how does this relate to maximum likelihood?
4. Given the high cost of wet-lab experiments, the runtime of a method for the drug discovery application is much less important than the quality of the obtained results. Is it actually more important to have a sampling time decrease from 100/400s to 4s or does the higher quality of results should matter more? If the latter, are there any other advantages of GraphNVP over other models?
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper presents a new reversible flow-based graph generative model wherein the whole graph i.e., representative attributes such as node features and adjacency tensor is modeled using seperate streams of invertible flow model. This allows training of generative model using exact likelihood maximization over the underlying graph dataset.The model avoids encoding any domain specific heuristics and thus can be applied to any structured graph data. The paper focusses it applicability for molecular graphs. Given that this approach avoids sequential generation of graph, it is faster by an order of magnitude than prior models for molecular generation. Empirical experiments on couple of molecular graph data suggets that GraphNVP approach performs as well as prior approach but albeit without any rule checker.

My major concern with such invertible models is "scalability". Given that flow-based model are required to retain the original dimension its applicability is limited to low dimensional feature vectors. In the case of GraphNVP, this means limited number of node labels as well as edge labels. Additionally, since it limits adjacency tensor, this would lead to modeling graphs with few nodes. However, if integrated with encoder-decder model some of these limitation can be overcome. Given this major weakness and with limited novelty (i.e., extending to adjacency tensor), I am inclined to reject this paper. I shall improve my rating if GraphNVP is applied to general graph structures - synthetic / real.

Few more limitations:
1. Although paper claims one-shot generation of graphs, in reality it seems otherwise. Since every layer processes only on single node, overall it operates sequentially from one node to another.
2. Moreover, this same sequential processing yet again limits it applicability to small graphs. 
3. As in MolGAN, the direct generation of adjacency tensor leads to training with fixed size graphs i.e., through the addition of virtual nodes.  It is not possible to train model with variable number of nodes. 
4. As pointed by authors, their model is not node permutation invariant. 

Clarification:
1. Are the function 's' and 't' fixed across time ? For QM9 with max of 9 atoms and 27 layers, each atom attribute is processed multiple times. Are they processed using same functionality of s and t ?
2. Is it possible to model permutation invariance by augmenting the training data using multiple permutation of nodes such as BFS, DFS, degree, k-node (see GRAN) ?
3. I understand GraphNVP can reconstruct perfectly. But I fail to note the actual significance of such metric. If it reconstruct 100% or not how does it matter ? What matters is unniqueness, validity and novelty.
4. Can you please compare inference time ?
5. How difficult is it to integrate validity checker with your generation process ? Can we have some comparison using it ?

Minor:
1. In eq (2) please use different notation for layer 'l' and node 'l'.
2. Page 4, penultimate line: So as functions s and t -? To model functions s and t
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Contributions:
1. This paper proposes an invertible flow-based method for the one-shot graph generation.
2. The paper demonstrates their method on a molecular graph generation task.
3. Empirical results show the effectiveness of the proposed method.

The merit of the proposed invertible flow method is two folds. First, it can guarantee a one hundred percent reconstruction accuracy. Second, it can be adapted to generate graphs with various types (such as molecules) without incorporating much domain knowledge. Below are my concerns regarding this paper.

[Page 7, Table 2] My first concern is: does the reconstruction performance matters in the graph generation case? Typically a lower reconstruction error does not mean a worse model to generate reasonable new graphs. So the reconstruction error should accompany with other criterions. In Table 2, I can see CD-VAE and JT-VAE does better in generating valid, novel and unique graphs. So I wonder whether it worth sacrificing novelty to pursue a perfect reconstruction.

[Page 7, Sec 4.2] The authors mention they cannot reproduce the decoder of CG-VAE and JT-VAE. So I expect they mention somewhere in this paper that they will release their code once published.

[Page 5, Sec 3.3.1] The authors should be explicit by saying we replace sliced matrices (z_X[l^-,:,] in Eq. 2 and z_A[l^-,:,:] in Eq. 4) with masked matrices rather than just saying "Eqs. (2,4) are implemented with masking patterns".

[Page 5, Sec 3.3.1] Can you explain the gain of masking? To my understanding, even with masking you still need a sequence of N coupling layers to update each node once.

[Page 5, Sec 3.3.1] The second paragraph in Sec 3.3.1 is confusing to me. The masking scheme indeed makes the whole process, not permutation invariant. But I'm confusing about the way you fix it. Can you explain your "permutation invariant coupling"? E.g., why you need to change the indexing on the non-node axis?

Overall, I think the method proposed in this paper sacrifices some more import aspects in graph generation such as novelty and uniqueness by introducing an invertible flow architecture. And some parts in the paper may require a significant re-writing, such as Sec 3.3.1. 
review_title: Official Blind Review #1

Meta-review: The authors propose an invertible flow-based model for molecular graph generation. The reviewers like the idea but have several concerns: in particular, overfitting in the model, need for more experiments and missing related work. It is important for authors to address them in a future submission
Decision: Reject