Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors of this paper propose to use a continuous-time normalizing flow and E(3)-Equivariant GNN as the differential function in the neural PDE for molecular generation. Experimental results on QM9 and ZINC show that the proposed model, ModFlow, obtained better validity and novelty in comparison with existing graph flow based models. 
strengths_and_weaknesses: Strengths:
1. The consideration of using continuous-time normalizing flow as encoder makes sense. The authors find the their proposed model is equivalent to a variant of temporal graph network (TGN). 

Weakness: 
1. The evaluation metrics used in this paper are validity, uniqueness, novelty, and reconstruction, which are usually used for sanity check. To further look into the consistency of between the generated samples and the training samples, more informative metrics should be used. For example, the metrics in MOSES can be considered. 
2. It seems that an intact test set was not used for result evaluation. 
3. Furthermore, the computational comparison should not be limited to flow-based methods. Other state-of-the-art deep learning based molecular generation approaches should be compared as well. 
4. Minors: (1) Moroever -> Moreover. (2) Check the capitalizations in each reference item. 
questions: The authors should (1) use more informative metrics, (2) dissect a test set for each dataset, and (3) compare with non-flow-based state of the art molecular generation models.  
limitations: Only technical limitations are mentioned as future work in one sentence in Section 5 Conclusion.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The presented method combines continuous E(3)-equivariant flows and PDEs on graphs to generate molecular structures.
strengths_and_weaknesses: # Strengths

- Figure 2 helped me get a high-level understanding of the model
- Experiment 4.1 show-cased the capabilities of the model with a simple example
- Sections 4.2 and 4.3 are easy to follow and fairly clear.
- The results seem impressive: the model's scores on "validity", "uniqueness", "novelty", and "reconstruction" seems impressive.

# Weaknesses

- The paper is not exceptionally well-written. I had a hard time following Sections 3.2 and 3.3 -- the main contributions of this paper.
- The reader is expected to be familiar with the EGNN model. If the reader doesn't happen to know the exact inputs and mode of operation of this model, the reader will get lost. The importance of this borrowed architecture in this approach warrants a short introduction.
questions: - Eq. (1) seems to encapsulate a central assumption of this probabilistic model. Why should it be that, given edge information, the nodes can be labeled independently?
- Sometimes, I am unsure whether you are working with 2D or 3D graphs. On the one hand, you use concepts such as bond orders and chemical validity; on the other, you are working with 3D structures. Do you convert generated 3D structures back to 2D graphs?
- Tree representation:
  - Is it always possible to decompose a molecular graph into a tree-like structure?
  - I understand that the model generates 3D molecular structures. How do you handle substructures, i.e., groups of atoms consisting of more than one atom?
- How is the neighborhood of an atom $\mathcal{N}_i$ defined? Do you recompute that throughout the generation process?
- Section 3.3: are you confusing invariant and equivariant in this section?
limitations: The authors adequately addressed the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new generative model for molecules based on normalizing flows. The network is based on a set of ODEs (one per node), that are coupled together to form a PDE, which lets the model graph densities accurately. The model is also E(3) equivariant, which is an important physical property for molecules.

The paper presents experiments on QM9 and Zinc250K datasets and evaluate results on validity, uniqueness, novelty, and reconstruction metrics. The ModFlow method performs well on these metrics.
strengths_and_weaknesses: Strengths
* As the ModFlow method is based on combining normalizing flows with neural ODEs, it can generate graphs in one-shot, and also provide density estimates (unlike GANs).
* The model is E(3) equivariant, which is a desirable property for molecules and other atomic systems.

Weaknesses
* It is not entirely clear how useful the metrics used in the paper are. As such they are weak metrics -- for example, validity simply measures the fraction of molecules that do not violate chemical valency rule. Most chemical applications require generating based on more complex properties, such as generating non-toxic drug molecules, generating synthesizable materials, etc. Therefore, it is unclear how useful this method is in practice.
questions: 1. Please include a discussion of the runtime of your model for training and inference.
2. How does this scale with the number of atoms? Can it effectively generate larger systems like materials or bio molecules?
limitations: Please discuss limitations in terms of computational efficiency, classes of atomic systems that can be generated by this method etc.
For example, this method cannot directly be used for materials where periodic boundary conditions need to be respected.

Also, potential negative societal impact (such as generating toxic materials, etc.) should be mentioned.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work introduces ModFlow, a modular continuous normalizing flow model parametrized by an E(3) equivariant GNN. To demonstrate the efficacy of ModFlow, the model is trained on QM9 and ZINC250K to the end of generating valid, unique, and novel samples. 
strengths_and_weaknesses: Strengths:
1. The writing is clear for the most part, and the model is well-presented. 
2. The validity, uniqueness, and novelty results for QM9 and ZINC250K look great. The generated samples also seem reasonable.

Weaknesses:
My main concerns with this work are related to its numerical experiments.
1. More metrics should be included for a comprehensive assessment of ModFlow's generation quality. This is because validity, uniqueness, and novelty scores may not correlate with how realistic the samples are. To this end, one may use the Frechet ChemNet Distance (FCD), which measures the similarity between the set of generated molecules and the set of training molecules. One can easily compute the FCD as well as a variety of other metrics through MOSES: https://github.com/molecularsets/moses, and the inclusion of these additional metrics will make the paper much more convincing.
2. More experiments are needed to fully demonstrate the effectiveness of ModFlow. For example, property optimization is conducted in e.g. JTVAE and GraphAF to find the best generated molecules with certain chemical scores, e.g. penalized LogP, QED. Such optimization showcases that the model is able to learn a meaningful latent space useful for downstream tasks. Other possibilities are latent space interpolation and property prediction.

Edit: in light of the added experimental results during the rebuttal, I have now raised my score to 6 as I believe the paper makes a solid contribution in cheminformatics.
questions: 1. Line 156, 157: should these be rotational equivariant and permutation equivariant instead of invariant?
2. Line 180-183: what does it mean for the forward and backward flows to be completely aligned?
3. To the best of my knowledge, the ZINC250k dataset only contains 2D atom coordinates. Are the generated 3D structures shown in Figure 5 ZINC250K computed via additional libraries, or are they generated by ModFlow? Similarly, are the QM9 conformers generated by ModFlow, or are they computed based on the molecular graph?
4. How does ModFlow compare to other models in terms of training/inference time? How does the training/inference time of ModFlow scale with molecule size (QM9 \to ZINC250K)?
5. I understand that it is appealing to have a model that captures chemical composition laws from data rather than relying on e.g. valency checks from external libraries. However, I am wondering if there is an example application in which a model's intrinsic generative validity matters?
limitations: While ModFlow can generate molecules with high validity by directly sampling from the latent space, there does not seem to be a straightforward way to incorporate valency checks into the sampling procedure of ModFlow, should one desire to do so.
review_title: null

Meta-review: metareview: This paper introduces ModFlow, an E(3) equivariant normalizing flow for generating molecular conformations. A normalizing flow model for molecular systems could be extremely useful since normalizing flows have desirable properties including tractable densities and good sampling behavior. The authors demonstrated the effectiveness of their method compared to state of the art on two canonical datasets (QM9 and ZINC250K) although it might have been nice to see something harder.

All four referees (weakly) thought the paper should be accepted. All of the referees liked the idea of using normalizing flows for this problem and thought the approach laid out by the authors seemed reasonable. Some of the referees thought the writing was good, while others thought it could be improved. The main stumbling block was the choice of metrics reported on the QM9 and ZINC datasets. However, during the rebuttal the authors provided additional metrics which seemed to corroborate the success of the approach. Given the global support by the referees and improvement by the authors during the rebuttal, I think the paper ought to be accepted.

confidence: Certain
Decision: Accept