Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes a generative model for generating molecule with three dimensional structure. Given a graph, it leverages a variational autoencoder to embed the distance between two atoms into latent vectors (encoder) and then generate distance between two atoms based on the latent vector. Even if the problem is well motivated, I have several concerns:
I. What is the variable "x"? Is it possible to define a bit formally? It appears that x=(G,d) i.e. the graph with d.
II. It would be nice to develop the generative model of (G,d) i.e. both G and d. Simply generating d looks a bit restrictive. 
III. How do you model/learn \mathcal{O}(x)?

I would be curious to know the answer of these questions.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary:
The authors propose a generative model designed for molecules, which is essentially a conditional variational auto-encoder. The model learns to generate, when conditioning on a molecule graph, the distribution of distances between each of the atoms and its second and third neighbour. Finally, using these information new molecules can be generated, which satisfy the generated distances. In the experiments the authors show the effectiveness of the method.

General Comments:
The paper is ok written, but as a non-expert of the field, I find it quite hard to follow. More specifically, I think that the main content aims to readers which are very familiar with the problem of molecule modeling, which is of course understandable. However, I find the technical (machine learning) content a bit unclear, because I think that the proposed method is not presented properly in details (see comments):

1. I think that authors should specify the dimensionalities of each variable and function in the paper. Unfortunately, none of them is clearly defined, which makes the technical part a bit hard to follow.

2. As a non-expert of the filed, I would like to know what exactly is a conformation x? Does it refer to the Euclidean coordinates of each node (atom) v of the graph? (Related to comment #1.)

3. For each G, there are multiple possible conformations x_i, where each x_i specifies the Euclidean coordinates of all the nodes (atoms) v_j? Also, each node v_j has some attributes? So the goal is to represent each conformation x_i -implicitly- as a set of distances between the nodes v_j? I think that Sec 2.1. can be more clear about the definition of the data, such that to be accessible from non experts in molecule modeling. (Related to comment #1.)

4. I think that the definition of the model in Sec 2.2. could have been better. As before, I believe that it is hard to understand explicitly what the functions input and output is, since the dimensionalities are not defined. In my opinion, it would have been better if instead of this high level text you could provide explicitly all the steps of the model e.g. the concatenations, the appends, etc.

5. I could not understand if and how the model is able to handle graphs with different number of nodes, and different number of distances among them.

6. The experimental section seems solid enough, and shows that the proposed approach works better than the other methods. Also, I like the fact that some of the limitations of the method are stated. Moreover, the related work seems to be properly included. However, since I am not an expert of the field, I am not able to provide precise feedback for these two parts.

In general, I think that the proposed model solves good enough the problem that is designed for. Also, in my opinion, the manuscript aims for specialized readers, which is definitely understandable. However, my main concern regarding the current version, is that the technical (machine learning) content is a bit unclear and probably too high level. In case the technical content was presented properly, I think that it would have been a much better fit for this community.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 
In this paper, a new method for estimating the equilibrium measure of molecular compounds is proposed and tested on a new (enriched) version of an existing data set (CONF17 is derived from the pre-existing ISO17, itself extracted from QM9). 
The method takes as input a collection of graph structures of a given stoichiometry mixture, and learns the corresponding conformations.
This results in a generative model, taking as input a graph structure (or actually a single conformation instance, thus setting the stoichiometry AND the isomer/graph structure), to generate (in a one-shot fashion) other conformations of that isomer.
The distribution of conformations sampled in this way compare well with those of an extensive training set (CONF17, set of isomers and conformations of a single compound), and in particular they perform better than 2 alternative methods published recently (2015 and 2019, the former performing globally much better than the latter!).


Concretely, the model/method/algorithm consists in a combination of existing network architectures.  The main original contribution is to map the Cartesian coordinates of atoms into a pairwise distance space, which is naturally translationally and rotationally invariant.  Some of the second and third neighbors distances in the graph structure are considered, such that the conformation is completely set (this would not be the case if considering only 1st nearest neighbors distances).
The precise combination of architectures is explained in section 2.2. This explanation is a little fast for me (but I am new to this field). 
In that section, it could be nice to precise that the job of the CVAE is to learn (among others) the parameters mu,sigma of q_phi(z|d,G), using a batch of conformations of a given isomer (and then generalize to other isomers, each time guessing the correct mu,sigma parameters, even for a new isomer).
This is not obvious for the inexperienced reader (I hope I "guessed" correctly).

The "new dataset" shared in this work is indeed useful for reproducing results and for later comparison with other works which aim at sampling conformations.
However, it derives directly from ISO17, which is itself an extraction from QM9. ISO17 lists the Cartesian coordinates of 127 isomers of C7,H10,O2 (each having ~3000 conformations).  The "new" CONF17 is novel only in that it provides these conformations in distance-matrix space, i.e. it sets which pairs of atoms need to have their distances computed.  This is very useful because this choice of pairs is not unique, and so setting it once is necessary for comparison.  I would not stress too much that this is a new dataset: at first it actually sounds like authors chose a dataset such that their method outperforms others'!  Which turns out not to be the case (they chose C7,H10,O2, but this doesn't look like fine-tuning).  Indeed, their sharing of this augmented version of ISO17 is absolutely necessary: I would simply put it differently, stating from the beginning that they share an augmented version of a pre-existing data set.

About the experimental protocol/methodology:
the work carried out seems quite impressive. Indeed, the paper presents a rather extensive comparison of the new model with 2 other ones, on a large data set.
However sometimes the presentation rushes a bit towards results, in the sense that it is not easy to figure out over which ensemble the mean/median/max/min has been computed. This is related to the intrinsic complexity of estimating accuracy, given the encapsulation (compound-> isomer-> conformations-> each bond-> distribution over bond properties).  I think however it could be made clearer, and this would benefit the paper greatly.
A deeper comment: in the field of conformation sampling, there are two issues. One is to correctly sample states in each local minimum of the free energy. This is addressed in the paper. The second issue is to hunt for all possible local minima of the isomer, i.e. sample even unlikely but crucial conformations. This is not addressed here.  For computing average energy, dipole moment etc, the most likely conformations will dominate, most of the time.  However given the topic I think it would be fair to mention this point, and explain why in this specific case, the second issue is not crucial.
Because of this, also, I think 

Overall, the paper is well written and is very well situated within the literature (including the very recent literature).
It states clearly its contributions and results, and provides detailed presentation of experimental results.
I think it contributes to the field and provides an interesting new way of dealing with the problem of molecular conformations sampling.  The results obtained are much better than those of the compared ML method (RDkit).
However, some claims (in introduction or in section 3) are stated more strongly than they are supported by direct experimental evidence.
- ''... the first work on sampling molecular conformations for molecules of arbitrary size and shape based on deep learning.''
However, the experiments have been performed only on isomers of  C7,H10,O2.
Furthermore, there is no discussion on the computational cost of the method at all, so its extendibility to larger shapes is not obvious.
- '' We create a new, challenging benchmark dataset CONF 17 for conformation generation, which is made publicly available.''
I made remarks on this claim above.'' 
- '' We develop a rigorous experimental approach for evaluating and comparing the accuracy of conformation generation methods based on the mean maximum deviation distance metric.''
However the paper's methodology doesn't address the issue of ''hunting for new local minima''. 
The data set could be enriched also with a labeling of ''which conformation is in which local minimum of free energy''
- ''It is a fast alternative to resource-intensive approaches based on MCMC or MD.''
As there was no mention of speed of computation in the paper (although we can guess it is faster), this is unsupported. However it is easy to remedy, simply give an order of magnitude of sampling time (for thousands of conformations of 1 isomer).
- ''Our principled representation based on pair-wise distances does not restrict our approach to any type of system (e.g., proteins) or any particular graph structure. In addition, it is extendable in a systematic fashion.''
This suggests that the method would also work for dense packings. However this is not shown in the experiments.
Probably I misunderstood the claim, and it only refers to the ability to tackle general molecular compounds, not just linear ones?  However proteins are not always linear, but can present cycles, and other branch-style patterns, so the claim is not very clear to me.

Because some of the important claims are only weakly supported, and because of the relatively small novelty of the method, I lean on rejecting the paper.



Additional comments (for improving the paper):

There is no discussion of hyper-parameter tuning, or any detail of the intermediate GNN that constitute the encoder and decoder in the CVAE. 
So it was either not done at all, and then it should be mentioned; or it was done and it should be discussed quickly.
In particular, in section 2.2 there is no discussion on the impact of the latent space' dimension, here set to Nv, i.e. 7+2+10=19 (correct?).  I guess there is a balance between accuracy and generalizability, a short discussion on this point would be interesting.

The paper should clarify the hierarchy of molecular configurations:
for a given composition (%C, %O, %H, etc) (here, fixed throughout the paper at C7,H10,O2)
there are several structures possible (graphs or isomers),
for a given graph (isomer), there can be a couple of chiral variants (mirroring of (pieces of) the molecule,
for each chiral variant, there are conformations, which correspond to rather weak variations of the Cartesian coordinates, due to thermal fluctuations. Some of these variations can be important, e.g. when a bond can freely rotate. Some correspond to bending or torsion, etc, of the molecule.
The algorithm presented in the paper attempts to sample appropriately the equilibrium distribution of conformations for each given graph structure (at fixed chirality, if I understood correctly).

Add a comment on why (in your opinion) the distributions are not exactly correctly sampled, which after all is the goal (since the initial graph structure is given as input here).


Additional question, for curiosity. In section 2.3: only a single molecule output is taken from each distance matrix.
Could you comment on the possibility to sample several Cartesian coordinates, for each distance matrix ?  It seems like several solutions should (often) exist. 
Sampling continuously from that distribution would surely provide a more continuous (rich) output (am I correct) ?

review_title: Official Blind Review #3

Meta-review: The paper presents a solution to generating molecule with three dimensional structure by learning a low-dimensional manifold that preserves the geometry of local atomic neighborhoods based on Euclidean distance geometry. 

The application is interesting and the proposed solution is reasonable. The authors did a good job at addressing most concerns raised in the reviews and updating the draft. 

Two main concerns were left unresolved: one is the lack of novelty in the proposed model, and the other is that some arguments in the paper are not fully supported. The paper could benefit from one more round of revision before being ready for publication. 


Decision: Reject