Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new VAE-based generative model for generating molecular conformations from graphs. The whole formulation and training objective are similar to existing VAE-based models like CVGAE, and the main contrition lies in the architecture of the decoder. The architecture borrows the idea from the recent advance of AlphaFold: the proposed decoder is composed of several identical updating blocks, where they iteratively update the node/edge embeddings as well as the coordinates in each block. And the blocks are implemented with advanced graph neural networks. 

main_review: Strengths:
The paper is well-organized and clearly written. All content is easy to understand for me.
The experimental part is comprehensive. The author nicely compares the proposed method with competitive baselines on both small-scale and large-scale datasets. The results demonstrate that the model is better or comparable with state-of-the-art results with much less computational cost.
Weaknesses:
The main concern of this paper is the originality. The main contribution of this paper is the decoder part, which is very similar to the “structure module” in Alphafold that iteratively updates the folding and node/edge features. However, I appreciate the efforts to extend the progress of protein folding to molecular modeling, so I also agree the adaption is non-trivial.
A minor problem: the problem definition in Sec.2 seems not accurate enough. In your statement, it seems each graph only corresponds to a single conformation, but this is not true. Please restate it.
I’m also interested in the design of the decoder block. I understand the training objective is invariant, however, I’m curious whether updating the coordinates with equivariant networks will further help the performance. Several recent works such as ConfGF have shown that equivariance is an important inductive bias for 3D generative models.

summary_of_the_review: The paper proposed a VAE model for molecular conformation generation, where the decoder borrows the idea of Alphafold to iteratively refine the generated structure. Personally, the paper seems an extension of the recent progress of protein folding to the small molecule setting, but I appreciate the authors’ efforts to make it work. So currently I vote for a weak acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: 
The paper presents a direct method for generating a molecular
conformation conditional on the molecular graph.  Inspired by the
success of AlphaFold2, the authors demonstrated that one could
directly write out 3D coordinates for a small molecule rather than use
the previous ways of generating intermediate interatomic distance
matrices or iterate over forces. The method displays a decent
performance on the standard benchmark set of QM9.  It performs better
than past baseline methods for larger molecules.

main_review: 
Overall, this work appears to be an advance compared to the
Langevin-based method it mainly compares against.  It is not clear to
me if this is simply due to a different parameter setting (e.g. a
larger model than a year ago works better for larger molecules), or if
this improvement is due to a genuine algorithmic advance, and if so,
what part of the algorithm helps the larger molecules the most and why
couldn't the authors beat the previous model on the smaller-sized
molecules.  I appreciated the authors effort to document the change of
the results across the methods in Table 3, but it would help if they
could also modify their new model (or ConfGf, or both) to a
drastically different size.

I did not understand what was the contribution for the fine-grained
loss function used in training. Unless I misunderstand something
important, the particular matching loss is not invariant to
permutations of symmetric atoms---think of the different ways to
enumerate the atoms in a methyl group, or replace the hydrogen in a
methane by methyls, or take any symmetric groups in a molecule that
can generate a combinatorial number of equivalent coordinates that
represent the same molecule graph. So I don't understand the statement
of the authors that this loss is zero if and only if R1 and R2
represent the same molecule structure---what would the loss be if I
randomly permute the symmetric hydrogen identities but not change the
coordinates?

Regarding Secion 4.3, I understand that these types of tables have
been published previously, but I wanted to state that I strongly
disagree with the terminology of calling the average energy of the
generated conformers a "property prediction".  I don't think that
there exists a single experiment that can measure this particular
quantity, although arguably one could use a Quantum Mechanical code,
as these authors and others have done, and calculate a number for it.
From a physics perspective, the somewhat more meaningful property is
the Boltzmann-weighted average of the energies of the molecules---is
that perhaps the quantity that this benchmark is supposed to evaluate?
Furthermore, what is the meaning of switching to an MMFF force field
for optimization and then use Psi4---why not directly evaluate the
conformation with Psi4?  Finally, what are the units in the energies
of Table 4?  If these are in Hartree (which I thought were the default
units for Psi4, though I'm no expert in it), then the reported errors
are all huge to be meaningful one way or another (to the order of more
than 13 kcal/mol for the smallest number listed in that table); these
types of enormous errors have little meaning for any conformational
energy comparison, as a trivial small bond-length deviation could add
this level of energy, whereas a wrong dihedral angle couldn't (unless
parts of the molecule almost clashed onto itself).  Perhaps the
authors would want to either clarify section 4.3 further (it took a
while to parse what the authors actually calculate there, and from a
quick check of the repo, I couldn't locate the relevant code for it)
or simply drop it altogether since it probably distracts from the rest
of this work.

I also didn't understand the first two of the three future directions:
I am not clear what the authors mean when they suggest combining this
method with flow-based or Langevin dynamics models---can they clarify
this point?  Importantly, what are the cases that fail in the current
model---can the authors document some examples in an appendix?


summary_of_the_review: 
I think that this paper is an advance on previous work and probably represents
the state of the art in generating molecular conformations conditional
on the graph.  The paper could use a review of the language to clarify
certain aspects of it, but I don't think that it is strong enough for ICLR.  
Importantly, it should attempt to clarify if the improvement is mainly 
due to model size changes or due to algorithm changes.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose the generative model that constructs a conformation (set of 3d coordinates) of the molecular graph with a variational autoencoder (VAE) framework. Unlike several recent works on conformation generation based on generating of distance matrix and further conformation recovering by solving a time-consuming distance geometry problem, the proposed approach directly generates 3d coordinates with an iterative process inside the decoder part of the model.
main_review: **Originality:** The work suffers from the lack of originality. The described model shares the same idea as ConfVAE and the unnamed model, proposed in [1] and [2] respectively - the autoencoder-based framework for conformation generation, the iterative generation procedure, and RMSD objective were proposed earlier. The main novelty of the model is neural network architectures of encoder and decoder, that utilize advanced graph convolutions. Both of [1] and [2] works are missed in baselines, the model compared with.

**Clarity and Quality:** The authors clearly describe their motivation and provide good discussion and ablation studies section, still paper contains vague contributions part and missed related work section. The paper is written in a hard-to-read way. The paper contains plenty of inline equations, inconvenient formatting of enumerated statements that make it hard to navigate over text and understand the method. The paper focuses too much attention on technical details of the model, that are minor and can be moved to the Supplementary materials.

**Significance:** The novelty of the paper is minor. The experiment results show the increase of metric values between the proposed model and ConfGF baseline, still, it's hard to determine the significance of the performance gap due to the lack of standard deviation. Also, the significance of the results is doubtful since the important baselines [1] and [2] are missed in Experiment section.

**Drawbacks / questions**:

1. I would recommend using different Greek letters to denote the encoders/decoder parts instead of subscript text to increase the readability of formulas.
2. The ablation studies cover only architecture details. Still [2] provide loss function based on distance differences that is also roto-translation invariant but do not require additional subproblem solving.
3. The small-scale dataset results are redundant still large-scale dataset covers it. I would recommend training CVGAE and CGCF on the large-scale dataset and including the results of these models.
4. The choice of conformation colors in Figure 3 is misleading. I would recommend to add an additional figure with aligned conformation cloud to show the diversity of generated conformations.

[1] An End-to-End Framework for Molecular Conformation Generation via Bilevel Programming, Xu et al.

[2] Auto-Encoding Molecular Conformations, Winter et al.
summary_of_the_review: The idea of the paper is not novel. The experiment part misses several important baselines. The text should be revised to improve readability and clarity. The ablation study section should provide experiments to justify loss function choice.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a method that is able to accurately generate molecule confirmations, achieving competitive results across four datasets. 
Unlike previous methods that often rely on directly predicting various inter-atomic distances (through a distance matrix) (which in some cases results in physically impossible molecules), the presented work predicts the location of all atoms. The loss function is invariant to rotation, an important property of many molecules. 
main_review: Strengths:
* The results are strong and the presented methodology is clear.
* The ablations shown in Table 5 are very nice to see and I think are a useful part of the paper.

Weaknesses:
* Some of the discussion of results could be expanded, or at least included in an appendix. For example, "This is consistent with our conjecture, and also demonstrates the potential of directly generating coordinates for complex molecules with more heavy atoms." This would be nice to have been further expanded upon.
* The presented results are limited in scope, and largely rely on comparisons with ConfGF, though then some additional comparisons would be useful. For example, for the property prediction showing results comparing to Tables 3 and 4 in the ConfGF paper would be useful. 

Comments:
* In Fig. 3, what do the colors represent? Are two views shown?
*  A table with exact hyper parameters/etc in the appendix could be useful (in addition to the details in section 4.1). It's great that code is provided.
* Some methods, like G-SchNet (https://github.com/atomistic-machine-learning/G-SchNet) from my understanding do not produce confirmations which violate the triangle inequality. Is this a misunderstanding on my end, or is this approach of successively predicting distances an alternate approach that produces valid molecules?  I realise it is a different problem being approached, but molecules are still being generated. 
* In Table 2, what is your interpretation of the COV(%) being 100%? 
* For the property prediction, on QM9 various properties can be predicted. Could these all be shown?  
* The text is at times a bit difficult to follow, but it does not detract from the ability to follow the work as a whole. 
* I don't quite follow the "ours with /FF" in Table 5, and why this is not the "default" flavour?
summary_of_the_review: The work presents strong results on molecular confirmation prediction. A novel method is developed that mitigates concerns of many previous methods by ensuring adherence to geometric rules. The results shown are very competitive, though I would like to see slightly broader comparisons and more exposition on various points that are raised in discussion.
review_title: null

Meta-review: The paper proposed a new VAE-based generative model for generating molecular conformations from graphs. Reading the paper, the reviews and the rebuttal, it looks like this project is a work in progress and not yet ready for publication. Some reviewers indicate that the paper lacks significant technical novelty. 

This is a worthy application. I encourage the authors to keep working and resubmit to another venue when they feel the work is ready and they have addressed all points raised by the reviewers.
Decision: Reject