Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work studies the problem of molecular conformation optimization and proposes a neural "energy minimization" formulation, where a neural network is parametrized to learn the gradient fields of a conformational energy landscape.
main_review: Concerns about technical parts  
- My major concern is the formulation of "energy minimization". Although the main claim of the work is energy minimization, I don't find any relation between the mathematical formulation and energy minimization. 
    - The potential energy function $u()$ in eqn (2) is not specified.
    - $f_x()$ in eqn (3) is not trained to minimize the energy function (which is not specified); instead, it is trained to minimize the L2 distance between the generated conformer and its closest reference conformer.
- The basic idea of this work needs justifications. The authors seem to suggest that casting "the prediction problem into an unrolled optimization process" is better than previous approaches such as ConfGF. This is not clear to me. 
   - "While our model is also trying to estimate the gradient field, we instead employ an unrolled optimization architecture (Domke, 2012; Liu et al., 2018) aiming to model the entire optimization process. By learning to minimize a parametric energy function in a fixed number of steps, our models enjoy a much faster inference time."  This claims that the proposed method is faster but not better (or more accurate). Then it is better to explain why it is better than ConfGF in Table 3, actually much much better in terms of MIS(%).
    - Seems the number of steps is a key hyperparameter in the proposed method, which is unfortunately not studied in experiments. For example, how many steps are used in experiments? Are the final results sensitive to the number of steps?  

Concerns about experiments  
- In Section 4.1, Table 1 shows that the three atom model is better than the two atom model for conformation optimization. However, only the two atom model is studied and compared in Section 4.2 for conformation generation. I expect the comparison with the three atom model rather than the two atom model in Section 4.2.
- As shown in Table S1, the results of the proposed method are (very) sensitive to the initialization of molecular conformation, i.e., Ours-TwoAtom is much better than Ours-TwoAtom-Random. Does this mean the proposed model can only work well with a good initialization? If yes, this may limit the application scope of the proposed model. 
- Similar concern about "instead of training the model to optimize a single initialization towards one reference, we train the model to optimize K initialization such that these K optimized conformers can be exactly matched to K different sampled references." Are the K initializations different from each other? If so, how to get the K different initializations?  
summary_of_the_review: I have several concerns about the technical part and empirical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present an equivariant graph network approach for optimizing and generating molecular conformers.
main_review: Regarding the machine learning approach and experiments, the comparison to existing experiments is unfair at points, since conformer ensembles generated with RDKit are often used as starting guesses, while other approaches rely on random positions. Since the authors have provided experiments with random starting positions in the SI (and still find good, although less impressive performance) this is only a minor issue. However, this difference between model evaluation should be clearly stated in the main text. A discussions on how hydrogen atoms are treated would also be helpful for putting the results into perspective. The presented approach appears to only generate heavy atom positions, with the hydrogens added via RDKit. How does this compare to other approaches and how does it influence the search for stable conformers (e.g. are additional optimizations with ab initio methods necessary)? Since a general correspondence between properties and structure can be expected, the downstream application experiments appear to offer no new insights on the quality of the method. Nevertheless, it would still be helpful to know how the labels for these experiments were generated. Were new computations performed for the new confomers or did the experiments reuse the orginal QM9 labels for molecules with the same graphs? There is also a slight confusion regarding the comparison to CGCF in Table 3. The numbers reported there diverge from [Xu 2020]. In addition, it was never specified which delta value was used for evaluating the COV and MIS scores.

One of the main points of criticism is the proposed neural energy minimization framework for constructing equivariant networks. It essentially recapitulates already established basics of SE(3) equivariant models and offers no new insights. It is well known, that the Cartesian gradients of functions are one way to systematically construct equivariant filters for interactions/updates. This ansatz was e.g. used in  [Schütt 2020] to derive continuous convolutions of an invariant input with an equivariant filter (order l=1). The resulting update has a strong resemblance with Eq 3. In general, updates and interaction can be expressed as a multipole expansion of the energy, and if this is done e.g. in Cartesian coordinates the interaction tensors take the form of first and higher order derivatives w.r.t. atomic positions. Irreducible representations and spherical harmonics are another way to systematically construct equivariant models frequently and are closely related. For example, the update formula in Eq. 3 could also be derived as the interaction between scalar (l=0) and vector (l=1) geometric tensors in this   framework [Batzner 2021]. The statement, that conventional SE(3) equivariant networks are either based on complex mathematical frameworks, which necessitate expensive coefficient calculations, or derived ad hoc from message passing is highly misleading. In general, there is a limited way on how equivariance can be introduced into a model properly. If these operations are to be formulated in a principled and general way, tensor algebra is helpful. This does not automatically imply, that operations are inefficient. If only lower order features/interactions (e.g. l <= 2) are used, no significant computational overhead compared to conventional message passing is introduced (see e.g. [Batzner 2021]). On the other hand, if one were to expand the proposed derivative mechanism to higher order interactions, similar problems would be encountered. E.g. interactions between two dipole vectors require second order derivative tensors (3x3), between quadrupoles a fourth order derivative (3x3x3x3) tensor, etc, thus greatly increasing computational cost.

Finally, there are several problems regarding the introduction and related work sections. 
With respect to molecular conformer generation, full ab initio molecular dynamics is neither the standard approach nor required for conformer search. Most of the time, heuristic approaches as e.g. used in RDKit followed by ab initio optimization are used. These are much more tractable and usually work well for small organic molecules, as can be seen based on the performance of RDKit in the conformer generation experiment. The main problem of such approaches is the energetic weighting of the conformers, for which said ab initio optimizations are required.
In the same paragraph (and introduction), a series of generative approaches is described as using distance matrices as intermediates which are then translated into 3D coordinates. However, many of the mentioned approaches do not rely on distance matrices as intermediates. G-SchNet, for example, uses an iterative approach based on factorized probabilities to directly generates a new 3D structure.
The description of ML based force fields is also misleading. While it is true, that sGDML requires retraining for every new molecular graph, this neglects a plethora of machine learning potentials, which do not suffer from this problem and scale linearly with molecule size (see e.g. [Unke 2021] for an overview).
It would also be helpful to provide an in depth discussion on the parallels between ConfGF [Shi 2021] and the present approach in the related works section.


Schütt, Kristof T., Oliver T. Unke, and Michael Gastegger. "Equivariant message passing for the prediction of tensorial properties and molecular spectra." arXiv preprint arXiv:2102.03150 (2021).

Batzner, Simon, et al. "Se (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials." arXiv preprint arXiv:2101.03164 (2021).

Unke, Oliver T., et al. "Machine learning force fields." Chemical Reviews (2021).
summary_of_the_review: While sharing conceptual similarities to the gradient fields by [Shi 2021], unrolling the structure optimization process in the form of a graph network is an interesting approach and casting the model as a fixed point search has parallels to deep equilibrium networks. Its potential is demonstrated by good performance in different experiments. The proposed theoretical framework, on the other hand, is a restatement of common concepts in equivariant architectures, which have been presented in a similar or more general form elsewhere. In addition, the different sections (primarily related work) suffer from several fundamental problems which further detract from the quality of the work. Due to the two latter points, I tend towards rejecting the work in its present form.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new deep neural network (DNN) architecture to generate molecular 3D conformation. It derives its architecture starting from gradient-based updates for minimization of the molecule's energy function. Then the architecture becomes a repetition of vertex-wise aggregation layer of SE(3)-equivariant Transformer-based DNN. The architecture takes a molecule and 3D conformation pre-calculated from RDkit as input. The authors show how the proposed architecture generalize existing DNN architectures for molecular conformation generation. Empirical results demonstrate superiority of the proposed method. 
main_review: Pros:
1. The proposed method seems to be simple and easy to implement.
2. The proposed method outperforms the existing methods in the experiments.

Cons:
1. I think the paper could use some clarifications in their writing. I point out some suggestion in what follows.
- First, I am not sure if the proposed method can be called an "neural energy minimization" method and an "unrolled optimization algorithm." There is no energy or optimization objective for the given DNN architecture. Without any regularization, the proposed DNN architecture is not a proper gradient field. 
- The "energy function E()" is defined only on the conformation X in Equation (1), but defined on the conformation X and atom features V in Equation (2).  
- This paper uses the terminology "V" as the set of atom nodes and set of atom features. I think the authors should add a new terminology to denote the set of atom features. 
- I am a little concerned that the paper propose to solve "conformation optimization", but the proposed architecture is trained using supervised learning and the evaluation is done using RMSD. In other words, there is no optimization in the proposed task. I understand that the task is to "predict the optimized conformation." Maybe the authors could make minor clarifications or changes to clarify this aspect?
2. Additional experimental results can strengthen this paper, especially for future works to build on this paper. For example, can the authors provide their experimental results on predicting the ensemble property (Table 5 in ConfGF paper)? 

Minor:
1. Given the training scheme used in the paper, it seems that the DNN-based "molecular conformation generation" methods are applicable to "molecular conformation optimization". Is this correct? 
2. How did the authors perform hyper-parameter tuning? 
summary_of_the_review: This paper shows solid empirical performance with the proposed DNN architecture for molecular conformation generation. However, there are some logic or details that was not very convincing to me, e.g., the proposed method being a neural energy minimization. I would like to see them clarified for me to raise my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new method to generate molecular conformations (i.e., the spatial arrangements of atoms belonging to a given molecule). The method works by updating initial atom positions (which can be initialized either randomly or using an alternative conformer generation method) through an iterative process. Each iteration involves predicting the gradients associated with atoms' positions and then changing their coordinates by taking a step in this direction (Eqn. 1 & Fig. 1b). 

The authors:
1. show how the parameterization of the gradients relates to how one models the conformational energy (Section 3.3);
2. demonstrate competitive performance on optimization (finding a single best conformer; Section 4.1) and generation (finding a set of relatively stable conformers; Section 4.2) tasks compared to previous benchmarks. This is judged in terms of (a) how well the generated conformers match up to ground truth conformers (e.g., Table 1) as well as (b) how well the generated conformers enable the prediction of molecular properties (Table 2).

main_review: # 1. Positives
i. The paper has strong empirical results.
By considering both optimization and generation baselines, the authors are able to compare against a wide range of previous baselines over several different forms of evaluation. They obtain improved performance over these models using multiple different metrics. This improved performance also seems to be obtained with reduced compute costs (e.g., approx. 170x speedup over ConfGF).

ii. I found the paper easy to follow.
Figure 1 is helpful in understanding the approach. I found Section 3 interesting and well-organized; describing previous methods through considering the implied form of the potential energy function (see also Appendix C) seems like a useful lens in which to compare and view different models.


# 2. Negatives
i. It would be nice to have been provided with some evidence for what aspect of the proposed approach is important for improving upon the previous SOTA ML model, ConfGF, in Table 3, given that (at a high-level) ConfGF is also predicting the gradient of the energy function. For instance, is it because of the unrolled optimization routine, different architectures, the optimal transport loss, etc.? Could this be teased out with an ablation study?

ii. The number of time steps (i.e. the max value of $t$) used/required in the different experiments is not very well discussed/evaluated. Questions I would like to see answered include: Is the authors' model sensitive to this value? Does the value differ for the two datasets? How can one choose this value?


# 3. Clarification Questions to the Authors

i. Table S1 indicates that the RDKIT initialization is important for performance. I just wanted to check that you also used RDKIT initializations for the baseline approaches you compared to (where applicable)? As opposed to RDKit initializations, have you thought about predicting the initial positions?

ii. I'm surprised that in Table 1 "Ours-TwoAtom" does so much better than EGNN, given on p.5 you describe the similarities between these models. Do you have any intuition for why this is?

iii. Sorry if I missed this, but in Table 3 are you using the two atom or three atom model?

iv. In Table 2 is SchNet really the best reference model? From Table S2 it seems that one can do much better with an equivariant model (when you know the conformation). 

v. How do you choose the step size in Eqn 1, $\gamma$?
 
 



# 4. Very minor comments (e.g., typos)
i. The grammar/typos could be fixed in a few places, e.g.:
- p.1 "could be required to encode the redundancy observed in such matrix" --> "could be required to encode the redundancy observed in such _matrices_" ?
- Fig1, caption: "the gradient updates for the 3D coordinate long with the updated atom" --> "the gradient updates for the 3D coordinate _along_ with the updated atom"
- p.5 "perspective of neural energy minimization, these are equivalent to optimizing an energy function that depends on both the atom presentations and inter-atomic distances just like our basic" --> "perspective of neural energy minimization, these are equivalent to optimizing an energy function that depends on both the atom _representations_ and inter-atomic distances just like our basic" ?
- p.15 "we ﬁnd many of them are pretty closed, as" --> "we ﬁnd many of them are pretty _close_, as"

ii. On p.9 when introducing terms COV, MIS, and MAT you could refer to their definitions in Appendix G. Also what value does $\delta$ take? (apologize if I missed this)

iii. The experiments in Appendix I are interesting and impressive! Maybe it would have been nice to have more explicitly referred to this section in the main paper?

iv. How were the hyperparameters described in Appendix E chosen?


summary_of_the_review: Overall, I thought this was an interesting paper and the proposed method achieves impressive empirical results. However, for now, I have gone with an overall score of 6, as I would like to see some of the experimental aspects better resolved (see 2.i-ii & 3.i-ii). Happy to consider raising my score if this gets addressed in the rebuttal!

I went with a lower confidence score as I have not gone through the proofs in the appendix in detail and only have a higher-level knowledge of some of the pieces of related work.

##  Rationale behind the significance scores
I put _"3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."_ for the technical contributions question below. I did not choose the top score, given that Shi et al. (2021) also consider parameterizing the gradients of the energy function directly. (I realize that there are differences however, such as the training routine and optimal transport loss).

I put _"2: The contributions are only marginally significant or novel."_ for the empirical novelty and significance. This paper's method is evaluated on existing datasets (Ramakrishnan et al., 2014; Axelrod & Gomez-Bombarelli, 2020) in a way that is similar to existing work. To introduce a new empirical evaluation the authors could consider evaluating how well the models trained on one dataset transfer to another or consider directly comparing the predicted gradients (from the NN) to gradients derived from traditional approaches. (Although in my opinion making more of a technical contribution and less of an empirical one (or vice versa) is fine).
review_title: null

Meta-review: All reviewers except one agreed that this paper should be accepted because of the strong author response during the rebuttal phase. Specifically the reviewers appreciated the new ablation study showing that improvements are not due to minor architectural changes, the new experiment on the number of time steps required for experiments, the agreement to change language around "neural energy minimization", the improvements to the related work, the novelty of the unrolled optimization approach, and the nice experimental results. Given this, I vote to accept. Authors: please carefully revise the manuscript based on the suggestions by the reviewers: they made many careful suggestions to improve the work and stressed that the paper should only be accepted once these changes are implemented. Once these are done the paper will be a nice addition to the conference!
Decision: Accept (Poster)