Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors develop new kinds of deep learning models for predicting energies and forces from 3D molecular structures. Their core approach is based on representing node embeddings as spherical signals ("spherical channels" via coefficients in a spherical harmonic basis) and iteratively updating them in a message-passing process. While this is similar to some prior equivariant networks for 3D molecular structure, they make a few changes: (1) they orient the spherical signals during message passing in a way where the ij direction vector is consistent (non equivariant because the part of the rotation is underdetermined and randomized) and (2) they use more flexible pointwise processing to process the  signals by converting them back and forth to grid representations (approximately equivariant based on sampling resolution). They find that this approach gives strong empirical performance on predicting forces, energies, and relaxed energies on the Open Catalyst dataset.
strengths_and_weaknesses: Strengths

Clear and practical ideas. While the author's modifications break rigid equivariance, they do so in an interesting way. Predictably orienting the spherical signals during message passing intuitively makes sense and makes it possible to process the spherical harmonic coefficients in a very flexible and unconstrained way (FC layers) while still having some of the rotational degrees of freedom fixed. I found the paper well-written and easy to follow. The authors adequately position with respect to prior work.

Strong evaluation results. The results are compelling, especially for the relaxed energy tasks (IS2RE). These are interesting because in some ways they would in principle require more high-order geometric reasoning and feature processing than direct energies which can often mostly be predicted from lower-order geometric features (i.e. it would need to be able to predict how energy will change upon coordinate re-configurations).

Weaknesses

Loss of equivariance. As the authors acknowledge, their final best performing models are not equivariant  but they compare to models that are equivariant by construction. I appreciate that the authors did a supplemental analysis on the errors incurred to forces and energies upon random rotation (which were non-trivially large), and even included an analysis based on summing over multiple rotation. It would be helpful to have more discussion on alternative strategies to achieving or improving equivariance, which without summing over multiple rotations seemed to give non-trivially large errors. I am curious if there would be non-randomized, canonicalization approaches to deal with the issue, for example orienting the rotation around the d_st axis in a way that points to the molecular center of mass, or something like that.

questions: My main request to authors would be that they address the Loss of equivariance issue described in the Weaknesses above. It would be helpful to have further discussion on how the force and energy errors under rotation might be further mitigated and also how much it would affect compounding error for something like molecular relaxation.

UPDATE:
8/9 - I read the authors rebuttal and appreciate their additional section about message equivariance. I stand by my review and I like the paper. I think it is sufficiently different from prior works for chemistry ML, shows compelling performance, and will challenge the the field to continue developing models that can realize these benefits but with exact equivariance.
limitations: The authors adequately discuss both model and societal limitations in the discussion section.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors present a novel deep learning architecture to predict scalars and non-conservative vector fields for applications in molecular modeling.
strengths_and_weaknesses: # Strengths

- The model architecture is simple.
- The (enormous) model seems to perform better than other (smaller) models on the OC20 task
- The model seems to be faster than other models (despite its size).

# Weaknesses

- The manuscript fails to relate the presented model to existing ones on an architectural level. While the related work section mentions other models, it is unclear what it is about SCN that other approaches should adopt to improve predictive performance and reduce computational time.
- It is unclear why the presented model outperforms existing ones on the OC20 dataset. The presented model contains up to 10x more parameters than many other (competitive) models. There is no ablation study investigating the effect of model's size.
- The model is not run on standard benchmarks such as (rev)MD17 or 3BPA. As a result, it is difficult to compare this model to existing ones and pinpoint the source of its performance.
- The model is not implementing a conservative force field. As a result, the applicability of the model to problem in computational chemistry mentioned in the abstract is severely limited. For instance, it is unsuitable as a potential in MD simulations.
questions: - How is the orientation $\hat{r}$ chosen for the node embeddings? Is this choice of $\hat{r}$ independent for all atoms? How do you compute (4) efficiently?
- Do you integrate over all rotations in (4) and (5). I believe a $d\hat{r}$ is missing.
- Have you considered reporting standard results on datasets like (rev)MD17 or 3BPA?
- What is the effect of the model's size on its performance?
- Did you study the effect of the fact that the predicted forces are not conservative? Did you try building a conservative force field from your model for the energy?
limitations: The authors have adequately addressed the potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a graph neural network (GNN) they name the "Spherical Channel Network" (SCN) to map molecular configurations to atomic energies and forces. They model the atom embeddings as a set of spherical channels using spherical harmonics. The authors relax the equivariance constraint in both message passing and aggregation and demonstrate SOTA results on the Open Catalyst dataset for the prediction of energies and forces. The proposed methodology aims to replace the computationally expensive Density Functional Theory (DFT) calculations necessary to compute an estimate of atomic energies and forces.
strengths_and_weaknesses: Strengths:
* The use of spherical harmonics to reason about directional information for all neighboring atoms as opposed to atom pairs.
* Apparent superior out-of-domain generalization compared to baseline models.
* Model allows for the complex mixing of different degrees of spherical harmonics enabling improvements in representation learning that can improve generalization performance.

Weaknesses:
* Relaxation of equivariance constraints can make reproducibility difficult and introduce variance in production.
* Energy is not conserved which hinders applicability to finite chemical systems in which equilibrium dynamics is concerned. 
* Requires arbitrary specification of rotation around vector $d_{st}$.
questions: * Can the authors compute the significance of the reported results as compared to the other baseline models reported in the manuscript?
* For the non-equivariant scenario, how is the variance in the predictions affected by the number of translations in the training set?
* Have the authors tried evaluating the models on larger molecules than seen in the training set? Is there any reasoning behind outliers or failures seen during testing?
limitations: Yes limitations adequately specified.
review_title: null

Meta-review: metareview: The paper presents a new model with state-of-the-art results on the IS2RE task on the OpenCatalyst dataset. 2 of 3 reviewers recommended acceptance, while the 3rd reviewer raised issues with lack of baseline comparisons on small standard molecular potential energy surface datasets. I am inclined to agree with the authors on this one - if the empirical results on a large, challenging benchmark are good enough, baseline comparisons on small benchmarks are not always necessary (did the AlexNet paper include benchmarks on MNIST?) It is slightly unnerving that the results are so good with a non-equivariant model, but it sounds like the authors discuss this as well. Overall this seems like a solid, if not revolutionary, improvement in machine-learned potentials for OpenCatalyst. I recommend acceptance.
confidence: Less certain
Decision: Accept