Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: **Summary**
This paper proposes ForceNet which is capable of capturing highly complex and non-linear interaction of atoms in 3D molecular space in order to accurately predict atomic forces. On the outset, the ForceNet framework seems extension of recent work on physics-based simulator for fluid dynamics using deep network called GNS (Sanchez-Gonzalez et al, 2020) but with carefully designed message passing network. Unlike GNS, the ForceNet computes messages using specifically designed continuous filter convolution. Inspired by SchNet, the filter weights in continuous filter are dynamically computed by first lifting edge feature input to new basis space followed by multi layer MLP. However, unlike SchNet, the filter computation is conditioned on the source and target node features as well as the 3D angular information.

Apart from this, ForceNet deploys spherical harmonic basis functions and smooth activation function - Swish, to realised better accuracy. Experimental evaluation on large-scale catalyst dataset OC20 shows that ForceNet reduces force estimation error by 30% in comparison to recently popular ML models for quantum chemistry simulation. When employed for quantum chemistry simulation, wherein DFT forces are replaced with ForceNet computed forces, it converges to similar energy structure as DFT in $10^3 \times$ lesser time.

**Quality**
The paper is very well written and easy to follow. The description of their model, experimental setup, comparison to existing baselines and their architectural choices are precisely described (as well as evaluated). Although their largest but best model is $3\times$ slower than existing work,  it produces highly accurate force prediction.

**Originality**
As mentioned above, this work combines best of two world proposed for quantum chemistry simulation in the past - GNN based framework and continuous convolution. From the outset, the architecture seem to me simple but effective extension of SchNet work with modifications in interaction layer such as,
1. Use of directional vector and atomic radii for feature input to basis function instead of mere scalar distances.
2. Convolution filter generation conditioned on source and target node features
3. Inclusion of self-message $m_t$ and scalar multiplier $\alpha$ and
4. Different basis and activation function

ForceNet achieves this by trading off rotation-invariant property while leveraging random rotation data augmentation to achieve rotation covariant.

**Significance**
The carefully designed message passing computation will motivate future work to design better layers for targeted application. Moreover, the speed + accuracy achieved on DFT task is very encouraging for future deployment of ML model for quantum chemistry simulation.

**Clarity**
1. Like previous works, can you compare errors w.r.t. energy output ?
2. SchNet uses shifted softplus as smooth activation. Ofcourse the motivation there was twice differentiability. Have you experimented ForceNet with shifted softplus as well as other non-smooth activation such as Leaky ReLU etc. ?
3. For Table 2, can you share results obtained by only omitting $m_t$.
4. Have you tried spherical bessel basis as in DimeNet ? Comparison to this was missing under basis function choices.
review_title: GNN + Continuous Convolution for Quantum Chemistry Simulation

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################
Summary:
This article presents the force fields predictor ForceNet. The authors show that ForceNet reduces the estimation error of atomic forces by 30% compared to existing ML models. Reconstructing molecular force fields is a very active and thriving field, hence this paper is highly relevant. In short, this article presents a very good model based a selection of very good methodologies and combining them in a clever way.

##########################################################################
Reasons for score: 
Overall, I vote for not accepting, but if the authors address most of the comments I don't have a problem if it is accepted. As mentioned before, this problem is of high relevance and is likely to have a considerable impact in the physics and simulations community. Nevertheless, this is a non-energy conserving and non-rotationaly covariant force field, meaning that for most of the applications of a force field, this will provide unphysical results.

##########################################################################

Pros:  
1. ForceNet reduces the estimation error of atomic forces by 30% compared to existing ML models.
 
2. It is based on strong and reliable previous results, which makes it a robust model.
 
##########################################################################
Comments: 

-The authors state ”However, if successful, accurate and fast ML-based models may lead to significant practical impact by accelerating simulations from O(hours-days) to O(ms-s), which in turn accelerates applications such as catalyst discovery.” And “If successful, ML could be applied to problems such as catalyst discovery which is key to solving many societal…” These statements hint that this has not been done yet, which is a half truth. As exposed in the references cited in the paper, there are already very successful ML-methodologies in this field, but it has to be stressed that the successes come from applications in local domains. This mean that errors of sub-0.1 kcal/mol have been achieved for molecules already couple of years ago (Chmiela et al Nat Commun 9, 3887 (2018), Christensen et al J. Chem. Phys. 152, 044107 (2020), and Unke et al J. Chem. Theory Comput. 2019, 15, 6, 3678–3693). Hence, phrasing the manuscript like this, it could be misleading for the reader.
-The authors state “This is possibly because the force-centric models capture the dependency of atomic interactions on atomic forces more explicitly than the energy-centric models.” This topic has been extensively analysed by Prof. Müller’s group (Sci. Adv. 3 (5), e1603015, 2017; Nat Commun 9, 3887 (2018)) and recently by Prof. von Lilienfeld (Mach. Learn.: Sci. Technol. 1 045018, 2020). In short, learning forces is equivalent to learn linearisation of the energy surface which is much more informative.
-MLP is not defined prior usage.
-The directed message e_{st} is a vector or scalar? In some parts appeared just as e_{st} and on other as \textbf{e}_{st}. 
-The atomic radii is a given physical value or is it a learning parameter?
-The authors state: “As we demonstrate in Section 4, the replacement of ReLU with Swish consistently and significantly improves the predictive accuracy while maintaining scalability across all choices of basis functions.” This is not relevant, since it is a direct statement. A more interesting comparison would be to shifted tanh function.

-From my point of view, the main downside of this article is the fact that the ForceNet is neither exactly covariant not energy conserving.
review_title: Report on ForceNet

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper propose a neural network force field that predicts atomic forces directly. This has the benefit of not requiring to differentiate an energy model and may be more flexible. The approach is well motivated and the paper is well structured and written. A strong point of the paper is the extensive discussion of model design choices.

A major weakness of the model is its lack of rotational covariance. In the presented application to relaxation of molecules on surfaces, this might be acceptable (since rotation only occurs in 2d here and relaxation path usually do not have a lot of structural variance), but for more flexible systems this will certainly lead to problems. I doubt that data augmentation can make up for this and the deviation of force predictions under rotation should be shown. In particular for MD simulations this might be a deal breaker, since the resulting model might not be energy conserving. For these reasons, the proposed approach might be of limited practical use beyond the demonstrated application. To prove otherwise, additional experiments with more flexible systems would be required.

Finally, the authors state: "However, practical models for real large-scale and complex problems remain out of reach." This is a bit of an overstatement as there exist ML force field approaches for practical problem ranging from nuclear quantum effect over reactions and spectroscopy to thermodynamics with thousands of atoms (e.g. refer to Table 1 in https://arxiv.org/pdf/2010.07067.pdf for an overview). For the particular problem of structure relaxation with ML, there is previous work by Shapeev et al (Phys. Rev. B, 2019) on accelerating crystal structure prediction.

Pros
------
- better and faster on the task than previous models
- extensive model selection and ablation studies

Cons
-------
- not energy conserving / rotational covariant
- thus probably limited applicability -> evaluate on a dataset with more variance and 3d rotation, e.g. MD trajectories


Update:
I read the response of the authors.
review_title: Fast and accurate model with limited applicability

final_score: rating: 7: Good paper, accept
review_content: This paper presents ForceNet, a graph neural network, for estimating per-atom forces based on the 3D molecular structure. The authors argue that force-centric learning could be better in terms of prediction accuracy compared to energy-centric learning. As such, they focused on force-centric simulation and did several simple, but effective optimizations to make their model more accurate and easier to scale to large size. Experiments validate the effectiveness of ForceNet.

Overall, I think it is a nice application paper. It is easy to follow and gives clear reasoning behind their designs. Though many of the design options look simple and may not be strong enough from a machine learning perspective, these designs are tailored towards
large-scale quantum chemistry simulation and would be nice to showcase the potential of machine learning models, especially GNNs, to these application domains. From this perspective, I am happy with its contribution.

Questions:
1. Would you make your OC20 dataset open-source upon acceptance? This would be a big plus for this paper, as it will promote more ML experts to pursue this field.
2. Section 3.1.3 is a little bit vague. Why would Swish be a good choice for the activation function? Little background and insights were given.
review_title: Relatively new application domain + nice results + ok technical contribution. 

Meta-review: The model presented here may be of use to others in running quantum chemistry simulations, and it may well lead to new advances, but the authors did not sufficiently address the key concerns around the model not being energy conserving and rotation covariant. The approach proposed could be learning such physical rules, and the authors in their general response provide some very preliminary evidence for this, but a much more thorough discussion with a full range of experiments is needed. ICLR is a broad conference where non-experts who may have never heard of DFT simulations must parse this work and decide on if/how to follow up. This is a critical missing piece for anyone that wants to do so.

The above is exacerbated by the fact that the work is not well-situated against prior work as pointed out by two reviewers. Together these two issues conspire to make understanding this model, the contribution of the work, and what followup is possible, untenable for an ICLR reader. For example, one would not be able to surmise from the manuscript and its brief discussion of rotation covariance that this is likely to result in ForceNet having limited applicability to other DM problems; which one reviewer pointed out and the authors generally agreed with. While the authors respond that perhaps the architecture itself may be useful for other applications, why this would be and what the specific advantage of the current model relative to the state of the art in those fields is unclear.

Decision: Reject