VIMPNN: A physics informed neural network for estimating potential energies of out-of-equilibrium systems
Authors: Jay Morgan
Adeline Paiement
Christian Klinke
Venue:  (2020)
Keywords: neural network
chemical energy estimation
density functional theory

Abstract: Simulation of molecular and crystal systems enables insight into interesting chemical properties that benefit processes ranging from drug discovery to material synthesis. However these simulations can be computationally expensive and time consuming despite the approximations through Density Functional Theory (DFT). We propose the Valence Interaction Message Passing Neural Network (VIMPNN) to approximate DFT's ground-state energy calculations. VIMPNN integrates physics prior knowledge such as the existence of different interatomic bounds to estimate more accurate energies. Furthermore, while many previous machine learning methods consider only stable systems, our proposed method is demonstrated on unstable systems at different atomic distances. VIMPNN predictions can be used to determine the stable configurations of systems, i.e. stable distance for atoms -- a necessary step for the future simulation of crystal growth for example. Our method is extensively evaluated on a augmented version of the QM9 dataset that includes unstable molecules, as well as a new dataset of infinite- and finite-size crystals, and is compared with the Message Passing Neural Network (MPNN). VIMPNN has comparable accuracy with DFT, while allowing for 5 orders of magnitude in computational speed up compared to DFT simulations, and produces more accurate and informative potential energy curves than MPNN for estimating stable configurations.

Under review as a conference paper at ICLR 2020
VIMPNN: PHYSICS INFORMED DNN TO ESTIMATE POTENTIAL ENERGIES OF UNSTABLE SYSTEMS
Anonymous authors
Paper under double-blind review
ABSTRACT
Simulation of molecular and crystalline systems enables insight into interesting chemical properties that benefit processes ranging from drug discovery material synthesis. However, these simulations can be computationally expensive and time consuming despite the approximations through Density Functional Theory (DFT). We propose the Valence Interaction Message Passing Neural Network (VIMPNN) approximate DFT's potential energy calculations. VIMPNN integrates prior physics knowledge, namely the information exchange between atoms being driven and modulated by different interatomic bonds, and the relevance of specific physical properties to the problem of estimating more accurate energies. Furthermore, while many previous machine learning methods consider only stable systems, our proposed method is demonstrated on unstable systems different interatomic distances. VIMPNN predictions can be used to determine the stable locations for atoms, a necessary step for the future simulation of crystal growth. Our method extensively evaluated on an augmented version of the QM9 dataset that includes unstable molecules, well new datasets of periodic and growing crystals. VIMPNN has comparable accuracy with DFT, while allowing for orders accurate and
of magnitude in computational speed up. It also produces informative potential energy curves than Message Passing Neural Network.
INTRODUCTION
Chemical simulations have many useful industrial applications ranging from drug discovery to the production of materials for daily use (Schütt et al.) (017). Simulating crystal systems in particular provides useful properties such as surface absorption, chemical reactions, and surface magnetism (Bilek & Skála 1978). Quantum Mechanical (QM) simulations can be used for the calculation of potential energies based upon the interaction of atoms. By simulating a chemical system at different interatomic distances, it is possible determine stable configuration where the atomic interaction is at an equilibrium (minimum potential energy). While QM simulations can be less time consuming than physical experimentation, they typically require large amounts of computing resources and do not scale well to larger system sizes (Jiang et al. 2003 Erba et al. 2017).
To address this difficultly in determining the potential energy, the Shann Density Functional Theory (DFT) may be used to simplify the calculations by considering the electronic density in good trade-off between speed of computation and chemical accuracy (Cohen et al., 2012). However, as DFT calculations are proportional the number of interacting electrons (Lanyon al., 2010). computation for large systems remains intractable. As many interesting and realistic systems are formed from large number of atoms, a computationally efficient and accurate method for chemical property estimation that scales well to these large systems would prove significantly useful in practical applications (Gomes) ctt al. 2008). Moreover, several simulations are often necessary to identify stable configurations away from local minima. Thus, fast approximation methods are desirable in such scenarios.
place of individual electrons. DFT has proved useful in QM due
Machine learning (ML) was proposed to efficiently classify (stable) molecules and estimate chemical properties by reproducing the results of DFT, with recent advancements through a Message Passing Neural Network (MPNN) (Gimer et al. 2017). However, unstable molecules and crystals are often not considered, although this would allow discovering new stable configurations.

---

Under review as a conference paper at ICLR 2020
In this article we demonstrate the applicability of ML for the energy prediction of unstable molecular and crystalline systems, where the atoms are at positions which are not at the DFT calculated minimum of potential energy, in the following referred to either "unstable" or "out-of-equilibriimm These energies are determined at the electronic ground-state at given positions of atoms for static systems. We propose new DNN framework, the VIMPNN (Valence Interaction Message Passing Neural Network), that considers the different interatomic interactions driven by different valences, It produces comparable accuracy to that of DFT while also improving the computation time by 5 orders of magnitude. We demonstrate that our method also produces more accurate energy estimations than that of MPNN, while being 30% slower 0.007 seconds per energy estimate, which is marginal compared to the gain from DFT and may be improved with some obvious code optimisation.
MI. in quantum chemistry requires the availability of large datasets of chemical structures and their (measured or computed) chemical properties. Online repositories such as Quantum Machine store a collection of datasets resulting from various QM simulations. QM9 (Blum Reymond 20099] Montavon et al. 2013) is one such dataset containing 134K molecules using carbon, hydrogen, oxygen, nttrogen, and fluorine atoms. However, as all molecules in this dataset are at their stable configuration, it is not suitable for designing and testing accurate energy prediction methods for unstable systems. We address this shortcoming by augmenting the QM9 dataset with out-of equilibiiu configurations for 10K of its molecules a [90%, 150%] range of interatomic distances. In addition, create two datasets composed of infinite-size periodic crystals and finite-size growing crystals of aluminium (Al) and copper (Cu) atoms.
In summary, the contributions of this work are:
A new VIMPNN model that accounts for the physics of atomic bonds within a molecule crystal to improve the estimation of chemical properties. This model also introduces auxiliary chemical property estimations to help learn descriptors that are closer to the physics of the problem.
A new use of ML for potential energy estimation of unstable molecules and crystals.
The public release of new periodic crystal and growing crystal datasets and an augmented QM9 dataset.
The remainder of this article is organised as follows: Section 2 properties estimation by ML. We describe the creation of new datasets Section 3 In Section we introduce our new physics informed DNN for accurate energy estimation, which we evaluate in Section 5
discusses previous works on molecular
We conclude in Section K
2 RELATED WORK
Several MI. methods have been used for estimating a variety of chemical properties from different representations of chemical systems. [L et al. (2015) constructed a covariance matrix from Fuclidean distances of atoms within molecules, then use Bayesian regression to estimate interatomic forces. Wang et al. (2013) used kernel ridge regression on the numbers of coarse 'building blocks' (such as CH2) in molecules to estimate many chemical properties such electron affinity and atomization energy. Recent research by Shi et al. (2019) used strain information and a Fourier transform- based representation of crystal lattice for ferw-rorward neural network (NN) estimate an electronic bandgap structure in silicon. The bandgap refers to the energy difference between the valence and conduction bands in insulators and semiconductors, and therefore the energy required to transition between stable excitation states. Silicon crystal in a known equilibruum state is strained in the range of -10% to 10% in each strain component (such as thermal properties) search the bandgap space for another stable state, for example transforming silicon from semiconductor to metal. Therefore, this work uses the NN to describe the bandgap as a function of the strain tensors. In contrast, our method infers the effect of spatial deformations on a system's 's potential energy.
Some approaches focused on the prediction of stable geometries for chemical systems. Timoshenko et al. (2018) used experimental spectra data as input for an NN to estimate the probability distribution of the stable bonding distance between pairs of atoms. They considered different combinations of
http://quantum-nna ne.
2

---

Under review as a conference paper at ICLR 2020
Pd and Au atoms (i.e. NN for Au-Au bonds, another for Pd-Au). When considering crystals made of two different atom types, Takahashi & Takahashi (2019) used quantities associated with each atom type (such as atomic radūi, electronegativity, and number of atoms) to perform a random forest classification on how the resulting stable crystal structure would form among set of 492 different possible structures. The predicted lattice type requires further optimísation using DFT to find its exact size/scaling. However, this method may also predict metastable configurations (i.e. local minima). Whereas the aforementioned methods attempted directly obtain the stable distance between atoms, we estimate potential energies for various geometries with the aim to minimise them into stable configurations. While this approach may require more training data and more complex modelling to handle arbitrary configurations, we argue that it has stronger generalisation potential. also has the added benefit of allowing the (future) observation of progressive crystal growth.
Previous works on estimating the potential energy of chemical systems include that of Rupp et al. (2012) who used a feed-forward NN on the Coulomb matrix (a pairwise matrix that describes the low-level electrostatic interaction between atoms) of molecules. However, predictions
this matrix are sensitive to permutations of atoms which can result in different property values being estimated invariant solution by training the NN
for same crystal. Montavon et al. (2012) provided of randomly permuted matrices. To show the benefit of their method, they train different ML models on the QM7 dataset and improved advancement by Gilmer et al.] (2017) provided more versatile representation for molecules as a chemical graph where nodes describe atoms and edges encode the distances and bond type between them. Their Message Passing Neural Network (MPNN) simulates the atomic interactions through the passing of messages between the nodes. So far, this work has only been evaluated on stable configurations of molecules from the QM9 dataset. We further evaluate MPNN on unstable molecules and crystals. Furthermore, we extend on their approach allow for more accurate predictions by better accounting for the physics of atomic interactions.
set
the accuracy of previous approaches by a factor of 3. A recent
DATASETS
Many previous ML approaches and associated datasets only consider the case of molecules and crystals at their stable configuration, therefore learning the interaction of atoms only at an equilibrium state. The estimation of potential energies for unstable systems needs to be supported by a dataset of unstable systems. We created such datasets train and benchmark VIMPNN. The first is an augmented QM9 dataset (Section 3.1) to investigate the complexity of atomic interactions from variety of atom types in small but diverse molecules. (Section 3.2) allow learning regular bonding patterns that arise in periodic structures. Thirdly, dataset containing growing crystals of an increasing (finite) size and complexity enables experimenting with large scale interactions (Section 3.3
the second dataset, infinite-size crystals
3.1 AUGMENTED QM9 DATASET
An augmented QM9 dataset created by taking the first 10,000 molecules of QM9 and modifying the interatomic distances at 10 regular intervals between 90 and 150% of the original stable configuration. This dataset therefore contains 100K different systems. At each interval, the potential energy is calculated using D to serve the target energy for our supervised learning task.
3.2 INFINITE CRYSTAL DATASET
Learning to estimate the potential energy for an infinitely sized crystal might benefit from the regular pattern in the lattice structure, possibly reducing the complexity of modelling for the MI. algorithm. dataset of infinite-sized crystals, although not necessarily relevant from a physical point of view, is therefore an interesting case study for the design and training of ML methods. We use the simple Face-Centred Cubic (fec) Bravais lattice illustrated in Fig. left for both Al and Cu atoms. We iteratively compress/dilate the cell size (thus, the interatomic distances) and compute the potential energies using DFT at 20,000 uniform intervals from 90-150% of the stable configuration. The change of atomic distances are performed isometrically (i.e. the same change in all spatial axes).
We used CP2K hhttps: http:://wwwwwpp.. org// calculate the potential energies with DFT.
3

---

Under review as a conference paper at ICLR 2020
Figure 1: Crystal structures for the Infinite Crystal (left) and Crystal Growth (centre: seed, right: growing) datasets.
Auxiliary Properties
Estimations
Optimal Distance
Stable Configuration
Prewss
Tota Enery (al
Single Gond
Mc,d)
Double Bond
Readour
Lowes:
Energy
Update internal states through specialised message passing
Potential energies
Atom Coordinates & Bond Types
Input Graph representation
Parallel readout functions
estimated at different
interatomic distances
Figure 2: Outline VIMPNN: physics-informed informed DNN for estimating potential energies from a graph representation of chemical systems.
3.3 CRYSTAL GROWTH DATASET
In contrast to the regular lattice pattern created by an infinite crystal, crystal growth allows for learning over more complex atomic interactions in non-regular systems. Starting from basic fee crystal seed of 14 atoms (Fig. centre), new crystalline systems are generated by iteratively placing atoms random location the surface of the growing crystal following its lattice pattern (Fig. right), with sizes ranging from 15 to 14 atoms. Such dataset enables evaluation of an ML method's ability learn how each atom contributes the final potential energy.
We generate two subsets of this dataset: one that includes compressionddilation of the interatomic distances regular intervals between 90 150% of the stable distance, referred the Unstable Crystal Growth subset, and another that contains only stable configurations, referred to the Stable Crystal Growth subset. The latter aims to evaluate an ML model's capability to estimate energy values varying system sizes (i.e. number of atoms) without the added complexity of learning energy as function distance. Both subsets contain Al and Cu crystalline systems. For the Stable subset, we 20 random seeds for each atom type, thus creating 40 varied crystal growths and 4,000 crystalline systems. For the Unstable subset, we select 5 of these random seeds for each atom type to consider 1,000 basic configurations, each being compresseddtilated for total of 10,000 crystalline systems.
4 METHODOLOGG
VIMPNN estimates the potential energies for molecular or crystaline systems through learning the energy as a function of the geometry of bonded atoms. By querying VIMPNN at various interatomic distances, we may find a stable configuration (Fig D VIMPNN extends MPNN (Gimmer et al. 2017) that simulates atomic interactions through the passing of messages between pairs of atoms (Section
However, VIMPNN further accounts for the physics of the system through an architecture that passes information according to the different forms of bonding between atoms (Section 4.2), and through reinforcing the physical relevance of the analysis and learnt features using auxiliary physical properties (Section 4.3) A representation of the VIMPNN model is depicted in Fig.
4

---

Under review as a conference paper at ICLR 2020
4.1 PREREQUISITE: MPNN
MPNN learns to predict potential energy from an undirected graph G. where atoms are nodes and edges enw describe the relation between two nodes and (e.g. their distance). MPNN applies three functions for a fixed number of iterations (recommended between 3 and 8): 1) the Message function (M4 in Eq. II implemented with perceptron) creates a message packet that symbolises the action of neighbouring atom on node v. This action does not replicate the physics of atomic interactions happening in real chemical systems, but it enforces the NN model to process information in physics-inppired way. A final message vector m., for node v is created in Eq. 1 by combining the actions of all neighbours. 2) An update function (U, Eq. 2 updates the hidden state hi, of node v using its previous hidden state ht- and message m It accounts for the effect of atomic interactions on the (information) properties of the node's atom. It is implemented by a Gated Recurrent Unit (GRU). 3) The Readout function (R in Eq. 3] implemented with a perceptron) uses the set of hidden state of all nodes at all timesteps to estimate the system's 's potential energy. Thus, it exploits the properties of atoms and of their exchanges of information, to learn to estimate the energy through minimising a mean-squared error loss.
m
m
Σ wEN(v)
Me (Niws Cvw)
h+t1 Ur (h m 1
(2)
Venergy = R (( lV E G})
(3)
Full details on the implementation of these three functions can be found in the original paper (sce Gilmer et al. (2017). Next, we detail how VIMPNN adapts these functions to further incorporate
physics principles in order to better handle out-of-quuilbrium systems.
4.2 ACCOUNTING FOR BOND TYPES
The type of bonds between different atoms are important factor when considering their interaction and its contribution to the energy of the system. Chemical bonds between atoms can be characterised by the number of valence electrons exchanged in the process of bonding. For example, ionic bonding requires one donor electron and one acceptor, while in situations where the bonding atoms have similar electronegativity, some electrons would be shared between the bonding atoms resulting in a covalent bond. Moreover, most atomic systems have minimum potential energy (stable bond) at an optimal atom separation (Feinberg & Ruedenberg 197I). Therefore, the potential energy of a system may be better estimated by accounting for the contribution of different bond types. Gilmer et al. (017)) acknowledged this fact and introduced bond type information as a feature in the edges eew of the undirected graph representing the chemical system. This information was used together with interatomic distance by the message function M, in Eq. to estimate the influence of neighbouring atoms node. Thus, messages exchanged by the NN nodes are impacted by the bond type. Gilmer 2017) demonstrated this approach results in better energy estimates over using distance alone. et al.
These encouraging results inspired us take this physics integratioon principle further and to modify the design of information exchange within the NN to better follow the physics of atomic interactions. We propose to introduce different specialised communication channels based on bond rype, with a direct effect on the update of nodes (Eq. 2}. This is supported by separate messages m.l for each bond type, computed as in Eq. T to combine all message packets of same bond type. Bond type is quickly predetermined based on the atoms' valency and electronegativity using the RDKit softward" In practice, the principle having different messaging channels and node update based on bond type may be implemented in various ways. We experimented with: a) combining the messages m. |BT as a weighted sum, where weights are learnt by the NN to control the influence of each mylBT:
my Σ molBT BT ABT
BT
where ÄBT is the weight for bond type BT, and is omitted for readability. We tried with ABT being i) a vector of same size as the message (with an element-wir wise product in Eq. 4 and ii) a simple scalar. The final combined message m. is provided to the GRU of Eq. 2 to compute node
https: wwwp///wwww/.c www.wwwwt.t...kooooo.om... org/
5

---

Under review as a conference paper at ICLR 2020
update where different bond types contribute differently. Another possibilitty is: b) concatenating the messages my before providing them to the GRU of 2 We experimented with i) letting the GRU handle the concatenated m. BT freely, and with ii) imposing similar handling of the different my BT through using the same GRU parameters, duplicated for each my m er with a simple (learnt) scaling to allow different weightings of the m. BT messages in the node update. Finally, also experimented with: c) having one GRU for each bond type, handling its corresponding message my BT. The updates of all GRUs were summed compute the final nodes' update:
1o Σ LUB (bes MolBT), BT
(5)
where UBT is the update function implemented by the GRU associated with bond type BT, and t being again omitted for readability. As demonstrated the Appendix (Table we found that the best results were obtained in the case a.ii) with combining all messages m, BT in simple way while letting the NN weight the influence of each bond type specialised message my [TT
4.3 ESTIMATING AUXILIARY PHYSICAL PROPERTIES
We hypothesise that, by encouraging the VIMPNN's hidden states to relate more to basic physical properties, we may obtain a more accurate energy estimator that generalises better to new systems. In addition, as the bond type is characterised by the valence property of atoms involved in the bonding process, a more physically relevant hidden state may better support the differentiation of messages and node updates per bond type (Section 4.2). In practice, we encourage this greater relationship with physical parameters through the estimation of auxiliary properties through parallel readout functions similar to Eq. 3 We present this strategy as a second and easy way of integrating knowledge (of task-related physical properties) into the model. We experimented with: 1) the number of atoms of each type present in the system, 2) the number of orbitals associated with each atom type a property that is directly relevant to the determination of bond type- -, and 3) probability distribution for the scaling to the stable distance, estimated as Gaussian function. For each auxiliary estimation, a mean-squared error loss term (weighted by an 0.3 hyperparameter) is minimised during training.
5 RESULTS AND DISCUSSION
In this section, the performance impact for each of the physics integration strategies introduced in Section 4 are first measured in turn our augmented QM9 dataset (Section 5.1). The VIMPNN model, that includes all augmentations, is further evaluated on all datasets to demonstrate its versatility on different scenarios (Section 3) using MPNN particular, we test the generalisation ability of VIMPNN by training it on crystalline systems of up to 25 atoms predict the energies of systems of up to 75 atoms (Section (.3) We finally explore the physics relevance and interpretability of VIMPNN by visualising its internal states (Section 5.4).
baseline for comparison (Section (.2)). In
In all experiments, the dataset is split into training, testing, and validation sets in proportions appropriate for the complexity of the data. The Augmented QM9 training set consists of all data augmentations, i.e. 10 compression/dilation operations, for 80% of the QM9 molecules present in the dataset, so 8,000 molecules, totalling 80,000 training samples. Following the same rules, the testing and validation sets each contain 10% of the dataset, so 10,000 samples. As the Infinite Crystal data points are uniformly sampled from their respective range of compression/dilation for training. The rest of data is split evenly between validation and test. For the Crystal Growth dataset, use 60/20/20 split with the same procedure.
dataset has 2 differentt crystal types (Cu and Al crystals), 8
5.1 PHYSICS INTEGRATION PERFORMANCES
We first evaluate the effectiveness of the physics integration strategies listed in Section by comparing them, in turn, against the baseline of no physics integration (i.e. Cow only contains distance and no bond type information) and MPNN (i.e. bond type information used in eow) We report, in Table [ average and standard deviation metries for the maan-absolute error (MAE), the mean-squared error (MSE), and the relative error (RE-
between the true and predicted 9 energy values.
All proposed methods have a positive effect on energy estimation, with the introduction of bond type information (rows 2-3) having particularly positive impact. This confirms Gilmer al. 2017)'s

---

Under review as a conference paper at ICLR 2020
Table 1: Evaluation of the impact of each physics integration strategy on the accuracy of energy
estimation on the Augmented QM9 dataset. Results are presented in the format: mean (std).
Strategy
MAE
MSE
RE
No bond type (BT) information
0.220 (1.169)
1.416 (17.913)
0.0027 (0.013)
MPNN (BT specialised messages)
0.091 (0,476)
0.235 (6.679)
0.0012 (0.005)
BT specialised node updates (case a.ii)
0.067 (0.141)
0.024 (0.406)
0.0009 (0.002)
# atoms
0.171 (0.986)
1.002 (13.940)
0.0021 (0.011)
Auxiliary estimates of
orbitals
0.195 (0.545)
0.335 (7.223)
0.0025 (0.006)
BT distance scaling
0.119 (0.698)
0.501 (9.548)
0.0015 (0.008)
Table 2: Evaluation of MPNN and VIMPNN on all proposed datasets.
Model
Dataset
MAE
MSE
RE
DSC
Augmented QM9
0.091 (0.476)
0.235 (6.679)
0.0012 (0.005)
0.039 (0.056)
Infinite Crystal
0.034 (0.032)
0.002 (0.005)
0.0012 (0.002)
0.201 (0.211)
MPNN
Stable CG
2.906 (4.200)
26.069 (71.750)
0.0047 (0.005)
Unstable CG
3.447 (4.401)
31.231 (71.743)
0.0058 (0.006)
0.497 (0.499)
Augmented QM9
0.065 (0.197)
0.043 (1.093)
0.0008 (0.002)
0.033 (0.047)
Infinite Crystal
0.034 (0.044)
0.003 (0.009)
0.0015 (0.002)
0.048 (0.035)
VIMPNN
Stable CG
0.513 (0.537)
0.552 (1.162)
0.0016 (0.002)
Unstable CG
2.387 (3.557)
18.336 (56.050)
0.0042 (0.004)
0.162 (0.237)
observation that the bond type is a good feature for the estimation of energy. Furthermore, the use of bond type information in the design of the NN architecture (case a. in Section 4.2) has stronger positive impact than using it as a feature in MPNN. This suggests that having specialised messaging channels based on bond type better captures the physics of atomic interactions than just including bond type information in generic messages. However, MPNN does improve significantly on the baseline, therefore in future works it may be interesting to combine these two strategies.
The auxiliary estimation of physical properties also improve on the baseline model, but less so than the integration of bond type information. We interpret this a result of the model having to implicitly discover and encode the useful physics representations required for accurate predictions. Indeed, the auxiliary quantities used are related to the global properties of the molecule, and in the case of the distance scaling to high level system optimisation. In future works, it may be interesting to investigate the effects of auxiliary quantities linked to the physics of individual atoms/nodes. The more modest effect of auxiliary properties may still be beneficial in combination to the bond type strategy, as we demonstrate in the next experiment and the Appendix, thus we continue using them.
5.2 EVALUATION OF VIMPNN ON OUT-OF LUT-OO-EQUILLBRIUM MOLECULAR AND CRYSTAL DATA
In this experiment, we update the baseline model of Section 5.1 with our physics integration strategies, namely bond type-specialised communication channels and node updates (case a.ii of Section 4.2) and all 3 of the auxiliary estimations to create VIMPNN. The energy estimation assessed on the different scenarios provided by our 3 datasets. On the Crystal Growth subsets, we consider system sizes ranging from 15 75 atoms, instead of the available 114, due memory restrictions. The performance of MPNN and VIMPNN are reported in Table Table[]using uusing the same metrics as previously, and the distance to stable configuration (DSC) where the interatomic distance at the minimum of the predicted energies is compared to the known at-equilibrium distance by absolute difference.
Our first observation, when comparing with Table [ is that the auxiliary estimates, which were found to be beneficial to the model in the previous experiment when used in isolation, also prove to be beneficial when used in conjunction with the bond type-specialised communication channels.
VIMPNN generally obtains better scores than MPNN all datasets, meaning that energy estimates are similar or more accurate and that their minimum is closer to the true stable interatomic distance (Fig. 3 left). The improvement over MPNN is particularly strong on the Crystal Growth dataset, hinting that VIMPNN is able to generalise better to new geometries. We also note that MPNN's metrics, especially DSC, suffer from a high standard deviation. Upon examination, find that
7

---

Under review as a conference paper at ICLR 2020
DET sturalated MPNN
Patertial emergy (ax..)
76.5
Polential enarg (aux)
66650
VIMPNN
-7700
-66.75
-77.5
siumlated
67.25
MPNN VIMPNN
-75.0
1.2
14
1.6
1.8
1.0
1.3
1.4
1.5
1.6
Interatoniic distance
Interationic distance
Figure 3: Energy estimations at different interatomic distances for MPNN and VIMPNN. Two examples cases are shown for the C3H403 and C5H7N30 molecules in the Augmented QM9 dataset.
60
MPNN
VIMPNN
VIMPNN
50
Do BIX atom
40
Absolute
30
20
19
AI
Cu
Al
Atom lype
Figure 4: Evaluation of MPNN and VMPNN's ability to handle different atom types.
there exists some more difficult cases" which may explain these results. Indeed, when examining these cases more closely (Fig. right), find that MPNN's estimations are usually close to the true energy values but do not contain a minimum. On the other hand, while VIMPNN's estimates slightly worsened by an offset, it is still able to account for the physies of the system, such as the repulsive effect of atoms getting too close to each other, to predict an energy minimum at a reasonable interatomic distance. The energy offset may be addressed future works, for example by a different loss term or new auxiliary estimates. The Infinite Crystal dataset is the only case where VIMPNN doesn't 't improve on MPNN in term of the accuracy of energy estimation, both methods obtaining similar scores, but with VIMPNN still producing better DSC measures by almost an order of magnitude. As we only consider two mono-atomic infinite crystals in our experiments, we may see further improvements by expanding this dataset to include higher number of structures outside of the fee lattice and wider variety of atom types.
Furthermore, when trained on the Crystal Growth dataset, the results of MPNN on Cu crystals are noticeably worse than for Al (see Fig. 4 We interpret this as MPNN not being able to learn how to handle both crystal types simultaneously. VIMPNN, however, maintains similarly good accuracy in both cases. We hypothesise that the auxiliary estimations of number of each type of atom may help VIMPNN to better differentiate between these atom types. We test this hypothesis by removing the atom-type estimation from VIMPNN. The resulting high errors for Cu crystals demonstrate that the auxiliary estimation has been an effective way integrate knowledge into the model, in this case by directing the attention of the model to atom type, a very relevant physical property for this problem. Other combinations of auxiliary estimations have been tested in the Appendix (Table 4
We further evaluate how VIMPNN scales to larger systems by comparing energy estimates on crystalline systems of various sizes (i.e. number of atoms) from the Stable Crystal Growth subset. The increasing energy errors for larger sizes in Fig. leeft suggest that the model struggles with handling many simultaneous interactions at a higher range of interatomic separations. When examining examples of such systems more closely, e.g. in Fig. 5 we observe that VIMPNN finds an energy minimum the correct stable interatomic distance for the small system (Fig. left, 15 atoms) even the estimated energy has offset, but in extreme cases (Fig right, 75 atoms) the predicted energies
*The cause these more difficult cases is still unknown and will be investigated future work.
8

---

Under review as a conference paper at ICLR 2020
Poteriial energy (a.u.)
30.0
Potential enrry (a.n..)
1400
130
VIMPNN
-30.5
150
160
-31.0
170
VIMPNN DFT sonalated
180
-31.5
190
3.5
4.5
5.5
3.5
4.5
5.0
5.5
Interatomiin distanen
Interatotnic distance
Figure 5: Energy estimations for a small (15 atoms, left) and large (75 atoms, right) crystalline system.
Pearson correlation: 0.635
Pearson correlation: 0.695
12
12
Absolute urror
Absohute srror
9
6
8
3
m
20
50
60
30
10
70
Systeem atoms)
System size (# atorss)
Figure 6: MAE of energy estimations w.r.t system size from the Stable Crystal Growth dataset. VIMPNN is trained on (left) all system sizes of 15-75 atoms, and (right) systems of up to 25 atoms.
no longer present conrectly located minimum. These results indicate that further work is needed towards better accounting for atomic interactions within large scale structures.
5.3 GENERALISATION TO LARGER SYSTEMS
In this section we investigate VIMPNN's ability to learn basic principles of atomic interactions in small chemical systems that are transferable to arbitrary (especially larger) system sizes. This ability would allow for quicker training times while generalising well irrespective of the graph size. To test whether VIMPNN is capable of such task, we train the model on filtered version of the Stable Crystal Growth subset that includes only systems of up to 25 atoms. The validation and testing are done on the full subset, thus the model estimates energies for systems of up to 75 atoms Fig. 6 (right) shows that the MAE increases with the system size above 25 atoms, but this trend only marginally stronger than when the model is trained on all sizes (Fig. left, of the previous experiment). This indicates that VIMPNN is learning some basic principles about the atomic interactions that are transferable larger, unfamiliar, systems. However, for VIMPNN make best use of this positive effect, future work will need to address the overall performance on large system sizes.
5.4 INTERPRETATION OF THE NODES' HIDDEN STATES
When using VIMPNN in-place of DFT, we are placing trust in the model for computational speed with minimal impact in accuracy. While DFT includes approximations to make computations more tractable, it is grounded in fundamental principles of atomic interaction. VIMPNN however makes inferences about the system's 's energy by searching for a generalised function that matches well the input (chemical graph) and output (energy) data. It is therefore helpful to investigate the learnt representations of atomic interactions within the NN, to understand how to interpret and trust the model's 'ss estimates and its ability to generalise to new systems. To do so, we visualise the hidden states of the nodes during the readout phase (step of Section 4. that conveniently reduces the nodes' vector to scalar values before summing to produce the energy estimate. Thus, these reduced
"As in Scction 5.2 larger systems up 14 not used due memory constraints.

---

Under review as a conference paper at ICLR 2020
1,0
-3..0
Normaiised contribut
Potential energy (a.u.
-37.5
-0.8
Potential energy (e.u.)
-37.5
0.850
Nor Normaliane melised mean
-40.0
-40.0
0.845
42.5
0.4
-42.5
0.840
-55.0
Potential energy
Patential energy
-47.5
moving atom
Contribution
0.0
-47.5
Mean contribution of static atoms
0.835
-50.0
-50.0
16
17
18
19
17 coordinate
coordinate
Figure 7: Contributions of moving atom (left) and static crystal (right) toward the energy estimate.
node values may be considered as the contributions of the nodes to the energy estimate, and these values, normalised across the nodes, are referred to as node's 's contribution the rest of this section.
After training VIMPNN on the Unstable Crystal Growth subset, we consider the node of an atom that newly added to a (stable) crystal seed of atoms. When compressing and dilating the distance of this (single) atom to the rest of the crystal, we examine its node contribution and that of the rest of the (static) atoms (Fig. 7 At its stable location, the atom's contribution marginal, in line with the crystal being its minimum energy driven by the regular structure of the crystal. As the atom is moved closer or pulled away, its contribution increases simultaneously with the potential energy. At the same time, the relative contributions of the static atoms slightly decrease to give way to the perturbation of the moving atom. This strongly suggests that the model learnt to pay attention to the location of individual atoms, although it was trained on systems that isometrically compressed/dilatee. This confirms the previous experiment's conclusion that VIMPNN learns transferable principles of atomic interaction.
This visualisation method could inform further improvements in the model, such as related to the aforementioned issues of large range atomic interactions. may also be helpful the simulation of crystal growth where lattice is not imposed, by pinpointing stable atomic positions that entail equal contributions of all atoms the system's energy.
6 CONCLUSION
We have shown how integrating physics knowledge into data driven learning models may help improve accuracy and generalisation. We proposed two integration strategies, namely specialised information passing channels within a DNN to better account for bond type in the simulation of atomic interactions, and estimating auxiliary physical properties that further relate the learnt representations to the underlying physics of the studied phenomenon. Our methods are incorporated into a Valence Interaction Message Passing Neural Network (VIMPNN) that estimates the potential energy both molecular and crystalline systems. The energy estimate is shown to be fine enough to identify stable spatial configurations that correspond to an energy minimum. Three datasets are introduced to benchmark the performance of DNN models: augmented QM9 dataset to include new unstable configurations of molecules, infinite-size periodic crystals, and a more complex stable and unstable Crystal Growth dataset with crystalline systems at varying sizes and different spatial configurations. We evaluate both VIMPNN and MPNN on these datasets, and find that our model performs as well or better than MPNN. VIMPNN proves able learn principles of atomic interactions that are transferable to new system sizes and spatial configurations. However, more work is needed to attain a more robust generalisation larger chemical systems. This may be achieved in future works by integrating more physics knowledge on long-range atomic interactions into the model.
'Note that this scenario is different to previous experiments with compressing/dilating the whole system. Still, VIMPNN produces very plausible energy estimates in Fig.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
O Bilek and I. Skála. From Finite to Infinite Crystals: Analytic Solution of Simple Tight Binding Model of Finite SC, FCC and BCC Crystals of Arbitrary Size. Czechoslovak Journal of Physies. 2809):1003-1019 1978.
Blum and Reymond. 970 Million Druglike Small Molecules for Virtual Screening in the Chemical Universe Database (GDB-13). J. Am. Chem. Soc., 131:8732, 2009.
Cohen, P Mori--Sácheez, and Yang. Challenges for density functional theory. Chemical Reviews, 112(1):28--320, 2012.
Erba, J Baima, Bush, Orlando, R Dovesi, and Erba. Large-Scale Condensed Matter DFT Simulations: Performance and Capabilities of the CRYSTAL Code. Journal of Chemical Theory and Computation, 13(10))5019-5027, 2017.
M Feinberg and K Ruedenberg. Paradoxical Role of the Kinetic-Energy Operator in the Formation of the Covalent Bond. The Journal of Chemical Physics, 54(4):1495-1511, 1971.
J Gilmer, S Schoenholz, P Riley, O Vinyals, and G Dahl. Neural Message Passing for Quantum Chemistry. In International Conference on Machine Learning, pp. 1263-1272, 2017.
A Gomes, Jacob, and L Visscher. Calculation of local excitations in large systems by embedding wave-function theory in 5353 5363,,36,,222 2008
theory. Physical Chemistry Chemical Physics, 10(35):
Jiang, H Baranger, and W Yang. Physical Review B Condensed Matter and Materials Physics, 68(16):1-9, 20000:09,0000.... 2003.
theory simulation of large quantum dots.
Lanyon, J Whitfield, G Gillett, M Goggin, M Almeida, I. Kassal, J. D. Biamonte, M. Mohseni, B. J. Powell, M. Barbieri, A. Sapuru-Guzik, and A. G. White. Towards quantum chemistry quantum computer. Nature Chemistry, 2(2): 106-11,, 2010.
Z Li, J. Kermode, and A De Vita. Molecular dynamics with on-the-fly fly machine learning of quantum- mechanical forces. Physical Review Letters. 1 11((9)1155, 2015.
G Montavon, K Hansen, S Fazli, M Rupp, F Biegler, A Ziehe, A Tkatchenko, OA Von Lilienfeld, and K Müller. Learning Invariant Representations of Molecules for Atomization Energy Prediction. In Advances in Neural Information Processing Systems, pp. 440-448, 2012.
G Montavon, M Rupp. V Gobre, A Vazquez-Mayagoitia, K Hansen, Alexandre Tkatchenko, Klaus Robert Müller, and O. Anatole Von Lilienfeld. Machine learning of molecular electronic properties in chemical compound space. New Journal of Physics, 15:095003, 2013.
M Rupp, Tkatchenko, Müller, and OA Von Lilienfeld. Fast and accurate modeling of molecular atomization energies with machine learning. Physical Review Letters, 108(5):1-5, 2012.
Schütt, F Arbabzadah, Chmiela, K Müller, and Tkatchenko. Quantum-chemice insights from deep tensor neural networks. Nature Communications, 8:6-13, 2017.
Shi, Tsymbalov, M Dao. Suresh, Shapeev, and Ju Li. Deep elastic strain engineering of bandgap through machine learning. National Academy of Sciences, 1 11::4117-412., 2019.
Takahashi and L Takahashi. Creating Machine Learning Daaning--rivee Material Recipes Based on Crystal Structure. The journal of physical chemistry letters, 10:283-288 2019.
J Timoshenko, C Wrasman, Luneau, Shirman, M Cargnello, S Bare, Aizenberg, Friend, and A Frenkel. Probing atomic distributions in mono- and bimetallic nanoparticles by supervised machine learning. Nano Letters, 19:520-529, 2018.
C Wang, R Ramprasad, S Rajasekaran, G Pilania, and X Jiang. Accelerating materials property predictions using machine learning. Scientific Reports, 3(1):16,, 2013.
11

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.
FURTHER ASSESSMENT OF THE PROPOSED PHYSICS INTEGRATION STRATEGIES
Table 3: Comparison of the different implementations of bond type specialised communication
channels (see Section 4.2 tested on the Augmented QM9 dataset.
Implementation
MAE
MSE
RE
a.i 0.1214 (0.2952)
0.1019 (1.1220)
0.0016 (0.0036)
a.ii 0.0670 (0.1408)
0.0243 (0.4059)
0.0009 (0.0016)
b.i 0.1816 (0.5816)
0.3712 (3.8593)
0.0023 (0.0061)
b.ii
0,1079 (0. (
0.0223 (1.1093)
0.0012 (0.0015)
0.1144 (0.1418)
0.0332 (0.0920)
0.0015 (0.0019)
Table 4: Evaluation of the auxiliary estimations combined with the selected bond type integration strategy (a.ii), on the Augmented QM9 dataset.
Auxiliary combination
MAE
MSE
RE
BT # atoms
0.0690 (0. 1921)
0.0409 (0. (0.9880)
0.0009 (0.0021)
BT # orbitals
0.0796 (0. 1357)
0.0220 (0.4670)
0.0009 (0.0015)
BT + distance scaling
0.0742 (0.1450)
0.0265 (0.4647)
0.0010 (0.0017)
A.2 THE EFFECT OF VARYING TIMESTEPS
In all experiments of this paper, three iterations of information passing between nodes were used as the basis for both VIMPNN and MPNN following the default recommendation from Gilmer et al. (2017). However, also experimented with both and timesteps, with results reported in Table We found empirically that iterations works best for VIMPNN as well.
Table 5: Performance on the Augmented QM9 dataset with varying number of timesteps T.
Model
# timesteps
MAE
MSE
RE
0. 0.1860 (0.338)
0. 0.1888 (1.341)
0.0026 (0.004)
VIMPNN
0.0646 10.1
0.0430 (1.093)
0.0008 (0.002)
8
0.0944 (0.1
0.0243 (0.067)
0.0013 (0.001)
0.1737 (0.298)
0.1191 (1.438)
0.0024 (0.003)
MPNN
3
0.0909 (0.476)
0.2348 (6.679)
0.0012 (0.005)
8
0.5550 (0.352)
0.4321 (0.496)
0.0077 (0.005)
A.3. IMPLEMENTATION DETAILS
All models were trained using an Nvidia Tesla V-100 with PyTorch 1.2.0, Python 3.7.2, and CUDA 10.1. Each model was trained for maximum of 360 epochs, with 50 epochs of no validation loss improvement as the criteria for early stopping. During training, the best model (determined by the validation loss) was saved to be used for testing.
The following hyperparameters were used for all experiments:
Batch size: 20 for the Augmented QM9 and Infinite Crystal datasets, and 5 for the Crystal Growth dataset due to memory constraints.
Learning rate: le-4
Learning rate decay: 0.6
Early stopping: 50 epochs
Auxiliary alpha: 0.3
12

---

Under review as a conference paper at ICLR 2020
Size of bond type specialised message mulBr: 73
Number of bond types:
Number of message passing iterations: 3
When training with the different implementations the bond type integration, though the size the bond type specialised message BT remains consistent at the aforementioned 73 dimensions, in case b) the concatenation of bond type messages entails that the nodes' hidden state size 219. For clarity, we detail the size of nodes' hidden state here for each of the methods:
73
b: 219
c: 73
13


Reviews:
[{"review_id": "rkeUP-b8oB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies approximation of the potential energy of molecules by a message passing architecture. The work builds upon [Gilmer et al., 2017] and the contributions are two-fold:\n1) The creation of new datasets to learn and test such architectures on and the augmentation of an existing dataset in order to account for energies at non-equilibrium states.\n2) A proposed modification to the MPNN architecture proposed in [Gilmer et al., 2017], in order to account for physical properties in the message-passing procedure.\nThe performances of the architectures are studied with numerous numerical experiments.\nThe paper is overall well-written and clear.\n\nThe new dataset utility is sound and well-motivated. Unfortunately I can not further motivate upon this, as I am not familiar with this area.\n\nFrom the point of view of the proposed architecture, the work is quite incremental. The bond type information, previously included as feature, is now transferred to an architectural modification. On the other hand, many different modifications (although no substantially different from each other) are proposed and tested (although no results about the different modifications are reported - it could be nice to have them in an appendix). This motivates the ‘weakly accepted’.\n\nThe authors also considered the idea of adding additional learning modules (and a related loss) to help the model learn more 'physics interpretable' hidden states. While it does not seem to give notable gains here, it is an interesting idea and I believe deserves further experimentations in the future.\n\nThe experiments are numerous and various, and they offer a very good overview on the goodness of the model (and its limitations). They first compare with the baseline on the augmented dataset, and they show notable gains on the MPNN baseline. The ability of the network to reproduce the energy curve at different interatomic distances is then studied on the different dataset and in different settings, showing gains over the baseline. The authors also report some negative results and experimental interpretation of the model hidden states, which are also an important contribution in my opinion. \n\nFurther comments:\n\n1. No details are given about the training of the models. I think a small paragraph (or larger and reported in the appendix) should be added.\n\n2. Even if it builds upon previous work, the (VI)MPNN model may be further explained. For example, what type of functions are M_t and R? The explanation on the considered modifications of MPNN may be clearer (maybe with the introduction of a more mathematical notation). \n\n3. How long is the message diffusion (T)? What is the effect of larger / smaller T’s?\n\n4. What’s the point of equations (5) (6) (7)? They are exactly the same and they do not add any information. It would be more useful to explain what type of function R is in my opinion.\n\n5. Table 1, Auxiliary estimates: Are these the results obtained by the model a.ii trained to jointly learn the energy and the properties i) ii) iii) ? In what sense they improve on the baseline? This part was not clear to me.\n\n6. Section 5.2: ‘[…] we combine our proposed physics integration strategies, namely bond type\nspecialised\nnode updates (case a.ii) of Section 4.2) and auxiliary estimations of physical properties,\ninto the VIMPNN model […]’. I do not understand this sentence. Isn’t in fact the VIMPNN architecture the same as MPNN with the modification a.ii? In what sense do you integrate a.ii in it?\n\n7. If I understood correctly, the main final objective is to be able to characterize the minima of the energy. In this case, could you asses the performances of the two methods (MPNN) and (VIMPNN) by measuring some kind of distance from the approximated minima from the actual one?\n\nTypos:\n\nSection 5.3 first line: ‘We investigate the VIMPNN’s the ability …’ -> ‘We investigate the VIMPNN’s ability’\n\nSection 5.2 Fourth sentence: ‘The first seeks to demonstrates a the model’s …’ -> ‘The first seeks to demonstrates the model’s…’", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1lomgQcYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the problem of estimating the electronic structure's ground state energy of a given atomic system by means of supervised machine learning, as a fast alternative to conventional explicit methods (DFT).\nThis is done by improving on a previous method, MPNN, Message Passing Neural Networks, and in particular by including information on bond type as input, so that the NN can learn the appropriate weight for messages going through bonds of that type.  In addition, training on several target labels (multi-regression) is attempted, with the idea that more physical outputs may help building better hidden representations (on this, there are mixed results).  Separately, training sets are enriched with non-equilibrium structures, so as to confront the NN with more diverse data.\n\nThe method is tested on 3 kind of training sets. The first is a simple but time-consuming augmentation of QM9, with inter-atomic distances varied, so as to increase the training set's size (and in particular, including non equilibrium configurations). The second consists in a periodic and thus infinite simple crystal structure (with, again, variations in inter-atomic distances, enriching the dataset). The third is a pseudo-cristalline structure with atoms randomly placed on a regular grid, forming a somewhat random structure, also named crystal (this is not a very good name).\n\nThe paper is overall rather well written, sometimes being a bit cumbersome (long sentences), but mostly it is stating clearly what is done or discovered.  The work is situated within the existing literature (that I am not familiar with at all).  The idea of using physics to guide architecture choices is gaining a lot of attention recently and seems to be well-suited to this particular problem, and well applied.  Several ways to use the bond type information have been attempted in this work, and several of them are reported and compared (a couple of them are discarded).  The results convincingly show that using bond type information indeed increases performance, both for small systems and for regular crystals.  The impact of performing multi-regression is less important, but still positive.  For large ''random crystals'', the method does not perform very well, and this represents a challenge for future work. Such a confession on the method's limitations is welcomed.\n\nGiven the idea (using physical information as bond type) is clearly and honestly presented, produces significant improvement compared with previous works, and has perspective for multiple future developments, I recommend acceptation of this paper.\n\n\nThere are however a number of points that could be improved.\n\n1. There is a physical mistake that is not crucial but should be corrected, when talking about the ground state, and in particular before this sentence: ''accurate ground-state energy estimation of out-of-equilibrium molecule''. Ground state means minimal, T=0K energy level, so by definition it is at equilibrium. Thus, the sentence seems quite contradictory to a physicist.\nWhat DFT and VIMPNN actually compute is the electronic structure's ground state's energy (at fixed positions of the atom kernels). I think this distinction should be mentioned just once, and then you could proceed with saying ground state energy. \nBecause of this, I would recommend to edit the title so as to suppress ''out of equilibrium'' from it.  Otherwise readers may think the method deals with non-equilibrium electronic structures (non ground states), which clearly it does not at all, or they may think that it is especially good at estimating energies for out of equilibrium systems, which is not its primary goal.\n\n2. I do not understand very well the training procedure.  Also there are some tests that seem to be interesting and that are not performed (as far as I understood).\nDoes each training set contains the 90%-150% data augmenations, for each non-augmented training configuration ?\nWhy is training performed separately for each kind of data set ? Wouldn't the ultimate goal be to transfer learning from a type to another, e.g. from QM9-style to crystal style, etc ?  (as far as I understood, this was not done)\nIsn't it interesting to see how much training on an augmented data set (let's say QM9) improves performance on the non-augmented data (the ''true data'' in a sense) ? Although the augmented data is obtained by DFT, and comparing models trained on different data sets is unfair, I think it may be interesting to see if VIMPNN benefits more than MPNN from this strategy (so compare the performance gains of both algorithms obtained by augmenting a data set). This kind of comparison may also be done for the ''augmentation'' of a training set by the concatenation of it with another one (although in that case it may be detrimental to the test accuracy?).\nIf you actually did some of this, then I misunderstood and I am sorry, but then this also means you should clarify.\n\n3. Section 3.3:\nI would not call this a crystal, but more something like ''random finite structure''. This should be done everywhere in the paper.\n\n4. Section 4.1 is a bit too short for the inexperienced reader. I suggest to be a bit more explicit on what is learned\n\n5. Section 4.2: It is nice to say you tried other ways, keep that.\nHowever, try to be more explicit on what is shared and what isn't, in the architecture you finally pick. Is lambda(v,w) a common value for all bonds of the same type, like C-C ? Maybe you could provide an example or some more detailed notation to make your choice fully explicit (after all this is the core of the paper).\n\n6. Section 4.3: could you quickly comment on why you don't use more of the 13 physical observables available in QM9 ?\n\n7. Table 1: do the three last lines correspond to ''no BT information + a single auxiliary estimate'' ?  It seems to be the case, but then you say you will continue with the auxiliaries, in addition to the BT information.  Why don't you display the result of using BT information AND the 3 auxiliary estimates ?  If you did, then I misunderstood, but it would also mean you did not explain well enough a counter-intuitive result (which would be that adding auxiliary information actually hurts the performance of the VIMPNN).\n\n8. Please include a couple of explicit numbers of your training/test/validation sets sizes.\n\n9. section 5.2 could be made more concise. In particular, there is no need in repeating what can be seen directly in the figures (stating numbers). It is useful to comment on the meaning of the results however (as you currently do).\n\n10. section 5.4  is a good idea, promising, however it does not really conclude into a very strong statement, and takes essentially 1 full page, which could be used to better clarify the architecture and/or the training procedure (or to reduce towards the ideal page length).\n\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hket-APeFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a number of new / extended datasets for the evaluation of ML-based prediction of energies of unstable systems, as well as a network (VIMPNN) that includes a new and better way of including bond-type information. It is also proposed to use auxiliary losses (predicting other chemical properties).\n\nAlthough I am not an expert in chemistry, the new datasets seem fairly well thought out and their utility is well motivated. The proposed change to the MPNN network architecture is rather simple and hardly physics inspired, but the empirical improvement seems substantial, so this too is a nice contribution. So I have decided to give the “weak accept” rating.\n\nIn 4.2 it is explained how different ways of incorporating bond information were evaluated, and it is stated that “best results were obtained in the case a.ii). However, no results are presented to support this claim, leaving the reader to wonder how rigorous this exploration was. I would suggest systematically evaluating the different options and including the results in an appendix. \n\nIn table 1 results are shown for the MPNN baseline, baseline with specialised node updates a.ii, and with auxiliary estimates. However, combinations of these are not evaluated. Nevertheless, if I understood correctly, the VIMPNN method tested later includes all of the separate improvements. It would be good to include experimental results to motivate this.\n\nAs acknowledged in the paper, the idea of using bond-type information was already in Gilmer et al. Also, I think the different ways of including bond-type explored in this paper are not really informed by physics. The choice for method a.ii is made based on empirical results. This is not a problem in itself, but I would suggest that the authors change the wording to not over-promise on the physics-inspiredness. E.g. the abstract says “VIMPNN integrates prior knowledge such as the existence of different interatomic bonds”, suggesting that there is more prior knowledge being exploited than just bonds.\n\n\nComments:\n\n“It produces comparable accuracy to that of DFT while also improving computation time by 5 orders of magnitude”. \nI assume this speedup is relative to DFT. It would be good to be explicit about that, and also discuss the speed relative to the MPNN baseline (I suppose MPNN and VIMPNN are similar).\n\n“The change of atomic distances are performed isomorphically” - I would say “isometrically”. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of estimating the electronic structure's ground state energy of a given atomic system by means of supervised machine learning, as a fast alternative to conventional explicit methods (DFT). For this purpose, it modifies the neural message-passing architecture to account for further physical properties, and it extends the empirical validation to also include unstable molecules. 

Reviewers acknowledged the valuable experimental setup of this work and the significance of the results in the application domain, but were generally skeptical about the novelty of the machine learning model under study. Ultimately, and given that the main focus of this conference is on Machine Learning methodology, this AC believes this work could be more suitable in a more specialized venue in computational/quantum chemistry. 
Decision: Reject