Anisotropic Message Passing: Graph Neural Networks with Directional and Long-Range Interactions
Authors: Moritz Thürlemann
Sereina Riniker
Venue: ICLR 2023 poster (2023)
Keywords: Message Passing
Graph Neural Networks
Directional
Long-Range
Equivariant
Quantum Chemistry
QM/MM

Abstract: Graph neural networks have shown great potential for the description of a variety of chemical systems.
However, standard message passing does not explicitly account for long-range and directional interactions, for instance due to electrostatics.
In this work, an anisotropic state based on Cartesian multipoles is proposed as an addition to the existing hidden features.
With the anisotropic state, message passing can be modified to explicitly account for directional interactions.
Compared to existing models, this modification results in relatively little additional computational cost. 
Most importantly, the proposed formalism offers as a distinct advantage the seamless integration of (1) anisotropic long-range interactions, (2) interactions with surrounding fields and particles that are not part of the graph, and (3) the fast multipole method.
As an exemplary use case, the application to quantum mechanics/molecular mechanics (QM/MM) systems is demonstrated.

Published as a conference paper at ICLR 2023
ANISOTROPIC MESSAGE PASSING: GRAPH NEURAL NETWORKS WITH DIRECTIONAL AND LONG-RANGE
INTERACTIONS
Moritz Thürlemann, Sereina Riniker
Department of Chemistry and Applied Biosciences ETH Zürich
Wadimir-Prelog- Weg 2, 8093 Zürich, Switzerland peritzt@ethz.cco srinikereethh.co
ABSTRACT
Graph neural networks have shown great potential for the description of a vari- ety of chemical systems. However, standard message passing does not explicitly account for long-range and directional interactions, for instance due to electrostat- ies. In this work, an anisotropic state based Cartesian multipoles is proposed as an addition to the existing hidden features. With the anisotropic state, message passing can be modified to explicitly account for directional interactions. Com- pared to existing models, this modification results in relatively little additional computational cost. Most importantly. the proposed formalism offers as a distinct advantage the seamless integration of (1) anisotropic long-range interactions, (2) interactions with surrounding fields and particles that are not part of the graph, and (3) the fast multipole method. As an exemplary use case, the application quantum mechanics/molccclar mechanics (QM/MM) systems is demonstrated.
INTRODUCTION
Message passing graph neural networks (GNN) have shown great potential for the description of a wide range of chemical systems (Scarselli et al. 2009 Battaglia al.) 2016 2018). Particularly the description of quantum molecular (QM) systems with machine learning (ML) potentials has received a of interest (Gilmer et al. 2017 Unke et al. 2021b). However, its general form, message passing does not explicitly account for directionality. which plays an important role in many physical interactions (Glotzer & Solomon, 2007] Kramer et al.) 2014). In recent years, range of models which include directional information have been proposed (Anderson et al. 2019 Klicpera al. 2020; Miller et al., 2020 Schüit et al., 202T). Especially models based on Clebsch- Gordan tensor products have shown superior data efficiency (Batzner et al., 2022 Batatia et al. 20226 Musaclian et al., 2022). However, the relatively high cost of these operations might hinder the application to larger systems such biomolecules in solution. This difficulty is compounded by growing evidence that message passing cannot accurately resolve long-range interactions (Alon) & Yahav 2020; Dwivedi et al.. 2022). Note that consider here interactions long-range convergence in real space is slow or non-existent, e.g. electrostatic interactions or polarization.
In light of these challenges, which are exemplary illustrated in Figure I propose a model with the aim to (1) include directional information while (2) retaining computational efficiency and (3) incorporating (anisotropic) long-range interactions. Specifically, the addition of an anisotropic state to the existing hidden features is proposed. This anisotropic state is based on Cartesian multipoles and expressed as linear combination of local frames. As a result, these multipoles are equivariant under rotations and can be used to describe anisotropy of interactions. The formalism is developed analogously to the concept of atomic multipoles commonly used in computational chemistry (Stone) 2013). The proposed modification results in relatively little computational overhead compared to standard message passing models. Most importantly, the formulation based on multipoles allows for the hybrid treatment of particles and fields. This is of particular interest for two cases: (1) Systems of particles embedded an external field, and (2) systems with large numbers of particles where long-range interactions may be treated with the fast multipole method (Rokhlin) 1985).

---

Published as a conference paper at ICLR 2023
As practical case, consider quantum mechanics/moleculle mechanics (QM/MM) simula- tions (Warshel & Levitt, 1976). In QM/MM simulations, a QM system embedded in an external electrostatic field generated by MM particles, allowing for an efficient description of large systems, for instance protein-ligand complexes or enzymatic reactions in solution. As such, the QM/MM formalism might be ideally suited to apply MI potentials to extended systems. Consequently, ML/MM formalism within the message passing framework is formulated analogous to QM/MM.
B
D
Exemplary situations arising in molecular systems for which message passing with dis- tance labelled edges fails. (A): Rotation of one molecule by angle cannot be recognized without adding more edges. However, this change can be resolved through the addition of a dipole (blue ar- row). (B): Similarly, rotation by angle around the axis described by the aligned dipoles ('torsion') ) can be resolved by including quadrupole (blue/red crossed artows). (C): An electric field caused by charges (+) surrounding the molecule. While the electrostatic interaction addi- tive, the polarization on the black atom due to the opposing charges cancels out in the left case but adds up in the right case. (D): 'Long-range' interactions (dashed lines): Message passing can nei- ther distinguish distance nor direction for the interactions between the red and the two blue-dotted particles.
Figure 1:
RELATED WORK
ML potentials have sparked interest possible solution to the steeply scaling computational cost of quantum chemical methods (Schuch & Verstraete 2009 Unke et al. 20216). with nigh-dimensionss neural network potentials being some the earliest proposed methods (Behler 201T). With the introduction of GNN (Scarselli et al. 2009 BBttaglia et al. 2016 2018), the development of ML po- tentials has shifted away from handcrafted features to end-to-end differentiable models which learn features solely from distances and element types (Gilmer et al. 2017| Unke et al.) Tb2 Rec- ognizing that GNN may fail to distinguish certain graphs has spurred the development of modified message passing schemes (Morris et al., 2019; Maron et al. 2019 Pozdnyakov & Ceriotti 2022). As a solution, the explicit integration of bany-body interactions in the message passing (Kondor 2018 Klicpera al. 2020 Shui & Karypis 2020 Zhho et al.) (2021), directional messages (Ander- son et al. 2019 Schütt et al. (001). and combinations thereof (Gasteiger et al. 2021 Batatia et al,
20226) have been proposed. Building on the concept of equivariant CNNs (Cohen Welling 2016
Weiler et al. 2018), equivariant message passing models based on Clebsch-Gordan tensor products
have received increased attention (Anderson al., 2019 Miller et 2020 Satorras et al. 1021
Brandstetter et al. 2021 Batzner et al.) 2022] Musaclian et al. (002) A categorization based on employed architectural features was recently proposed by Batatia et al. (2022a). offering succinct overview. Deficiencies in the description of long-range interactions have led to models that include explicit interaction terms (Unke Meuwly| 2019 Ko et al. 2021). as well as models that integrate this information in the representation itself (Grisafi & Ceriotti 2019 Grisafi et al. 202 Unke al.) 2021a). As an alternative to the ML potentials mentioned so far, ML within the QM/MM formalism might facilitate the description of condensed phase systems. Recent work has highlighted the chal- lenges and potential of ML for the simulation of QM/MM systems (Zhang et al. 2018 Böselt et al.
2021 Pan al. 2021 Hofstetter et al. 2022 Giese et al. 2022).
2

---

Published as a conference paper at ICLR 2023
3 METHODS
3.1 MESSAGE PASSING GRAPH NEURAL NETWORKS
Message passing GNN process graph-structured data. In its commonly used form, node, edge, and/or global features are iteratively refined. Various models have been proposed in recent years with differences found in the construction of the graph and the employed message passing mecha- nism (Scarselli et al. 2009; Battaglia et al. 2016 Gilmer et al., 2017 Battaglia et al.) 2018).
Considering a graph G (V,8) with nodes v, E and edges Eij E E, message passing can be defined as
4+ = oh
Σ JEN(O
&e(h', R Uiz))
with h IR" describing the hidden-feature vector of node v; after / iterations, Wi IK" the edge feature of the edge Eij between node i and j. .ii)) denoting the set of neighbours of Us and de and dh describing edge and node update functions. The superscript denotes the current message passing iteration with n being the total number of message passing layers.
In the following paragraphs, we will refer QM and MM particles. In the context of this work, QM particles are nodes v; of graph G, which are labelled with hidden feature h,. MM particles, on the other hand, are not part of the graph, and therefore do not hold hidden feature. Instead, only a scalar partial charge yi (monopole) is attached to MM particles. More generally, MM particles can be understood as specific example source of an external field in which the graph embedded. In addition, both QM and MM particles hold position E R3 in Cartesian space.
3.2 THE ANISOTROPIC STATE
The present work proposes the introduction of an internal anisotropic state M, composed of Carte- sian multipoles of order to the existing hidden features. Cartesian multipoles are ideally suited for this task they result in relatively little additional computational costs and conceptual complexity. Specifically, M E R33 refers the k-th moment of the anisotropic state of node i composed of the monopole MD. the dipole M, and so on. For clarity, the iteration superscript will be omitted. However, in practice, the anisotropic state can be updated during each iteration much like the hidden feature h/.
The anisotropic state is expressed as linear combination of a local basis (Thürlemann et al. 2022) This formulation preserves equivariance under rotations and invariance to translations. As the local basis, the traceless tensor product of order lt of the unit vector î. pointing from the position of node i the position of its neighbour is used
rig riy 8 ...
(2)
with i referring to the order of the multipole and R 1. Each component of order of the anisotropic state M is then expressed as
M
Σ R
(3)
(EN()
with coefficients et predicted for each interaction as
= (M( (h, hy, Uzy)
(4)
where (b)()) refers learnable function. The scalar coefficient ct is based the hidden features of the interacting nodes i and j as well as an edge feature uij- The coefficient ct; can be interpreted as the contribution of the node j to the polarization of order & of node i analogous to (hyper- ppolarizability. We note that the multipoles discussed here serve as a tool to introduce anisotropy. Albeit possible, they do not necessarily carry physical meaning.
3

---

Published as a conference paper at ICLR 2023
3.3 ANISOTROPIC INTERACTION FEATURES
Given the anisotropic state M, proposed in the previous section, multipole-multipol interactions can be defined. Formulation of the multipole interaction follows the notation and formalism described in (Smith) 1998. Lin 2015; Burnham & English 2020) which decomposes the interaction between multipoles on distinct sites into product radial component by (1]y)) and coefficients of the multipole interactions giy (ri)), resulting in the following terms
Waj
(((rrrill) (riy)
(5)
with by (x[ ((]l) (2k 11)1//1rr 2 In the context machine learning, by can be reinterpreted filters analogous to the continuous-fille convolutions introduced by |Schütt al.] (2018). Con- sequently, the coefficients zi encode the angular information based on the relative orientation of
the multipoles centered on the interacting nodes. In the notation of Burnham English (2020), the
coefficients can be generated with
g
A d,td+
(M)detde) d. (M) ( +de) d).R
(6)
given multipoles M (d++de) of order (dj + de) site and the previously defined traceless tensor products of the vector fig. + refers to the concatenation the scalar coefficients. All d, dj, de where d,y +d k and di, dj, de 2 0 are included. d denotes the number of contractions, with d; being the number of contractions over the indices of the first bracket, d, the number contractions in the second bracket and de the number of contractions between the two brackets, indicated by (..-)- Analogous to the multipole interaction defined in Eq. 5 the anisotropic feature aij is defined as
ai ob (r G(faj, M., M,)
where we use to denote the aforementioned element-wise multiplication with radial weights gen- crated by db and G(ft) M,, M2) for the concatenated multipole interaction coefficients g of all considered orders k.
3.4 ANISOTROPIC MESSAGE PASSING (AMP)
Based on the anisotropic feature defined in the previous section, the modified message passing is obtained
M Σ
(sscky(h's hJy anj) R
JEN())
a.j = @o(1lrggl o Gay(Fig, M. M,)
(8)
h !+1 (n Σ
Se(hl, h/y a,)))
JEN()
Thus, two additional steps are added to the standard message passing formalism: (1) Expansion of the multipoles on each node, and (2) contraction of the multipoles for each pair interacting nodes. The resulting feature is used to incorporate directional information in the message maj. For the use as a ML potential, the total potential energy is expressed as a sum of atom- based contributions of each QM particle, i.c. VeM -QQ Φv (h,"). We will refer to this model as AMP(k) where k denotes the highest order of the multipoles used and is the number of graph layers.
3.5 QUANTUM MECHANICS/MOLECULAN MECHANICS (QM/MM)
QM/MM has become integral part of computational chemistry since its inception in 1976 (Warshel & Levitt 1976). QM/MM offers an efficient way to incorporate interactions with an environment, for instance solvent molecules and/or proteins, through combination of a QM and a classical (MM) description of a system. As such it retains the fidelity of QM-based methods for the region of interest, e.g. the solute, and the computational efficiency of classical force fields for the surroundings. Here, a brief description of the electrostatic embedding, one of several possible QM/MM schemes, is given. We refer the interested reader to Senn & Thiel (2007) for further details.
4

---

Published as a conference paper at ICLR 2023
Within an electrostatic embedding, designated QM zone interacts with the partial charges of the surrounding MM particles. For this purpose, the molecular Hamiltonian Hoa is extended by an interaction with the MM zone
HQM/MM- - Σ
MM EL
Σ
r. ryll ΣΣ
4i
QM M.
Z:43
r. ryl
where qi are the partial charges of the MM zone and Z; refer to the core charges of the nuclei. MM refers to the MM particles, EL to the electrons in the QM zone and QM to the QM nuclei. While the first term incorporates the interaction of the surrounding partial charges analogous to the core- electron interaction, the second term describes the Coulomb interaction between the nuclei within the QM zone and the partial charges. In other words, the QM system interacts with point charges representing the MM particles. The point charges polarize the QM system and interact with the resulting charge distribution. Remaining interactions between QM and MM particles, i.c. exchange- repulsion and dispersion, are described with classical force-field terms such as the Lennard-Jones potential (Jones & Chapman 1924).
3.6 QM/MM FOR MESSAGE PASSING: DIRECT POLARIZATION
As synthesis of the two previous sections, we propose a further modification to anisotropic message passing, which incorporates interactions with a surrounding field or particles that are not part of the graph.
Σ
(Mar (h h/ any R
Σ (EN
MMar (1l any R
(10)
JENqar(1)
Here, NQa (i) refers to the neighbours of i in the QM zone and (i) to the neighbours of in the MM zone. In addition to the previously described polarization due to neighbouring atoms that are part of the graph, second term is introduced, which adds contribution caused by the MM par- ticles. The second term is conditioned on the hidden feature of the QM particle and the anisotropic feature azj. which is based on the multipoles on the QM particle and the partial charge on the MM particle. Evidently, this formulation permits also the application to polarizable embeddings schemes, use with the fast multipole method, and use of higher-order multipoles. We note that MM particles contribute only to the polarization of multipoles with order k 2 1, analogous to the defi nition of polarr-ppoarzzabiiiiiiss (Stone 2013). Since MM particles are only labelled with charge and not with hidden feature h". the interaction depends solely on the feature of the QM particle and the anisotropic interaction feature a.j. which encodes the strength of the interaction and the relative orientation of the MM particle with respect to the polarized QM particle. The advantages of the proposed interaction mechanism are twofold: First, the evaluation of messages between MM particles and between QM and MM particles is avoided. Instead, only the cheaper polarization term evaluated for the more numerous QM-MM interactions. Second, directional and long-range in- teractions are efficiently incorporated in addition to the proper cancellation of opposing polarization terms. The total potential energy is then obtained as the atomm based contribution due to the QM particles described previously and contribution due to the Coulomb interaction between QM and MM particles.
QM MM (pph)) -g |sylll
VOMIMM,ESP = ΣΣ
(11)
where qj denotes the partial charge of the respective MM particle and (p?) refers to a scalar charge density localized on the respective QM particle, analogous to discretized Coulomb interac- tion. In other words, a pairwise interaction between each QM atom and its neighbours in the MM zone is added. In principle. (p aig) may be used to incorporate anisotropy. Here, only the hidden feature h2 is used. Thus, asymptotically correct long-range behaviour can be guaranteed by enforcing that the resulting scalar charges conserve total charge within the QM zone by subtraction of the mean excess charge, i.e. LQ (h?) = 0 for neutral systems. Expansion of @, (h?)) it- self may be an alternative route to obtain anisotropic QM/MM electrostatic interactions, as shown by Grisañ& & Ceriotti (2019). The forces F, acting on the QM and MM particles are obtained as the derivative of the negative total energy (VOM) M, MSP + VQar) with respect to the positions r, of the QM and MM particles, respectively. We will refer to models that include this direct polarization terms AMP(k)-D. Use of the pairwise long-range potential in Eq. 11 will be denoted by '-P'.
5

---

Published as a conference paper at ICLR 2023
4 RESULTS
Results are split into three parts: First, two model systems are investigated to explore the proposed modifications. Second, the model applied to an existing dataset of QM/MM systems. Finally, results for QM9 are reported as comparison with existing models.
4.1 WATER DIMERS: DIRECTIONAL FEATURES
As an illustrative example of the discussed concepts, a water dimer as shown in Figure I is inves- tigated. Configurations were generated by rotating one molecule in steps of one degree around the axis described by the aligned dipoles, resulting in total of 360 configurations. The molecules were separated by a distance of 4 A. For such system, three cutoff regimes are significant: (1) The cutoff Teut used to construct the graph is larger than the maximal distance mmax in the system, resulting in a fully-connected graph (Tmax < Teur). (2) The cutoff is larger than the distance Too between the two opposing oxygen atoms but smaller than the maximal distance (roo Yeut < Tmmax). (3) The cutoff is smaller than the distance between the oxygen atoms, resulting in two disconnected graphs (Feut < roo). Particularly case (2) is interesting it can be related torsional degrees of freedom, which play an important role in molecular dynamics.
0.1
0.1
Energy [k]mml]]
k=1
-0.1-
MP2
k=
-0.1
Angle 270
270
270
Angle (")
Angle [']
Figure 2: Potential energy (y-axis) for the quadnupol-qqudrnpole interaction between two wa- ter molecules for the proposed cutoff regimes with Tout = 8.0 (A), Tou = 4.25 (B), and Teut = 2.0 A (C). The relative orientation is indicated by the angle on the æ-axis. The resulting graph topology for each cutoff is shown as inset. The reference (MP2) is shown in grey. Results for AMP(1), AM,( and AMP(2)-DP are shown as orange, red, and blue lines, respectively.
For all results shown in this section, a minimal 1-layer model was used. While the k = model still able to describe the interaction using a cutoff that results in fully connected graph (i.e. case (1) shown in Figure A). marginally larger error MAE (0.32 AMP(I) versus 0.05 J mol AMP(2)) was found for the AMP(() compared to the AMP(2) model. This observation could be an indication that the present task is also more challenging for AMP(() than AMP(2). Learning curves shown in Figure 5 add further evidence to this interpretation. For case (2) shown in Figure 2 the A 1 model (orange line) fails describe the interaction, remains invariant under the considered rotation. This issue can be traced back to the quadrupole- quadrupole interaction of the opposing oxygen atoms, motivating the need for A: > We note that this case is also interesting as it exemplary demonstrates the limitations of two existing directional message passing architectures, i.c. DimeNet and PaiNN Klicpera al. (2020); SShüttt et al.) (2021). These two models seem to fail to describe this ceadrrpool-quadrupor interaction despite the use of directional information. Our interpretation that for DimeNet this is due to the invariance of bond angles under the considered rotation, and for PaiNN because it only uses vectorial information (i.e. 1). Finally for case (3) shown Figure 2C, all message passing models without additional long-range interactions fail (red line), motivating the proposed pairwise interaction and contributions to the polarization (models with suffix 'D' and 'P'', shown in blue). We note that the long-ram range interaction would in principle also include dispersion, which do not address here due to the availability of robust and efficient classical models.
the dipole-dipole interaction
4.2 WATER CLUSTERS: EXTENSIVITY
For this task, clusters of water molecules, ranging in size from six to 20 molecules, were investigated. For each cluster, one molecule randomly selected the QM zone while all other molecules

---

Published as a conference paper at ICLR 2023
treated MM particles. The goal of this task is the prediction of the difference in energy and gradients caused by placing the QM system in the external field generated by the MM particles. A successful model must be able to discriminate between (1) the polarization of the QM zone by the external field, which does not directly scale with cluster size, and (2) the electrostatic interaction, which increases with cluster size. All models were trained 1'289 samples from clusters of size 10. All other cluster sizes were used for testing. Results are shown in Figure 3
Baseline Message Passing: As baseline message passing model, the PaiNN architecture is used. PaiNN was shown to perform accurately on wide variety of tasks through inclusion of directional information (Schütt al., 202T). For our QM/MM task, the MM particles are part of the same graph as the QM particles. However, energies are only predicted for the three QM particles of the water molecule the QM zone.
Ablation Studies Multipole Order and Polarization: In addition, the role of the multipole order k was investigated. To compensate for different sizes of a, due the multipole order, the number of radial weights was adjusted such that the total number of trainable parameters (~ 1M) remained constant.
MAE Gradieat kJ [JJllAA]]
MAE Energ [d/oool
8-
AMP(0)
AMP(T)
6-
AMP(2)
4-
AMP(3)
PaINN
2-
2-
10
12
14
16
Cluster
Cluster Sizee
Figure 3: Mean absolute errors (MAE) with respect to the reference (MP2) for the energy (EoM) and gradients (FOM) of QM/MM water clusters of different sizes. In all cases, one water was treated as a QM system within the electric field of the surrounding charges. All models were trained only on clusters of size 10. Energies are reported in kJ mol and forces in kJ mol A-1
Not surprisingly, AMP(O) fails at this task as it only interacts with MM particles through pairwise Coulomb potential of Eq. [ The results for AMP(0) provide rough estimate of the role of both polarization and anisotropy in the present task. Including the direct polarization (i.e., AMP(0) versus AMP(1)), results in the largest relative improvement. This result demonstrates that the proposed modification allows the model to incorporate information about the surrounding par- ticles without explicit message passing between the QM and MM zone. Interestingly, increasing from 2 to (i.e., AMP(2) versus AMP(3)), seems to diminish the extrapolation capabilities again, notably on the largest clusters. Overall, we find that all models that interact with the MM particles (i.e., AMP(1), AMP(2), AMP(3) and PaiNN), perform comparably on the training set and clusters with similar size, whereas differences in performance arise for much larger and smaller clusters. While the performance of the PaiNN model is comparable to the AMP(2) and AMP(3) models on gradients, larger deviations are observed for energies. This result could be an indication that message passing models like PaiNN may have difficulties to differentiate between the electro- static and polarization interaction in the present task. Building models that differentiate between interaction types might be important step for ML potentials to improve transferability to large and/or condensed-phan systems.
4.3
QM/MM SYSTEMS
As more realistic example, the model was tested on previously published datasets of QM/MM systems introduced by Böselt et al.) (202I). These datasets include QM energies, QM gradients, and MM gradients of five molecules of varying size solvated in water. In this work, focus on the three largest and most challenging systems: uracil, retinoic acid (RA) and the S-adenosyl methionine/cysteine dimer (SAM) with 12, 50 and 49 14 QM atoms, respectively. On average,

---

Published as a conference paper at ICLR 2023
each molecule is surrounded by approximately 1'491 (uracil), 2'391 (RA) and 2'553 (SAM) MM particles. Each water molecule is represented by three partial charges centered on each atom. These datasets are challenging due to the large number of surrounding MM partieles that polarize the QM system, the resulting electrostatic long-range interactions, as well as the size and flexibility of RA and SAM. A A-ML. approach, which learned correction on top of semi-empirical tight-binding
method (DFTB (Elstner et al. 1998)) ), was used in previous work (Böselt et al. 2021 Hofstetter al. 2022).
Table MAE for the QM energy (EOM). QM gradients (FOM), and MM gradients (FMM) of the three QM/MM systems uracil, RA and SAM in water as described in Böselt et al. (202T). Energies are reported in kJ mo! and forces in kJ mol - Á-! The terms "unshufiled" and "shuffled" refer to the dataset with and without temporal ordering, respectively.
AMP(3)-DP 50 A (unshuffled)
AMP(3)-DP DD 50 Á (shuffled)
System Uracil
EgM
FoM
FxM
Fom
FMM
0.2/3.1/2.5
0.4/7.3/7.1
0.04/0.0800.09 0.06/0.16/0.1
0.3/0.3/0.3
0.6/0.6/0.6
0.03/0.03/0.03
RA SAM
0.5/9.1/12.0
1,2/5.0/9.6
0.6/0.6/0.6
5.4/1.5/1.5
0.06/0.06/0.06
2/2/3
3.3/16.1/16.0
0.20/0.39/0.34
3.1/3.2/3.3
3.2/3.5/3.5
0.15/0.15/0.15
No A-learning scheme was used in this work. In Table [ results for a 3-layer AMP(3)-DP model reported. The performance is compared for two pre-processing steps: While (unshuffled) uses the temporally sequential ordering used in the initial work Böselt et al. (2021); Hofstetter et al. (2022). (shuffled) refers to shuffled (i.e. randomized) version of the same data. Comparing the same dataset (unshuffled), we observe in most cases larger errors compared to the previously reported A-models but smaller errors than previously reported results without A-learning. For uracil, (unshuffled) errors were reported without A-learnin:: Using a HDNNP architecture, [öscllt ctt al. (2021) reported MAEs of 2.9/6.6/4.8
mol for EqM, 6.38/ / 12.6/ 2..4/2.44 kJ mol A-1 for FOM and 0.62 1.6//0.58/0 /0.65 kJ mol 1 A for FMM respectively. With a GNN architecture, Hof- reported MAEs of 7.3 kJ mol for EQM, 13.52 kJ mol A for FoM and 0.58 kJ mol A for FMM on the test-set. These results suggest that the proposed model can suc- cessfully incorporate long-range polarization and electrostatic interactions. However, the relatively large drop in accuracy for the (unshuffled) dataset, which was previously observed and discussed in HHofsttter al. (2022). indicates limited capability to generalise. The difference between random splits or time splits might be important for commonly used benchmark datasets such as MD17 (Chmiela al., (217)), as random splits tend to give too optimistic performance assessment. This issue is already well known in biological activity prediction Sheridan (2013). Calculating QM energy, QM gradients, and MM gradients for uracil (12 QM atoms and an average of 1490 MM atoms) using a 3-layer AMP(3)DDP takes around 10 ms on a Nvidia Titan V. Additional results re- garding computational costs are reported in the Appendix A.I Thus, (QM)ML/MM simulation based on the AMP model could in principle remove the bottleneck of current QM/MM molecular dynamics (MD) simulations almost entirely. However, further improvements are required to bring the cost down to a level comparable to the calculation of classical force fields, which is on the order of 1 ms/step for a system with around 14'000 atoms (Eastman et al., 2017). Such improvements are clearly within reach, for instance through the use of the fast multipole method (Rokhlin, 1985). Nevertheless, it is important to mention that a low MAE does not guarantee a stable MD simula- tion (Stocker al.] 2022). In practice, A learning scheme might still be required to obtain stable simulations over long timescales. The robustness of the AMP(k: model for the propagation of (QM)ML/MM MD simulations will therefore be investigated in the future.
/0.58/0.6555
stetter et al. (2022)
Table 2: Comparison of long-range cutoffs. Energies are reported in kJ mol 1 and forces in kJ mol A- The models used here were trained for fewer epochs than those in Table [ re- sulting in slightly larger errors. The shuffled dataset was used. Results are given for training/test split (7'000/3'00k 000 data points).
AMP(3)-DP 30 Á
AMP(3)-DP 50 A
System
Egaı
FuM
FMM
Eqm
FуM
FAm
Uracil
0.3/0.4
0.8/0.9
0.06/0.06
0.3/0.3
0.8/0.9
0.03/0.03
RA
5.3/S.5
2.9/3.1
0.14/0.14
1.5/1.6
2.5/2.7
0.13/0.13
SAM
6.0/6.4
6.1/6.4
0.45/0.45
5.5/5.7
5.3/5.5
0.33/0.33
8

---

Published as a conference paper at ICLR 2023
In addition, two long-range cutoffs,, 30 Å and 50 Å, for the interactions between QM and MM par- ticles are investigated (Table 2). Notable performance differences are observed for the two investi- gated Long-range cutoffs, which reaffirms the importance of maage interactions. In both cases, a smooth cutoff function is used, thus the differences cannot simply be explained by cutoff artifacts. In this context, it is further important mention that increased long-range cutoff does not change the number of parameters and QM-QM interactions but only the number of QM-MM interactions. 50 A cutoff covers all observed QM-MM interactions for all systems. For 30 A cutoff, the same is only true for uracil. Particularly for retinoic acid, for which the solvent-shell is elongated, the difference of the MAE for EΩM between the two cutoffs is considerable. Hence, even if a message passing architecture with six layers and cutoff of Å, could accurately resolve long-range interac- tions, these interactions would still fall outside the perceptive radius. However, given that there is little evidence that message passing models can faithfully resolve and incorporate such Long-range interactions in the first place, this observation might be crucial for future work ML potentials.
4.4
QM9 DATASET
Results for 4-layer AMP(3) model with a cutoff of Á for the 12 properties of the QM9 dataset are given in Table 3 and compared with existing models reported in the literature. Inference of 4-layer AMP(3) model used here requires approximately 26 for batch of 100 samples with a total of 1'833 atoms (see Appendix A.1 NVIDIA Titan V), comparing favourably to existing models. Errors on properties related to the potential-energy surface (i.e., U, Up. H, G, and ZPVE) are higher than for other models (Allegro, PaiNN, PaxNet, DimeNet). For the remaining properties, performance comparable with other state-of the-art art architectures.
Table 3: Mean absolute егrоr (MAE) for the 12 properties of the QM9 dataset (Ramakrishnan et al., 2014.. AMP refers the model proposed in this work. Values for existing models the literature
are taken from: SchNet (Schütt et al. 2018), DimeNet (Klicpera et al. 2020), PaiNN (Schütt et al.
2021
Allegro (Musaelaan
2022). SEGNN (Brandstetter et 2021). and PaxNet (Zhang
et al.
2022).
Target
Unit
SchNet
DimeNet
PaiNN
Allegro
SEGNN
PaxNet
AMP
Uo
meV
14
8.02
5.85
4.7
15
5.90
11.3
U
meV
19
7.89
5.83
4.4
13
5.92
11.4
H
meV
14
8.11
5.98
4.4
16
6.04
11.3
meV
14
8.98
7.35
5.7
15
7.14
12.4
ZPVE
meV
1.7
1.29
1.28
1.62
1.17
4.1
éHOMO
meV
27.8
27.6
22.8
25.7
61UMO
meV
34
19.7
20.4
21
19.2
22.6
Ae
meV
34.8
45.7
31.0
44.9
mD
33
28.6
12.0
23.0
10.8
11.7
maj
235
46.9
45
60
44.7
66.8
(R?)
ma
73
33.1
66
660
93
249.2
C
24.9
24
23.1
31.7
mil K
OUTLOOK
In this work, modified message passing formalism that allows for the efficient incorporation of directional and long-range information was proposed. For future work, an implementation, which makes use of the fast multipole method, as well as the explicit inclusion of many-body interactions, as demonstrated by Klicpera et al. (020)) and Batatia et al. (2022b), and additional weights, as shown by Schütt al. (2021), could be promising avenues. As particular case, envision the application in (MM)ML/MM MD simulations, which could facilitate the study of large biomolecular systems and enzymatic reactions.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Code and model parameters used to produce the results in this work are made available at github. comrrb/er/iiiikkka/AAMPP All datasets used this work are either already publicly available or made available under the same URL.
ACKNOWLEDGMENTS
The authors thank Lennard Böselt for helpful discussions. The authors gratefully acknowledge NVIDIA for providing Titan V under the NVIDIA Academic Hardware Grant Program.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL https: h/wp://wwwwwoo/ wwwsorflow. org/ Software available from tensorflow.org.
Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. arXiv, pp. (xXi:20006.05055 2020.
Brandon Anderson, Truong-Sov Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Adv. Neural Inf. Process. Syst., 32, 2019.
Ilyes Batatia, Simon Batzner, Dávid Péter Kovács, Albert Musaelian, Gregor NC Simm, Ralf Drautz, Christoph Ortner, Boris Kozinsky, and Gábor Csányi. The design space e(3)-equivariant atom- centered interatomic potentials. arXiv, pp. arXiv:2205.06643 2022a.
Ilyes Batatia, Dávid Péter Kovács, Gregor NC Simm, Christoph Ortner, and Gábor Csányi. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. arXiv, pp. arXiv:2206.07669,, 2022b.
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. Adv. Neural Inf. Process. Syst., 29, 2016.
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez,, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. arXiv, pp. arXiv:1806.01266 2018.
Simon Batzner, Albert Musaclian, Lixin Sun, Mario Geiger, Jonathan P. Mailoa, Mordechai Korn- bluth, Nicola Molinari, Tess E. Smidt, and Boris Kozinsky. E(3)-equivariant graph neural net- works for data-efficient and accurate interatomic potentials. Nat. Commun., 13:2453, 2022.
Jörg Behler. Atom-centered symmetry functions for constructing high-dimensional neural network potentials. J. Chem. Phys, 134:074106, 2011.
Lennard Böselt, Moritz Thürlemann, and Sereina Riniker. Machine learning in QM/MM molecular dynamics simulations condensed- phase systems. J. Chem. Theory Comput.. 17:2641-2655. 2021.
10

---

Published as a conference paper at ICLR 2023
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Geometric and physical quantities improve e(3) equivariant message passing. arXiv, pp. arXiv:2110002005, 2021.
Christian J. Burnham and Niall J. English. new relatively simple approach to multipole interac- tions in either spherical harmonics cartesians, suitable for implementation into Ewald sums. Int. J. Mol. Sci., 21:277, 2020.
Stefan Chmiela, Alexandre Tkatchenko, Huziel E. Sauceda, Igor Poltavsky, Kristof T. Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Sci. Adv. 3:e1603015, 2017.
Taco S. Cohen and Max Welling. Group equivariant convolutional networks. In International
Conference on Machine Learning, pp. 2990-2999, 2016.
TensorFlow Developers. Tensorflow, May 2021.
Robert A. Distasio JR., Ryan P. Steele, Young Min Rhee, Yihan Shao, and Martin Head-Gordon. An improved algorithm for analytical gradient evaluation in resolution-of ofeenoo---hheeeee tettthe-diieee ieeeendnttyy second-order ppoller-plesset perturbation theory: Application alanine tetrapeptide conformational analysis.
J. Compul. Chem., 28:839-856, 2007.
Brett Dunlap. Robust and variational fitting. Phys. Chem. Chem. Phys., pp. 2113-2116. 2000.
Thom H. Dunning. Gaussian basis sets for use in correlated molecular calculations. the atoms boron through neon and hydrogen. J. Chem. Phys., 90:1007-1023, 1989.
Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. arXiv, pp. arXiv:2206.08 2022.
Peter Eastman, Jason Swails, John D. Chodera, Robert T. McGibbon, Yutong Zhao Kyle A. Beauchamp, Lee-Ping Wang, Andrew C. Simmonett, Matthew Harrigan, Chaya D. Stern, Rafal P. Wiewiora, Bernard Brooks, and Vijay S. Pande. OpenMM 7: Rapid development of high performance algorithms for molecular dynamics. PLoS Comput. Biol., 13:c1005659, 2017.
M. Elstner, D. Porezag, G. Jungnickel, J. Elsner, M. Haugk, Th. Frauenheim, S. Suhai, and G. Seifert. Seff-consistentcchargee density- gighiin method for simulations of com- plex materials properties. Phys. Rev. B, 58:7260-7268, 1998.
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Ady. in Neural Inf. Processing Syst., 34:6790-6802, 2021.
Timothy J. Giese, Jinzhe Zeng, Şölen Ekesan, and Darrin M. York. Combined qm/mm, machine learning path integral approach to compute free energy profiles and kinetic isotope effects in ma cleavage reactions. J. Chem. Theorr and Comput., 18:4304 4317, 2022.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George F. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pp. 1263-1272, 2017.
Sharon C. Glotzer and Michael Solomon. Anisotropy of building blocks and their assembly into complex structures. Nature Materials, 6:557-562, 2007.
Andrea Grisafi and Michele Ceriotti. Incorporating long-range physics atomic-scale machine learning. J. Chem. Phys., 151()))200105, 2019.
Andrea Grisafi, Jigyasa Nigam, and Michele Ceriotti. Multi-scale approach for the prediction of atomic scale properties. Chem. Sci., 12:2078-2090 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference Computer Vision (ICCV), December 2015.

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proc. IEEE Int. Conf. Comput. Vis., pp. 770-778, 2016.
Albert Hofstetter, Lennard Böselt, and Sereina Riniker. Graph convolutional neural networks for (qm)ml//m molecular dynamics simulations. arXiv, pp. arXiv:2206. 14422, 2022.
J. E. Jones and Sydney Chapman. On the determination of molecular fields. I. From the variation of the viscosity of a gas with temperature. Proc. R. Soc. Lond.,
1924.
Rick A. Kendall, Thom H. Dunning, and Robert J. Harrison. Electron affinities of the first-row atoms revisited. systematic basis sets and wave functions. J. Chem. Phys., 96:6766-6800 1992.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv, pp-
arXiv:1412.6980, 2017.
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molec- ular graphs. arXiv, pp. arXiv:2003.03123, 2020.
Tsz Wai Ko, Jonas A. Finkler, Stefan Goedecker, and Jörg Behler. A Nourth-generatinn high- dimensional neural network potential with accurate electrostatics including non-local charge transfer. Nat. Commun., 12:398, 2021.
Risi Kondor. N-body networks: A covariant hierarchical neural network architecture for learning atomic potentials. arXiv, pp. 1rXiv:1803.0155 2018.
Christian Kramer, Alexander Spinn, and Klaus R. Liedl. Charge anisotropy: Where atomic multi- poles matter most. Chem. Theory Comput., 10:4488-4496, 2014.
Dejun Lin. Generalized and efficient algorithm for computing multipole energies and gradients based cartesian tensors. J. Chem. Phys., 143:1 14115, 2015.
Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett (eds.), Adv. Neural Inf. Process Syst. volume 32, 2019.
Benjamin Kurt Miller, Mario Geiger, Tess Smidt, and Frank Noé. Relevance of rotationally equivariant convolutions for predicting molecular properties. arXiv, pp. arXiv:2008.0840 2020.
Chr. Møller and M. S. Plesset. Note an approximation treatment for many-electron systems.
Phys. Rev, 18662,,,,, 1934.
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4602-4609, 2019.
Albert Musaclian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J Owen, Mordechai Ko- mbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. arXiv, pp. arXiv:2204.052.4 2022.
Xiaoliang Pan, Junjie Yang, Richard Van, Evgeny Epifanovsky, Junming Ho. Jing Huang, Jingzhi Pu, Ye Mei, Kwangho Nam, and Yihan Shao. Machine-learinng-asistee free energy simulation of solution-phase and enzyme reactions. J. Chem. Theory and Comput., 17:5745-577 2021.
Robert M. Parrish, Lori A. Burns, Daniel G. A. Smith, Andrew C. Simmonett, A. Eugene DePrince, Edward G. Hohenstein, U. Bozkaya, Alexander Yu. Sokolov, Roberto Di Remigio, Ryan M. Richard, F, Gonthier, Andrew M. James, Harley R. McAlexander, Ashutosh Kumar, Masaaki Saitow, Xiao Wang, Benjamin P. Pritchard, Prakash Verma, Henry F. Schaefer, Konrad Patkowski, Rollin A. King, Edward F. Valeev, Francesco A. Evangelista, Justin M. Turney. T. Daniel Craw- ford, and C. David Sherrill. Psi4 1.1: An open-source electronic structure program emphasizing automation, advanced libraries, and interoperability. Chem. Theory Comput., 13:3185-3199 3197,
2017.
12

---

Published as a conference paper at ICLR 2023
Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. Understanding the exploding gradient prob- lem. arXiv, pp. 1tXiv:1211.5066 2012.
Sergey N Pozdnyakov and Michele Ceriotti. Incompleteness of graph convolutional neural networks for points clouds in three dimensions. arXiv, pp. arXiv:2201.077 ..0000.....,..... 2022.
Avijit Rakshit, Pradipta Bandyopadhyay, Joseph Heindel, and Sotiris S. Xantheas. Atlas of pu- tative minima and low-lying energy networks of water clusters n = 3-255. J. Chem. Phys., 151: 214307, 2019.
Prajit Ramachandran, MarretZopp, and Quoe V. Le. Searching for activation functions. arXiv, 2017.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quan- tum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1:140022, 2014.
V Rokhlin. Rapid solution of integral equations of classical potential theory. J. Comput, Phys., 60: 187-207, 1985.
Víctor Gareia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural net- works. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 9323-9332, 2021.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Trans. Neural Netw. Learn. Syst., 20:61-80, 2009.
Norbert Schuch and Frank Verstracte. Computational complexity of interacting electrons and fun- damental limitations of density functional theory. Nat. Phys., 5:732-735, 2009.
Kristof T. Schütt, Oliver T. Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning. pp. 9377-9388. PMLR, 2021.
K. T. Schütt, learning architecture for molecules and materials. Journal of Chem. Phys., 148:241722, 2018.
Sauceda, P.-J Kindermans, Tkatchenko, and K.-R. Müller. Schnet deep
K. T. Schütt, P. Kessel, M. Gastegger, K. A. Nicoli, A. Tkatchenko, and K.-R. Müller. Schnetpack:
A deep learning toolbox for atomistic systems. J. Chem. Theory and Comput., 15:448-455, 2019.
Hans Martin Senn and Walter Thiel. QM/MM Methods for Biological Systems, pp. 173-290. Springer, 2007.
Robert P Sheridan. Time-split cross-validation method for estimating the goodness of prospec- tive prediction. J. Chem. Inf. Model., 53:783-790, 2013.
Zeren Shui and George Karypis. Heterogeneous molecular graph neural networks for predicting molecule properties. In 2020 IEEE International Conference on Data Mining (ICDM), pp. 492- 500, 2020.
Daniel G. A. Smith, Lori A. Burns, Andrew C. Simmonett, Robert M. Parrish, Matthew C. Schieber, Raimondas Galvelis, Peter Kraus, Holger Kruse, Roberto Di Remigio, Asem Ale- naizan, Andrew M. James, Susi Lehtola, Jonathon P. Misiewicz, Maximilian Scheurer, Robert A. Shaw, Jeffrey B. Schriber, Yi Xie, Zachary L. Glick, Dominic A. Sirianni, Joseph Senan O'Brien, Jonathan M. Waldrop, Ashutosh Kumar, Edward G. Hohenstein, Benjamin P. Pritchard, Bernard R. Brooks. Henry F. Schaefer, Alexander Yu. Sokolov, Konrad Patkowski. A. Eugene DePrince, U. Bozkaya, Rollin A. King, Francesco Evangelista, Justin M. Turney, Daniel Crawford, and C. David Sherrill. PSI4 1.4: Open-source software for high-throughput quantum chemistry. J. Chem. Phys., 152: 152:184108. 84108. 2020.
William Smith. Point multipoles in the ewald summation (revisited). Information Newsletter for Computer Simulation of Condensed Phases, pp. 15-25, 1998.
Sina Stocker, Johannes Gasteiger, Florian Becker, Stephan Günnemann, and Johannes Margraf. How robust are modern graph neural network potentials in long and hot molecular dynamics simulations? ChemRxiv, 2022.
13

---

Published as a conference paper at ICLR 2023
Anthony Stone. The Theory of Intermolecular Forces. Oxford University Press, 2013.
Moritz Thürlemann, Lennard Böselt, and Sereina Riniker. Learning atomic multipoles: Prediction of the electrostatic potential with equivariant graph neural networks. J. Chem. Theory Comput. 18: 18:1701-77 2022.
Justin M. Turney, Andrew C. Simmonett, Robert M. Parrish, Edward G. Hohenstein, Francesco A. Evangelista, Justin T. Fermann, Benjamin J. Mintz, Lori A. Burns, Jeremiah J. Wilke, Micah L. Abrams, Nicholas J. Russ, Matthew L. Leininger, Curtis L. Janssen, Edward Seidl, Wesley D. Allen, Henry F. Schaefer, Rollin A. King, Edward F. Valeev, C. David Sherrill, and T. Daniel Crawford. Psi4: An open-source ab initio electronic structure program. WIREs Comput. Mol.
Sci., 2:556-565, 2012.
Oliver T. Unke and Markus Meuwly, PhysNet: A neural network for predicting energies, forces, dipole moments, and partial charges. J. Chem. Theory Comput., 15:3678 77988-363,--1 2019.
Oliver T. Unke, Stefan Chmiela, Michael Gastegger, Kristof T. Schütt, Huziel E. Sauceda, and Klaus-Robert Müller. Spookynet: Learning force ffields with electronic degrees of freedom and nonlocal effects. Nat. Commun., 12:7273, 2021a.
Oliver Unke, Stefan Chmiela, Huziel E. Sauceda, Michael Gastegger, Igor Poltavsky, Kristof T. Schütt, Alexandre Tkatchenko, and Klaus-Robert Müller. Machine learning force fields. Chem. Rev., 121:10142-10186, 2021b.
Max Veit, David M. Wilkins, Yang Yang, Robert A. DiStasio, and Michele Ceriotti. Predicting molecular dipole moments by combining atomic partial charges and atomic dipoles. J. Chem. Phys, 153:024113, 2020.
Toon Verstraelen, Steven Vandenbrande, Farnaz Seidar-Zadehh Louis Vanduyfhuys, Veronique Van Speybroeck, Michel Waroquier, and Paul W. Ayers. Minimal basis iterative stockholder: Atoms in molecules for force-field development. J. Chem. Theory Comput., 12:3894-3912, 2016.
A. Warshel and M. Levitt. Theoretical studies of enzymatic reactions: Dielectric, electrostatic and steric stabilization of the carbonium ion in the reaction lysozyme. J. Mol. Biol., 103:227-249, 1976.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: earning rotationally equivariant features in volumetric data. Adv. Neural Inf. Process Syst., 31, 2018.
Shuo Zhang, Yang Liu, and Lei Xie. Efficient and accurate physics-aware multiplex graph neural networks for 3d small molecules and macromolecule complexes. arXiv, pp. arXiv:2206.02789, 2022.
Yin-Jia Zhang, Alireza Khorshidi, Georg Kastlunger, and Andrew A. Peterson. The potential for machine learning in hybrid qm/mm calculations. Chem. Phys., 148(24):241740, 2018.
Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. arXiv, pp. arXiv:21 2021.
A APPENDIX
A. COMPUTATIONAL COST
The modifications proposed this work will result in additional computational costs. However, important to note that the proposed changes will only increase the constant cost of each message but not the overall complexity. The following two paragraphs show empirical running times pure QM and in QM/MM setting. All timings reported based Nvidia Titan V.
14

---

Published as a conference paper at ICLR 2023
Table 4: Relative inference time for single batch of 100 molecules of the QM9 dataset with m-layer AMP model. The absolute cost is given in parenthesis. The relative cost in percentage is given with respect to the message passing without the proposed modifications defined in Eq.
n
AMP(O)
AMP(1)
AMP(2)
AMP(3)
138% (5.7 ms)
146% (6.0 ms)
164% (6.8 ms)
207% (8.5 ms)
2
151% (8.1 ms)
164% (8.8 ms)
200% (10.7 ms)
267% (14.3 ms)
3
162% (10.6 ms)
178% (11.6 ms)
221% (14.4 ms)
307% (20.1 ms)
4
167% (12.9 ms)
187% (14.4 ms)
236% (18.2 ms)
333% (25.7 ms)
A.1.1 COMPUTATIONAL COST: QM9
Here, the computational cost for various orders and number of layers is reported for a single batch of the QM9 dataset containing 100 samples (1'833). This task only includes single forward pass. Timings were averaged over 100 calls. In addition to absolute costs (in brackets), relative costs with respect the baseline GNN described in Eq. Tare reported in Table
A.1.2 COMPUTATIONAL COST: QM/MM
For QM/MM simulations, which generally include large number of MM particles and a large num- ber of repeated calls, the cost of the ML potential will significantly contribute the simulation cost. In general. the computational cost will be dominated by the QM/MM interactions due to the larger number of MM particles. Since the QM/MM interaction is based wise interaction, these costs will be proportional to NQM & x MM., M. i.e., the number of MM particles times the number of QM particles. The dominating N M can be reduced further based on the fast multipole method Rokhlin (1985). Here, we report timings for two settings. Setting considers the computational cost without interactions with the MM particles (i.e., the isolated molecule). In setting 2, the computational cost due to the QM/MM interaction is investigated. In both cases, the three layer AMP(3) model used for the systems described in Section 4.3 used.
one-time unidirectional pair-
In this paragraph, computational costs for the isolated molecule are reported. Batches of size 60 were used. Timings were averaged over all batches using 100 runs and are reported per molecule. A cutoff of 5 Aand three layers were used. Results are reported in Table A.1.2 We find that for the
Table Cost for the prediction of QM energies and QM gradients for the isolated QM systems. The number of particles is shown in the second column. GNN refers to message passing model based on Eq. [ i.e., without the proposed modifications.
System
Size
GNN
AMP(0)
AMP(1)
AMP(2)
AMP(3)
Uracil
12
0.51 ms
0.65
0.82 ms
1.04
1.52 ms
RA
50
0.76 ms
1.33 ms
1.53 ms
2.07 ms
2.99 ms
SAM
63
0.87 ms
1.67 ms
1.95 ms
2.28 ms
3.15 ms
two larger systems, the computational cost per edge saturates roughly jss per edge shown in Figure 4 For both molecules, the cost per edge is roughly 50% higher with AMP(3) compared to AMP(2).
Finally, we report the total computational cost to predict energies, QM gradients, and MM-gradients for varying numbers of MM particles in Table A.1.2 Results were averaged over batches of size 10 using 100 runs. Reported are the costs per system.
A.2 GENERAL IMPLEMENTATION DETAILS
If not noted otherwise, the following architecture and hyperparameters were used. Three message passing layers were used except for the QM9 dataset, where we used four message passing layers. The architecture looked as follows:
15

---

Published as a conference paper at ICLR 2023
AMP(3) SAM
AMP(2) SAM
Cost per Edge [us]
8
AMP(3) RA
AMP(2) RA
6
2-
2
Cutoff [A]
Figure 4: Computational cost per edge for the isolated molecules.
Table 6: Cost for the prediction of QM energies, QM gradients, and MM-gradients for varying numbers of MM particles (M) Results are reported for the three layer AMP(3) model used in Section 4.3
MM
Uracil
RA
SAM
10
5.6 ms
20.1 ms
20.5 ms
100
5.7 ms
22.3 ms
22.8 ms
1'000
8.4 ms
37.7 ms
37.4 ms
1. Generation of edge weights with [Linear(16)], which parametrized ds. As input, the onehot features of the two connecting atoms and Bessel function (n = 20) expanded distance features were used.
2. Embedding of node features: [Linear(128).
3. Message layers with [Linear(128). Swish, Linear(128), Swish] for oh and [Linear(128] Swish, Linear(128). Swish, Linear(k+1)] for the prediction of the co- efficients Cij, ie.. (M(()) o (k) and @Mars(()) The coefficients for all orders were predicted with the same module.
4. A readout layer [Linear(128), Swish, Linear(128), Swish, Linear(1))-
Linear(n) refers to standard fully connected network layer with bias. No bias was used for the last layer of the output. For each message passing iteration, independent modules were used. Swish was employed as the non-linearity (Ramachandran al. (017). Layer weights were initialized based on the method proposed by He et (2015). Coeflicients Cij were scaled by 10- to support training stability during early epochs.
Node features were initialized as onchot vectors of the element. Norms of the multipoles were concatenated with the hidden feature. Edges were added for all atom pairs within distance of 5 Å. For interactions between QM and MM particles, an independent long-range cutoff of 50 A was used. Radial weights were conditioned on the initial onehot features, i.e., dy (h. h ||r Distances |ral] were encoded with the enveloped Bessel functions proposed by Klicpera et al. (2020) with 20 functions and 5 for the envelope. The frequencies were not optimized. A Res update of the hidden feature was used, i.e., h ++ h + on (h), m.; Hee et al. 2016 Model parameters were optimized using ADAM (Kingma & Ba (017)) with default parameters (3) 0.9, 32 = 0.999, = 10-7) and an exponentially decaying learning rate (5 10-4 10-5). Gradients were clipped by their global norm with clip norm of 1 (Pascanu et al. 2012). Mean-squared errors were optimized. the model jointly trained energies and gradients, the following loss function was used
L (1 1-))
Σ(vo (Vb vo)²
A
ΣΣ
BN 2
ab,
Fia))
(12)
B
3BN
Ori.
a=0
16

---

Published as a conference paper at ICLR 2023
with the potential energy V and the force component Fi.c for each Cartesian dimension À 0.8 was used to balance the contributions of the energies and gradients to the loss. The loss term for the MM gradients was obtained in the same way but scaled by an additional factor 100 proposed in (Hofstetter et al., 2022). Models were implemented with TensorFlow (2.6.2 and 2.9.1) (Abadi
al. 2015 Developers 202T). Single precision float32 used.
A.3
WATER DIMER
Calculations were performed with PSI4 (1.6) (Turney et al. 2012: Parrish et al. 2017 Smith et al.,
2020 on MMP2//uu-cc--P// level of theory using density fitted MP2 (Dunlap 2000 Distasio JR. et al. 2007). The molecules were placed such that the two oxygen atoms were separated by A Structures were assigned training and test sets in an alternating manner. Single layer models were used for all experiments, Models were trained for 10'000 steps using batch size of five. 64- dimensional features were used as node features and four edge weights. Since whole system was simulated with MP2, partial charge embedding was used. Instead, the pairwise interaction for the model 'DP'' was implemented as
(p(h)) @p(h;)
VESP ΣΣ
(13)
|ragll
j>
for all long-range pairs, i.e., ||r/|| 2 Teut- Double precision float64 was used for this system.
A.3.1 WATER DIMER: LEARNING CURVE
Work on equivariant and directional message passing models has shown that inclusion of directional information can improve data efficiency (Schütt et al. 2021 Batzner al., 2022 [attatia et al.
20226). Figure illlstrrtes Filustees the convergence of AMP models for the fully connected topology of the water dimer in Figure
MAE log EnEngy Dog log.ools.J m
2-
0-
-2-
AMP(O)
AMP(1)
AMP(2)
500
1000
1500
2000
Training Step
Figure 5: Mean absolute error (MAE) of the energy (0gol 11 on the validation set during training for the (fully-connected) dimer topology show in Figure 2 The bold line indicates the mean over runs while the shaded regions indicate one standard deviation. The black horizontal line indicates a null model (prediction mean).
In addition, shown in Figure 6 we also explored the training behaviour for the AMP(2) model on the sparsely-connected topology shown Figure B Interestingly, there appear to be cases where training proceeds in phase-transition like manner in contrast to the consistent convergence observed for the fully connected topology in Figure 2A.
A.4 WATER CLUSTERS
The dataset with the water clusters created based the geometries presented in (Rakshit al., 2019) For each set of clusters ranging from 6 to 20 water molecules, up to 2'000 clusters were ran- domly sampled. For each cluster, water molecule randomly designated the QM zone with
17

---

Published as a conference paper at ICLR 2023
MAE Energy in logio [logsa(3]]] [mmo)))]]
-2-
1000
2000
3000
4000
5000
Training Step
Figure 6: MAE of the energy in logio(kJmol-" on the validation set during training for the (sparsely connected) dimer topology show in Figure2B. Each line represents the training trajectory of of 10 randomly initialized models. The black horizontal line indicates null model (prediction mean).
remaining molecules forming the MM zone. The long-range cutoff was set to Á including all QM-MM interactions. All accessible clusters were used, i.e., all known clusters within 5 kcal/mol energy window above the putative minimum. All clusters of size n = 10 were used for training while the other clusters were used for testing. For the PaiNN, SchNet, and GNN models, the energy was predicted for the QM particles. Furthermore, addition to the existing atom types (oxygen-QM) Pargen--MMaaa and hydrogen-QMa MM), two tion provided the SchNetPack (Schütt
introduced: oxygen-MM and hydrogen-MM. The implementa- et al.) 2019) was used for PaiNN and SchNet (Schütt et al., 2021 Schütt et al. 2018). For PaiNN, the size of the scalar representation (160) was increased to provide a comparable number of trainable parameters L.M.. For SchNet, the size of layers and the number of filters was set 256. The same cutoff (5 Å), readout and radial basis functions were used as in the AMP model. Numerical results for all models are reported in Table A.4.1 Partial charges of MM zone were assigned based on MBIS monopoles (Verstraelen et al.] 2016). i.e., 0.9141977 for oxygen and 0.457098e for hydrogen. Calculations were performed
types
MP2/aug-cc-pVTT level
of theory (Møler & Plesset 1934 Dunning 1989 Kendall et al. 1992) using density fitted MP2 (Dunlap) 2000; Distasio JR. al. 2007) as implemented in PSI4 (1 (1.6) TTrnny al. 2012: Parrish ctt al., |017:|Smith [mith et al. 2020). Gradients and energies of the isolated QM zone were subtracted. The models were trained over 50'000 steps, presenting each time batch of 16 randomly drawn samples.
A.4.1 WATER CLUSTERS: RESULTS
Table 7: Mean absolute error (MAE) of the energies in kJ mol-1 presented in Figure 3
AMP(D)
7.5
6.25
3.57
3.4
5.11
5.70
4.4
5.14
5.00
5.13
4.57
4.71
5303
5.02
3.11
AMP(1)
2.66
2.22
1.44
0.37
0.25
0.53
0.86
0.92
1.02
1.36
2.03
3.01
4.59
6.2
8.1
AMP(2)
0.26
0.47
AMBY3)
PaINN
1.14
1.83
0.41
0.51
0.38
0.75
0.65
0.5
0.89
1.01
12
2.08
191
2.19
18
SchNet
1.79
1.42
9.5
0.75
0.19
1.05
1.02
1.28
133
1.5
175
3,61
3.61
4.1
4.11
GNN
0.8
A.5 QM/MM SYSTEMS
The datasets were taken from Böselt et al. 2021 and randomly split into training/validatiionnest sets 11000002000/1111 samples, respectively. Polarization due to the MM particles described in Eq. 10 was only incorporated during the last message passing step. In addition, only eight distance
18

---

Published as a conference paper at ICLR 2023
Table 8: Mean absolute errors (MAE) of the gradients in kJ mol- presented in Figure
17
AMF(O
8,31
1.92
6.45
6.27
5.4
6.14
6.35
5.2
5.2
8.23
5.21
5.25
5.29
5.13
8.52
AMP(1)
0.61
1.39
5.48
AMP(3)
0.57
0.59
0.25
0.3
0.1
0.44
0.61
123
1.32
161
1.98
2.19
PuiNN
1.3
1.36
01.
0.51
0.05
0.55
0.08
0.73
0.77
0.79
0.88
1366
1.44
1.52
1.35
SchNet
2.25
0.97
0.12
0.96
1.16
1.18
2.4
2.36
2.77
GNN
1.89
GNN-P
1.76
1.49
0.08
0.73
0.08
0.97
0.95
1.29
1.61
195
222
274
3.21
3.78
3.30
weights were used for interactions between QM and MM particles while the default was used for edges within the graph, i.e., between QM particles. The models were trained over 400'000 steps using batches of size 5 for uracil and batches of size 1 for retinoic acid and SAM//yssteine. For the comparison between the long-range cutoffs, only 200'000 training steps were used.
A.6
QM9 DATASET
The dataset was taken from Ramakrishnan et al. (2014). An independent 4-layer AMP(3) model was trained for each property over 1'250 epochs with an exponentially decaying learning rate 10-4, 5 6) A 4-layer AMP(3) results in 1'153'79' trainable parameters. Properties in units of Eh were converted and trained in units of keal mol-! Each property was predicted as the sum of atomic contributions except for the dipole moment μ.. For all properties, except for the dipole moment μ4, the mean of the training set was subtracted. In addition, the energy of the isolated atoms was subtracted for Up. U, H, and G. 3'054 molecules for which the geometry consistency checks failed were removed from the dataset [Ramakrishnan et al. (2014). The remaining data points were sets (110'000, 10'000, 10'831) using batch size of
randomly split into rraining/validation/// 100. As proposed by Veit al. 2220),, the molccular dipole moment μe was modeled as the sum of the contribution from atomic monopoles qi and atomic dipoles μ4i
Σ (h))rr
(14)
with each atomic dipole μr being predicted according to Eq. tracted from each qe enforce charge conservation.
The mean excess charge was sub-
19


Reviews:
[{"review_id": "nOB-bq1KoJ-", "reviewer": "Reviewer_YdbF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a 3D geometric GNN, and develops a novel message passing scheme considering the directional effects from an external field or from a set of particles unreachable in a graph. If we place a molecule in an electric field, we need to take it into account when calculating the potential energies of the molecule. The developed scheme is a perfect fit to the cases, for example, (A) a geometric graph in R^3 with edge connections by distance cutoff, where long-range interactions beyond the cutoff are ignored during message passing, and (B) QM/MM situations where we consider a set of particles in quantum mechanics (QM), but other particles treated in molecular mechanics (MM). The idea for this is to prepare a state tensor (anisotropic state) M_i^k for each node i, and used in message passing. M_i^k is defined by linear combinations of basis, i.e., the tensor product of order k of the unit vector from node_i to a neighbor j, where coefficients are learned by neural nets. This corresponds to multipole expansions of an angle-dependent function on R^3, which is often used in computational chemistry (monopoles, bipoles, dipoles, ...). The paper also provides case studies of water dimer cases (interactions between 2 H2O molecules) and water cluster (a set of many water molecules) as well as QM9 benchmarking with geometric graphs with an edge cutoff of 5 angstrom. ", "summary_of_the_review": "Basically, I liked the paper. In order to consider the effects from an external field (directional interactions) or those from unreachable nodes (long-range interactions), this paper develops a novel message passing scheme called Anisotropic message passing. The directional or long-range contributions are formulated by a multipole series expansion with learnable coefficients. The use of series expansion is also a good fit for me. Then the entire long-range or directional effects are modeled via message passing with multipole-multipole interactions as the proposed Anisotropic Message Passing (AMP) scheme of order k. Also, the empirical evaluations with four nice use cases are very convincing.\n", "strength_and_weaknesses": "[Strength]\n\n- The proposed scheme of Anisotropic Message Passing (AMP) is an interesting and natural scheme to consider any directional effects from an external field or any long-range effects from a set of particles unreachable in the graph used for message passing. When we use any 3D geometric GNN, message passing is often done within cutoff ranges rather than complete graph having edges between all pairs of nodes for efficient computation. But this ignores long-range interactions between atoms or particles. Not only this is suitable to considering the interactions to an external field such as QM/MM or multi-molecules cases, but also suitable for a single molecule case if we use distance-based edge cutoff (as demonstrated in QM9 cases of 4.4).\n\n- The idea to introduce anisotropy is based on multipole expansions with respect to introduced \"internal anisotropic states\", which sounds convincing and neat. It would be well motivated when we consider such directional/angular effects in computational chemistry. The use of series expansion sounds well suited because, similarly to Taylor series, the first term has a good approximation of the original function, and higher-order terms are correction terms to represent deviations of the approximation due to the k-order directional effects (we can control the approximation precision by setting the order k). Also, the use of such anisotropic states in message passing corresponds to multipole-multipole interactions that are also well discussed in the computational chemistry literature as cited.\n\n- The paper comes with three nice case-study demonstrations: (i) 2 of water molecules (water dimers), (ii) QM/MM calculations of 6-20 water molecules (water cluster), (iii) five molecules of varying size solvated in water (a realistic QM/MM system example from Böselt et al. 2021), (iv) QM9 dataset with edge cutoff of 5 angstrom.\n\n\n[Weaknesses]\n\n- As we can see in Figure 2 (water dimer cases), if we have a complete graph (edges are formed between all nodes), then we don't need to use this AMP and just use standard message passing. So I felt that one of big values of this paper would be practical efficiency. Given that we often apply distance cutoff for geometric message passing, this point is also appealing, but the paper can have any discussions and concrete results on how inefficient if we consider a complete graph for message passing.\n\n- \"directional interactions\" in the title was easy to get, but \"long-range interactions\" in the title were unclear to me at first glance. When we use a 3D geometric GNN such as SchNet and DimeNet (unlike 2D non-geometric GNNs), we can apply message passing to all pairs of nodes in theory. We don't necessarily need to apply distance cutoff to edges if the practical efficiency is OK (in the case of small molecules?). It would be nice to have descriptions on this to make clear this point.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "n67MnG1HGqP", "reviewer": "Reviewer_2uEo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focused on the message-passing schemes of a GNN model. It integrates the anisotropic state based on Cartesian multiples to the message-passing models to compensate for the hidden features. Such operation enhances the interactions, i.e., (1) anisotropic long-range, (2) the surrounding fields and particles not part of the graph.", "summary_of_the_review": "In general, it is a  technically sound paper, which proposes a novel message-passing scheme in the Quantum Chemistry fields. ", "strength_and_weaknesses": "1. Strength:\n\nS1. The paper is technically sound.\n\nS2. Extensive experiments are designed to verify the performance of the proposed method.\n\nS3. The paper is well-written and easy to follow.\n\n2. Weaknesses:\n\nW1. There are many other GNN models. I other GNN models? Can you be more clear? I am not sure which GNN model you used.\n\nW2. I noticed that the performance for \"shuffled\" data in Table 2 is relatively bad. Can you explain it? Is there any way to improve it? Since we always use shuffled data for training and test to evaluate deep learning models.\n\nW3. Can you offer the time complexity analysis or add more experiments on the running time?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iHhHgv8WdX", "reviewer": "Reviewer_yByC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propeses a  modified message passing GNN that allows for the efficient incorporation of directional and long-range information. Based on the anisotropic feature, a new message passing functions are designed. ", "summary_of_the_review": "The view of the idea for GNN is novel. The theoretical modeling seems meaningful for some datasets. However, this also brings the underlying computation burdens, especially considering limited performance promotion.", "strength_and_weaknesses": "Strength:\n1. The introduction of QUANTUM MECHANICS/MOLECULAR MECHANICS sees some novel.\n2. Experimental results on WATER DIMERS show the effectiveness of the proposed model.\n\nWeakness:\n\n1. The description of the developed methodology is unclear. It is hard to follow the work. The notation and equation formulation are complicated. Together with the corresponding explanation, they should be largely improved.\n2. The long range information is the main advantage of the proposed method, however, there are no corresponding experimental discuss or theoretical analyses in the manuscript. \n3. The computation complexity analysis is a big concern. The proposed method uses more parameters and computation steps during message passing, this would bring lots of computation burden. However, from the real dataset experimental results in table 4, the performance promotion is slight. \n4 The novelty isn't particularly high. Most techniques used were proposed and demonstrated before. \n5. How about experimental results on the PDBbind dataset.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MN91bmSmT0R", "reviewer": "Reviewer_cgSu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, a message passing GNN is proposed by incorporating an anisotropic state based on Cartesian multiples to address long-range and directional interactions in chemical systems. The proposed model is evaluated on two model systems and two existing datasets for quantum mechanics/molecular mechanics (QM/MM) systems and small organic molecules. ", "summary_of_the_review": "Given that there are several existing papers addressing a similar question, the authors should cover and discuss these related works. The experiments should be better designed with more compared baselines and more suitable datasets. \n\n----------------------------------------------------------------------\nBased on the authors' response:\nThe authors have addressed my concerns about the related works, comparisons to other models, computational cost, and dataset. The updated version has been improved to better demonstrate the proposed method. ", "strength_and_weaknesses": "Pros:\n1. The paper is well-organized.\n2. The paper proposes a practical way to address long-range and directional interactions for QM/MM systems. \n\nCons:\n1. The novelty of the proposed model is limited. There are existing works [1-4] using GNN to deal with QM/MM systems or force fields. There is also work [5] addressing multiples using GNN.\n2. In the experiments, the proposed model is only compared to limited or simple baselines without including enough related and state-of-the-art models. \n3. For the experiment about water dimers, the seems to be no difference between the curves when k=1 and k=2 in Figure 2A. However, the authors mention that \"the learning task seems to be more challenging for the AMP(1) compared to the AMP(2) model, indicated by marginally larger errors\", which is misleading. Moreover, the authors claim that Dimenet and PaiNN \"seem to fail to describe this quadrupole-quadrupole interaction\" without having convincing experimental results. Lastly, the authors use the wrong name (\"Dime\") when referring to DimeNet.\n4. For the experiment about QM/MM systems, the comparison to the previous models is unclear. For example, Table 2 and 3 only show the results of the proposed model without having the results of other models. \n5. For the experiment on QM9 dataset, I'm feeling that this experiment is not very suitable to be shown here, since QM9 is mainly for the prediction of scalar molecular properties. I would suggest using MD17, which is a commonly used benchmark dataset that includes MD trajectories of small organic molecules. The goal is to predict energies and forces, which is more suitable for the proposed model.\n6. Although the authors claim that their proposed modifications require little additional computational cost, there are no empirical results addressing that. \n\n[1] Qiao, Zhuoran, et al. \"Informing geometric deep learning with electronic interactions to accelerate quantum chemistry.\" Proceedings of the National Academy of Sciences 119.31 (2022): e2205221119.\n\n[2] Haghighatlari, Mojtaba, et al. \"Newtonnet: A newtonian message passing network for deep learning of interatomic potentials and forces.\" Digital Discovery (2022).\n\n[3] Unke, Oliver T., et al. \"SpookyNet: Learning force fields with electronic degrees of freedom and nonlocal effects.\" Nature communications 12.1 (2021): 1-14.\n\n[4] Batzner, Simon, et al. \"E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials.\" Nature communications 13.1 (2022): 1-11.\n\n[5] Thürlemann, Moritz, Lennard Böselt, and Sereina Riniker. \"Learning atomic multipoles: Prediction of the electrostatic potential with equivariant graph neural networks.\" Journal of Chemical Theory and Computation 18.3 (2022): 1701-1710.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster