Directional Message Passing for Molecular Graphs
Authors: Johannes Gasteiger
Janek Groß
Stephan Günnemann
Venue:  (2020)
Keywords: GNN
Graph neural network
message passing
graphs
equivariance
molecules

Abstract: Graph neural networks have recently achieved great successes in predicting quantum mechanical properties of molecules. These models represent a molecule as a graph using only the distance between atoms (nodes). They do not, however, consider the spatial direction from one atom to another, despite directional information playing a central role in empirical potentials for molecules, e.g. in angular potentials. To alleviate this limitation we propose directional message passing, in which we embed the messages passed between atoms instead of the atoms themselves. Each message is associated with a direction in coordinate space. These directional message embeddings are rotationally equivariant since the associated directions rotate with the molecule. We propose a message passing scheme analogous to belief propagation, which uses the directional information by transforming messages based on the angle between them. Additionally, we use spherical Bessel functions and spherical harmonics to construct theoretically well-founded, orthogonal representations that achieve better performance than the currently prevalent Gaussian radial basis representations while using fewer than 1/4 of the parameters. We leverage these innovations to construct the directional message passing neural network (DimeNet). DimeNet outperforms previous GNNs on average by 76% on MD17 and by 31% on QM9. Our implementation is available online.

Published as a conference paper at ICLR 2020
DIRECTIONAL MESSAGE PASSING FOR MOLECULAR GRAPHS
Johannes Gasteiger, Janek Groß & Stephan Günnemann Technical University of Munich, Germany (j gastet gross ja, guennemann) @in, tum. de
ABSTRACT
Graph neural networks have recently achieved great successes in predicting quan- tum mechanical properties of molecules. These models represent a molecule as graph using only the distance between atoms (nodes). They do not, however, consider the spatial direction from one atom to another, despite directional infor- mation playing a central role in empirical potentials for molecules, e.g. in angular potentials. To alleviate this limitation we propose directional message passing, which we embed the messages passed between atoms instead of the atoms themselves. Each message is associated with a direction in coordinate space. These directional message embeddings are rotationally equivariant since the associated directions rotate with the molecule. We propose message passing scheme analo- gous to belief propagation, which uses the directional information by transforming spherical Bessel messages based the angle between them. Additionally, well-founded, orthog- functions and spherical harmonics to construct theoretically onal representations that achieve better performance than the currently prevalent Gaussian radial basis representations while using fewer than 1/4 the parameters. We leverage these innovations to construct the directional message passing neural network (DimeNet). DimeNet outperforms previous GNNs on average by 76 % on MD17 and by 31 % QM9, Our implementation available online.
INTRODUCTION
In recent years scientists have started leveraging machine learning to reduce the computation time required for predicting molecular properties from matter of hours and days to mere milliseconds. With the advent of graph neural networks (GNNs) this approach has recently experienced a small revolution, since they do not require any form of manual feature engineering and significantly outperform previous models (Gilmer et al.] 2017]] Schütt et al., 2017). GNNs model the complex interactions between atoms by embedding each atom high-dimensional space and updating these embeddings by passing messages between atoms. By predicting the potential energy these models effectively learn an empirical potential function. Classically, these functions have been modeled as the sum of four parts: (Leach 2001)
Evonds + Eangle + Evossin + Emon-booned-
where Evoods models the dependency on bond lengths, Eangle on the angles between bonds, Evosion on bond rotations, i.e. the dihedral angle between two planes defined by pairs of bonds, and Frse bonded models interactions between unconnected atoms, e.g. via electrostatic or van der Waals interactions. The update messages in GNNs, however, only depend on the previous atom embeddings and the pairwise distances between atoms not on directional information such bond angles and rotations. Thus, GNNs lack the second and third terms of this equation and can only model them via complex higher-order interactions of messages. Extending GNNs to model them directly is not straightforward since GNNs solely rely on pairwise distances, which ensures their invariance to translation, rotation, and inversion of the molecule, which are important physical requirements.
In this paper, we propose to resolve this restriction by using embeddings associated with the directions to neighboring atoms, i.e. by embedding atoms set of messages. These directional message
https:// /wt::////wwww.....o dam1. tum. www.tu.mmee.......... menett

---

Published as a conference paper at ICLR 2020
embeddings equivariant with respect the above transformations since the directions move with the molecule. Hence, they preserve the relative directional information between neighboring atoms. We propose to let message embeddings interact based on the distance between atoms and the angle between directions. Both distances and angles are invariant to translation, rotation, and inversion of the molecule, required. Additionally, we show that the distance and angle can be jointy represented in principled and effective manner by using spherical Bessel functions and spherical harmonics. We leverage these innovations construct the directional message passing neural network (DimeNet). DimeNet can learn both molecular properties and atomic forces. It is twice continuously differentiable and solely based on the atom types and coordinates, which are essential properties for performing molecular dynamics simulations. DimeNet outperforms previous GNNs on average by 76% % on MD17 and by 31 % on QM9, Our paper's main contributions are:
1. Directional message passing, which allows GNNs incorporate directional information by connecting recent advances in the fields of equivariance and graph neural networks as well as ideas from belief propagation and empirical potential functions such as Eq. [
Theoretically principled orthogonal basis representations based on spherical Bessel functions and spherical harmonics. Bessel functions achieve better performance than Gaussian radial basis functions while reducing the radial basis dimensionality by 4x or more.
3. The Directional Message Passing Neural Network (DimeNet): A novel GNN that leverages these innovations to set the new state of the art for molecular predictions and is suitable both for predicting molecular properties and for molecular dynamics simulations.
RELATED WORK
ML for molecules. The classical way of using machine learning for predicting molecular properties is combining an expressive, hand-crafted representation of the atomic neig neighborhood (Bartók et al.. 2013) with Gaussian processes (Bartók et al. 2010: 2017, Chmiela et al.) 2017) or neural networks (Behler & Parrinello 2007p. Recently, these methods have largely been superseded by graph neural networks. which do not require any hand-crafted features but learn representations solely based on the
atom types and coordinates molecules (Duvenaud et al.) 2015 Gilmer al. 2017 Schhitt et al.) 2017]
Hy et al.] 2018 Unke & Meuwly] 2019). Our proposed message embeddings can also be interpreted as directed edge embeddings or embeddings on the line graph (Chen et al. 2019b). (Undirected)
edge embeddings have already been used in previous GNNs for molecules (Jørgensen et al.) 2018
Chen et al., 2019a). However, these GNNs use both node and edge embeddings and do not leverage any directional information.
Graph neural networks. GNNs were first proposed the 90s (Baskin et al. 1997) and 00s (Gori et al. 2005 Scarselli et al. 2009). General GNNs have been largely inspired by their application to molecular graphs and have started to achieve breakthrough performance in various tasks at around the same time the molecular variants did (Kipf & Welling, [017]] Gasteiger jet al. 2019| Zambaldi et al. 2019). Some recent progress has been focused on GNNs that are more powerful than the I-
1997 Sperduti & Starita
test of isomorphism (Morris et al., 2019 Maron et 2019). However, for molecular predictions these models are significantly outperformed by GNNs focused on molecules (see Sec. 7.. Some recent GNNs have incorporated directional information by considering the change in local coordinate systems per atom (Ingraham et al., 2019). However, this approach breaks permutation invariance and is therefore only applicable to chain-like molecules (e.g. proteins).
Equivariant neural networks. Group equivariance as principle of modern machine learning was first proposed by Cohen & Welling (0016). Following work has generalized this principle to spheres (Cohen et al.) 2018). molecules (Tomas et al.) 2018), volumetric data (Weiler et al., 2018). and general manifolds (Cohen et al.) 2019). Equivariance with respect to continuous rotations has been achieved so far by switching back and forth between Fourier and coordinate space in each layer (Cohen et al. 2018 or by using a fully Fourier space model (Kondor et al., 20188] Anderson et al. 2019). The former introduces major computational overhead and the latter imposes significant constraints on model construction, such as the inability of using non non-linearities. Our proposed solution does not suffer from either of those limitations.
2

---

Published as a conference paper at ICLR 2020
3 REQUIREMENTS FOR MOLECULAR PREDICTIONS
In recent years machine learning has been used to predict a wide variety of molecular properties, both low-level quantum mechanical properties such as potential energy, energy of the highest occupied molecular orbital (HOMO), and the dipole moment and high-level properties such as toxicity, permeabilitty, and adverse drug reactions (Wu al. 2018). In this work we will focus on scalar regression targets, i.e. targets E R. A molecule is uniquely defined by the atomic numbers = (..,.... zN } and positions X {æi..... Øv Some models additionally use auxiliary information @ such as bond types or electronegativity of the atoms. We do not include auxiliary features in this work since they are hand-engineered and non-essential. In summary, we define ML model for molecular prediction with parameters 0 via fe {X, > + IR
Symmetries and invariances. All molecular predictions must obey some basic laws of physics, either explicitly or implicitly. One important example of such are the fundamental symmetries of physics and their associated invariances. In principle, these invariances can be learned by any neural network via corresponding weight matrix symmetries (Ravanbakhsh et al. 2017). However, not explicitly incorporating them into the model introduces duplicate weights and increases training time and complexity. The most essential symmetries are translational and rotational invariance (follows from homogeneity and isotropy), ) permutation invariance (follows from the indistinguishabiiiy of particles), and symmetry under parity, i.c. under sign flips of single spatial coordinates.
Molecular dynamics. Additional requirements arise when the model should be suitable for molecular dynamics (MD) simulations and predict the forces F; acting on each atom. The force field is conservative vector field since it must satisfy conservation of energy (the necessity of which follows from homogeneity of time (Noether, (918)). The easiest way of defining a conservative vector field is via the gradient of a potential function. We can leverage this fact by predicting a potential instead of the forces and then obtaining the forces via backpropagation the atom coordinates, i.e. F2 x) = öm, fo X, 2). We can even directly incorporate the forces in the training loss and directly train model for MD simulations (Pukrittayakamee et al. 2009):
3
N
3
( dx.
LMD xx 2) = fo x 2) 8
2)
Fia (x)
(2)
3N
2=1
a=1
where the target = is the ground- energy (usually available as well). F are the ground-truth forces, and the hyperparameter sets the forces' loss weight. For stable simulations F; must be continuously differentiable and the model So itself therefore twice continuously differentiable. We hence cannot discontinuous transformations such as ReI.U non-linearities. Furthermore, since the atom positions X can change arbitrarily we cannot use pre-computed auxiliary information e such bond types.
DIRECTIONAL MESSAGE PASSING
Graph neural networks. Graph neural networks treat the molecule graph, in which the nodes are atoms and edges are defined either via predefined molecular graph or simply by connecting atoms that lie within cutoff distance Each edge is associated with pairwise distance between atoms d, ||ær, x, I|2- GNNs implement all of the above physical invariances by construction since they only pairwise distances and not the full atom coordinates. However, note that predefined molecular graph or a step function-like cutoff cannot be used for MD simulations since this would introduce discontinuities in the energy landscape. GNNs represent each atom via an atom embedding h; E HR The atom embeddings are updated in each layer by passing messages along the molecular edges. Messages are usually transformed based on an edge embedding e(j) E ff. and summed over the atom's 's neighbors Nº, i.c. the embeddings are updated in layer I via
h (I+1) fupdaat (1) Σ fim(h (1) (())).
(3)
JEN;
with the update function fupdate and the interaction function f.n. which are both commonly imple-
(1)
mented using neural networks. The edge embeddings ((g) usually only depend on the interatomic
distances, but can also incorporate additional bond information (Gilmer et al. 2017) or be recursively
updated in each layer using the neighboring atom embeddings (Jørgensen et al. 2018).
3

---

Published as a conference paper at ICLR 2020
Directionality. In principle, the pairwise distance matrix contains the full geometrical information of the molecule. However, GNNs do not use the full distance matrix since this would mean passing messages globally between all pairs of atoms, which increases computational complexity and can lead to overfitting. Instead, they usually use cutoff distance c, which means they cannot distinguish between certain molecules (Xu et al. 2019). E.g. at a cutoff of roughly Å a regular GNN would not be able to distinguish between hexagonal (e.g. Cyclohexane) and two triangular molecules (e.g. Cyclopropane) with the same bond lengths since the neighborhoods of each atom are exactly the same for both (see Appendix, Fig. (). This problem can be solved by modeling the directions to neighboring atoms instead of just their distances. A principled way of doing so while staying invariant transformation group G (such as described in Sec. B) is via group-equivariance (Cohen & Welling 2016). A function f X + Y is defined as being equivariant if f(P (r)) = sy (f(z)) with the group action in the input and output space " and yo However, equivariant CNNs only achieve equivariance with respect to a discrete set of rotations (Cohen & Welling 2016). For a precise prediction of molecular properties we need continuous equivariance with respect to rotations, i.e. to the SO(3) group.
Directional embeddings. We solve this problem by noting that an atom by itself is rotationally invariant. This invariance is only broken by neighboring atoms that interact with it, i.c. those inside the cutoff Since each neighbor breaks up to one rotational invariance they also introduce additional degrees of freedom, which we need to represent in our model. We can do so by generating separate embedding my for each atom and neighbor j by applying the same learned filter in the direction of each neighboring atom (in contrast to equivariant CNNs, which apply filters in fixed, global directions). These directional emheddings are equívariant with respect to global rotations since the associated directions rotate with the molecule and hence conserve the relative directional information between neighbors.
Representation via joint 2D basis. We use the directional information associated with each embed- ding by leveraging the angle (k(jj.j)) = Lxgw æ,æ; when aggregating the neighboring embeddings m&j of j.-- We combine the angle with the interatomic distance dæj associated with the incoming (kj.j)) E Nsus NNaa using a 2D representation based on spherical Bessel functions and spherical harmonies, as explained in Sec. 5 We empirically found that this basis representation provides a better inductive bias than the raw angle alone. Note that by only using interatomic distances and angles our model becomes invariant rotations.
message myj and jointly represent both in ass
Message embeddings. The directional embedding myj associated with the atom pair ji can thought of message being sent from atom atom i. Hence, in analogy to belief propagation, we embed each atom i using a set of incoming messages mjs i.e. h, = EEEN, mjjs and update the message mjj based on the incoming messages mkj (Yedidia et al. 2003). Hence, as illustrated in Fig. [ aggregation scheme for message embeddings as
l22
mii
km
miki
we define the update function and
m&ji
j
ka
Mika
(1+1) fupdane (m miji
(1) js
Σ kEN
fia(mm)
(!) CRBF O
(kj.t))
)),
(4) Figure I: Aggregation scheme for message embeddings.
where (it) denotes the radial basis function representation of the inter-
atomic distance dji, which will be discussed in Sec. 5 We found this aggregation scheme to not only have a nice analogy to belief propagation, but also to empirically perform better than altermatives. Note that since Jinr now incorporates the angle between atom pairs, or bonds, we have enabled our model to directly learn the angular potential Eangle. the second term in Eq. T Moreover, the message embeddings essentially embeddings of atom pairs, used by the provably more powerful GNNs based on higher-order distinguish molecules that regular GNN cannot (c.g. the previous example of a hexagonal and two triangular molecules) (Morris et al. 2019).
Phisfciler-Lhhmale tests of isomorphism. Our model can therefore provably
5 PHYSICALLY BASED REPRESENTATIONS
Representing distances and angles. For the interaction function fin in Eq. we use joint (kj.jt) of the angles ((jj)8) between message embeddings and the interatomic representation assF
4

---

Published as a conference paper at ICLR 2020
distances d ||a - 225122 well representation GRBF of the distances dji- Earlier works have used a set of Gaussian radial basis functions to represent interatomic distances, with tightly spaced means that are distributed e.g. uniformly (Schütt et al. 2017) exponentially (Unke & Meuwly] 2019). Similar in spirit to the functional bases used by steerable CNNs (Cohen & Welling 2017: Cheng et al. 2019) we propose to use an orthogonal basis instead, which reduces redundancy and thus improves parameter efficiency. Furthermore, basis chosen according to the properties of the modeled system can even provide helpful inductive bias. We therefore derive a proper basis representation for quantum systems next.
(ji)
From Schrödinger to sourier-Bessel. To construct a basis representation in a principled manner we first consider the space of possible solutions. Our model aims at approximating results of density functional theory (DFT) calculations, i.c. results given by an electron density ( with the electron wave function (d) and d = 3y Z.. The solution space of Y (d) is defined by the (d)
time-indeppndent Schrödinger equation
(d), with constant mass m and energy E. We do not know the potential ((d) and so choose it in an uninformative way by simply setting it to 0 inside the cutoff distance c (up to which we pass messages between atoms) and to oc outside. Hence, we arrive at the Helmholtz equation (V2 + ) (d) = 0, with the wave 22mF and the boundary condition ((c) = at the cutoff c. Separation of variables in polar coordinates (d, a, ()) yields the solution (Griffiths & Schroeter 2018
V + V(d)
number k
ad,
(atmJi (kd) + bmm bmm.u((kd))
(5)
1=0 m=-!
with the spherical Bessel functions of the first and second kind ji and y and the spherical harmonics Y.m. As common in physics we only use the regular solutions, i.. those that do not approach -co the origin, and hence set him- = Recall that our first goal is to construct joint 2D basis for dep and (((kjj))- i.e. function that depends on d and a single angle a. To achieve this we set m = 0 and obtain ss a) >i (a))())) The boundary conditions are satisfied by setting k = Sin. where 2In is the n-th root of the l-order Bessel function, which precomputed numerically. Normalizing SSB inside the cutoff distance e yields the 2D spherical Fourier-Bessel
:
basis SAF by
(kj.jt) E R NNmm--Namm which is illustrated in Fig. and defined
Figure 2: 2D spheri-
cal Fourier-Bes Bessel basis
2 Fin ()), j2z (2t))
ÖSBF.to (d, ox)
ÖSBF in (d,
ji(
(6)
with E (0..NNHBr. - 1] and [1R.Ns Our second goal is constructing a radial basis for dzj, i.e. a function that solely depends on d and not on the angles a and y. We achieve this by setting
[1..
20
ēxnr (dje)
10
m = 0 and obtain v (d) = ajo(" 20.n (<=="dd d), with roots at 20,x nı.
Normalizing this function on [0. c] and using ju(d) sin(d)/ /d gives
the radial basis êRAF E R NRBE as shown in Fig. 3 and defined by
0.0
0.5
1.0
d/c
2 sin 2M
ERAT, (d) =
Figure 3: Radial Bessel ba-
with n E NRBE). Both of these bases are purely real-valued and orthogonal in the domain of interest. They furthermore enable us to bound the highest-frequenc
sis for NRBF 5.
2T Wdg; < Naur and Wa,, < Naur This restriction is an effective way of regularizing the model and ensures that predictions stable to small perturbations. We found NSRer and NRBF = 16 radial basis functions to be more than sufficient. Note that NRBF 4x lower than PhysNet's 's 64 (Unke & Meuwly 2019) and 20x lower than SchNer's 300 radial basis functions (Schütt et al. 2017).
components by wa < Namr
(Rsjj)) Continuous cutoff. dSBF and (d)r(d) are not twice continuously differentiable due to the step function cutoff at c. To alleviate this problem we introduce an envelope function u(d) that has a root of multiplicity 3 at d c, causing the final functions agar (d) u(dd)Ran(( and eRBr(d)
5

---

Published as a conference paper at ICLR 2020
Embedding:
Interaction:
Output:
33 F4
550
(Rout)
may
UP
Model:
3 M.
da
W
Emnbedding
(WW)+
o(W +-))
Wn
10
RBF
Fibeeding
F(WL+
Wn
WD
Interaction
mi
Output
Directional message passing
Residual
#(WT
a(W
Interaction
Residual:
WD
Interaction
(W
Residual
(WL-M))
Residual
Output
Figure 4: The DimeNet architecture. D denotes the layer's 's input and || denotes concatenation. The distances djs are represented using spherical Bessel functions and the distances dæj and angles (K(k))H)) are jointly represented using 21) spherical Fourier-Besse the inital message embeddings m,j- These embeddings are updated in multiple interaction blocks via directional message passing, which uses the neighboring messages maj,, k E NN {i), the
basis. An embedding block generates
(k.jj))) and the distance representations ERBF. Each block passes the resulting 2D representations OSBF
(fi)) embeddings an output block, which transforms them using the radial basis egar and sums them up atom. Finally, the outputs of all layers are summed up to generate the prediction.
u(d) (d) and their first and second derivatives to go to at the cutoff. We achieve this with the polynomial
u(d) 1 (p (p+ +) d² p(p + 2)d
p(p p(p+ + 2t
2
where p E No. We did not find the model to be sensitive to different choices of envelope functions and choose p = Note that using an envelope function causes the bases to lose their orthonormality, which we did not find to be problem in practice. We furthermore fine-tune the Bessel wave numbers kn = az used in ÊRUF E R Neur via backpropagation after initializing them to these values, which we found to give a small boost in prediction accuracy.
DIRECTIONAL MESSAGE PASSING NEURAL NETWORK (DIMENET)
The Directional Message Passing Neural Network's (DimeNet) design is based on a streamlined version of the PhysNet architecture (Unke & Meuwly 2019), in which we have integrated directional message passing and spherical Fourier-Bessed representations. DimeNet generates predictions that invariant atom permutations and translation, rotation and inversion of the molecule. DimeNet is suitable both for the prediction of various molecular properties and for molecular dynamics (MD) simulations. is twice continuously differentiable and able to learn and predict atomic forces via backpropagation, as described in Sec. B] The predicted forces fulfill energy conservation by construction and are equivariant with respect to permutation and rotation. Model differentiability in combination with basis representations that have bounded maximum frequencies furthermore guarantees smooth predictions that are stable to small deformations. Fig. 4 ggives an overview the architecture.
Embedding block. Atomic numbers are represented by learnable, randomly initialized atom type em- beddings h E F that are shared across molecules. The first layer generates message embeddings from these and the distance between atoms via
m (1) = ((h
(0) |/
(o)
( | W + b),
(9)
where || denotes concatenation and the weight matrix W and bias b are learnable.

---

Published as a conference paper at ICLR 2020
Table 1: MAE on QM9, DimeNet sets the state of the art on targets, outperforming the second-best
model on average by 31 % (mean std. MAE).
Target
Unit
PPGN
SchNet
PhysNet
MEGNet-s
Cormorant
Neme
D
0.047
0.033
0.0529
0.05
0.13
0.0286
4n
0.131
0.235
0.0615
0.081
0.092
0.0469
"HOMO
meV
40.3
32.9
43
36
27.8
ELUMO
meV
32.7
34
24.7
44
36
19.7
Ae
meV V
60.0
63
42.5
66
60
34.8
R2)
ao 2
0.592
0.073
0.765
0.302
0.673
0.331
ZPVE
meV 3.12
1.7
1.39
1.43
1.98
1.29
Uo
me V
36.8
14
8.15
12
28
8.02
U
meV 36.8
19
8.34
13
7.89
H
me V
36.3
14
8.42
12
8.11
G
meV
36.4
14
9.40
12
8.98
cal
0.055
0.033
0.0280
0.029
0.031
0.0249
K
std. MAE logMAE
%
1.84
1.76
1.37
1.80
2.14
1.05
4.64
-5.17
-5.35
-5.17
-4.75
-5.57
Interaction block. The embedding block is followed by multiple stacked interaction blocks. This (kigi)
block implements fint and fupdate of Eq. as shown in Fig. Note that the 2D representation a first transformed into an Nbilinear-iiman dimiess-dimensionnall representation via a linear layer. The main purpose of (kj.jt) independent of the subsequent bilinear layer, which uses
this is to make the dimensionality of agF a comparatively large Nutinar X x F -ddimensional weight tensor. We have also experimented with using bilinear layer for the radial basis representation, but found that the element wise multiplication (ft) @RAF W C Mig performs better, which suggests that the 2D representations require more complex transformations than radial information alone. The interaction block transforms each message embedding my, using multiple residual blocks, which are inspired by ResNet (He et al. 2016) and consist of two stacked dense layers and skip connection.
Output block. The message embeddings after each block (including the embedding block) are passed output block. The output block transforms each message embedding mjj using the (ji) radial basis CRBE which ensures continuous differentiability and slightly improves performance. Afterwards the incoming messages are summed up per atom obtain h, = E, mjj. which is then
transformed using multiple dense layers to generate the atom-wiss output (!) These outputs are then summed up obtain the final prediction / Σ. 4.
Continuous differentiability. Multiple model choices were necessary achieve twice continuous model differentiabillity First, DimeNet uses the self-gated Swish activation function (xx) sigmoid(z) (Ramachandran et al. 2018) instead of a regular ReLU activation function. Second, we multiply the radial basis functions (R)r(d) with envelope function u(d) that has root of multiplicity at the cutoff c. Finally, DimeNet does not use any auxiliary data but relies on atom types and positions alone.
EXPERIMENTS
Models. For hyperparameter choices and training setup see Appendix B We use 6 state-of-the- art models for comparison: SchNet (Schütt al.) (017). PhysNet (results based on the reference implementation) (Unke & Meuwly 2019). provably powerful graph networks (PPGN, results provided by the original authors) (Maron et al. 2019), DEGNee-simple (without auxiliary information) (Chen) al. 2019a). Cormorant (Anderson al. 2019), and symmetrized gradient-domale machine learning (sGDML) (Chmiela et 2018). Note that sGDML cannot be used for QM9 since can only be trained single molecule.
QM9, We test DimeNet's performance for predicting molecular properties using the common QM9 benchmark (Ramakrishnan et al. 2014). It consists of roughly 130 000 molecules in equilibrium
7

---

Published as a conference paper at ICLR 2020
Table 2: MAE on MD17 using 1000 training samples
(energies in kcal forces in _eck)). DimeNet outper-
mai' forms SchNet by large margin and performs roughly on par with sGDML.
sGDML
SchNet
DimeNet
Energy
0.19
0.37
0.204
Aspirin
Forces
0.68
1.35
0.499
Energy
0.10
0.08
0.078
Renzene
Forces
0.06
0.31
0.187
Energy
0.07
0.08
0.064
Ethanol
Forces
0.33
0.39
0.230
Figure 5: Examples of DimeNet filters. They
Energy
0.10
0.13
0.104
exhibit a clear 2D structure. For details see
Malonaldehyde
Forces
0.41
0.66
0.383
Appendix D
Energy
0.12
0.16
0.122
Naphthalene
Forces
0.11
0.58
0.215
Energy
0.12
0.20
0.134
Table 3: Ablation studies using multi-task learning on QM9. All of our contributions have significant impact on performance.
Salicylic acid
Forces
0.28
0.85
0.374
Energy
0.10
0.12
0.102
Toluene
Forces
0.14
0.57
0.216
Uracil
Energy
0.11
0.14
0.115
Variation
MAE MAE DemeNei
AlogMAE
Forces
0.24
0.56
0.301
Energy
2.53
3.32
2.49
Gaussian RBF
110 %
0.10
MAE (%)
Forces
1.01
2.38
1.10
Nster 1
126 % 168 %
0.11
Node embeddings
0.45
with up to 9 heavy C, O, N, and F atoms. We use 110000 molecules in the training, 10000 in the validation and 10 831 in the test set. We only use the atomization energy for Up. U. H. and G, i.e. subtract the atomic reference energies, which are constant per atom type, and perform the training using eV. In Table we report the mean absolute error (MAE) of each target and the overall mean standardized MAE (std. MAE) and mean standardized logMAE (for details see Appendix C.. We predict Ae simply by taking fLUMO <HOMO, since it is calculated in exactly this way by DFT calculations. We train a separate model for each target, which significantly improves results compared to training a single shared model for all targets (see App. E DimeNet sets the new state of the art on 11 out of 12 targets and decreases mean std. MAE by 31 % and mean logMAE by 0.22 compared to the second- best model.
MD17. We use MD17 (Chmiela et al., 2017 to test model performance in molecular dynamics simulations. The goal of this benchmark is predicting both the energy and atomic forces of eight small organic molecules, given the atom coordinates of the thermalized (i.e. non-equilibrium, slightly moving) system. The ground truth data is computed via molecular dynamics simulations using DFT. A separate model is trained for each molecule, with the goal of providing highly accurate individual predictions. This dataset is commonly used with 50 000 training and 100 validation and test samples. We found that DimeNet can match state-of-the-ar performance in this setup. E.g. for Benzene, depending the force weight p. DimeNet achieves 0.035 kcal mol- MAE for the energy
or 0.07 kcal mol and 0.17 kcal mol A- for energy and forces, matching the results reported by Anderson al. (2019) and Unke & Meuwly (2019). However, this accuracy is two orders of magnitude below the DFT calculation's 's accuracy (approx. 2.3 kcal mol for energy (Faber et al. 2017)), any remaining difference real-world data is almost exclusively due to errors in the DFT simulation. Truly reaching better accuracy can therefore only be achieved with more precise ground-truth data, which requires far more expensive methods (e.g. CCSD(T)) and thus ML models that are more sample-efficient (Chmiela et al.) 2018). We therefore instead test our model on the harder task of using only 1000 training samples. As shown in Table DimeNet outperforms SchNet by a large margin and performs roughly on par with SGDML. However, sGDML uses hand engineere descriptors that provide strong advantage for small datasets, can only be trained on a single molecule (a fixed set of atoms), and does not scale well with the number of atoms or training samples.
Ablation studies. To test whether directional message passing and the Fourier- Besssel basis are the actual reason for DimeNet's improved performance, ablate them individually and compare the mean standardized MAE and logMAE for multi-task learning on QM9. Table 3 that both of our contributions have significant impact the model's performance. Using 64 Gaussian RBFs
8

---

Published as a conference paper at ICLR 2020
instead of 16 and 6 Bessel basis functions to represent djs and dæj increases the error by 10 %, which shows that this basis does not only reduce the number of parameters but additionally provides a helpful inductive bias. DimeNet's error increases by around 26 % when we ignore the angles between messages by setting Nsuar = 1, showing that directly incorporating directional information does indeed improve performance. Using node embeddings instead of message embeddings (and hence also ignoring directional information) has the largest impact and increases MAE by 68 %. at which point DimeNet performs worse than SchNet. Furthermore, Fig. S shows that the filters exhibit structurally meaningful dependence on both the distance and angle. For example, some of these filters are clearly being activated by benzene rings (120" angle, 1.39 Á distance). This further demonstrates that the model learns leverage directional information.
CONCLUSION
In this work we have introduced directional message passing, a more powerful and expressive interac- tion scheme for molecular predictions. Directional message passing enables graph neural networks to leverage directional information in addition to the interatomic distances that are used by normal GNNs. We have shown that interatomic distances can be represented in a principled and effective manner using spherical Bessel functions. We have furthermore shown that this representation can be extended to directional information by leveraging 2D spherical Merrier-Bessel basis functions. We have leveraged these innovations to construct DimeNet, GNN suitable both for predicting molec- ular properties and for use in molecular dynamics simulations. We have demonstrated DimeNet's performance on QM9 and MD17 and shown that contributions are the essential ingredients that enable DimeNet's ttae-of-thheart performance. DimeNet directly models the first two terms in Eq. [ which are known as the important "hard" degrees of freedom in molecules (Leach) 200T). Future work should aim at also incorporating the third and fourth terms of this equation. This could improve predictions even further and enable the application molecules much larger than those used in common benchmarks like QM9.
ACKNOWLEDGMENTS
This research was supported by the German Federal Ministry of Education and Research (BMBF), grant no. 011S18036D 18036B, and by the Deutsche Forschungsgemeinscaat (DFG) through the Emmy Noether grant GU 1409/2-1 and the TUM International Graduate School of Science and Engineering (IGSSE), GSC 81. The authors of this work take full responsibilities for its content.
REFERENCES
Brandon M. Anderson, Truong-Sor Son Hy, and Risi Kondor. Cormorant: Covariant Molecular Neural Networks. In NeurlPS, 2019.
Albert P. Bartók, Mike C. Payne. Risi Kondor, and Gábor Csányi. Gaussian Approximation Potentials: The Accuracy of Quantum Mechanics, without the Electrons. Physical Review Letters, 104(13): 136403, April 2010.
Albert Bartók, Risi Kondor, and Gábor Csányi. On representing chemical environments. Physical Review B, 87(18):184115 May 2013.
Albert P. Bartók. Sandip De, Carl Poelking, Noam Bernstein, James R. Kermode, Gábor Csányi, and Michele Ceriotti. Machine learning unifies the modeling of materials and molecules. Science Advances, 3(12) 1701616,, December 2017.
Igor I. Baskin, Vladimir A. Palyulin, and Nikolai S. Zefirov. Neural Device for Searching Direct Correlations between Structures and Properties of Chemical Compounds. Journal of Chemical Information and Computer Sciences, 37(4):715-721, July 1997.
Jörg Behler and Michele Parrinello. Generalized Neural-Network Representation of High- Dimensional Potential-Energy Surfaces. Physical Review Letters, 98( (8(14): 146401, April 2007.
Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph Networks Universal Machine Learning Framework for Molecules and Crystals. Chemistry of Materials, 31 (9):3544-3572, May 2019a.

---

Published as a conference paper at ICLR 2020
Zhengdao Chen, Lisha Li, and Joan Bruna. Supervised Community Detection with Line Graph Neural Networks. In ICLR, 2019b.
Xiuyuan Cheng, Qiang Qiu, A. Robert Calderbank, and Guillermo Sapiro. RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariin Deep Networks. In ICLR, 2019.
Stefan Chmiela, Alexandre Tkatchenko, Huziel E. Sauceda, Igor Poltavsky, Kristof Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science Advances, 3(5)xe May 2017.
Stefan Chmiela, Huziel E. Sauceda, Klaus- Müller, and Alexandre Tkatchenko. Towards exact molecular dynamics simulations with machine-learned force fields. Nature Communications, 9(1): 1-10, September 2018.
Taco Cohen and Max Welling. Group Equivariant Convolutional Networks. In ICML, 2016.
Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge Equivariant Convolutional Networks and the Icosahedral CNN. In ICML, 2019.
Taco S. Cohen and Max Welling. Steerable CNNs. In ICLR, 2017.
Taco Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. ICLR, 2018.
David K. Duvenaud, Dougal Maclaurin, Jorge Aguilera-pparaguurre, Rafael Goomez-oomarrelee Timothy Hirzel, Alán Aspuru-Guzik, and Ryan Adams. Convolutional Networks on Graphs for Learning Molecular Fingerprints. In NIPS, 2015.
Felix A. Faber, Luke Hutchison, Bing Huang, Justin Gilmer, Samuel S. Schoenholz, George E. Dahl, Oriol Vinyals, Steven Kearnes, Patrick F. Riley, and O. Anatole von Lilienfeld. Machine learning prediction errors better than DFT accuracy. CoRR, 1702.05532, February 2017.
Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then Propagate: Graph Neural Networks Meet Personalized PageRank. In /CLR, 2019.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Message Passing for Quantum Chemistry. In ICML, 2017.
M. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In IEEE International Joint Conference on Neural Networks, July 2005.
David J. Griffiths and Darrell F. Schroeter. Introduction to Quantum Mechanics. edition, August 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
Truong Son Hy, Shubhendu Trivedi, Horace Pan, Brandon M. Anderson, and Risi Kondor. Predicting molecular properties with covariant compositional networks. The Journal of Chemical Physics, 148(24):241755, June 2018.
John Ingraham, Vikas K. Garg, Regina Barzilay, and Tommi S. Jaakkola. Generative Models for Graph-Based Protein Design. In ICLR workshop, 2019.
Peter Bjørn Jørgensen, Karsten Wedel Jacobsen, and Mikkel N. Schmidt. Neural Message Passing with Edge Updates for Predicting Properties of Molecules and Materials. CoRR, 1806,03146, 2018.
Thomas N. Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR, 2017.
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-Gordee Nets: a Fully Fourier Space Spherical Convolutional Neural Network. In NewrIPS, 2018.
Andrew R. Leach. Molecular Modelling: Principles and Applications. 2001.
10

---

Published as a conference paper at ICLR 2020
Haggai Maron, Heli Hamu, Hadar Serviansky, and Yaron Lipman. Provably Powerful Graph Networks. NeurIPS, 2019.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and Leman Go Neural: Higher-Order Graph Neural Networks. In AAAI, 2019.
Emmy Noether. Invariante Variationsprobleme.. Nachrichten von der Gesellschaft der Wissenschaften 2u Göttingen, Mathematisch- saahhemmtisch-PPysskalsssi Klasse, 1918:235 257-257,, 1918.
A. Pukrittayakamee, M. Malshe, M. Hagan, L. M. Raff, R. Narulkar, S. Bukkapatnum, and R. Ko- manduri. Simultaneous fitting of potential-energy surface and its corresponding force fields using feedforward neural networks. The Journal of Chemical Physics, 130(13): April 2009.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for Activation Functions. In ICLR workshop, 2018.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld, Quantum chemistry structures and properties of 134 kilo molecules. Scientific Dala, 1(1):1-7, August 2014.
Siamak Ravanbakhsh, Jeff G. Schneider, and Barnabás Póczos. Equivariance Through Parameter- Sharing. In JCML, 2017.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the Convergence of Adam and Beyond. In ICLR, 2018.
Scarselli, M. Gori, Ah Chung Tsoi, Hagenbuchner, and G. Monfardini. The Graph Neural Network Model. JEEE Transactions on Neural Networks, 20(1):61-80) January 2009.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Rooe Robert Müller. SchNet: A continuous- filter convolutional neural network for modeling quantum interactions. In NIPS, 2017.
A. Sperduti and A. Starita. Supervised neural networks for the classification of structures. IEEE Transactions on Neural Networks, 8(3):714-735, May 1997.
Nathaniel Thomas, Tess Smidt, Steven M. Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor Field Networks: Rotation- and Transiation-EEuiivarian Neural Networks for 3D Point Clouds. CoRR, 1802.08219, 2018.
Oliver T. Unke and Markus Meuwly. PhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments, and Partial Charges. Journal of Chemical Theory and Computation, 15(6): 1678-3693, June 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D Stecrable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. In NeurlPS, 2018.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: a benchmark for molecular machine learning, Chemical Science, 9():513-530. 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are Graph Neural Networks? In ICLR, 2019.
Jonathan S Yedidia, William T Freeman, and Yair Weiss. Understanding belief propagation and its generalizations. In Exploring artificial intelligence in the new millennium, volume 8, pp. 236-239. 2003.
Vinícius Flores Zambaldi, David Raposo. Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter W. Battaglia. Deep reinforcement learning with relational inductive biases. In ICLR, 2019.
11

---

Published as a conference paper at ICLR 2020
A INDISTINGUISHABLE MOLECULES
Figure 6: A standard non-directional GNN cannot distinguish between hexagonal (left) and two triangular molecules (right) with the same bond lengths, since the neighborhood of each atom is exactly the same. An example of this would be Cyclohexane and two Cyclopropane molecules with slightly stretched bonds, when the GNN either uses the molecular graph or cutoff distance of с S 2.5 Å. Directional message passing solves this problem by considering the direction of each bond.
B EXPERIMENTAL SETUP
The model architecture and hyperparameters were optimized using the QM9 validation set. We use 6 stacked interaction blocks and embeddings of size F 128 throughout the model. For the basis functions we choose NSHBF = 7 and NSRBF NRAF 6. For the weight tensor in the interaction block
Notlinear 8. We did not find the model be very sensitive to these values as long as they were large enough (i.e. at least 4).
We found the cutoff c = Å and the learning rate x 10 10-3 to be rather important hyperparameters. We optimized the model using AMSGrad (Reddi et al.] 2018) with 32 molecules per mini-batch. We use a linear learning rate warm-up over 3000 steps and an exponential decay with ratio 0.1 every 2 000 000 steps. The model weights for validation and test
obtained using exponential moving average (EMA) with decay rate 0.999. For MD17 we use the loss function from Eq. 2 with foree weight p 100, like previous models Schütt et al. (2017). Note that p presents trade-off between energy and force accuracy. It should be chosen rather high since the forces determine the dynamics of the chemical system (Unke & Meuwly 2019). We use early stopping on the validation loss. On QM9 train for most 000 000 and on MD17 for at most 100 000 steps.
C SUMMARY STATISTICS
We summarize the results across different targets using the mean standardized MAE
M
N
(m)
X,, 2;) ((()
1
std. MAE
(10)
M
N
m=1
and the mean standardized logMAE
c(m)
[X,, x2) (((m))
logMAE 1 M
Σ
log
(11)
0m
with target index m, number of targets M 12, dataset size N. ground truth values fim) model
ff ((m) inputs X, and 2,, and standard deviation om ((m) Std. MAE reflects the average error compared to the standard deviation of each target. Since this enror is dominated few difficult
targets (c.g. (HOMO) we also report logMAE, which reflects every relative improvement equally but sensitive outliers, such as SchNet's result on (R2).
12

---

Published as a conference paper at ICLR 2020
D DIMENET FILTERS
To illustrate the filters learned by DimeNet we separate the spatial dependency in the interaction function fint via
mm dit daj. (kk.))))) = Σ (o m b)l, filter, (dje)22nte22. (dks, ((kjjo))
(12)
The filters futett, R
R and f R x [0, 2 + R are given by
fülterl,n (d) (RRFFeRF(d)))
(13)
f (d, (ds,) W,.:
(14)
where WRBF, WSB,, and W are learned weight matrices/tensors, eRaF(d) is the radial basis represen- tation, and asar(d, (d, ax) the spherical Fourier-Besse representation. Fig. [ shows how the first 15 elements of f (d, a:) vary with d and when choosing the tensor slice n (with at the top the figure).
E MULTI-TARGET RESULTS
Table MAE on QM9 with multi-target learning. Single-target learning significantly improves performance on all targets. Using separate output block per target slightly reduces this difference with little impact on training time.
Target
Unit
Multi-target
Sep. output blocks
Single-target
D
0.0775
0.0815
0.0286
a
ag²
0.0649
0.0616
0.0469
meV
45.1
45.5
27.8
eLUMO
meV
41.1
33.9
19.7
A
meV
59.2
63.6
34.8
(R²)
ag²
0.345
0.348
0.331
ZPVE
mcV
2.87
1.44
1.29
U
meV
12.9
10.6
8.02
U
me V
13.0
10.5
7.89
H
meV
13.0
10.4
8.11
G
me V
13.8
10.8
8.98
4
mol K
0.0309
0.0283
0.0249
std. MAE
%
1.92
1.90
1.05
logMAF
-5.07
55.21
-5.57
13


Reviews:
[{"review_id": "rJx3xOahYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper beneficially incorporates directional information into graph\nneural networks for molecular modeling while preserving equivariance.\n\nThis paper is a tour de force of architecture engineering.  Continuous\nequivariance, potential field representation, and bandwidth limited\nbasis functions are synthesized in a compelling manner.\n\nI found the exposition extremely intelligible despite my lack of\nfamiliarity with molecular modeling.\n\nThe contribution is clear, although applicability beyond the specific\ndomain of molecular modeling is possibly limited.  Being continuously\nequivariant with respect to rotations is interesting, but seems to require\nproblems where the input is encoded as a point cloud in a vector space;\nI'm not familiar with such problems.  Nonetheless, the domain of molecular\nmodeling is sufficiently important in isolation.\n\nI recommend acceptance, because the contribution is strong, the writing\nis excellent, the ideas are well-motivated, and the experiments support\nthe empirical claims.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Sklq1fBatH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis is a sophisticated paper on predicting molecular properties at atom as well as molecular levels using physics inspired, extended GNN architectures. Two key extensions are provided above and beyond previous GNN models that operated on graphs derived from pairwise distances between atoms. First, the encoding of atom distances for use in neural messages is no longer done in terms of Gaussian radial basis function representations but in terms of spherical Bessel functions. The provide an orthogonal decomposition at resolutions controlled by associated frequencies. The orthogonality is though lost due to the use of an envelop function to ensure differentiability at cutoff distance defining the graph (essential for molecular simulations) but this appears not to affect performance. The second and the primary contribution of the paper, beyond the architecture itself, is the use of directional embeddings of messages where angles are transformed into cosine basis for neural mappings. In other words, the message sent from atom i to j aggregates messages from i's other neighbors in a manner that takes into account the angle formed by i, j, and k's respective atom positions. Since local directions are equivariant with an overall molecular rotation, the message passing architecture in this new representation remains invariant to rotations/translations. The added directional information, embedded in local basis representation, clearly makes the network more powerful (able to distinguish higher order structures). \n\n- the authors suggest that the radial information can be transformed simply by element-wise multiplication while angular information requires more complex transformations in message calculations. Is there a physical insight to this or is this simply an empirical finding?\n\n- there are many layers of transformations introduced from the atom embeddings before reaching the property of interest. Are so many layers really necessary? \n\n- it seems models for QM9 data were trained separately for each different physical target. Is this really necessary? Given the many layers of transformations until the properties are predicted, couldn't the message passing component be largely shared? \n\n- what exactly is the training data for the molecular simulation tests? The description in the paper is insufficient. A separate model is trained for each molecule, presumably based on samples resulting from physical simulations (?). What is provided to the models based on each \"sample\"?\n\n- the ablation studies are helpful to assess the impact of the main differences (directionality, bessel vs Gaussian, node embeddings vs message) though I would wish to see what the degradation effect is on QM9 if one used a shared message passing architecture (just sharing the messages, resulting embeddings could be transformed separately for different predictions). \n\nThere's a recent workshop paper also making use of directional information (local coordinate transformations along a protein backbone chain) in message passing/transformer architectures: Ingraham et al., Generative models for graph-based protein design, ICLR workshop 2019\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eUT3mRtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n-- The paper is well written and easy to follow\n-- The authors proposed a new approach called directional message passing to model the angles between atoms, which is missing in existing graph neural networks for molecule representation learning\n--  The proposed approach are effective on some targets. \n\nWeakness:\n-- From the point of view of graph neural networks, the novelty of the proposed techniques is marginal\n-- The performance of the proposed method are only better than existing methods on some of the targets. \n\nThis paper studied learning the graph representation of molecules by considering the angles between atoms. The authors proposed a specific type of graph neural network called directional message passing. Experimental results on the QM9 data set prove the effectiveness of the proposed approach over existing sota algorithms such as Schnet for some of the targets. \n\nOverall, the paper studies a very important problem, which aims to learn the representation of molecules. Modeling the angles of atoms is indeed less explored in existing literature. From the view of graph neural networks, the proposed technique is not that new since edge embedding has already been studied in existing literature. But for the technique could be particular useful for molecule representation learning, especially with the BESSEL FUNCTIONS. One question is that the schnet also leverages the coordinates of the atoms, which may also implicitly model the angles between the edges. In terms of the experiments, the proposed approach does not seem that strong, only achieving the best performance on 5 targets out of 12. \n\nOverall, I feel this paper is on the borderline. Now I will give weak accept and will revise my score according to other reviews and comments.  ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies Graph Neural Networks for quantum chemistry by incorporating a number of physics-informed innovations into the architecture. In particular, it considers directional edge information while preserving equivariance.

Reviewers were in agreement that this is an excellent paper with strong empirical results, great empirical evaluation and clear exposition. Despite some concerns about the limited novelty in terms of GNN methodology ( for instance, directional message passing has appeared in previous GNN papers, see e.g. https://openreview.net/forum?id=H1g0Z3A9Fm , in a different context). Ultimately, the AC believes this is a strong, high quality work that will be of broad interest, and thus recommends acceptance. 
Decision: Accept (Spotlight)