Directional Message Passing on Molecular Graphs via Synthetic Coordinates
Authors: Johannes Gasteiger
Chandan Yeshwanth
Stephan Günnemann
Venue: NeurIPS 2021 Poster (2021)
Keywords: GNN
graph neural network
graphs
molecules
personalized pagerank

Abstract: Graph neural networks that leverage coordinates via directional message passing have recently set the state of the art on multiple molecular property prediction tasks. However, they rely on atom position information that is often unavailable, and obtaining it is usually prohibitively expensive or even impossible. In this paper we propose synthetic coordinates that enable the use of advanced GNNs without requiring the true molecular configuration. We propose two distances as synthetic coordinates: Distance bounds that specify the rough range of molecular configurations, and graph-based distances using a symmetric variant of personalized PageRank. To leverage both distance and angular information we propose a method of transforming normal graph neural networks into directional MPNNs. We show that with this transformation we can reduce the error of a normal graph neural network by 55% on the ZINC benchmark. We furthermore set the state of the art on ZINC and coordinate-free QM9 by incorporating synthetic coordinates in the SMP and DimeNet++ models. Our implementation is available online.

Directional Message Passing on Molecular Graphs via Synthetic Coordinates
Johannes Gasteiger, Chandan Yeshwanth, Stephan Günnemann Technical University of Munich, Germany
{j Dogasteiige,, wewwwant, www.gguunnnemnnnn.mmm tum. de
Abstract
Graph neural networks that leverage coordinates via directional message passing have recently set the state of the art on multiple molecular property prediction tasks. However, they rely atom position information that is often unavailable, and obtaining it is usually prohibitively expensive or even impossible. paper propose synthetic coordinates that enable the use of advanced GNNs without requiring the true molecular configuration. We propose two distances as synthetic coordinates: Distance bounds that specify the rough range of molecular configurations, and graph- based distances using a symmetric variant of personalized PageRank. To leverage both distance and angular information we propose method of transforming normal graph neural networks into directional MPNNs. We show that with this transformation we can reduce the error of a normal graph neural network by 55 % on the ZINC benchmark. We furthermore set the state of the art ZINC and coordinate-free QM9 by incorporating synthetic coordinates in the SMP and DimeNet models. Our implementation is available online.
this
Introduction
Graph neural networks (GNNs) have set the state of the art many tasks of molecular machine learning, such as the prediction of quantum mechanical properties (Gilmer et al.) (017)). solubility (Wu et al. 2018), or the generation of new molecules (Jin al. 2020). Thanks their fast inference time, good generalization and scalability, GNNs are thus promising to revolutionize large parts of chemistry, from ab-initio quantum mechanical simulations and reaction kinetics to synthesis planning and drug discovery. Atom positions are central to many of these tasks, but unavailable in most cases. Many tasks in chemistry instead use a more coarse- -rariee representation: The molecular graph. Unfortunately, this representation makes many predictive tasks substantially harder, and GNNs have performed significantly better when they have access the exact molecular configuration (Gimmer et al. 2017). Missing atom positions furthermore preclude the use of many advanced GNNs that were developed with coordinates in mind.
In this work we aim to fill in this information with well-defined coordinates constructed purely from the molecular graph. Regular approximation methods for generating atom positions often do not benefit model performance, due to the fundamental ambiguity of molecular configurations. The energy landscape of molecules can have multiple local minima, and molecule can be in any of multiple different minima, known as conformers. In this work, we propose to circumvent this problem by incorporating the conformational ambiguity via empirical distance bounds. Instead of yielding potentially wrong configuration, these bounds only estimate the range of viable molecular geometries. They are thus valid regardless of which state the molecule was in when generating the data.
https://ww..dds daml. wtt..iiiww....egpteeeieoooooooo
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Molecular graph
Featurized line graph
Pairwise distances
GNN
Distances and angles
Directional MPNN
Figure 1: Illustration of transforming regular molecular graph (ethanol) line graph with synthetic coordinates. We first calculate all (bounds of) pairwise distances using our synthetic coordinates. We then calculate the (bounds of) distances and angles for the molecular graph. Finally, we convert the molecular graph to its line graph and embed the distances and angles as features. This process allows us convert regular GNN to directional MPNN, which improves its accuracy and allows to incorporate angular information.
A molecular configuration is fully specified by the pairwise distances between all atoms, due to rotational, translational, and reflectional invariance. We can thus obtain a molecular geometry from any method that provides pairwise distances between atoms. Since directional message passing does not require the full molecular geometry, these distances do not need to correspond to an actual three- dimensional configuration. We leverage this generality and propose purely graph-based distances calculated from a symmetric variant of personalized PageRank (PPR) as second set of coordinates. This distance performs surprisingly well, despite incorporating chemical knowledge. Both the distance bounds and the symmetric PPR distance require no hand-tuning and can be calculated efficiently, even for large molecules.
leverage these two variants of synthetic coordinates to transform regular GNNs into directional message passing, as illustrated in Fig. We first calculate the synthetic, pairwise distances for the given molccular graph. Based on these, we calculate the edge distances and angles between edges. Finally, compute the molecule's 's line graph. Executing GNN the line graph improves its expressivity (Garg et al. 2020) and allows us to incorporate angular information. We use the original node and edge attributes together with the distances as node attributes, and the obtained angles edge attributes. The GNN is then executed on this featurized line graph instead of the original graph. Our experiments show this transformation GNN, across multiple models and datasets. Incorporating synthetic coordinates reduces the error of a normal GNN by 55 %, putting it on par with the current state of the art. Our enhanced version of the SMP model (Vignac et al.) 2020) improves upon the current state of the art on ZINC by 21 %, and DimeNer++ (Gasteiger al. 2020a) with synthetic coordinates outcompetes previous methods coordinate-fre QM9 by 20 %. In summary, our core contributions are:
significantly improve the performance of the underlying
Well-defined synthetic coordinates based node distances and simple molecular bounds, which significantly improve the performance GNNs for molecules.
general scheme of converting a normal GNN into a directional MPNN, which can improve performance and allows incorporating both distance and angular information.
2 Directional message passing
Graph neural networks. To use GNNs for molecules we represent them as graphs g = 8), where the atoms define the node set V and the interactions the edge E. These interactions usually the bonds of the molecular graph, but they can also be all atoms pairs within a given cutoff e.g. Å. In this work we focus on an extension of message passing neural networks (MPNNs) (Gimerr al. 2017). MPNNs embed each atom u separately as h., E H and can additionally use interaction embeddings ((ue) E IR H. These embeddings are updated in each layer using messages
passed between neighboring nodes, starting with the atom features (0) (.()) (e.g. its type) and the interaction features (0) (E)
(ur) (uo)) (e.g. the bond type or a distance representation). Extended

---

MPNNs can be expressed via the following two equations:
h (L+1) Supdnte (h (1) Agg Mmag (h h (
(uo))))
(1)
((+1) (uv)
fedge
(h (141), ((+1), (0) (uv)).
The atom and interaction update functions fnode and fedge and the message function fmsg are learnable functions, such as simple linear layers or arbitrarily complex neural networks. The aggregation Agg over the atom's 's neighbors N₂ is usually simple summation.
Line graph. The directed line graph [(g) (Ve. Eg.) expresses the adjacencies between the directed edges in G. Its nodes are the directed edges of the original graph Vc. = { {u, v) E V, v E N.... For undirected graphs like molecular graphs, every undirected edge {u, v) is split into two directed edges (u, u) and (v. u). Two nodes in L()) are connected if the corresponding edges in G share a
node, i.c. & {((u, v), (v, w)) (u, v), (v, w) E Vc}. We obtain node features for the line graph by embedding the original node and edge features as a (Vc)
(uv) = Jemh (V) xv
(V)
(mo)) The line
graph can furthermore incorporate additional features for atom triplets as edge features x (Ec) such (wvw))
as the angle between bonds or interactions.
Directional message passing. Directional MPNNs improve upon regular MPNNs two ways. First, they embed the directed messages instead of the nodes in the graph, essentially operating the directed line graph. Models using only this first step are also known as directed MPNNs or line graph neural networks (Dai et al. 2016 Yang al. 2019 Chen et al., 2019). Directed MPNNs strictly more expressive than regular MPNNs (Morris et al. 2020). We can transform any MPNN to a directed MPNN simply by executing
the directed line graph instead of the original graph. Second, for graphs with nodes that are embedded in an inner product space (such as molecules in 3D space) the directed edges correspond directions in that space, via ((()) (V) x!?) Directional MPNNs leverage this connection to better represent the molecular configuration, usually by using the angles in x (&g)
(wew) GGsttigger et al. 2020b). To fully leverage both aspects directional MPNNs we therefore need some form of coordinates.
Expressivity of GNNs. A central limitation of GNNs is their inability of distinguishing between certain -oniisooorpccc graphs. For example, GNNs are not able to distinguish between a hexagon and two triangles if all nodes and edges have the same features. More specifically, Xu et al. (019);
Morris et al. (019)) have shown that GNNs are only as powerful as the I-Weisfeiler-Lehmee test of isomorphism. While it is still possible to construct indistinguishable examples for directional MPNNs, this significantly more difficult (Garg et al. 2020). Dym & Maron (20T)) have shown that MPNNs using SO(3) group representations and atom positions are even universal, i.c. able to
(WL)
approximate any continuous function to arbitrary precision. This demonstrates that coordinates can alleviate and even solve this central limitation of GNNs.
3 Molecular configurations
To prevent any pitfalls when constructing synthetic coordinates for GNNs based on chemical knowl- edge we first need to consider the properties of atomic positions in a molecule and how they are obtained. At first glance these positions might seem like an obvious and straightforward property, However, molecular configurations are actually ambiguous and difficult to obtain, even for small molecules. This misconception has even led some works to suggest remi-supervised learning meth- ods leveraging positions, effectively treating them as abundant input features (Hao et al., 2020). To clarify this issue will next describe the complexity behind molecular configurations and how to approximate them efficiently.
Finding molecular configurations. The atoms of a molecule can in principle be at any arbitrary position. However, most of these configurations will lead to an extremely high energy and are thus very unlikely to be observed in nature. A molecular configuration thus usually refers to the atom positions at or close to equilibrium, i.e. at the molecule's 's energy minimum. To find these positions we have to search the molecule's ' energy landscape and solve a non-convex optimization problem. This in fact a bilevel optimization problem, where the atom positions are optimized in the outer and the electron wavefunctions in the inner task. These wave functions can then be ignored in the outer task;
3

---

they only influence the energy and the forces F; = ae acting on each atomic nucleus. We can Sw:
then use these forces for gradient-based basedd optimization, and avoid saddle points by using quasi-Newton methods.
Difficulties. The above optimization process is very expensive due to the quantum mechanical (QM) computations required for optimizing the electron wavefunction each gradient step. It is orders of magnitude more expensive than calculating the energy of a given molecular configuration, since we need to calculate the energy's gradient for each optimization step. Furthermore, the optimization will only converge to local, and not the global minimum. And in fact, the global minimum is not the only state of interest any reasonably low local minimum of the energy landscape is valid configuration, known as a conformer. A molecule thus does not have a unique configuration; it can be in any of these states. Their statistical distribution and the interaction between them central for many molecular properties. This ambiguity of atom positions poses fundamental limit on how precise molecular predictions can be without knowing the exact (ensemble of) configurations. For example, without knowing the molecule's ' conformer we can not reasonably predict its energy at precision below roughly 60 Ve (Grimme 2019) except for small, rigid molecules that do not have multiple conformers (e.g. benzene).
Approximating energies and forces. The most prominent way of accelerating the process of finding a valid molecular configuration is by approximating its most expensive part: The quantum mechanical optimization of the electron wavefunction. There is large hierarchy of methods with various runtime versus accuracy trade-offss (Folmsbee & Hutchison 2021). The cheapest class of methods are force fields. They allow running molecular dynamics simulations with millions of atoms, and can estimate the equilibrium structure of small molecule in less than one second. Force fields approximate the quantum- mechanical interactions via a closed-form, differentiable function that only depends on the atom positions. One common example is the Merck Molecular Force Field (MMFF94) (Halgren 1996). MMFF94 calculates the molecular energy based on interatomic distances, angles, dihedral angles, and long-range interaction terms. Each term is approximated using an analytic equation with empirically chosen coefficients that depend on the involved atom types. Forces are obtained via the analytical gradients F. BE
C... and conformers via gradient-based optimization. Generating
configurations with force fields is fast enough to even generate large ensemble of conformers.
However, the resulting conformers are highly biased and require corrections based on expensive
QM-based methods for reasonably approximating the molecule's i's true distribution (Ebejer et al., 2012
Kanal et al. 2018).
Directly predicting the configuration. There are multiple methods that circumvent the optimization process to quickly generate low-energy conformers for a given molecular graph. Distance geometry methods generate conformers using an experimental database of ideal bond lengths, bond angles, and torsional angles (Havel 2002). The ETKDG method combines this with empirical torsional angle preferences (Riniker & Landrum 2015). Multiple machine learning methods for generating conformers have also recently been proposed (Weinreich et al., 2021 Lemm et al. 2021).
Restrictions for ML. All of the above methods yield reasonable molecular configurations. However, they often require many initializations and a considerable amount of sand-tuning to yield a good result for every molecule in a dataset. Furthermore, the obtained conformer might not even be the correct one for the data of interest. The data could have been generated by a different conformer or by statistical ensemble of multiple conformers. The configuration of wrong conformer can cause our model to overfit to the false training data and cause bad generalization Sec. 6). To solve this issue we could try generate an ensemble of conformers and embed their distribution. However, cheap generation methods yield strongly biased ensembles and would thus require expensive post- processing, defeating the purpose of fast and scalable machine learning (ML) methods (Ebejer et al.) 2012] Kanal et al. 2018). We propose to instead solve this issue by using less precise synthetic coordinates that are easier and cheaper obtain.
4 Synthetic coordinates
Molecular distance bounds. To circumvent the issues associated with conformational ambiguity, we propose to use pairwise distance bounds instead of simple coordinates, i.e. minimum and maximum distances demin) and d(max) for every pair of atoms. These bounds only provide the chemical information we are certain of, without being falsely accurate. Specifically, we use the distance bounds

---

provided by RDKit (RDKit 2021). These bounds provide different estimates depending on how the atoms are bonded the molecular graph. The edges in the molecular graph correspond directly bonding atoms, whose bounds are calculated as the equilibrium distance (as parametrized in the universal force field (UFF) (Rappe et al. 1992)) plus or minus a tolerance of 0.01 A. The angles between triplets of atoms are estimated based on bond hybridization and whether an atom is part of ring. The distance bounds between two-hop neighbors are then calculated based on this angle, the bond length, and tolerance of 0.04 Á, or 0.08 A for atoms larger than Aluminium. Pairwise distances between higher-order neighbors are not relevant for our method, since we only use the distances and angles of the molecular graph. The distance bounds are then refined using the triangle inequality. Note that these bounds depend almost exclusively on the directly involved atoms. They thus only provide local structural information.
Based on these distance bounds calculate three different angles for directional MPNNs: The maximally and minimally realizable angles, and the center angle. We obtain them using standard trigonometry, via
(iD)A + (
de (a).ik
(a) iik
arccos
2dbb),s
where (a) = (max) and (b) = (min) for the maximally realizable angle, (a) = (min) and (b) = (max) for the minimally realizable angle, and (a) (b) (center) for the center angle, with the center distance d(center) = (d(min) + d) )/2. These distance and angle bounds hold for all reasonable molecular structures and thus provide valuable, general information for model. Their calculation requires no hand-tuning, takes only a few milliseconds, and worked out-of-the-boo for every molecule investigated.
Graph-based distances. Directional MPNNS only use the distances interactions and the angles between interactions; they do not require full three-dimensional geometry. We leverage this gener- ality to propose a second distance based on a common graph-baste proximity measure: Personalized PageRank (PPR) (Page al. 1998), also known random walks with restart. PPR measures how close two atoms in the molecular graph are by calculating the probability that a random walker starting atom i ends up atom j. At each step, the random walker jumps to any neighbor of the current atom with equal probability, and teleports back to the original atom with probability a. To satisfy the symmetry property of metrie we variant of PPR that uses the symmetrically normalized transition matrix, i.e.
IT = aIIN - (1- -1- 1/2 1/2)-1,
4))
with the teleport probability E (0, 1]. the adjacency matrix A, and the diagonal degree matrix Di - Ee dsddj.. We found that this method works well even without considering any bond type information in A. We convert Ipe to distance via
dsppr.i
PPr
SI 211 spr
(5)
Note that IP defines a positive definite kernel, and this is the induced distance in its reproducing kernel Hilbert space. It therefore satisfies all properties of a metric, i.c. identity indiscernibles, symmetry, and the triangle inequality (Berg et al., 1984] Chapter 3, $3). However, deppe.i ij is a general metric and does not yield atom positions in 3D. This is purely graph-based measure that does not incorporate any chemical knowledge. It reflects how central an atom in the molecular graph, and how important another atom is to this one, based on the overall network of bonds. It thus only helps the GNN better reflect and process the molecular graph structure. Fig. 2 shows an example of deppr on ethanol. Since the law of cosines holds for any inner product space we can calculate the angles for directional message passing via
0.58
0.66
0.53
Figure 2: dappe distance be- tween direct neighbors ethanol.
dP + d3P
dg
Oiik- arccos
(6)
2dijdij
5

---

Note that the bounds- and graph-based distances encode orthogonal information. The former is solely based on the global molecular graph structure, while the latter provides purely local chemical knowledge. Instead of just choosing one or the other we can therefore combine both to obtain the benefits of both.
Representing distances and angles. The additional structural information can directly be incor- porated into existing models as edge features. For this purpose, we propose to first represent the distances using NRBF Gaussian radial basis functions (RBF), i.c.
BRBF,. (diz) exp 1xpp11/(d.--a)
where the Gaussian centers Gp are set uniformly between 0 and the overall maximum distance, n E [0, NRRF], and a Ci Og is set as the distance between two neighboring centers. The angles similarly represented using NABF cosine angular basis functions (ABF), i.c.
hABF. (Asgg) = cos(nac))):
(8)
with n [0, NABll We then transform these features using two linear layers. The first layer is global and uses small output dimension to force the model to learn well-generalizing intermediate representation. The second layer specific to each GNN layer, enabling more flexibility. Overall, we obtain the distance-based edge features Gij and angle-based triplet features Aijk in layer I via
will) RBF2 WRBFI (/RBF (1)) ) 11 | (8)),
(9)
(! ABF2 WARFI (ABF((GGH))
(10)
where wilh) RBF2 and ((()) ABF2 are layer-wise learned weight matrices, WRBFI and WABFI are global learned weight matrices, || denotes concatenation, and (8) are bond (edge) features. We can furthermore combine multiple synthetic coordinates by concatenating their representations hRaF and hABF- Note that for DimeNer+ we use the original basis transformation instead of the one described here.
Related work
Graph neural networks. Sperduti & Starita (997); Baskin al. (1997) proposed the first models resembling modern GNNs. Gori et al. (2005 Scarselli et al. (2009) were the first to use the name GNN, but these models are quite different to current GNNs, described in See.. 2 GNNs became widely adopted after their potential in a wide range of graph- tasks was shown by Kipf & Welling (207);; Veličković al. (2018); Gasteiger et al. 2019); Defferrard et al.) (2016); Bruna et al. (2014). Notably, [Baini et (2021) use the Laplacian eigenvectors of a graph to enable anisotropic aggregation in MPNNs. This approach is related to our synthetic coordinates. However, it is not rotationally invariant w.r.t. the directions induced by the eigenvectors, and unsuited for enabling existing directional MPNNs.
GNNs for molecules. Molecules have always played a central role in the development of GNNs, both for the very first GNNs (Baskin al., 1997) and during the modern era of GNNs (Duvenaud et al. 2015:Gilmer Gilmer et al. 2017). GNNs have been particularly successful when leveraging coordinates
SSchitt al. 2017
Unke & Meuwly 2019), but many variants only rely the molecular graph (Fey
ct al. 2020).
Directionality in GNNs. Incorporating directionality in molecular MPNNs is currently a very active and successful area of research. These methods can roughly be divided into two classes: Models based on SO(3) group representations (Thomas et al.) 2018 Anderson et al. 2019), and models incorporating directional information directly (Gasteiger et al. 2020b). Multiple promising models have recently been proposed for both classes (Fuchs et al. 2020; Batzner al. 2021 Schüttt et al.
2021 Liu et al. 2021 Satorras et al. 2021). While we focus on directional message passing in this paper, all of these methods can benefit from synthetic coordinates.
Molecular representations. Molecular fingerprints are a useful tool for comparing molecules, e.g. for machine learning. Popular examples include extended connectivity fingerprints (ECFP), also known as Morgan or circular fingerprints (Rogers Hahn, 2010), MACCS keys (Durant al. 2002), MHFP (Probst & Reymond 2018), the subgraph-based RDKit fingerprint (DDKit 2021). and

---

the SELFIES string representation (Krenn et al. 2020). These can be viewed as an alternative or supplement to synthetic coordinates. However, unlike synthetic coordinates they do not leverage the peculiarities of directional MPNNs, and can usually only be used with regressors that are not graph-based. Another class of molecular representations aims at better encoding the geometry of a molecule (Faber et al.) 2017). Examples include FCHL (Faber et al., 20188 8; Christensen et al. (020), smooth overlap of atomic positions (SOAP) (Bartók et al. 2013). and atomic spectrum of London and ATM potential (aSLATM) (Huang & von Lilienfeld| 2020). OrbNet is example of GNN that enhances its input with such a representation (Qiao et al. 2020). Obviously, none of these can be used without access to the molecular configuration.
Experiments
6.1 Experimental setup
We use three common benchmarks to evaluate the proposed synthetic coordinates: Coordinate-free QM9 (Ramakrishnan et al. 2014, CCO license), ZINC (Irwin al. 2012), and ogbg-molhiv (Hu) et al. 2020, MIT license). QM9 contains various quantum mechanical properties of equilibrium conformers of small molecules with up to nine heavy atoms. To exclude effects from regular chemical information we use all available edge (bond types) and node features (acceptor/donor, aromaticity, hybridization). However, unlike previous work we do not use the Mulliken partial charges. These are computed by quantum mechanical calculations that use the molecule's 's configuration. They thus lead to information leakage and defeat the purpose of QM9's ' regression task. We use the same data split (Brockschmidt (2020) for QM9, i.e. 10000 molecules for the validation and test sets, and the remaining ~ 000 molecules for training. Note that the properties in QM9 fundamentally depend the molecular configuration. The predictions in coordinate-fres QM9 should thus be viewed as estimates for the equilibrium configurations. There are fundamental limits to the accuracy achievable in this setup, as discussed in [Sec. 3 The goal in ZINC is to predict the penalized logP (also called "constrained solubility" in some works), given by = logP SAS cycles (Jin et al.) 2018), where logP is the water-octanol partition coefficient, SAS is the synthetic accessibility score (Ertl & Schuffenhauer 2009), and cycles denotes the number of cycles with more than six atoms. Penalized is score commonly used for training molecular generation models (Kusner et al. 2017). We use 10 000 training. 1000 validation, and 1000 test molecules, as established by Dwivedi al. (2020) and provided by PyToreh Geometric (Fey & Lenssen 2019). For ogbg-molhir we need to predict whether a molecule inhibits HIV virus replication. It contains 41 127 graphs, out of which 80 % are training samples, and 10 % each are validation and test samples, as provided by the official data splits. We report the mean and standard deviation across five runs for ZINC and ogbg-mot ogbg-molhiv. Due to computational constraints we only report single results on QM9. The experiments were run on GPUs using an internal cluster equipped mainly with NVIDIA GeForce GTX 1080Ti.
We aim to answer two questions with our experiments: 1. Do synthetic coordinates improve the performance of existing GNNs? 2. Does transforming existing GNNs to directional MPNNs improve accuracy? To answer these questions we investigate three GNNs: DeeperGCN (L et al. 2020 MIT license). structural message passing (SMP) (Vignac et al. 2020 MIT license), and DimeNet (Gasteiger et al., 2020al Hippocratic license). We show step-by-step how the changes affect the resulting error of each model. For easier comparison we also provide published results of other
state-of-the-art GNNs: Gated GCN, MPNN-JT (Fey et al. 2020). GIN (Xu et al. 2019 Fey et al.
2020), PNA (Corso et al., 2020), DGN (Beaini et al. 2021). SMP (Vignac et al.) 2020). GNN-FILM (Brockschmidt 2020), and GNN-
(Alon Yahav 2021).
6.2 Model hyperparameters
To prevent overfitting we use the SMP and + models and hyperparameters largely as-is, without any further optimization. Similarly, we chose the DeeperGCN variant and hyperparameters based on the ogbg-molhie dataset, and did not further tune on ZINC. More specifically, we use the DeeperGCN (Li et al.] 2020) with 12 ResGCN+ blocks, mean aggregation in the graph convolution, and average pooling to obtain the graph embedding. For SMP (Vignac et al. 2020) we use 12 layers, 8 towers, an internal representation of size 32 and no residual connections. For both DeeperGCN and SMP we use an embedding size of 256, and distance and angle bases of size and 18, respectively, with bottleneck dimension of 4 between the global basis embedding and the local embedding in
7

---

Table I: Ablation study for transforming DeeperGCN and SMP to directional MPNNs (MAE on ZINC). Every step improves the error of DeeperGCN, resulting in a 55 improvement. The combined bounds+ PPR encoding performs best. *Replicated using the reference implementation.
Table MAE on ZINC. SMP with
synthetic coordinates outcompetes previous models by 21 %, without any hyperparameter tuning.
Model
MAE
SMP
DeeperGCN
Gated GCN
0.282
Basic
0.139 + 0.028*
0.317 + 0.021
GIN
0,252
Bounds
0.124 + 0.002
0.264
PNA
0.188
+distance
PPR
0.151 + 0.008
0.006
DGN
0.168
Bou
0.121 0.006
0.228
MPNN-JT
0.151
+distance
Bounds
0.112 - 0.004
0.212
SMP
0.138
& line graph PPR
0.150 0.003
0.194
DeeperGCN-SC
0.142 + 0.006
+distance,
Bounds
0.113 + 0.003
0.007
SMP-SC
0.109 + 0.004
line graph PPR
0.153 + 0.005
0.158 + 0.005
& angle
Bounds+PPR
0.109 + 0.004
0.142 + 0.006
Table 3: Comparison of different distance generation methods for DeeperGCN ZINC (MAE). Our simpler, faster, and more principled methods (bounds, PPR) perform better than more sophisticated conformer generation methods.
MMFF94
ETKDG
Bounds
PPR
Bounds+PPR
Distance
0.324 = 0.012 0.329 = 0.022 0.264 + 0.003 0.227 + 0.006 0.228 + 0.005
Distance & line graph
0.232 + 0.008 0.234 + 0.007 0.212 + 0.008 0.194 + 0.009
0.178 + 0.009
Distance, line graph & angle 0.236 + 0.011 0.274 + 0.012 0.180 + 0.007 0.158 0.005
0.142 + 0.006
each layer. We train all models on ZINC with the same training hyperparameters as SMP, particularly the same learning rate schedule with a patience of 100 and minimum learning rate of 1 x 10-5
For DimeNer++ we use a cutoff of 2.5 A, radial and spherical bases of size 12, embedding size 1 128, output embedding size 256, basis embedding size 8 and 4 blocks. We use the same optimization parameters learning rate 0.001, 3000 warmup steps and a decay rate of 0.01.
6.3 Results
Transforming existing GNNs. Table shows that DeeperGCN's errors improve for each step of the transformation: Adding distance information, switching to the line graph, and adding angles. Interestingly, the PPR distance reduces the error more than molecular distance bounds do. This suggests that this structural information more relevant for the GNN than the rough bounds. SMP benefits less from using the line graph and angles. This is likely due to SMP already encoding structural information as part of its architecture. Using both the PPR distance and molccular distance bounds improves the performance further for both models. Table 3 shows that using more expensive methods of generating conformers yields a higher error than our simple and fast methods. As discussed Sec.. 3, this can be attributed to the ambiguities of different molecular conformers. DeeperGCN with synthetic coordinates performs similarly well the best models proposed previously, while the enhanced SMP sets a new state of the art on this dataset, as shown in Table 2]
Enhancing DimeNet + DimeNet was originally developed for molecular dynamics and other use cases that provide the true atom positions, such as the full QM9 dataset. Despite this, we can still use it as-is without available positional information by setting the used distance and angle embeddings to constants. DimeNet" still performs surprisingly well in this form, as shown in Table 4 However, its performance increases significantly if we provide it with the proposed synthetic coordinates. Notably, the PPR distance again causes larger improvement than the molecular distance bounds. Combining both distances still performs best, though. Table 5 furthermore shows that DimeNet ++ sets the state of the art for coordinate- lesss QM9 on eight out of twelve targets without any further hyperparameter optimization. Interestingly, the achieved energy error lies significantly below the limit of 60 V we mentioned in [Sec. 3 This is likely due to two reasons. First, QM9 only contains small molecules, many of which are very rigid. These molecules do not have multiple conformers and their energy will thus be more deterministic. Second, QM9's data was generated by initializing
8

---

Table 4: MAE on coordinate-free QM9 (meV) for DimeNet*+ with synthetic coordinates. Both syn- thetic distances and angles yield sig- nificant improvements, together re- ducing the error by 24 % on average.
Table 5: MAE on coordinate-free QM9. DimeNer++ with syn- thetic coordinates outperforms previous models by 20 %, with- out any hyperparameter tuning. *Uses Mulliken partial charges.
Unit GNN-FiLM* GNN-FiLMM+FA* DimeNet
D
0.238
0.226
0.303
ag
0.375
0.193
0.171
<HомO Up
CHMM meV
52.5
47.7
61.7
No dist/angle
74.1
41.9
CLUMO
me V
55.9
54.3
No angle
Ae meV
84.3
77
86.2
63.5
32.1
(bounds+PPR)
R2)
ag²
18.7
14.3
12.7
distance & angle:
ZPVE
meV
13.2
5.62
2.98
Bounds
63.6
29.4
Uo
meV
233
68.8
28.7
PPR
63.0
29.5
meV
256
75.2
29.6
Bounds+PPR
61.7
28.7
H
meV
240
83
29.6
G
meV
222
76.1
28.2
Gy
molk
0.173
0.082
0.076
each molecule's position with fast force field method. This bias the final conformer towards deterministic state, which might be learnable by a GNN.
Synthetic coordinates without direc-
tional message passing. In some cases found that using synthetic coordi- nates yields performance improvements while transforming the model to direc- tional MPNN does not. Table demon- strates this using DeeperGCN and ogbg-molhiv. Using the line graph significantly impairs performance both datasets. Whether directional MPNNs pro- vide benefit thus the underlying model and the dataset. This likely due to the directional MPNN's different training dynamics, which require further architectural and hyperparameter changes. Moreover, directional MPNNs are likely more prone overfitting due their better expressivity. This affects ogbg-molhiv in particular, since uses scaffold split for the test set. However, the additional information provided by synthetic coordinates still yields improvements in both cases.
Table 6: Ablation of DeeperGCN QM9 Uo (MAE, meV) and ogbg-molhiv (ROC-AUC), Using the line graph does not always provide benefits. However, syn- thetic coordinates help even in these cases.
QM9
QM9, Un oghg-molhiy
Basic
106
0.728 = 0.008
Bounds
114
0.724 = 0.014
+distance
PPR
88
0.734 = 0.014
depend on both
Bounds+PPR
100
0.733 = 0.024
+distance
Bounds
233
0.705 = 0.01T
& line graph PPR
204
0.697 + 0.009
+distance, Bounds
205
0.703 = 0.021
line graph PPR
164
0.700 = 0.014
& angle Bounds+PPR
186
0.767 + 0.016
Limitations and societal impact
Limitations. Converting a GNN to directional MPNN incurs significant computational overhead, since the line graph is usually substantially larger than the molecular graph. However, just incorporat- ing the information provided by graph distances or molecular distance bounds without transforming directional message passing can also provide benefits, with almost no computational overhead. We furthermore found that transforming GNN to directional MPNN does not yield improvements in many cases, while synthetic coordinates still do (see Sec. 6 for details). There are likely also cases where synthetic coordinates lead overfitting and do not improve accuracy. Directional MPNNs appear to be most successful when the molecular configuration is directly relevant.
Societal impact. Improving the predictions of molecular models can positively affect various applications chemistry, biology, and medicine. Our research is general and not focused on field where malicious use should be expected. However, similar to most methodological research, our improvements can misused to accelerate the development of chemical agents and biological weapons. We do not think that this potential for harm goes beyond regular research in theoretical chemistry and related fields. Still, to slightly reduce these negative effects our code will be published under the Hippocratic license (Ehmke) 2020).

---

8 Conclusion
We proposed two methods for providing synthetic coordinates: Molecular distance bounds based on the interacting atom types, and graph-based distances based on personalized PageRank scores. Both of these methods provide well-defined pairwise distances, which can then be used to calculate distances for edge features in the molecular graph, and angles for edge features in its line graph. These synthetic coordinates improve GNN performance various models and datasets, and allow transforming regular GNN into directional MPNN. This transformation leads to substantial improvements, resulting in state-of-the-ae accuracies on multiple datasets.
Acknowledgments and Disclosure of Funding
We would like to thank Johannes Margraf, Nicholas Gao, and Oleksandr Shehur for their invaluable advice and comments.
This research was supported by the Deutsche Forschungsemminscchatt (DFG) through the Emmy Noether grant GU 1409/2-1 and the TUM International Graduate School of Science and Engineering (IGSSE), GSC 81.
References
Uni Alon and Eran Yahav. On the Bottleneck of Graph Neural Networks and its Practical Implications. In ICLR, 2021.
Brandon M. Anderson, Truong-Son Hy, and Risi Kondor. Cormorant: Covariant Molecular Neural Networks. In NeurIPS, 2019.
Albert Bartók, Risi Kondor, and Gábor Csányi. On representing chemical environments. Physical Review 87(18):184115, 2013.
Igor Baskin, Vladimir A. Palyulin, and Nikolai S. Zefirov. A Neural Device for Searching Direct Correlations between Structures and Properties of Chemical Compounds. Journal of Chemical Information and Computer Sciences, 37(4):115-721 1997.
Simon Batzner, Tess E, Smidt, Lixin Sun, Jonathan P. Mailoa, Mordechai Korbluth, Nicola Molinari, and Boris Kozinsky. Equivarant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials. arXiv, 2101.03164 2021.
Dominique Beaini, Saro Passaro, Vincent Létourncau, William L. Hamilton, Gabriele Corso, and Pictro Liò. Directional Graph Networks. In ICML, 2021.
Christian Berg, Jens Peter Reus Christensen, and Paul Ressel, Harmonic Analysis on Semigroups. Number 100 in Graduate Texts in Mathematics. 1984.
Mare Brockschmidt. GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation. In ICML. 2020.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral Networks and Deep Locally Connected Networks on Graphs. In /CLR, 2014.
Zhengdao Chen, Lisha Li, and Joan Bruna. Supervised Community Detection with Line Graph Neural Networks. In ICLR, 2019.
Anders S. Christensen, Lars A. Bratholm, Felix A. Faber, and O. Anatole von Lilienfeld. FCHL revisited: Faster and accurate quantum machine learning. The Journal of Chemical Physics, 152(4):044107, 2020.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pictro Liö, and Petar Veličković. Principal Neighbourhood Aggregation for Graph Nets. In NeurlPS, 2020.
Hanjun Dai, Bo Dai, and Le Song. Discriminative Embeddings of Latent Variable Models for Structured Data. In ICML, 2016.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In NeurIPS, 2016.
10

---

Joseph L. Durant, Burton A. Leland, Douglas R. Henry, and James G. Nourse. Reoptimization of MDL Keys for Use in Drug Discovery. Journal of Chemical Information and Computer Sciences, 1276::127- 2002.
David K. Duvenaud, Dougal Maclaurin, Jorge Aguilera- Iparraguirre, Rafael Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional Networks on Graphs for Learning Molecular Fingerprints. In NewrlPS, 2015.
Timothy
Vijay Prakash Dwivedi, Chaitanya K. Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmark- ing Graph Neural Networks. arXiv, 2003.00982, 2020.
Nadav Dym and Haggai Maron. On the Universality of Rotation Equivariant Point Cloud Networks. In ICLR, 2021.
Jean-Paul Ebejer, Garrett M. Morris, and Charlotte M. Deane. Freely Available Conformer Generation Methods: How Good Are They? Journal of Chemical Information and Modeling, 52(5): (153::144-1115 1158, 2012.
Coraline Ada Ehmke. The Hippocratic License 2.1: An Ethical License for Open Source., 2020. URL
httra://firatddooham..c dev
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of Cheminformattic, 1(1):8, 2009.
Felix A. Faber, Luke Hutchison, Bing Huang. Justin Gilmer, Samuel Schoenholz. George Dahl. Oriol Vinyals, Steven Kearnes, Patrick F. Riley, and O. Anatole von Lilienfeld. Prediction Errors of Molecular Machine Learning Models Lower than Hybrid DFT Error. Journal of Chemical Theory and Computation, 13 (11):5255-5264, 2017.
Felix A. Faber, Anders S. Christensen, Bing Huang, and O. Anatole von Lilienfeld. Alchemical and structural distribution based representation for universal quantum machine learning. The Journal of Chemical Physics, 148(24)2241717, 2018.
Matthias Fey and Jan E. Lenssen. Fast Graph Representation Learning with PyTorch Geometric. In Workshop on Representation Learning on Graphs and Manifolds, ICLR, 2019.
Matthias Fey, Jan-Gin Yuen, and Frank Weichert. Hierarchical Inter-Message Passing for Learning on Molecular Graphs. In Workshop on Graph Representation Learning and Beyond, ICML, 2020.
Dakota Folmsbee and Geoffrey Hutchison. Assessing conformer energies using electronic structure and machine learning methods. International Journal of Quantumn Chemistry, 121(1):26381, 2021.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE(3)- TEans-rransforme:::: 3D Roto-Translate Equivariant Attention Networks. In NeurlPS, 2020.
Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. Generalization and Representational Limits of Graph Neural Networks. In ICML, 2020.
Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then Propagate: Graph Neural Networks Meet Personalized PageRank. In /CLR, 2019.
Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and Uncertainty-Awaat Directional Message Passing for Non-Equilibrium Molecules. In Machine Learning for Molecules Workshop, NewrlPS, 2020.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional Message Passing for Molecular Graphs. In ICLR, 2020.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Message Passing for Quantum Chemistry. In ICML, 2017.
M. Gori, G. Monfardini, and F. Scarselli, new model for learning in graph domains. In IEEE International Joint Conference on Neural Networks, 2005.
Stefan Grimme. Exploration of Chemical Compound, Conformer, and Reaction Space with Meta- -aaaaDDnmmicss Simulations Based on Tight-Binding Quantum Chemical Calculations. Journal of Chemical Theory and Computation, 15(5):2847-2866 2019.
Thomas A. Halgrnn. Merck molecular force field. I. Basis, form, scope, parameterization, and performance of MMFF94. Journal of Computational Chemistry, 17(5-6):490-5 519, 1996.
11

---

Zhongkai Hao, Chengqiang Lu, Zhenya Huang, Hao Wang, Zheyuan Hu, Qi Liu, Enhong Chen, and Cheekong Lee. ASGN: An Active Semi-supervised Graph Neural Network for Molecular Property Prediction. In KDD, 2020.
Timothy F. Havel. Distance Geometry: Theory, Algorithms, and Chemical Applications. Encyclopedia of Computational Chemistry. 2002.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In NeurlPS, 2020.
Bing Huang and O. Anatole von Lilienfeld. Quantum machine learning using atom-in-mmolcall-based fragments selected on the fly. Nature Chemistry, 12(10):945-99) 951, 2020.
John J. Irwin, Teague Sterling, Michael M. Mysinger, Erin S. Bolstad, and Ryan G. Coleman. ZINC: A Free Tool to Discover Chemistry for Biology. Journal of Chemical Information and Modeling, 1277)1757--1668, 2012.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction Tree Variational Autoencoder for Molecular Graph Generation. /CMI,, 2018.
Wengong Jin, Dr Regina Barzilay, and Tommi Jaakkola. Hierarchical Generation of Molecular Graphs using Structural Motifs, In JCML, 2020.
alaaa Y. Kanal, John A. Keith, and Geoffrey R. Hutchison. A sobering assessment of small-molecule force field methods for low energy conformer predictions. International Journal of Quantum Chemistry, 11555),25512, 2018.
Thomas N. Kipf and Max Welling. Semi- Sume-Supervised Classification with Graph Convolutional Networks. In ICLR, 2017.
Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik. Self-referencing embedded strings (SELFIES): 100% robust molecular string representation. Machine Learning: Science and Technology, 1(4):045024, 2020.
Matt J. Kusner, Brooks Paige, and José Miguel Mernandez-Lobato. Grammar Variational Autoencoder. In ICML, 2017.
Dominik Lemm, Guido Falk von Rudorff, and O. Anatole von Lilienfeld. Machine learning based energy-free structure predictions of molecules, transition states, and solids. Nature Communications, 12(1):4468, 2021,
Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. DeeperGCN: All You Need to Train Deeper GCNs. arXiv, 2006.07739, 2020.
Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical Message Passing for Graph Networks. arviv, 2102.05013. 2021.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and Leman Go Neural: Higher-Order Graph Neural Networks. In AAAI, 2019.
Christopher Morris, Gaurav Rattan, and Petra Mutzel. Weisfeiler and Leman go sparse: Towards scalable higher-order graph embeddings. In NeurlPS, 2020.
Lawrence Page. Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Report, Stanford InfoLab, 1998.
Daniel Probst and Jean-Louis Reymond. A probabilistic molecular fingerprint for big data settings. Journal of Cheminformaties, 10(1):66, 2018.
Zhuoran Qiao, Matthew Welborn, Animashree Anandkumar, Frederick R. Manby, and Thomas F. Miller. OrbNet: Deep learning for quantum chemistry using symmetry-adapted atomic-orbital features. The Journal of Chemical Physics, (33(12):124111, 2020.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp. and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1(1):1-7, 2014.
A. K. Rappe, C. J. Casewit, K. S. Colwell, W. A. Goddard, and W. M. Skiff, UFF, a full periodic table force field for molecular mechanics and molecular dynamics simulations. Journal of the American Chemical Society. 114225):1002411000 1992.
12

---

RDKit RDKit: Open-Source Cheminformatics Software, 2021. URL httpa::/rdkit.orrg
Sereina Riniker and Gregory A. Landrum. Better Informed Distance Geometry: Using What We Know To Improve Conformation Generation. Journal of Chemical Information and Modeling, 55(12):2562-2574, 2015.
David Rogers and Mathew Hahn. Extended-Coonncctvity Fingerprints. Journal of Chemical Information and Modeling, 50(5):742-75,, 2010.
Victor Gareia Satorras, Emiel Hoogehoom, and Max Welling. E(n) Equivariant Graph Neural Networks. In ICML, 2021.
F. Scarselli, M. Gori, Ah Chung Tsoi, M. Hagenbuchner, and G. Monfardini. The Graph Neural Network Model. IEEE Transactions Neural Networks, 20( (0(1):61-80, 2009.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. SchNet: A continuous-filter convolutional neural network for modeling quantum interactions. In NeurlPS, 2017.
Kristof T. Schütt, Oliver T. Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In JCML, 2021.
A. Sperduti and A. Starita. Supervised neural networks for the classification of structures. IEEE Transactions on Neural Networks, 8(3):714-735, 1997.
Nathaniel Thomas, Tess Smidt, Steven M. Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor Field Networks: Rotation- and Translation- Eqquivvrann Neural Networks for Point Clouds, arXiv, 1802.08219, 2018.
Oliver T. Unke and Markus Meuwly. PhysNet: Neural Network for Predicting Energies, Forces, Dipole Moments, and Partial Charges. Journal of Chemical Theory and Computation, 15()):367833633, 2019.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio. Graph Attention Networks. In ICLR, 2018.
Clement Vignac, Andreas Loukas, and Pascal Frossard. Building powerful and equivariant graph neural networks with structural message-passing. In NeurIPS, 2020.
Jan Weinreich, Nicholas Browning, and O. Anatole von Lilienfeld. Machine learning of free energies chemical compound space using ensemble representations: Reaching experimental uncertainty for solvation. The Journal of Chemical Physics, 154(13):13411 13, 2021.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: benchmark for molecular machine learning. Chemical Science, 9 (2):513-530, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How Powerful are Graph Neural Networks? In ICLR, 2019.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, Tommi Jaakkola, Klavs Jensen, and Regina Barzilay. Analyzing Learned Molecular Representations for Property Prediction. Journal of Chemical Information and Modeling,
59(8):3370-338 59tdlling.998)3337777
2019.
13


Reviews:
[{"review_id": "ot4UMHkQ1C", "reviewer": "Reviewer_4tVD", "summary": "The paper proposes a method for estimating molecular properties without 3D information about the molecules. The main contribution of the paper is an approach to approximate the local distance and angular information. This may then be fed into other GNNs for calculating the various properties. The authors also explore improvements related to using line graphs.\n\nResults are shown on QM9, ZINC and ogbg-molhiv datasets.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose an interesting approach to approximating local geometry information in molecules. It is a novel and potentially useful technique when geometry information cannot be obtained, as the authors describe in Section 3. However, the authors never compare against other techniques for computing approximate geometry information, such as force fields or direct approaches. Without these experiments the paper is below the acceptance threshold.\n\n\\+ Interesting approach that sidesteps the need for 3D geometry information when predicting molecular properties. \n\n\\+ Initial results look promising.\n\n\\-- No comparison with other approximate 3D geometry estimation techniques. Comparison should be done with force field methods or direct methods (i.e., other computationally efficient approaches). Otherwise the usefulness of the author’s technique is unknown.\n\n\\- ML models are also trained to predict atomic forces (SchNet, DimeNet, etc.). These may then be used to predict relaxed structures for the molecules (potentially finding multiple conformers). These methods are also computationally very fast and may be more accurate than force fields, especially for small molecules.\n\n\\- The authors state that conformers are a significant problem. However, the models can be run on multiple potential conformers and the ones with lowest energy selected. This won’t work if the energy isn’t estimated, but may be true for many applications.\n\n\\- On line 193, the authors state “The angles in ring systems are handled separately based on a set of hand-crafted rules.” It’d be better is a principled method was used in all cases.\n\n\\- The results in Table 5 contradict the earlier results. It would greatly strengthen the paper if the discrepancy was explained.\n\nOther comments:\n\n* Move limitations after the results section. This will give the reader better context on the discussion.\n\t\n* Recommend not emphasizing the 49% improvement with DeeperGCN since it’s not the SOTA approach (Table 1). Better to state the improvement on the best method, SMP.\n\n* The labels in the tables are confusing as to which settings are used. E.g., in Table three, do PPR and bounds use angles (I assumed so)?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Good.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "mXr_rHEVJt", "reviewer": "Reviewer_FwTj", "summary": "This paper aims to improve molecular property prediction by generating synthetic\n3D coordinates. The method is based on the key observation that the true 3D\ncoordinates are difficult to obtain and oftentimes undecidable for molecular\nstructures. The paper proposes synthetic coordinates, which then enables using\nGNNs without requiring the true molecular configuration. The experimental\nresults on several data sets have shown very promising results using this\nmethod.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work generates synthetic 3D coordinates to improve molecular property\nprediction. The key observation is that many molecule properties are related the\ntheir 3d structures, but the true 3D coordinates are 3D coordinates are\ndifficult to obtain and oftentimes undecidable. The authors thus propose to\ngenerate \"synthetic\" coordinates, which enables using GNNs without requiring the\ntrue molecular configurations.\n\nThe synthetic coordinates provide pairwise distances between atoms, which can\nthen be used to calculate distances and angles as edge features for the 2D\nmolecule graphs. This can be then used to enhance GNNs into directional MPNNs to\nimprove their predictive power. The authors propose two methods for generating\nsynthetic coordinates: 1) graph-based distances based on personalized PageRank\nscores; and 2) molecular distance bounds based on the interacting atom types.\n\nThe overall idea of incorporating synthetic coordinates into GNNs is well\nmotivated, and has shown very good performance. However, the design of the two\nproposed methods for obtaining synthetic coordinates are quite arbitrary and\nheuristic. This is especially true for the personalized pagerank method for\ngenerating distances. The authors first calculate PPR scores and then turn that\ninto distances using the arccos function. But why does this conversion makes\nsense? How can the arccos of PPR scores approximate the true 3D distance?\n\nThe experimental results are quite positive. The author have shown that such\n\"synthetic\" coordinates enhancements can improve the performance of regular GNNs\non multiple benchmarks.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "D26jtnSuGsR", "reviewer": "Reviewer_oh4Z", "summary": "The authors use a generic algorithm to embed molecular graph into 3D space using a distance matrix computed from PageRank. Then they use this 3D information as additional features to do property prediction. Results show that a) this approach is better than using GNNs without this information b) the augmented structures can even be applied to on Euclidean GNNs such as DimNet and get better results compared to simple GNNs.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper attempts to construct a learning approach that is not meaningful for molecular setting. Specifically, the authors are trying to predict properties that are a function of chemical graph *and* configuration (3D info) from the chemical graph alone. This is done by adding pseudocoordinate information extracted from the chemical graph itself. However, since no real coordinate information is added this just seems to indicate that some inefficiency in the GNNs compared to was exploited - it must be possible to get similar performance with suitable GNNs because they use exactly the same input information eventually.\n\nMore fundamentally, it is simply meaningless to predict a property of the 3d structure independent of the 3d structure. What *would* be meaningful is to predict an ensemble average over 3d structures such as a solvation free energy as a function of the chemical graph alone, but the prediction of the QM9 properties certainly isn't. As a result, I am not convinced of any of the results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "9onNONXYL9A", "reviewer": "Reviewer_u3W7", "summary": "This paper focuses on the task of molecular property prediction and proposes novel approaches to build synthetic coordinates (geometric information) in molecules when atom positions are not available for advanced GNNs. Since the performance of those GNNs highly relies on atom coordinates, the applicable cases of advanced GNNs are limited without reliable and efficient approaches to generate coordinates before training.\n\nThe authors propose two ways to generate the synthetic coordinates: 1. Graph-based distance and angles via Personalized Page-Rank; 2. Molecular distance bounds and corresponding angles based on chemical knowledge. To embed the resulting distances and angles, the authors propose to convert the original molecular graphs to line graphs. With line graphs as input, regular GNNs work as directional MPNNs, which exhibit better performance in practice. Experiments on benchmarks validate the improvements achieved by using synthetic coordinates in molecular graphs with several baseline models. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed synthetic coordinates address the gap between the performance of GNNs on molecular property predictions when the atom coordinates are provided and not provided. The design of advanced GNNs with available atom positions is also different from regular GNNs when only molecular graphs are achievable. With synthetic coordinates and line graphs, regular GNNs can be converted to directional MPNNs, which have the potential to gain better performance with distance and angles as features. The proposed idea is novel and contributes to the related field especially molecular property prediction is an important subarea now. \n\nOverall the paper is well-written but can be better organized: I feel that the \"Molecular configurations\" section is a bit too long for discussing the background chemical knowledge and motivations. In contrast, the \"Synthetic coordinates\" section that describes the proposed methods is too concise without touching enough details. Some other suggestions for \"Synthetic coordinates\" section:\n\n(1) The proposed methods can be explained with values computed on certain example molecules for better understanding. \n\n(2) It would be interesting to compare the synthetic coordinates achieved by the two methods to find any possible relations. \n\n(3) As for the implementation, although the authors mention combining both two synthetic coordinates, it is not clear how the two coordinates are implemented together in GNNs. \n\nThe results in the experiment section clearly demonstrate the improvements achieved by introducing synthetic coordinates and line graphs. I would suggest that Table 1 and Table 5 can be better written since it seems that each ablated model only contains one modification (distance, line graph, or angle), while \"+angle\" actually means \"+disrance & line graph & angle\". By only reading the tables, it is unknown that the changes are done step-by-step.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations and potential negative social impact of their work in section 5.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The ratings were 3, 6, 4, 6.

The reviewer who gave the 3 wrote "it is simply meaningless to predict a property of the 3d structure independent of the 3d structure", however the authors gave a detailed response that suggests there might have been some confusion, and the reviewer didn't follow up. Moreover, this approach effectively uses carefully chosen inductive biases, which seems to go beyond "some inefficiency in the GNNs compared to was exploited" as the reviewer suggests.

For the reviewer who gave the 4, since the authors provide some new results in response, the score probably should be elevated.

While this is still a borderline paper, I recommend "accept" given that the results are pretty good, there doesn't seem to be any technical problem, and there is clear novelty.
Decision: Accept (Poster)