Directional Message Passing on Molecular Graphs via Synthetic Coordinates
Year: 2022

Abstract:
Graph neural networks that leverage coordinates via directional message passing have recently set the state of the art on multiple molecular property prediction tasks. However, they rely on atom position information that is often unavailable, and obtaining it is usually prohibitively expensive or even impossible. In this paper we propose synthetic coordinates that enable the use of advanced GNNs without requiring the true molecular configuration. We propose two distances as synthetic coordinates: Distance bounds that specify the rough range of molecular configurations, and graph-based distances using a symmetric variant of personalized PageRank. To leverage both distance and angular information we propose a method of transforming normal graph neural networks into directional MPNNs. We show that with this transformation we can reduce the error of a normal graph neural network by 55 % on the ZINC benchmark. We furthermore set the state of the art on ZINC and coordinate-free QM9 by incorporating synthetic coordinates in the SMP and DimeNet models. Our implementation is available online. 1

heading: 1 Introduction
text: Graph neural networks (GNNs) have set the state of the art on many tasks of molecular machine learning, such as the prediction of quantum mechanical properties (Gilmer et al., 2017), solubility (Wu et al., 2018), or the generation of new molecules (Jin et al., 2020). Thanks to their fast inference time, good generalization and scalability, GNNs are thus promising to revolutionize large parts of chemistry, from ab-initio quantum mechanical simulations and reaction kinetics to synthesis planning and drug discovery. Atom positions are central to many of these tasks, but unavailable in most cases. Many tasks in chemistry instead use a more coarse-grained representation: The molecular graph. Unfortunately, this representation makes many predictive tasks substantially harder, and GNNs have performed significantly better when they have access to the exact molecular configuration (Gilmer et al., 2017). Missing atom positions furthermore preclude the use of many advanced GNNs that were developed with coordinates in mind.
In this work we aim to fill in this information with well-defined coordinates constructed purely from the molecular graph. Regular approximation methods for generating atom positions often do not benefit model performance, due to the fundamental ambiguity of molecular configurations. The energy landscape of molecules can have multiple local minima, and a molecule can be in any of multiple different minima, known as conformers. In this work, we propose to circumvent this problem by incorporating the conformational ambiguity via empirical distance bounds. Instead of yielding a potentially wrong configuration, these bounds only estimate the range of viable molecular geometries. They are thus valid regardless of which state the molecule was in when generating the data.
1https://www.daml.in.tum.de/synthetic-coordinates
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
A molecular configuration is fully specified by the pairwise distances between all atoms, due to rotational, translational, and reflectional invariance. We can thus obtain a molecular geometry from any method that provides pairwise distances between atoms. Since directional message passing does not require the full molecular geometry, these distances do not need to correspond to an actual threedimensional configuration. We leverage this generality and propose purely graph-based distances calculated from a symmetric variant of personalized PageRank (PPR) as a second set of coordinates. This distance performs surprisingly well, despite incorporating no chemical knowledge. Both the distance bounds and the symmetric PPR distance require no hand-tuning and can be calculated efficiently, even for large molecules.
We leverage these two variants of synthetic coordinates to transform regular GNNs into directional message passing, as illustrated in Fig. 1. We first calculate the synthetic, pairwise distances for the given molecular graph. Based on these, we calculate the edge distances and angles between edges. Finally, we compute the molecule’s line graph. Executing a GNN on the line graph improves its expressivity (Garg et al., 2020) and allows us to incorporate angular information. We use the original node and edge attributes together with the distances as node attributes, and the obtained angles as edge attributes. The GNN is then executed on this featurized line graph instead of the original graph. Our experiments show this transformation can significantly improve the performance of the underlying GNN, across multiple models and datasets. Incorporating synthetic coordinates reduces the error of a normal GNN by 55 %, putting it on par with the current state of the art. Our enhanced version of the SMP model (Vignac et al., 2020) improves upon the current state of the art on ZINC by 21 %, and DimeNet++ (Klicpera et al., 2020a) with synthetic coordinates outcompetes previous methods on coordinate-free QM9 by 20 %. In summary, our core contributions are:
• Well-defined synthetic coordinates based on node distances and simple molecular bounds, which significantly improve the performance of GNNs for molecules.
• A general scheme of converting a normal GNN into a directional MPNN, which can improve performance and allows incorporating both distance and angular information.

heading: 2 Directional message passing
text: Graph neural networks. To use GNNs for molecules we represent them as graphs G = (V, E), where the atoms define the node set V and the interactions the edge set E . These interactions are usually the bonds of the molecular graph, but they can also be all atoms pairs within a given cutoff of e.g. 5 Å. In this work we focus on an extension of message passing neural networks (MPNNs) (Gilmer et al., 2017). MPNNs embed each atom u separately as hu ∈ RH , and can additionally use interaction embeddings e(uv) ∈ RHe . These embeddings are updated in each layer using messages passed between neighboring nodes, starting with the atom features h(0)u = x (V) u (e.g. its type) and the interaction features e(0)(uv) = x (E) (uv) (e.g. the bond type or a distance representation). Extended
MPNNs can be expressed via the following two equations:
h(l+1)u = fupdate(h (l) u , Agg
v∈Nu [fmsg(h
(l) u ,h (l) v , e (l) (uv))]), (1)
e (l+1) (uv) = fedge(h (l+1) u ,h (l+1) v , e (l) (uv)). (2)
The atom and interaction update functions fnode and fedge and the message function fmsg are learnable functions, such as simple linear layers or arbitrarily complex neural networks. The aggregation Agg over the atom’s neighbors Nu is usually a simple summation. Line graph. The directed line graph L(G) = (VL, EL) expresses the adjacencies between the directed edges in G. Its nodes are the directed edges of the original graph VL = {(u, v) | u ∈ V, v ∈ Nu}. For undirected graphs like molecular graphs, every undirected edge {u, v} is split into two directed edges (u, v) and (v, u). Two nodes in L(G) are connected if the corresponding edges in G share a node, i.e. EL = {((u, v), (v, w)) | (u, v), (v, w) ∈ VL}. We obtain node features for the line graph by embedding the original node and edge features as x(VL)(uv) = femb(x (V) u ,x (V) v ,x (E) (uv)). The line graph can furthermore incorporate additional features for atom triplets as edge features x(EL)(uvw), such as the angle between bonds or interactions.
Directional message passing. Directional MPNNs improve upon regular MPNNs in two ways. First, they embed the directed messages instead of the nodes in the graph, essentially operating on the directed line graph. Models using only this first step are also known as directed MPNNs or line graph neural networks (Dai et al., 2016; Yang et al., 2019; Chen et al., 2019). Directed MPNNs are strictly more expressive than regular MPNNs (Morris et al., 2020). We can transform any MPNN to a directed MPNN simply by executing it on the directed line graph instead of the original graph. Second, for graphs with nodes that are embedded in an inner product space (such as molecules in 3D space) the directed edges correspond to directions in that space, via x(VL)(uv) = x (V) u −x(V)v . Directional MPNNs leverage this connection to better represent the molecular configuration, usually by using the angles in x(EL)(uvw) (Klicpera et al., 2020b). To fully leverage both aspects of directional MPNNs we therefore need some form of coordinates.
Expressivity of GNNs. A central limitation of GNNs is their inability of distinguishing between certain non-isomorphic graphs. For example, GNNs are not able to distinguish between a hexagon and two triangles if all nodes and edges have the same features. More specifically, Xu et al. (2019); Morris et al. (2019) have shown that GNNs are only as powerful as the 1-Weisfeiler-Lehman (WL) test of isomorphism. While it is still possible to construct indistinguishable examples for directional MPNNs, this is significantly more difficult (Garg et al., 2020). Dym & Maron (2021) have shown that MPNNs using SO(3) group representations and atom positions are even universal, i.e. able to approximate any continuous function to arbitrary precision. This demonstrates that coordinates can alleviate and even solve this central limitation of GNNs.

heading: 3 Molecular configurations
text: To prevent any pitfalls when constructing synthetic coordinates for GNNs based on chemical knowledge we first need to consider the properties of atomic positions in a molecule and how they are obtained. At first glance these positions might seem like an obvious and straightforward property. However, molecular configurations are actually ambiguous and difficult to obtain, even for small molecules. This misconception has even led some works to suggest semi-supervised learning methods leveraging positions, effectively treating them as abundant input features (Hao et al., 2020). To clarify this issue we will next describe the complexity behind molecular configurations and how to approximate them efficiently.
Finding molecular configurations. The atoms of a molecule can in principle be at any arbitrary position. However, most of these configurations will lead to an extremely high energy and are thus very unlikely to be observed in nature. A molecular configuration thus usually refers to the atom positions at or close to equilibrium, i.e. at the molecule’s energy minimum. To find these positions we have to search the molecule’s energy landscape and solve a non-convex optimization problem. This is in fact a bilevel optimization problem, where the atom positions are optimized in the outer and the electron wavefunctions in the inner task. These wave functions can then be ignored in the outer task;
they only influence the energy and the forces Fi = − ∂E∂xi acting on each atomic nucleus. We can then use these forces for gradient-based optimization, and avoid saddle points by using quasi-Newton methods.
Difficulties. The above optimization process is very expensive due to the quantum mechanical (QM) computations required for optimizing the electron wavefunction at each gradient step. It is orders of magnitude more expensive than calculating the energy of a given molecular configuration, since we need to calculate the energy’s gradient for each optimization step. Furthermore, the optimization will only converge to a local, and not the global minimum. And in fact, the global minimum is not the only state of interest — any reasonably low local minimum of the energy landscape is a valid configuration, known as a conformer. A molecule thus does not have a unique configuration; it can be in any of these states. Their statistical distribution and the interaction between them is central for many molecular properties. This ambiguity of atom positions poses a fundamental limit on how precise molecular predictions can be without knowing the exact (ensemble of) configurations. For example, without knowing the molecule’s conformer we can not reasonably predict its energy at a precision below roughly 60 meV (Grimme, 2019) — except for small, rigid molecules that do not have multiple conformers (e.g. benzene).
Approximating energies and forces. The most prominent way of accelerating the process of finding a valid molecular configuration is by approximating its most expensive part: The quantum mechanical optimization of the electron wavefunction. There is a large hierarchy of methods with various runtime versus accuracy trade-offs (Folmsbee & Hutchison, 2021). The cheapest class of methods are force fields. They allow running molecular dynamics simulations with millions of atoms, and can estimate the equilibrium structure of a small molecule in less than one second. Force fields approximate the quantum-mechanical interactions via a closed-form, differentiable function that only depends on the atom positions. One common example is the Merck Molecular Force Field (MMFF94) (Halgren, 1996). MMFF94 calculates the molecular energy based on interatomic distances, angles, dihedral angles, and long-range interaction terms. Each term is approximated using an analytic equation with empirically chosen coefficients that depend on the involved atom types. Forces are obtained via the analytical gradients Fi = − ∂E∂xi , and conformers via gradient-based optimization. Generating configurations with force fields is fast enough to even generate a large ensemble of conformers. However, the resulting conformers are highly biased and require corrections based on expensive QM-based methods for reasonably approximating the molecule’s true distribution (Ebejer et al., 2012; Kanal et al., 2018).
Directly predicting the configuration. There are multiple methods that circumvent the optimization process to quickly generate low-energy conformers for a given molecular graph. Distance geometry methods generate conformers using an experimental database of ideal bond lengths, bond angles, and torsional angles (Havel, 2002). The ETKDG method combines this with empirical torsional angle preferences (Riniker & Landrum, 2015). Multiple machine learning methods for generating conformers have also recently been proposed (Weinreich et al., 2021; Lemm et al., 2021).
Restrictions for ML. All of the above methods yield reasonable molecular configurations. However, they often require many initializations and a considerable amount of hand-tuning to yield a good result for every molecule in a dataset. Furthermore, the obtained conformer might not even be the correct one for the data of interest. The data could have been generated by a different conformer or by a statistical ensemble of multiple conformers. The configuration of a wrong conformer can cause our model to overfit to the false training data and cause bad generalization (see Sec. 6). To solve this issue we could try to generate an ensemble of conformers and embed their distribution. However, cheap generation methods yield strongly biased ensembles and would thus require expensive postprocessing, defeating the purpose of fast and scalable machine learning (ML) methods (Ebejer et al., 2012; Kanal et al., 2018). We propose to instead solve this issue by using less precise synthetic coordinates that are easier and cheaper to obtain.

heading: 4 Synthetic coordinates
text: Molecular distance bounds. To circumvent the issues associated with conformational ambiguity, we propose to use pairwise distance bounds instead of simple coordinates, i.e. minimum and maximum distances d(min) and d(max) for every pair of atoms. These bounds only provide the chemical information we are certain of, without being falsely accurate. Specifically, we use the distance bounds
provided by RDKit (RDKit, 2021). These bounds provide different estimates depending on how the atoms are bonded in the molecular graph. The edges in the molecular graph correspond to directly bonding atoms, whose bounds are calculated as the equilibrium distance (as parametrized in the universal force field (UFF) (Rappe et al., 1992)) plus or minus a tolerance of 0.01 Å. The angles between triplets of atoms are estimated based on bond hybridization and whether an atom is part of a ring. The distance bounds between two-hop neighbors are then calculated based on this angle, the bond length, and a tolerance of 0.04 Å, or 0.08 Å for atoms larger than Aluminium. Pairwise distances between higher-order neighbors are not relevant for our method, since we only use the distances and angles of the molecular graph. The distance bounds are then refined using the triangle inequality. Note that these bounds depend almost exclusively on the directly involved atoms. They thus only provide local structural information.
Based on these distance bounds we calculate three different angles for directional MPNNs: The maximally and minimally realizable angles, and the center angle. We obtain them using standard trigonometry, via
α (a) ijk = arccos
( d2(b),ij + d 2 (b),jk − d 2 (a),ik
2d(b),ijd(b),jk
) , (3)
where (a) = (max) and (b) = (min) for the maximally realizable angle, (a) = (min) and (b) = (max) for the minimally realizable angle, and (a) = (b) = (center) for the center angle, with the center distance d(center) = (d(min) + d(max))/2. These distance and angle bounds hold for all reasonable molecular structures and thus provide valuable, general information for our model. Their calculation requires no hand-tuning, takes only a few milliseconds, and worked out-of-the-box for every molecule we investigated.
Graph-based distances. Directional MPNNs only use the distances of interactions and the angles between interactions; they do not require a full three-dimensional geometry. We leverage this generality to propose a second distance based on a common graph-based proximity measure: Personalized PageRank (PPR) (Page et al., 1998), also known as random walks with restart. PPR measures how close two atoms in the molecular graph are by calculating the probability that a random walker starting at atom i ends up at atom j. At each step, the random walker jumps to any neighbor of the current atom with equal probability, and teleports back to the original atom i with probability α. To satisfy the symmetry property of a metric we use a variant of PPR that uses the symmetrically normalized transition matrix, i.e.
Πsppr = α(IN − (1− α)D−1/2AD−1/2)−1, (4)
with the teleport probability α ∈ (0, 1], the adjacency matrix A, and the diagonal degree matrix Dij = ∑ k Aikδij . We found that this method works well even without considering any bond type information in A. We convert Πsppr to a distance via
dsppr,ij = √ Πspprii +Π sppr jj − 2Π sppr ij . (5)
Note that Πsppr defines a positive definite kernel, and this is the induced distance in its reproducing kernel Hilbert space. It therefore satisfies all properties of a metric, i.e. identity of indiscernibles, symmetry, and the triangle inequality (Berg et al., 1984, Chapter 3, §3). However, dsppr,ij is a general metric and does not yield atom positions in 3D. This is a purely graph-based measure that does not incorporate any chemical knowledge. It reflects how central an atom is in the molecular graph, and how important another atom is to this one, based on the overall network of bonds. It thus only helps the GNN better reflect and process the molecular graph structure. Fig. 2 shows an example of dsppr on ethanol. Since the law of cosines holds for any inner product space we can calculate the angles for directional message passing via
αijk = arccos
( d2ij + d 2 jk − d2ik
2dijdjk
) . (6)
Note that the bounds- and graph-based distances encode orthogonal information. The former is solely based on the global molecular graph structure, while the latter provides purely local chemical knowledge. Instead of just choosing one or the other we can therefore combine both to obtain the benefits of both.
Representing distances and angles. The additional structural information can directly be incorporated into existing models as edge features. For this purpose, we propose to first represent the distances using NRBF Gaussian radial basis functions (RBF), i.e.
hRBF,n(dij) = exp −1/2(dij−cn)2/σ2 , (7)
where the Gaussian centers cn are set uniformly between 0 and the overall maximum distance, n ∈ [0, NRBF], and σ = c1 − c0 is set as the distance between two neighboring centers. The angles are similarly represented using NABF cosine angular basis functions (ABF), i.e.
hABF,n(αijk) = cos(nαijk), (8)
with n ∈ [0, NABF]. We then transform these features using two linear layers. The first layer is global and uses a small output dimension to force the model to learn a well-generalizing intermediate representation. The second layer is specific to each GNN layer, enabling more flexibility. Overall, we obtain the distance-based edge features eij and angle-based triplet features aijk in layer l via
e (l) ij = W (l) RBF2WRBF1(hRBF(dij)∥x (E) ij ), (9)
a (l) ijk = W (l) ABF2WABF1hABF(αijk), (10)
where W (l)RBF2 and W (l) ABF2 are layer-wise learned weight matrices, WRBF1 and WABF1 are global learned weight matrices, ∥ denotes concatenation, and x(E)ij are bond (edge) features. We can furthermore combine multiple synthetic coordinates by concatenating their representations hRBF and hABF. Note that for DimeNet++ we use the original basis transformation instead of the one described here.

heading: 5 Related work
text: Graph neural networks. Sperduti & Starita (1997); Baskin et al. (1997) proposed the first models resembling modern GNNs. Gori et al. (2005); Scarselli et al. (2009) were the first to use the name GNN, but these models are quite different to current GNNs, as described in Sec. 2. GNNs became widely adopted after their potential in a wide range of graph-related tasks was shown by Kipf & Welling (2017); Veličković et al. (2018); Klicpera et al. (2019); Defferrard et al. (2016); Bruna et al. (2014). Notably, Beaini et al. (2021) use the Laplacian eigenvectors of a graph to enable anisotropic aggregation in MPNNs. This approach is related to our synthetic coordinates. However, it is not rotationally invariant w.r.t. the directions induced by the eigenvectors, and unsuited for enabling existing directional MPNNs.
GNNs for molecules. Molecules have always played a central role in the development of GNNs, both for the very first GNNs (Baskin et al., 1997) and during the modern era of GNNs (Duvenaud et al., 2015; Gilmer et al., 2017). GNNs have been particularly successful when leveraging coordinates (Schütt et al., 2017; Unke & Meuwly, 2019), but many variants only rely on the molecular graph (Fey et al., 2020).
Directionality in GNNs. Incorporating directionality in molecular MPNNs is currently a very active and successful area of research. These methods can roughly be divided into two classes: Models based on SO(3) group representations (Thomas et al., 2018; Anderson et al., 2019), and models incorporating directional information directly (Klicpera et al., 2020b). Multiple promising models have recently been proposed for both classes (Fuchs et al., 2020; Batzner et al., 2021; Schütt et al., 2021; Liu et al., 2021; Satorras et al., 2021). While we focus on directional message passing in this paper, all of these methods can benefit from synthetic coordinates.
Molecular representations. Molecular fingerprints are a useful tool for comparing molecules, e.g. for machine learning. Popular examples include extended connectivity fingerprints (ECFP), also known as Morgan or circular fingerprints (Rogers & Hahn, 2010), MACCS keys (Durant et al., 2002), MHFP (Probst & Reymond, 2018), the subgraph-based RDKit fingerprint (RDKit, 2021), and
the SELFIES string representation (Krenn et al., 2020). These can be viewed as an alternative or supplement to synthetic coordinates. However, unlike synthetic coordinates they do not leverage the peculiarities of directional MPNNs, and can usually only be used with regressors that are not graph-based. Another class of molecular representations aims at better encoding the geometry of a molecule (Faber et al., 2017). Examples include FCHL (Faber et al., 2018; Christensen et al., 2020), smooth overlap of atomic positions (SOAP) (Bartók et al., 2013), and atomic spectrum of London and ATM potential (aSLATM) (Huang & von Lilienfeld, 2020). OrbNet is an example of a GNN that enhances its input with such a representation (Qiao et al., 2020). Obviously, none of these can be used without access to the molecular configuration.

heading: 6 Experiments

heading: 6.1 Experimental setup
text: We use three common benchmarks to evaluate the proposed synthetic coordinates: Coordinate-free QM9 (Ramakrishnan et al., 2014, CC0 license), ZINC (Irwin et al., 2012), and ogbg-molhiv (Hu et al., 2020, MIT license). QM9 contains various quantum mechanical properties of equilibrium conformers of small molecules with up to nine heavy atoms. To exclude effects from regular chemical information we use all available edge (bond types) and node features (acceptor/donor, aromaticity, hybridization). However, unlike previous work we do not use the Mulliken partial charges. These are computed by quantum mechanical calculations that use the molecule’s configuration. They thus lead to information leakage and defeat the purpose of QM9’s regression task. We use the same data split as Brockschmidt (2020) for QM9, i.e. 10 000 molecules for the validation and test sets, and the remaining ∼110 000 molecules for training. Note that the properties in QM9 fundamentally depend on the molecular configuration. The predictions in coordinate-free QM9 should thus be viewed as estimates for the equilibrium configurations. There are fundamental limits to the accuracy achievable in this setup, as discussed in Sec. 3. The goal in ZINC is to predict the penalized logP (also called “constrained solubility” in some works), given by y = logP−SAS− cycles (Jin et al., 2018), where logP is the water-octanol partition coefficient, SAS is the synthetic accessibility score (Ertl & Schuffenhauer, 2009), and cycles denotes the number of cycles with more than six atoms. Penalized logP is a score commonly used for training molecular generation models (Kusner et al., 2017). We use 10 000 training, 1000 validation, and 1000 test molecules, as established by Dwivedi et al. (2020) and provided by PyTorch Geometric (Fey & Lenssen, 2019). For ogbg-molhiv we need to predict whether a molecule inhibits HIV virus replication. It contains 41 127 graphs, out of which 80 % are training samples, and 10 % each are validation and test samples, as provided by the official data splits. We report the mean and standard deviation across five runs for ZINC and ogbg-molhiv. Due to computational constraints we only report single results on QM9. The experiments were run on GPUs using an internal cluster equipped mainly with NVIDIA GeForce GTX 1080Ti.
We aim to answer two questions with our experiments: 1. Do synthetic coordinates improve the performance of existing GNNs? 2. Does transforming existing GNNs to directional MPNNs improve accuracy? To answer these questions we investigate three GNNs: DeeperGCN (Li et al., 2020, MIT license), structural message passing (SMP) (Vignac et al., 2020, MIT license), and DimeNet++ (Klicpera et al., 2020a, Hippocratic license). We show step-by-step how the changes affect the resulting error of each model. For easier comparison we also provide published results of other state-of-the-art GNNs: Gated GCN, MPNN-JT (Fey et al., 2020), GIN (Xu et al., 2019; Fey et al., 2020), PNA (Corso et al., 2020), DGN (Beaini et al., 2021), SMP (Vignac et al., 2020), GNN-FiLM (Brockschmidt, 2020), and GNN-FiLM+FA (Alon & Yahav, 2021).

heading: 6.2 Model hyperparameters
text: To prevent overfitting we use the SMP and DimeNet++ models and hyperparameters largely as-is, without any further optimization. Similarly, we chose the DeeperGCN variant and hyperparameters based on the ogbg-molhiv dataset, and did not further tune on ZINC. More specifically, we use the DeeperGCN (Li et al., 2020) with 12 ResGCN+ blocks, mean aggregation in the graph convolution, and average pooling to obtain the graph embedding. For SMP (Vignac et al., 2020) we use 12 layers, 8 towers, an internal representation of size 32 and no residual connections. For both DeeperGCN and SMP we use an embedding size of 256, and distance and angle bases of size 16 and 18, respectively, with a bottleneck dimension of 4 between the global basis embedding and the local embedding in
each layer. We train all models on ZINC with the same training hyperparameters as SMP, particularly the same learning rate schedule with a patience of 100 and minimum learning rate of 1×10−5.
For DimeNet++ we use a cutoff of 2.5 Å, radial and spherical bases of size 12, embedding size 128, output embedding size 256, basis embedding size 8 and 4 blocks. We use the same optimization parameters - learning rate 0.001, 3000 warmup steps and a decay rate of 0.01.

heading: 6.3 Results
text: Transforming existing GNNs. Table 1 shows that DeeperGCN’s errors improve for each step of the transformation: Adding distance information, switching to the line graph, and adding angles. Interestingly, the PPR distance reduces the error more than molecular distance bounds do. This suggests that this structural information is more relevant for the GNN than the rough bounds. SMP benefits less from using the line graph and angles. This is likely due to SMP already encoding structural information as part of its architecture. Using both the PPR distance and molecular distance bounds improves the performance further for both models. Table 3 shows that using more expensive methods of generating conformers yields a higher error than our simple and fast methods. As discussed in Sec. 3, this can be attributed to the ambiguities of different molecular conformers. DeeperGCN with synthetic coordinates performs similarly well to the best models proposed previously, while the enhanced SMP sets a new state of the art on this dataset, as shown in Table 2.
Enhancing DimeNet++. DimeNet was originally developed for molecular dynamics and other use cases that provide the true atom positions, such as the full QM9 dataset. Despite this, we can still use it as-is without available positional information by setting the used distance and angle embeddings to constants. DimeNet++ still performs surprisingly well in this form, as shown in Table 4. However, its performance increases significantly if we provide it with the proposed synthetic coordinates. Notably, the PPR distance again causes a larger improvement than the molecular distance bounds. Combining both distances still performs best, though. Table 5 furthermore shows that DimeNet++ sets the state of the art for coordinate-less QM9 on eight out of twelve targets — without any further hyperparameter optimization. Interestingly, the achieved energy error lies significantly below the limit of 60 meV we mentioned in Sec. 3. This is likely due to two reasons. First, QM9 only contains small molecules, many of which are very rigid. These molecules do not have multiple conformers and their energy will thus be more deterministic. Second, QM9’s data was generated by initializing
each molecule’s position with a fast force field method. This can bias the final conformer towards a deterministic state, which might be learnable by a GNN.
Synthetic coordinates without directional message passing. In some cases we found that using synthetic coordinates yields performance improvements while transforming the model to a directional MPNN does not. Table 6 demonstrates this using DeeperGCN on QM9 and ogbgmolhiv. Using the line graph significantly impairs performance on both datasets. Whether directional MPNNs provide a benefit thus seems to depend on both the underlying model and the dataset. This is likely due to the directional MPNN’s different training dynamics, which require further architectural and hyperparameter changes. Moreover, directional MPNNs are likely
more prone to overfitting due to their better expressivity. This affects ogbg-molhiv in particular, since it uses a scaffold split for the test set. However, the additional information provided by synthetic coordinates still yields improvements in both cases.

heading: 7 Limitations and societal impact
text: Limitations. Converting a GNN to a directional MPNN incurs significant computational overhead, since the line graph is usually substantially larger than the molecular graph. However, just incorporating the information provided by graph distances or molecular distance bounds without transforming to directional message passing can also provide benefits, with almost no computational overhead. We furthermore found that transforming a GNN to a directional MPNN does not yield improvements in many cases, while synthetic coordinates still do (see Sec. 6 for details). There are likely also cases where synthetic coordinates lead to overfitting and do not improve accuracy. Directional MPNNs appear to be most successful when the molecular configuration is directly relevant.
Societal impact. Improving the predictions of molecular models can positively affect various applications in chemistry, biology, and medicine. Our research is general and not focused on a field where malicious use should be expected. However, similar to most methodological research, our improvements can be misused to accelerate the development of chemical agents and biological weapons. We do not think that this potential for harm goes beyond regular research in theoretical
chemistry and related fields. Still, to slightly reduce these negative effects our code will be published under the Hippocratic license (Ehmke, 2020).

heading: 8 Conclusion
text: We proposed two methods for providing synthetic coordinates: Molecular distance bounds based on the interacting atom types, and graph-based distances based on personalized PageRank scores. Both of these methods provide well-defined pairwise distances, which can then be used to calculate distances for edge features in the molecular graph, and angles for edge features in its line graph. These synthetic coordinates improve GNN performance for various models and datasets, and allow transforming a regular GNN into a directional MPNN. This transformation leads to substantial improvements, resulting in state-of-the-art accuracies on multiple datasets.
Acknowledgments and Disclosure of Funding
We would like to thank Johannes T. Margraf, Nicholas Gao, and Oleksandr Shchur for their invaluable advice and comments.
This research was supported by the Deutsche Forschungsgemeinschaft (DFG) through the Emmy Noether grant GU 1409/2-1 and the TUM International Graduate School of Science and Engineering (IGSSE), GSC 81.

Review summaries:
The authors use a generic algorithm to embed molecular graph into 3D space using a distance matrix computed from PageRank. Then they use this 3D information as additional features to do property prediction. Results show that a) this approach is better than using GNNs without this information b) the augmented structures can even be applied to on Euclidean GNNs such as DimNet and get better results compared to simple GNNs.

This paper focuses on the task of molecular property prediction and proposes novel approaches to build synthetic coordinates (geometric information) in molecules when atom positions are not available for advanced GNNs. Since the performance of those GNNs highly relies on atom coordinates, the applicable cases of advanced GNNs are limited without reliable and efficient approaches to generate coordinates before training.  The authors propose two ways to generate the synthetic coordinates: 1. Graph-based distance and angles via Personalized Page-Rank; 2. Molecular distance bounds and corresponding angles based on chemical knowledge. To embed the resulting distances and angles, the authors propose to convert the original molecular graphs to line graphs. With line graphs as input, regular GNNs work as directional MPNNs, which exhibit better performance in practice. Experiments on benchmarks validate the improvements achieved by using synthetic coordinates in molecular graphs with several baseline models. 

The paper proposes a method for estimating molecular properties without 3D information about the molecules. The main contribution of the paper is an approach to approximate the local distance and angular information. This may then be fed into other GNNs for calculating the various properties. The authors also explore improvements related to using line graphs.  Results are shown on QM9, ZINC and ogbg-molhiv datasets. 

This paper aims to improve molecular property prediction by generating synthetic 3D coordinates. The method is based on the key observation that the true 3D coordinates are difficult to obtain and oftentimes undecidable for molecular structures. The paper proposes synthetic coordinates, which then enables using GNNs without requiring the true molecular configuration. The experimental results on several data sets have shown very promising results using this method.