Simple GNN Regularisation for 3D Molecular Property Prediction and Beyond
Authors: Jonathan Godwin
Michael Schaarschmidt
Alexander L Gaunt
Alvaro Sanchez-Gonzalez
Yulia Rubanova
Petar Veličković
James Kirkpatrick
Peter Battaglia
Venue: ICLR 2022 Poster (2022)
Keywords: Graph Neural Networks
GNNs
Deep Learning
Molecular Property Prediction

Abstract: In this paper we show that simple noisy regularisation can be an effective way to address oversmoothing. We first argue that regularisers ad-dressing oversmoothing should both penalise node latent similarity and encourage meaningful node representations. From this observation we derive “Noisy Nodes”,a simple technique in which we corrupt the input graph with noise, and add a noise correcting node-level loss.  The diverse node level loss encourages latent node diversity, and the denoising objective encourages graph manifold learning.  Our regulariser applies well-studied methods in simple, straightforward ways which allow even generic architectures to overcome oversmoothing and achieve state of the art results on quantum chemistry tasks such as QM9 and Open Catalyst, and improve results significantly on Open Graph Benchmark (OGB) datasets.  Our results suggest Noisy Nodes can serve as a complementary building block in the GNN toolkit.

Published as a conference paper at ICLR 2022
SIMPLE GNN REGULARISATION FOR 3D MOLECULAR PROPERTY PREDICTION & BEYOND
Jonathan Godwin, Michael Schaarschmidt, Alexander Gaunt, Alvaro Sanchez-Gonzales, Yulia Rubanova, Petar Veličković, James Kirkpatrick & Peter Battaglia DeepMind, London
{ (onathangodwin n) edeepmind.com com
ABSTRACT
In this paper we show that simple noisy regularisation can be an effective way to address oversmoothing. We argue that regularisers addressing oversmoothing should both penalise node latent similarity and encourage meaningful node repre- sentations. From this observation we derive "Noisy Nodes", a simple technique which we corrupt the input graph with noise, and add a noise correcting node-level loss. The diverse node level loss encourages latent node diversity, and the denoising objective encourages graph manifold learning, Our regulariser applies well-studied methods in simple, straightforward ways which allow even generic architectures to overcome oversmoothing and achieve state of the art results on quantum chemistry tasks, and improve results significantly on Open Graph Benchmark (OGB) datasets. Our results suggest Noisy Nodes can serve complementary building block in the GNN toolkit.
INTRODUCTION
Graph Neural Networks (GNNs) are a family of neural networks that operate on graph structured data
by iteratively passing learned messages over the graph's ' structure (Scarselli et al. 2009 Bronstein
al. 2017 Gilmer et al. 2017 Battaglia et al. 2018 Shlomi et al. 2021) While Graph Neural
Networks have demonstrated success in a wide variety of tasks (Zhou et al. 2020a Wu et al. 2020
Bapst et al. 2020 (chüit et al, 2017 Klicpera et al. 2020a), it has been proposed that in practice "oversmoothing" limits their ability to benefit from overparametrizztion.
Oversmoothing is a phenomenon where a GNN's latent node representations become increasing indis- tinguishable over successive steps of message passing (Chen et al. 2019). Once these representations are oversmoothed, the relational structure of the representation is lost, and further message- message-passing cannot improve expressive capacity. We argue that the challenges of overcoming oversmoothing two fold. First, finding a way to encourage node latent diversity; second, to encourage the diverse node latents encode meaningful graph representations, Here propose simple noise regulariser, Noisy Nodes, and demonstrate how it overcomes these challenges across a range of datasets and architectures, achieving top results on OC20 IS2RS& & IS2RE direct, QM9 and OGBG-PCQM4MVL
Our "Noisy Nodes" method is a simple technique for regularising GNNs and associated training procedures. During training, our noise regularisation approach corrupts the input graph's 's attributes with noise, and adds per-node noise correction term. We posit that our Noisy Nodes approach is effective because the model rewarded for maintaining and refining distinct node representations through message passing to the final output, which causes it to resist oversmoothing. Like denoising autoencoders, encourages the model to explicitly learn the manifold on which the uncorrupted input graph's 's features lic, analogous to form of representation learning. When applied to 3D molecular prediction tasks, it encourages the model to distinguish between low and high energy states. We find that applying Noisy Nodes reduces oversmoothing for shallower networks, and allows us to improvements with added depth, even on tasks for which depth was assumed to be unhelpful.
This study's 's approach is to investigate the combination of Noisy Nodes with generic, popular baseline GNN architectures. For 3D Molecular prediction we use a standard architecture working on 3D point clouds developed for particle fluid simulations, the Graph Net Simulator (GNS) (Sonaee7

---

Published as a conference paper at ICLR 2022
al. 2020). which has also been used for molecular property prediction (Hu et al. 2021b). Without using Noisy Nodes the GNS is not a competitive model, but using Noisy Nodes allows the GNS to achieve top performance
three 3D molecular property prediction tasks: the OC20 IS2RE direct task by 43% over previous work, 12% on OC20 IS2RS direct, and top results on 3 out of 12 of the QM9 tasks. For non-spatial GNN benchmarks OGBG-MOLPCBA and OGBG-PCQMAM (Hu et al., 202 and again see significant improvements. Finally, we applied Noisy Nodes to a GCN (Kipt & Welling 2016), arguably the most popular and simple GNN, trained on OGBN- -ixiv and see similar results. These results suggest Noisy Nodes can serve as a complementary GNN building block.
test MPNN (Gilmer et al. (017)
2 PRELIMINARIES: GRAPH PREDICTION PROBLEM
Let G = (V, (V, g) be an input graph. The nodes are V (Vi..... uivi where vs R. The directed, attributed edges E {ei, -........ (IE) each edge includes sender node index, receiver node index, and edge attribute, ex (8,,TR, ek), respectively, where rk E ( [VI] and Eg E R The graph-level property is g E Rd
The goal is to predict target graph. G', with the same structure G. but different node, edge, and/or graph-level attributes. We denote G' as model's prediction of G'. Some error metric defines quality of G' with respect to the target G', Error(G', G'), which the training loss terms are defined to optimize. In this paper the phrase "message passing steps" is synonymous with "GNN layers".
3 OVERSMOOTHING
"Oversmoothing" is when the node latent vectors of a GNN become very similar after successive layers of message passing. Once nodes are identical there is relational information contained in the nodes, and no higher-order latent graph representations can be learned. It is easiest to see this effect with the update function of a Graph Convolutional Network with adjacency normalization v/ = -,, Woy with j E Neighhorhood W T& xdg and k the layer index. As the number of applications increases, the averaging effect of the summation forces the nodes to become almost identical. However, as soon as residual connections are added we can construct a network that need suffer from oversmoothing by setting the residual updates to zero similarity threshold. Similarly, multi-head attention Vaswani et al. (2017); Veličković et al. (2018) and GNNs with edge updates (Battaglia al. 2018 Gilmer al. (2017) modulate node updates. As such for modern GNNs oversmoothing is primarily a "training" problem i.e. how to choose model architectures and regularisers encourage and preserve meaningful latent relational representations.
We can discern two desiderata for a regulariser or loss that addresses oversmoothing. First, it should penalise identical node latents. Second, it should encourage meaningful latent representations of the data. One such example may be the auto-regressive loss of transformer based language models (Brown et al. (2020). In this case, each word (equivalent to node) prediction must be distinct, and the auto-regressive loss encourages relational dependence upon prior words. We can take inspiration from this observation to derive auxiliary losses that both have diverse node targets and encourage relational representation learning. In the following section derive one such regulariser, Noisy Nodes.
4 NOISY NODES
Noisy Nodes tackles the oversmoothing problem by adding a diverse noise correction target, modify- ing the original graph prediction problem definition in several ways. It introduces graph corrupted by noise, G (V. E. where i, E V is constructed by adding noise, d;, to the input nodes, 0: U/ o, The edges, E, and graph-level attribute, I, can either be uncorrupted by noise (i.e., E = E, ğ 9). calculated from the noisy nodes (for example in nearest neighbors graph), or corrupted independent of the problem setting.
are minor choices that can be informed by the specific
2

---

Published as a conference paper at ICLR 2022
toi
Per Layer Node Latent MAD
Vi
0.073
14
0.06
Ui
13
0.17
0.2
0.016
0.8
12
0.45
0.05
0.054
11
D.7
0.062
0.074
10
0.81
0.32
0.11
+Di '
Layer
0.45
0.57
0,032
0.46
0,015
0.12
0.57
0.5
0.7
+
Oi
0.56
0.67
0.6b
0.4
0.79
0.2
0.74
1/
0.67
0.7
0.73
0.95
0.85
0.92
Noisy Nodes
Drophdge
DropNode
Figure I: Noisy Node mechanics during training. Input positions are corrupted with noise o, and the training objective is the node-level difference between target positions and the noisy inputs.
Figure 2: Per layer node latent diversity, measured by MAD 16 layer MPNN trained on OGBG- MOLPCBA. Noisy Nodes maintains a higher level of diversity throughout the network than competing methods.
Our method requires noise correction target to prevent oversmoothing by enforcing diversity in the last layers of the GNN, which can be achieved with an auxiliary denoising autoencoder loss. For example, where the Error is defined with respect to graph-level predictions (e.g., predict the minimum energy value of some molecular system), a second output head can be added to the GNN architecture which requires denoising the inputs as targets. Alternatively, if the inputs and targets are in the same real domain as the case for physical simulations we can adjust the target for the noise. Figure [ demonstrates this Noisy Nodes set up. The auxiliary loss is weighted by constant coefficient À E R.
In Figure 2 we illustrate the impact of Noisy Nodes on oversmoothing by plotting the Mean Absolute Distance (MAD) (Chen al.) (2000) of the residual updates of each layer of an MPNN trained on the QM9 (Ramakrishnan et al. (014)) dataset, and compare it to alternative methods DropEdge (Rong al. 2019) and Drop (Do al. 2021). MAD is a measure of the diversity of graph node features, often used to quantify oversmoothing, the higher the number the more diverse the node features, the lower the number the less diverse. In this plot we can see that for Noisy Nodes the node updates remain diverse for all of the layers, whereas without Noisy Nodes diversity is lost. Further analysis of MAD across seeds and with sorted layers can be seen Appendix Figures 7 and 6 for models applied to 3D point clouds.
The Graph Manifold Learning Perspective. By using an implicit mapping from corrupted data to clean data, the Noisy Nodes objective encourages the model to learn the manifold on which the clean data lies- we speculate that the GNN learns to go from low probability graphs to high probability graphs. In the autoencoder provided, the GNN Iearns the manifold of the target data (c.g. the manifold of atoms at equilibrium). We speculate that such manifold may include commonly repeated substructures that are useful for downstream prediction tasks. A similar motivation can be found for denoising in (Vincent et al.) 2010 Song & Ermon 2019).
the GNN learns the manifold of the input data. When node targets
The Energy Perspective for Molecular Property Prediction. Local, random distortions of the geometry of molecule a local energy minimum are almost certainly higher energy configurations. As such, a task that maps from a noised molecule to a local energy minimum is learning a mapping from high energy to low energy. Data such as QM9 contains molecules at local minima.
Some problems have input data that is already high energy, and targets that are at equilibrium. For these datasets we can generate new high energy states by adding noise to the inputs but keeping the equilibrium target the same, Figure] T demonstrates this approach. To preserve translation invariance we use displacements between input and target A, the corrected target after noise is A a.
RELATED WORK
Oversmoothing, Recent work has aimed to understand why it is challenging to realise the benefits of training deeper GNNs (Wu et al.) (020). Since first being noted in ((i et al.) (018)) oversmoothing has been studied extensively and regularisation techniques have been suggested to overcome it (Chen
3

---

Published as a conference paper at ICLR 2022
et al.
2019:
Rong et al. 2019 Zhou et al. 2020b Yang et al. 2020: Do et al. Zhao & Akoglu 2020). A recent paper, (Li et al. 2021), finds, as in previous work, (Li et al.
Cai & Wang 2020;
2021
2019
2020), the optimal depth for some datasets they evaluate on to be far lower (5 for -GBN-
from the Open Graph Benchmark (Hu et al. 2020a), for example) than the 1000 layers possible.
Denoising & Noise Models. Training neural networks with noise has a long history (Sietsma & Dow 1991 Bishop 1995). Of particular relevance are Denoising Autoencoders (Vincent et al. 2008)
in which an autoencoder is trained to map corrupted inputs x to uncorrupted inputs x. Denoising
Autoencoders have found particular success as a form of pre-training for representation learning (Vincent al., 2010)) More recently, in research applying GNNs simulation (Sanchez- Gonzalez
et al.] Sanchez- Sanchez-Gonzallee Gonzalez* et al.) 2020: Plaff et al. 2020) Gaussian noise is added during training to input positions of ground truth simulator to mimic the distribution of errors of the learned simulator. Pre-training methods (Devlin et al.) 2019 You et al. 20200 (hakooor et al., 2021) are another similar approach; most similarly to our method Hu et al. (2020b) apply a reconstruction loss to graphs with masked nodes to generate graph embeddings for use in downstream tasks. FLAG KKng et al. 2020) adds adversarial noise during training to input node features as form of data augmentation for GNNs that demonstrates improved performance for many tasks. It does not add an additional auxiliary loss, which we find is essential for addressing oversmoothing. In other related GNN work, (Sato et al., (021) use random input features to improve generalisation of graph neaural networks. Adding noise to help input node disambiguation has also been covered in (Dasoulas et al. 2019
Loukas 2020
Vignac et al. 2020: Murphy et al. 2019). but there is no auxiliary loss.
Finally, we take inspiration from (Vincent et al. 2008 2010; Vincent 2011 Song & Ermon 2019) which use the observation that noised data lies off the data manifold for representation learning and generative modelling.
Machine Learning for 3D Molecular Property Prediction. One application of GNNs is to speed up quantum chemistry calculations which operate on 3D positions of a molecule (Duvenaud et al. 2015; Gilmer al., [017 Schütt et 2017 Hu et al.] (2021b). Common goals are the prediction of molecular properties (Ramakrishnan et al. 2014). forces (Chmiela et al. 2017), energies (Chanussot
ct al. )020) and charges (Unke & Meuwly y 2019).
A common approach to embed physical symmetries is to design network that predicts a rotation and translation invariant energy (Schltt et al. 2017: Klicpera et al. 2020a Liu et al. (022)) The input features of such models include distances (Schütt et 20.77) angles (Klicpera et al.) 2020b a) or torsions and higher order terms (Liu et al.] 2021). An alternative approach to embedding symmetries is to design rotation equivariant neural network that equivariant representations (Thomas et al.
2018 Köhler et al., 2019: Kondor et al. 2018 Fuchs et al. 2020 Batzner et al. 202 Anderson
al.
2019 Satorras et al. 2021).
Machine Learning for Bond and Atom Molecular Graphs. Predicting properties from molecular graphs without 3D points, such graphs of bonds and atoms, is studied separately and often used to benchmark generic graph property prediction models such as GCNs (Hu et al. (020a) or GATs (Veličković et al., 2018).. Models developed for 3D molecular property prediction cannot be applied to bond and atom graphs. Common datasets that contain such data are OGBG-MOLPCBA and OGBG-MOLHIV.
6 3D MOLECULAR PROPERTY PREDICTION EXPERIMENTS AND RESULTS
In this section we evaluate how a popular, simple model, the GNS (Sanchez-Gonzalezz et al. 2020 performs on 3D molecular prediction tasks when combined with Noisy Nodes. The GNS was originally developed for particle fluid simulations, but has recently been adapted for molecular property prediction (Hu et al., (0221b) We find that Without Noisy Nodes the GNS architecture is not competitive, but by using Noisy Nodes see improved performance comparable to the use of specialised architectures.
We made minor changes to the GNS architecture. We featurise the distance input features using radial basis functions. We group layer weights, similar to grouped layers used in [nmper et al. (202T) for reduced parameter counts; for group size of the first layer weights are repeated, i.e. the first layer with group size of 10 has the same weights the 11 th 21 31sd layers and so contiguous
4

---

Published as a conference paper at ICLR 2022
Shared Layer Weights
Shared Layer Weights
With Node Aux. Task
3x
Previous S01A
Previous SOTA
$x
1x30
3P
10 MP
0.8
Training Step
208
Unshared Weights Noisy Nodes
D)
50 Message Passing Steps Noisy Nodes
70200T
PN 1-"
MP
Shar ed Layer Weights
10 MP
Unshared are Layer Weights
50
6x1001
6-
Prestuu
Previuus 5OTA
OV
0.0 022
0.4
0.5
3.0 Training Stop
0.5
32
166
0.00
0:25
0.5
2.15
1.00
1.25
155
2.00
Trining Stop
Figure 3: Validation curves, OC20 IS2RE ID. A) Without any node targets our model has poor performance and realises no benefit from depth. B) After adding position node loss, performance improves as depth increases. C) As we add Noisy Nodes and parameters the model achieves SOTA, even with 3 layers, and stops overfitting. D) Adding Noisy Nodes allows a model with even fully shared weights to achieve SOTA.
blocks of layers are considered single group. Finally we find that decoding the intermediate latents and adding a loss after each group aids training stability. The decoder is shared across groups.
We tested this architecture on three challenging molecular property prediction benchmarks: OC20 (Chanussot* et al.) 2020) IS2RS & IS2RE, and QM9 (Ramakrishnan et al. (2014). These benchmarks are detailed below, but as general distinctions, OC20 tasks use graphs 2-20x larger than QM9. While QM9 always requires graph-level prediction, of OC20's two tasks (IS2RS) requires node-level predictions while the other (IS2RE) requires graph-level predictions. All training details may be found in the Appendix.
6.1 OPEN CATALYST 2020
Dataset. The OC20 dataset (Chanussot* et al. 2020) (CC Attribution 4.0) describes the interaction of small molecule (the adsorbate) and large slab (the catalyst). with total systems consisting of 20-200 atoms simulated until equilibrium is reached.
We focus on two tasks; the Initial Structure to Resulting Energy (IS2RE) task which takes the initial structure of the simulation and predicts the final energy, and the Initial Structure to Resulting Structure (IS2RS) which takes the initial structure and predicts the relaxed structure. Note that we train the more common "direct" prediction task that map directly from initial positions to target in a single forward pass, and compare against other models trained for direct prediction.
Models are evaluated on 4 held out test sets. Four canonical validation datasets also provided. Test sets are evaluated on a remote server hosted by the dataset authors with a very limited number of submissions per team.
Noisy Nodes in this case consists of a random jump between the initial position and relaxed position. During training first sample uniformly from point in the relaxation trajectory or interpolate uniformly between the initial and final positions (v; - 0;)y, y ~ U(0, 1), and then add L.I.D Gaussian noise with mean zero and 0.3. The Noisy Node target the relaxed structure.
5

---

Published as a conference paper at ICLR 2022
Table I:
OC20 ISRE Validation, eV MAE,
"GNS-Sharee indicates shared weights. "GNS-10" indicates a group size of 10.
Model
Layers
OOD Both
OOD Adsorbate
OOD Catalyst
ID
GNS
0.59 10.01
0.65 10.01
0.35 10.00
0.54 10.00
GNS-Shared + Noisy Nodes
50
0.49 10.00
0.54 10.00
0.51 10.01
0.51 10.01
GNS Noisy Nodes
50
0.48 10.00
0,53 10.00
0.49 10.01
0.48 10.00
GNS-10 Noisy Nodes
100
0.46.0000
0.51 10.00
0.48 10.00
0.47 10.00
Table 2: Results OC20 IS2RE Test
eV MAE +
SchNet
DimeNet++
SpinConv
SphereNet
GNS + Noisy Nodes
OOD Both
0.704
0.661
0.674
0.638
0,465 (-24.0%)
0OD Adsorbate
0.734
0.725
0.723
0.703
0.565 (-22.8% )
OOD Catalyst
0.662
0.576
0.569
0.571
0.437 (-17.2%
ID
0.639
0.562
0.558
0.563
0.422 (-18.8%
Average Energy within Threshold (AEwT)
SchNet
DimeNet++
SpinConv
SphereNet
GNS + Noisy Nodes
OOD Both
0.0221
0.0241
0.0233
0.0241
0.047 (+95.8 %%
0OD Adsorbate
0.0233
0,0207
0.026
0.0229
0.035 (+89.5%
OOD Catalyst
0.0294
0.0410
0.0382
0.0409
0.080 (+95.1
ID
0.0296
0.0425
0.0408
0.0447
0.091 (+102.0%)
We first convert to fractional coordinates (i.e. use the periodic unit cell as the basis) which render the predictions of our model invariant to rotations, and append the following rotation and translation invariant vector (a:3", , ary", lal, (8), hx]) e R to the edge features where a; B, y are vectors of the unit cell. This additional vector provides rotation invariant angular and extent information to the GNN.
show how using Noisy Nodes allows the GNS to achieve state of the art performance. Figure 3 A shows that without any auxiliary node target, an IS2RE GNS achieves poor performance even with increased depth. The fact that increased depth does result in improvement supports the hypothesis that GNS suffers from oversmoothing. As we add a node level position target in B) we better performance, and improvement as depth increases, validating our hypothesis that node level targets are key to addressing oversmoothing, In C) we add noisy nodes and parameters, and see that the increased diversity the node level predictions leads very significant improvements and SOTA, even for shallow 3 layer network. D) demonstrates this effect is not just due to increased parameters SOTA can still be achieve with shared layer weights.
IS2RE Results. In Figure
In Table I we conduct an ablation on our hyperparameters, and again demonstrate the improved performance of using Noisy Nodes. Results were averaged over seeds and standard errors on the best obtained checkpoint show little sensitivity to initialisation. All results in the table are reported using sampling states from trajectories. We conducted an ablation on ID comparing sampling from a relaxation trajectory and interpolating between initial & final positions which found that interpolation improved our score from 0.47 to 0.45.
Our best hyperparameter setting 100 layers which achieved 95.6% relative performance improvement against SOTA results (Table on the AEwT benchmark. Due to limited permitted test submissions, results presented here were from one test upload of our best performing validation seed.
IS2RS Results. In Table we see that GNS + Noisy Nodes is significantly better than the only other reported IS2RS direct result, ForceNet, itself GNS variant.

---

Published as a conference paper at ICLR 2022
Table OC20 IS2RS Validation, ADwT,
Model
Layers
OOD Both
OOD Adsorbate
0OD Catalyst
ID
GNS
50
43.0%=00.0
38.0%::0.00
37.5% 0.0
40.0%=#000
GNS + Noisy Nodes
50
10.1%
44.3%:0.0
44.1%:0.0
46.1% 10.0
GNS-10 Noisy Nodes
52.0%
46.2%:
46.1% 10.0
48.3% 10.0
GNS-10 Noisy Nodes + only
54.3%=10.0
48.3%
48.2% #0.0
50.0% 10.0
Table OC20 IS2RS Test, ADwT, 1
Model
OOD Both
OOD Adsorbate
OOD Catalyst
ID
ForceNet
46.9%
37.7%
43.7%
44.9%
GNS Noisy Nodes
52.7% %
43.9%
48.4%
50.9%
Relative Improvement
+12.4%
+16.4%
+10.7%
+13.3%
6.2
QM9
Dataset. The QM9 benchmark (Ramakrishnan et al. 2014) contains 134k molecules in equilibrium with up to heavy C, O, N and atoms, targeting associated chemical properties (License: CCBY 4.0). We use 114k molecules for training, 10k for validation and 10k for test. All results are on the test set. We subtract fixed per atom energy from the target values computed from linear regression to reduce variance. We perform training in eV units for energetic targets, and evaluate using MAE. summarise the results across the targets using mean standardised MAE (std. MAE) in which MAEs are normalised by their standard deviation, and mean standardised logMAE. Std. MAE is dominated by targets with high relative error such as Ae, whereas logMAE is sensitive to outliers such R² As is standard for this dataset, model is trained separately for each target.
For this dataset we add I.I.D Gaussian noise with mean zero and a = 0.02 to the input atom positions. A denoising autoencoder loss is used.
Results In Table 6 we can see that adding Noisy Nodes significantly improves results by 23.1 % relative for GNS, making it competitive with specialised architectures. To understand the effect of adding a denoising loss, we tried just adding noise and found no where near the same improvement (Table
A IONS-10+ Noisy Nodes with 30 layers achieves top results on 3 of the 12 targets and comparable performance the remainder (Table 6). On the std. MAE aggregate metric GNS Noisy Nodes performs better than all other reported results, showing that Noisy Nodes can make even a generic model competitive with models hund-crafted for molecular property prediction. The same trend is repeated for an rotation invariant version of this network that uses the principle axes of inertia ordered by eigenvalue as the co-ordinate frame (Table
(R²), the electronic spatial extent, is an outlier for GNS Noisy Nodes. Interestingly, we found that without noise GNS-10+ Noisy Nodes achieves 0.33 for this target. We speculate that this target particularly sensitive to noise, and the best noise value for this target would be significantly lower than for the dataset as a whole.
Table QM9, Impact of Noisy Nodes on GNS architecture.
Layers
std.
MAE
Change
logMAE
GNS
10
1.17
-5.39
GNS Noise But No Node Target
10
1.16
-0.9%
-5.32
GNS + Noisy Nodes
10
0.90
-23.1%
-5.58
GNS-10 10 + Noisy Nodes
20
0.89
-23.9%
-5.59
GNS-10 10-10+ + Noisy Nodes + Invariance
30
0.92
-21.4%
-5.57
GNS-10 10 + Noisy Nodes
30
0.88
-24.8%
-5.60

---

Published as conference paper at ICLR 2022
Table 6: QM9, Test MAE, Mean & Standard Deviation of 3 Seeds Reported.
Target
Unit
SchNet
E(n)GNN
DimeNet++
SphereNet
PaiNN
GNS Noisy Nodes
D)
0.033
0.029
0.030
0.027
0.012
0.025 +0.01
ao
0.235
0,071
0.043
0.047
0.045
0.052 #0.00
HOMO
mcV
41
29.0
24.6
23.6
27.6
20.4 #0.2
CLUM
meV
34
25.0
19.5
18.9
20.4
18.6 #0.4
Ae
meV
63
48.0
32.6
32.3
45.7
28.6 #0.1
(R2)
40²
0.07
0.11
0.33
0.29
0.07
0.70 +0.01
ZPVE
meV
1.7
1.55
1.21
1.12
1.28
1.16 #0.01
Up
meV
14.00
11.00
6.32
6.26
5,85
7.30 +0.12
U
meV
19.00
12.00
6.28
7.33
5.83
7.57 +0.03
H
meV 14.00
12.00
6.53
6,40
5,98
7.43=0.06
G
meV 14.00
12.00
7.56
8.0
7.35
8.30 10.14
Gv
molk
0.033
0.031
0.023
0.022
0.024
0.025 #0.00
std. MAE
%
1.76
1.22
0.98
0.94
1.00
0.88
logMAE
-5.17
-5.43
-5.67
-5.68
-5.85
-5.60
Table OGBG-PCQM4M Results
Model
Number of Layers
Using Noisy Nodes
MAE
MPNN Virtual Node
16
Yes
0.1249 0.0003
MPNN Virtual Node
50
No
0.1236 + 0.0001
Graphormer
Ying et al. 2021
0. 1234
MPNN Virtual Node
50
Yes
0.1218 + 0.0001
NON-SPATIAL TASKS
The previous experiments the 3D geometries of atoms, and models that operate 3D points. However, the recipe of adding a denoising auxiliary loss can be applied to other graphs with different types of features. In this section we apply Noisy Nodes additional datasets with 3D points, using different GNNs, and show analagous effects to the 3D case. Details of the hyperparameters, models and training details can be found in the appendix.
7.1 OGBG-PCQM4M
This dataset from the OGB benchmarks consists of molecular graphs which consist of bonds and atom types, and no 3D or 2D coordinates. To adapt Noisy Nodes to this setting, we randomly flip node and edge features a rate of 5% and add reconstruction loss. We evaluate Noisy Nodes using an MPNN + Virtual Node (Gilmer et al. 2017). The test set is not currently available for this dataset.
In Table 7
we that for this task Noisy Nodes enables 50 layer MPNN to reach state of the art results. Before adding Noisy Nodes, adding capacity beyond 16 layers did not improve results.
7.2 DGBG-MOLPCBA
The OGBG-MOLPCES dataset contains molecular graphs with no 3D points, with the goal of classifying 128 biological activities. On the OGBG-MOLPCES Virtual Node and random flipping noise. In Figure we see that adding Noisy Nodes improves the performance of the base model, accentuated for deeper networks. Our 16 layer MPNN improved from 27.6% 0.004 28. 1% 0.002 Mean Average Precision ("Mean AP"). Figure diguneeademonstrates how Noisy Nodes improves performance during training. Of the reported results, MPNN is most similar to GCN + Virtual Node and GIN + Virtual Node (Xu et al. (2088) which report results of 24.2% + 0.003 and 27.03% = 0.003 respectively. We evaluate alternative methods for
dataset we again use an MPNN +
Thee GCN implemented in the official OGB code hase has explicit edge updates, akin the MPNN.
8

---

Published as a conference paper at ICLR 2022
0.290
Noisy Effect,
Velidation Curve. MGBG-MMLPCBA, Layers
Noley Nodes
0.205
Without
0.29
Hean NM
0.280
0.25
0.275
VMl Mean
0.21
0.26
0.205
0.260
0.25
Noley Nooes
Withott
With
0.2
Global
Figure 4: Adding Noisy Nodes with random flipping of input categories improves the per- formance of MPNNs, and the effect is accen- tuated with depth.
Figure 5: Validation curve comparing with and without noisy nodes. Using Noisy Nodes leads to a consistent improvement.
oversmoothing, DropNode and DropEdge in Figure and find that Noisy Nodes is more effective at address oversmoothing, although all methods can be combined favourably (results in appendix).
7.3 OGBN-ARXIV
The above results use models with explicit edge updates, and are reported for graph prediction. To test the effectiveness with Noisy Nodes with GCNs, arguably the simplest and most popular GNN, we use OGBN-ARXIV: Adding Noisy Nodes, with noise as input dropout of 0.1, to 4 layer GCN with residual connections improves from 72.39% 0.002 accuracy to 72.52% + 0.003 accuracy. baseline layer GCN on this dataset reports 71.71% + 0.002. The SOTA for this dataset is 74.31% % (Sun & Wu 2020).
citation network with the goal of predicting the arxiv category of each paper.
7.4 LIMITATIONS
We have demonstrated the effectiveness of Noisy Nodes in small data regimes, which may be important for learning from experimental data. The representation learning perspective requires access local minimum configuration, which is not the case for all quantum modeling datasets. We have also not demonstrated the combination of Noisy Nodes with more sophisticated 3D molecular property prediction models such as DimeNet++(llipernr alternative reconstruction loss to position change, such pairwise interatomic distances. We leave this to future work.
2020a). such models may require an
Noisy Nodes requires careful selection of the form of noise, and balance between the auxiliary and primary losses. This can require hyper parameter tuning, and models can sensitive to the choice of these parameters. Noisy Nodes has particular effect for deep GNNs, but depth is not always an advantage. There are situations, for example molecular dynamics, which place a premium on very fast inference time. However even at 3 layers (a comparable depth to alternative architectures) the GNS architecture achieves state the art validation OC20 IS2RE predictions (Figure 33. Finally, returns diminish depth increases indicating depth is not the only answer (Table T).
8 CONCLUSIONS
In this work we present Noisy Nodes, novel regularisation technique for GNNs with particular focus on 3D molecular property prediction. Noisy nodes helps address common challenges around oversmoothed node representations, shows benefits for GNNs of all depths, but in particular improves performance for deeper GNNs. We demonstrate results on challenging 3D molecular property prediction tasks, and some generic GNN benchmark datasets. We believe these results demonstrate Noisy Nodes could be a useful building block for GNNs for molecular property prediction and beyond.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Code for reproducing OGB- DGB-PCQM4M results using Noisy Nodes is available on github, and was prepared as part of leaderboard submission. https://githut com/ deepmind/
We provide detailed hyper parameter settings for all our experiments in the appendix, addition to formulae for computing the encoder and decoder stages of the GNS.
10 ETHICS STATEMENT
Who may benefit from this work? Molecular property prediction with GNNs is a fast-growing area with applications across domains such as drug design, catalyst discovery, synthetic biology, and chemical engineering. Noisy Nodes could aid models applied these domains. We also demonstrate on OC20 that our direct state prediction approach is nearly as accurate as learned relaxed approaches at a small fraction of the computational cost, which may support material design which requires many predictions.
Finally, Noisy Nodes could be adapted and applied to many areas in which GNNs are used for example, knowledge base completion, physical simulation traffic prediction.
Potential negative impact and reflection. Noisy Nodes sees improved performance from depth, but the training of very deep GNNs could contribute to global warming. Care should be taken when utilising depth, and we note that Noisy Nodes settings can calibrated at shallow depth.
REFERENCES
Brandon M. Anderson, T. Hy, and R. Kondor. Cormorant: Covariant molecular neural networks. In NeurIPS, 2019.
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci, Jonathan Godwin, Chris Jones, Tom Hennigan, Matteo Hessel, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King, Lena Martens, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http: cwm/ deepmind]
V. Bapst, T. Keck, Agnieszka Grabska- Grabska-Barwinska. C. Donner, E. D. Cubuk, S. Schoenholz, A. Obika, Alexander W. R. Nelson, Back, Hassabis, and P. Kohli. Unveiling the predictive power of static structure in glassy systems. Nature Physics, 16:448-44 -444,,4444001 2020.
Battaglia, Jessica Hamrick, V. Bapst, A. Sanchez-Gonzaler V. Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, A. Santoro, R. Faulkner, Çaglar Gülçehre. H. Song. A. J. Ballard, Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charlie Nash, Victoria Langston, Chris Dyer, N. Heess, Daan Wierstra, P. Kohli, M. Botvinick, Oriol Vinyals, Y. Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. ArXiv, abs/180s 888.01261.. 2018.
Simon Batzner, T. Smidt, L. Sun, J. Mailoa, M. Kornbluth, N. Molinari, and B. Kozinsky. Se(3)- equivariant graph neural networks for data-efficient and accurate interatomic potentials. ArXiv, abs/2101.031 031644, 2021.
Charles M. Bishop. Training with noise is equivalent to tikhonov regularization. Neural Computation, 7:108-116, 1995.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL
http::/gitee com/ goog/ www.ooololoaaeeoo ja
10

---

Published as a conference paper at ICLR 2022
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18 42, 2017.
T. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, TT. Henighan, R. Child, A. Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005.141655 2020.
Chen Cai and Yusu Wang. A note on over-smoothhing for graph neural networks. CoRR, abs/2006. 13311006.33,, 2020. URL
Lowik Chanussot*, Abhishek Das*, Siddharth Goyal*, Thibaut Lavrile, Muhammed Shuaibi*, Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 0(0):
Ulissi.
6059-6072, 2020. doi:
URL https:://dll.or//111
acscatal 0c04525
Deli Chen, Yankai Lin, Wei Li, Peng Li. Jie Zhou, and Xu Sun. Measuring and relieving the over- smoothing problem for graph neural networks from the topological view. CoRR, 0tns/19090033 2019. htllhtttp///raeee
org abs 9099999900 C3211
Deli Chen, Yankai Lin, W. Li, Peng Li, J. Zhou, and Xu Sun. Measuring and relieving the over- smoothing problem for graph neural networks from the topological view. In AAAI, 2020.
Stefan Chmiela, A. Tkatchenko, H. E. Sauceda, I. Poltavsky, Kristof T. Schütt, and K. Müller molecular force fields. Science Advances, 3, 2017.
Machine learning of accurate
George Dasoulas, Ludovic Dos Santos, Kevin Scaman, and Aladin Virmaux. Coloring graph neural networks for node disambiguation. ArXiv, abs/1912.06058, 2019.
Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Tien Huu Do, Due Minh Nguyen, Giannis Bekoulis, Adrian Munteanu, and N. Deligiannis. Graph con- volutional neural networks with node transition probability-baa message passing and dropnode regularization. Expert Syst. Appl., 174:114711, 2021.
David Duvenaud, Dougal Maclaurin, Jorge Aguilera- Iaarrcra--parrrrrrreee Rafael Bömez-Bombarelli, Tim- othy Hirzel, Alán Aspuru-Guzia and Ryan P. Adams. Convolutional networks on graphs for learning molecular fingerprints. Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, NIPS' 15, pp. 2224-2232, Cambridge, MA, USA, 2015. MIT Press.
Fuchs, Daniel E. Worrall, Volker Fischer, and M. Welling. transfrrrassssmmrss: 3d roto-translation equivariant attention networks. ArXiv, habb/2000610503.. 2020.
Gilmer, S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. ArXiv, abs/ 704.01212, 2017.
Jonathan Godwin Thomas Keck*, Peter Battaglia, Victor Bapst, Thomas Kipf, Yujia Li, Kimberly Stachenfeld, Petar Veličković, and Alvaro Sanchez- Jraph: A library for graph neural networks in jax., 2020. URL http:://tt:tthee http:///gg..tabb.cooo//eeee/
Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020.
URL http::////gttt thub com/ hoom/deepmi.. nd/ / dm- immhaiku
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. ArXiv, abs/2005.00681 2020a.
11

---

Published as a conference paper at ICLR 2022
Weihua Hu, Bowen Liu, Joseph Gomes, M. Zitnik, Percy Liang, V. Pande, and J. Leskovec. Strategies for pre-training graph neural networks. arXiv: Learning, 2020b.
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-Isc: A large-scale challenge for machine learning on graphs. arXiv preprint #xXi:22133004430, 2021a.
Weihua Hu, Muhammed Shuaibi, Abhishek Das, Siddharth Goyal, Anuroop Sriram, J. Leskovec, Devi Parikh, and C. L. Zitnick. Forcenet: A graph neural network for large-scale quantum calculations. ArXiv, abs/2103.01432 2021b..
John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridg- land, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David A. Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Bergham- mer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:583 589, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/ Abs/1442,6980, 2015.
Thomas N. Kipf and Max Welling. Semi-supervised classsfiictinn with graph convolutional networks. 02907
CoRR, abs/1 abs//100900290,, 2016. [RL//ttp://axxw.or////b/b
Johannes Klicpera, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. and uncertainty-awee aware directional message passing for equilliiii molecules. CoRR, abs/2011.14115. 2020a. URL/ttts:://axxi wwtp://rgg//////////o//o /201 14115
Fast
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. ArXiv, abs/2003.031233 2020b.
Risi Kondor, Hy Truong Son, Horace Pan, Brandon M. Anderson, and Shubhendu Trivedi. Covariant compositional networks for learning graphs. CoRR, htt//800.0022 022144, 2018. URL http://
www.iioor//bb//1880 021444
Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, G. Taylor, and T. Goldstein. Flag: Adversarial data augmentation for graph neural networks. ArXiv, abs/2010.09891, 2020.
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: sampling configurations for multi-body systems with symmetric energies, 2019.
G. Li, M. Müller, Ali K. Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as cnns? 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp.
2019.
Guohao Li, C. Xiong, Ali K. Thabet, and Bernard Ghanem. Deepergen: All you need to train deeper gens. ArXiv, abs/2006.07739, 2020.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. CoRR, abs/2106.07476. 2021. URL http:::/axxd.ooo///aa/a
07476
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semii cemi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Yi Liu. Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint 100.03200..00013, 2021.
Andreas Loukas. How hard is to distinguish graphs with graph neural networks? arXiv: Learning. 2020.
12

---

Published as a conference paper at ICLR 2022
Ryan L. Murphy, Balasubramaniam Srinivasan, Vinayak A. Rao, and Bruno Ribeiro. Relational pooling for graph representations. In ICML, 2019.
T. Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and P. Battaglia. Learning mesh-based simula- tion with graph networks. ArXiv, abs/2010.03409, 2020.
R. Ramakrishnan, Pavlo O. Dral, M. Rupp, and O. A. von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. The truly deep graph convolutional networks for node classification. CoRR, 0bs/1907.10900, 2019. URL http:://axxiv.or/
abs /bs/1907.or 10903
Alvaro Sanchez-Gonzalle N. Heess, Jost Tobias Springenberg, J. Merel, Martin A. Riedmiller, R. Hadsell, and P. Battaglia. Graph networks as learnable physics engines for inference and control. ArXiv, 106.1222,,,, 2018.
Alvaro Sanchez-Gonzalezz Jonathan Godwin*, Tobias Pfaff*, Rex Ying*, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8459-8468 PMLR, 13-18 Jul 2020. URL htmll
htttp://proceet ngs .arssmmr.. prwpsr/sssss.. htsss//111//ancceeegoe/ee
R. Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. In SDM, 2021.
Victor Gareia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks, 2021.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2009. doi: 10. 200.66000...
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoe Sauceda Felix, Stefan Chmiela, A. Tkatchenko, and K. Müller. Schnet: A continuous-filte convolutional neural network for modeling quantum interactions. In NIPS, 2017.
Jonathan Shlomi, Peter Battaglia, and Roch Vlimant. Graph neural networks in particle physics. Machine Learning: Science and Technology, 2(2):021001, Jan 2021. ISSN 2632-2153. doi: 10.1088/233-21153abbb URL doi /ttp://.........1.///22221111131
Sietsma and Robert Dow. Creating artificial neural networks that generalize. Neural Networks, 4:67-79, 1991.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. ArXiv, habbs/19770056e. 2019.
Chuxiong Sun and Guoshi Wu. Adaptive graph diffusion networks with hop-wise attention. ArXiv, abs/2012.15024, 2020.
Shantanu Thakoor, C. Tallec, M. G. Azar, R. Munos, Petar Velivckovi'e, c, and Michal Valko. Boot- strapped representation learning on graphs. ArXiv, abs/2102.06514, 2021.
Nathaniel Thomas, Tess Smidt, Steven M. Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-qquiivrraat neural networks for 3d point clouds. CoRR, abs/ 0tts/1822.08199, 2018. URL
08219
Oliver T. Unke and Markus Meuwly. Physnet: A neural network for predicting energies, forces, dipole moments, and partial charges. Journal of Chemical Theory and Computation, (56)3367-333 May 2019. ISSN 1549-9626. doi: (0.1021/acs.jccccbb00133 URL http: //tp htttt::/dxx.do..rrz.. www.ddi...ggo.... I021/acs. jctc. 9b00181
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv,
2017.
13

---

Published as a conference paper at ICLR 2022
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks, 2018.
Cl'ement Vignac, Andreas Loukas, and Pascal Frossard. Building powerful and equivariant graph neural networks with structural gressage-passingg arXiv: Learning. 2020.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computa- lion, 23: 16666666611 11674, 2011.
Pascal Vincent, H. Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML '08, 2008.
Pascal Vincent, H. Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. Mach. Learn. Res., 11:3371- 14708, 2010.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? CoRR, abs/1810.00826, 2018. URLttp::///arxi> org abs 1wtp/11110.0022.
Chaoqi Yang, Ruijie Wang, Shuochao Yao, Shengzhong Liu, and Tarek Abdelzaher. Revisiting" over-smoothing" in deep gens. arXiv preprint arXiv:2003.13663 2020.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform bad for graph representation? ArXiv, abs/2106.05234, 2021.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. ArXiv, abs/2010. 139022 2020.
L. Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. ArXiv, abs/1909. 12223, 2020.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang. Changeheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020a.
Kuangqi Zhou, Yanfei Dong, Wee Sun Lee, Bryan Hooi, Huan Xu, and Jiashi Feng. Effective training strategies for deep graph neural networks. CoRR, abs/2006.07107, 2020b. URL httpst
7/ 7/arxiv.oree wrxxiv.org/abs/2066.. 07107
A
APPENDIX
The following sections include details on training setup, hyper- paper-parameters, input processing, as well as additional experimental results.
A.1 ADDITIONAL METRICS FOR OPEN CATALYST IS2RS TEST SET
Relaxation approaches IS2RS minimise forces with respect positions, with the expectation that forces at the minimum are close to zero. One metric of such model's success is to evaluate the forces at the converged structure using ground truth Density Functional Theory calculations and see how close they are to zero. Two metrics are provided by OC20 (Chanussot* et al., 2020) on the IS2RS test set: Force below Threshold (FbT), which is the percentage of structures that have forees below 0.05 eV/Angstrom, and Average Force below Threshold (AFbT) which is FbT caleulated at multiple thresholds.
The OC20 project computes test DFT calculations the evaluation server and presents a summary result for all IS2RS position predictions. Such calculations take 10- 12 hours and they are not available for the validation set. Thus, detail. Before application to catalyst screening further work may be needed for direct approaches to ensure forces do not explode from atoms being too close together.
not able to analyse the results in Tables 8 and ( in any further
14

---

Published as a conference paper at ICLR 2022
Table OC20 IS2RS Test, Average Force below Threshold %,
Model
Method
OOD Both
OOD Adsorbate
OOD Catalyst
ID
Noisy Nodes
Direct
0.09%
0.00%
0.29%
0.54%
Table 9: OC20 IS2RS Test, Force below Threshold %, 1
Model
Method
OOD Both
OOD Adsorbate
OOD Catalyst
ID
Noisy Nodes
Direct
0.0%
0.0%
0.0%
0.0%
A.2
MORE DETAILS ON GNS ADAPTATIONS FOR MOLECULAR PROPERTY PREDICTION.
Encoder.
The node features are learned embedding lookup of the atom type, and in the case of OC20 two additional binary features representing whether the atom is part of the adsorbate catalyst and whether the atom remains fixed during the quantum chemistry simulation.
The edge features, eg are the distances [d] featurised using Radial Bessel basis functions, ĒRBF. 2 sin( d) and the edge vector displacements, d, normalised by the edge distance:
V
cj = (onca(@xsF::(4))..... (RBF,, [d]
Our conversion to fractional coordinates only applied to the vector quantities, i.e. [d]
Decoder
The decoder consists of two parts, graph-level decoder which predicts a single output for the input graph, and node-level decoder which predicts individual outputs for each node. The graph-level decoder implements the following equation:
I
V
W Proc
MLPProc (a, (roc) + Prox W Enc
W
MLPEnee (a.
b Enc
y
Where a Proc are node latents from the Processor, Enc are node latents from the Encoder, W'Ene and Pro are linear layers, Fnc and Proe are biases, and [V| is the number of nodes. The node-level decoder simply an MLP applied to each Proc which predicts a²
A.3 MORE DETAILS ON MPNN FOR OGBG-PCQM4M AND OGBG-MOLPCBA
Our MPNN follows the blueprint of Gilmer et al. (2177). We use [(() denote the latent vector of node at message passing step f, and m!! be the computed message vector for the edge between nodes and message passing step f. We define the update functions as:
(1++1) = vi++ uv
[1]] T(0)
((()) + m))
(1)
g(++) = Φ4+1
I(t)
Σ
m
(t+1)
Σ
(++1) m
+ h
(2)
N.
BEN.
Where the message function 2/2+1 and the update function 61+1 MLPs. We "Virtual Node" which is connected to all other nodes to enable long range communication. Out readout function is an MLP. No spatial features are used.
15

---

Published as a conference paper at ICLR 2022
Layer Node Latent MAD
Per Layer Node Labent MAD
0.27
0.0047
0.40
0.18
0.015
0.10
0. 36
0.015
0.26
0.036
8.
0.098
0.30
0.27
0.05
0.30
Layerr
0.16
0.05
Layers
0.34
0.096
0.20
0.20
34
0.24
0.11
041
0.10
0.10
0.18
0.4
0.43
0.4
0.41
0.4
0.45
0:45
Nolpy
Figure 6: GNS Unsorted MAD per Layer Averaged Over 3 Random Seeds. Evidence of oversmoothing clear. Model trained on QM9.
Figure 7: GNS Sorted MAD Layer Av- eraged Over Random Seeds. The trend is clearer when the MAD values have been sorted. Model trained on QM9,
A.4 EXPERIMENT SETUP FOR 3D MOLECULAR MODELING
Open Catalyst. All training experiments were ran on cluster of TPU devices. For the Open Catalyst experiments, each individual run (i.c. single random seed) utilised TPU devices on 2 hosts (4 per host) for training, and 4 V100 GPU devices for evaluation (1 per dataset).
Each Open Catalyst experiment was ran until convergence for up to 200 hours. Our best result, the large 100 layer model requires days of training using the above setting. Each configuration was run at least 3 times in this hardware configuration, including all ablation settings.
We further note that making effective use of our regulariser requires sweeping noise values. These sweeps are dataset dependent and can be carried out using few message passing steps.
QM9. Experiments were also run TPU devices. Each seed was run using TPU devices on single host for training, and V100 GPU devices for evaluation. QM9 targets were trained between 12-24 hours per experiment.
Following Klicpera et al. (2020)) we define std. MAE as
M
| (m)
X;, 2i) - 15 7m
std. MAE
M
N
m
m=1
1=1
and logMAE as:
((m)
(X,, zi) -P/m0)
logMAE
log
M
N
om
with target index number of targets M 12, dataset size N. ground truth values f(m)) model p(m)) inputs X; and 2;, and standard deviation on of f(m).
A.5 OVER SMOOTHING ANALYSIS FOR GNS
In addition to Figure we repeat the analysis with a mean MAD over 3 seeds 7 Furthermore we remove the sorting layer by MAD value and find the trend holds.
A.6 NOISE ABLATIONS FOR OGBG-MOLPCB.
We conduct noise ablation the random flipping noise for OGBG-MOLPCBA with an 8 layer MPNN + Virtual Node, and find that our model is not very sensitive to the noise value (Table IO), but degrades from O.I.
16

---

Published as a conference paper at ICLR 2022
Flip Probability
Mean AP
0.01
27.8% +- 0.002
0.03
27.9% 0.003
0.05
28.1% +- 0.001
0.1
28.0% +- 0.003
0.2
27.7% + 0.002
Table 10: OGBG-MOLPCBA Noise Ablation
Mean AP
MPNN Without DropEdge MPNN With DropEdge
27.4% + 0.002
27.5% + 0.001
MPNN + DropEdge Noisy Nodes
27.8% + 0.002
Table 11: OGBG-MOLPCBA DropEdge Ablation
A.7 DROPEDGE & DROPNODE ABLATIONS FOR MOLG-MA
We conduct an ablation with our layer MPNN using DropEdge at rate of 0.1 an alternative approach to improving oversmoothing and find it does not improve performance for ogbg-molpcha (Table [T), similarly we find DropNode (Table 12 does not improve performance. In addition, we find that these two methods can't 't be combined well together, reaching a performance of 27.0% + 0.003. However, both methods can combined advantageously with Noisy Nodes.
We also measure the MAD of the node latents for each layer and find the indeed Noisy Nodes is more effective addressing oversmoothing in Figure
A.8 TRAINING CURVES FOR OC20 NOISY NODES ABLATIONS DEMONSTRATING OVERFITIIGG
Figure
Mean AP
MPNN With DropNode MPNN Without DropNode
27.5% + 0.001 27.5% + 0.004 28.2% + 0.005
MPNN + DropNode + Noisy Nodes
Table 12: OGBG-MOLPCBA DropNode Ablation
17

---

Published as a conference paper at ICLR 2022
Per Layer Node Latent MAD
15
0.56
0.073
0.045
14
0.14
0.06
0.036
13
0.17
0.2
0.016
0.8
12
0.43
0.05
0.054
11
0.7
0.062
0.074
10
0.81
0.32
0.11
0.6
9
0.45
0.032
0.12
Layer
8
0.67
0.46
0.015
7
0.57
0.5
0.2
6
0.66
0.66
0.6
0.4
5
0.67
0.67
0.76
4
0.62
0.79
0.83
3
0.59
0.81
0.84
0.2
2
0.76
0.74
0.87
1
0.67
0.7
0.73
0.85
0.85
0.92
Noisy Nodes
DropEdge
DropNode
Figure 8: Comparison of the effect of techniques to address oversmoothing MPNNs. Whilst Some effect can be seen from DropEdge and DropNode, Noisy Nodes is significantly better at preserving per node diversity.
A.9 PSEUDOCODE FOR 3D MOLECULAR PREDICTION TRAINING STEP
Algorithm 1: Noisy Nodes Training Step
G= (V, E,g) // Input graph
G == G// Initialize noisy graph
A // Noisy Nodes Weight
not provided(V") then V' +
end
predict differences then A= fot 0 E 1,..../VVI) end
for each E 1, .,.., do do sample_node_nneee [egee//ood___osehhpp__offff)))
0. = 1/4 + a3; if predict difieet-dffeeeeee then
Á A 03 end endfor
E recompute_ pe G' GNN(G);
if predict_differeeces differences then
V' = Å end
Loss = X No V') + PrimaryLoss((", V*)); Loss.minimise()
18

---

Published as a conference paper at ICLR 2022
Shared Layer Weights
Shared Layer Weights With Node Aux. Task
10-1
MAE
eV MAE
10
10 MP
50 MP
10
10 MP
50 MP
Training Step
Training Step
Unshared Weights Noisy Nodes
Message Passing Steps Noisy Nodes
Shared Layer Weights Layer Weights
10 MP
50 MP
10-1
10-1
eV MAE
MAE
10a2
10aa
0.00
0.25
0.50
0.75
1.00
1.25
150
1.75
2.00
Training Step
1n6
Training Step
166
Figure Training curves to accompany Figure 3 This demonstrates that even as the validation performance is getting worse, training loss is going down, indicating overfitting.
19

---

Published as a conference paper at ICLR 2022
Table 13: Open Catalyst training parameters.
Parameter
Value or description
Optimiser
Adam with warm up and cosine cycling
11
0.9
82
0.95
Warm up steps
5e5
Warm up start learning rate
le
Warm up/cosine max learning rate
le 4
Cosine cycle length
5e6
Loss type
Mean squared emror
Batch size
Dynamic max edge/node/graph count
Max nodes in batch
1024
Max edges in batch
12800
Max graphs in batch
10
MLP number of layers
3
MLP hidden sizes
512
Number Bessel Functions
512
Activation
shifted softplus
message passing layers
50
Group size
10
Node/Edge latent vector sizes
512
Position noise
Gaussian (μ = 0, o 0.3)
Parameter update
Exponentially moving average (EMA) smoothing
EMA decay
0,9999
Position Loss Co-efficient
1.0
A.10 TRAINING DETAILS
Our code base implemented in JAX using Haiku and Jraph for GNNs, and Optax for training (Bradbury et al., 2018 Babuschkin et al. 2020 Godwin et al. 2020 Hennigan et al. 2020). Model selection used early stopping.
All results reported as an average of 10 random seeds. OGBG-PCQM4M & OGBG-MOLPCBA were trained with 16 TPUs and evaluated with single V100 GPU. OGBN-Arxiv was trained and evalated with a single TPU
3D Molecular Prediction
We minimise the mean squared eтror loss on mean and standard deviation normalised targets and use the Adam (Kingma & Ba 2015)) optimiser with warmup and cosine decay. For OC20 IS2RE energy prediction we subtract learned reference energy, computed using an MLP with atom types input.
For the GNS model the node and edge latents well MLP hidden layers were sized 512, with layers per MLP and using shifted softplus activations throughout. OC20 & QM9 Models were trained on 8 TPU devices and evaluated on single V100 GPUs. We provide the full set of hyper-parameters and computational resources used separately for each dataset in the Appendix. All noise levels were determined by sweeping a small range of values (ze 10) informed by the noised feature covariance.
Non Spatial Tasks
A.11 HYPER-PARAMETEE
Open Catalyst. We list the hyper-parameters used train the default Open Catalyst experiment. not specified otherwise (e.g. in ablations of these parameters), experiments were ran with this configuration.
20

---

Published as a conference paper at ICLR 2022
Table 14: QM9 training parameters.
Parameter
Value or description
Optimiser
Adam with warm up and cosine cycling
11
0.9
82
0.95
Warm up steps
le4
Warm up start learning rate
3e -
Warm up/cosine max learning rate
le 4
Cosine cycle length
2e6
Loss type
Mean squared emror
Batch size
Dynamic max edge/node/graph count
Max nodes in batch
256
Max edges in batch
4096
Max graphs in batch
8
MLP number of layers
3
MLP hidden sizes
1024
Number Bessel Funtions
512
Activation
shifted softplus
message passing layers
10
Group Size
10
Node/Edge latent vector sizes
512
Position noise
Gaussian (μ = 0, a 0.02)
Parameter update
Exponentially moving average (EMA) smoothing
EMA decay
0.9999
Position Loss Coefficient
0.1
Dynamic batch sizes refers to constructing batches by specifying maximum node, edge and graph counts (as opposed to only graph counts) to better balance computational load. Batches constructed until one of the limits is reached.
smoothed using an EMA for the current training step with the current decay value computed through decay = min (decay, (1.0 + step)/(10.)) + step). As discussed in the evaluation, best results on Open Catalyst were obtained by utilising 100 layer network with group size 10.
Parameter updates
QM9 Table 14 lists QM9 hyper-parameters which primarily reflect the smaller dataset and geometries with fewer long range interactions. For Ug. U, H and G we use slightly larger number of graphs per batch 16 and smaller position loss co-efficient of 0.01.
OGBG-PCQM4M Table 15 provides the hyper parameters for OGBG-PCQMAM.
OGBG-MOLPCBA Table provides the hyper parameters for the OGBG-MOLPCBA experiments.
OGBN-ARXIV Table 17 provides the hyper parameters for the OGBN-Arxiv experiment.
21

---

Published as a conference paper at ICLR 2022
Table 15: OGBG-PCQM4M Training Parameters.
Parameter
Value or description
Optimiser
Adam with warm and cosine eycling
31
0.9
|8
0.95
Warm up steps
5e4
Warm up start learning rate
-e-5 5
Warm up/cosine max learning rate
le 1e-4
Cosine cycle length
5e5
Loss type
Mean absolute error
Reconstruction type
Softmax Cross Entropy
Batch size
Dynamic max edge/node/graph count
Max nodes in batch
20,480
Max edges batch
8,192
Max graphs in batch
512
MLP number of layers
2
MLP hidden sizes Activation
512
relu
Node/Edge latent vector sizes
512
Noisy Nodes Category Flip Fate
0.05
Parameter update
Exponentially moving average (EMA) smoothing
EMA decay
0.999
Reconstruction Loss Coefficient
0.1
Table 16: OGBG-MOLPCBA Training Parameters.
Parameter
Value or description
Optimiser
Adam with warm up and cosine cycling
81
0.9
82
0.95
Warm up steps
1e4
Warm up start learning rate
1e ---5 5
Warm up/cosine max learning rate
le-4
Cosine cycle length
le5
Loss type Reconstruction loss type
Softmax Cross Entropy
Softmax Cross Entropy
Batch size
Dynamic to max edge/node/graph count
Max nodes in batch
20,480
Max edges in batch
8,192
Max graphs in batch
512
MLP number of layers
2
MLP hidden sizes
512
Activation
relu
Batch Normalization
Yes, after every hidden layer
Node/Edge latent vector sizes
512
Dropnode Rate
0.1
Dropout Rate
0.1
Noisy Nodes Category Flip Fate
0.05
Parameter update
Exponentially moving average (EMA) smoothing
EMA decay
0,999
Reconstruction Loss Coefficient
0.1
22

---

Published as a conference paper at ICLR 2022
Table 17: OGBG-ARXIV Training Parameters.
Parameter
Value or description
Optimiser
Adam with warm up and cosine cycling
81
0.9
82
0.95
Warm up steps
50
Warm up start learning rate
1e-5
Warm up/cosine max learning rate
1e-3
Cosine cycle length
12. 000
Loss type
Softmax Cross Entropy
Reconstruction loss type
Mean Squared Error
Batch size
Full graph
MLP number of layers
Activation
relu
Batch Normalization
Yes, after every hidden layer 256
Node/Edge latent vector sizes
Dropout Rate
0.5
Noisy Nodes Input Dropout
0.05
Reconstruction Loss Coefficient
0.1
23


Reviews:
[{"review_id": "pSYfD6YOkRh", "reviewer": "Reviewer_yQtf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The NN method shows improved performance on a wide variety of tasks for different GNN architectures. The paper includes experiments with both graph level prediction tasks as well as node level prediction tasks and NN is shown to improve for both kinds of tasks.\n2. It has generally been hard to train very deep GNNs for many problems because of oversmoothing problem. The NN method is shown to work well with upto a depth of 50 in the paper.\n3. The paper clearly demonstrates that a model trained with NN is able to overcome oversmoothing by showing the MAD statistic at each layer (fig 2).\n\nWeaknesses\n1. NN adds additional hyper parameters like noise standard deviation and weight of the auxiliary loss. From reading the paper, it is unclear how sensitive the final performance is to these hyper-params. It would be good to shed some light on this because the practical utility of a method like this depends on the sensitivity to such hyper-params.\n2. In table 7, the 16 layer model is only shown with NN. Please add a column to show performance without NN.\n3. While the current results are very impressive, some of the best models for molecular prediction use higher order terms (e.g. Senet, Dimenet++, Gemnet). It is unclear if the NN method can work with such models.\n\nA few more things that could be added to the paper to improve it:\n1. The paper shows benefits of scaling the depth up to 50 layers. It would be good to show what happens beyond this depth. For example, does performance keep improving and plateau at some point, or does it deteriorate beyond a limit?\n2. NN can be seen as both a regularization method (which is most helpful in the small data setting), and a way to reduce oversmoothing to enable training very deep models (which is most helpful in the large data setting). It is, therefore, unclear how the performance of NN would vary with the size of a dataset. It would be good if the authors can shed some light on this.\n", "summary_of_the_paper": "The paper presents a simple regularization method based on denoising, called Noisy Nodes (NN) for Graph Neural Networks to reduce oversmoothing. The NN method adds a small noise to the input node representation (and optionally edge representations) during training. An auxiliary loss is added for predicting the uncorrupted node representation. The paper presents experiments using this method for a few popular GNNs (GNS, GCN etc) on different datasets and show improved performance, especially for very deep GNNs.\n", "summary_of_the_review": "The paper presents a simple new method for improving GNNs. The proposed method is based on existing ideas used in other domains, but their application to GNNs is novel. Through extensive experiments, this method was shown to benefit a variety of GNN architectures across different datasets and tasks. The simplicity and generality of this method is likely to have a big impact on the field of GNNs, which prompts me to lean towards accepting the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "i3zX8TS3FZ", "reviewer": "Reviewer_Y3os", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The simplicity of the proposal.  \n- Solid empirical results.\n\n**Weaknesses**\n- It is not entirely clear where the gain is coming from (W1);\n- No comparison against other ways to achieve deep GNNs, i.e., ways to tackle over-smoothing (W2);\n- Limited technical novelty with non-concrete claims (W3);\n- Experimental setup: not clear why the authors focus on molecules (W4).\n\n**Detailed comments**\n- (W1) The authors fail to show that GNS (same setup but without noisy nodes and additional loss) actually suffers from over-smoothing and overfitting (the paper's initial motivation). Figure 3 is not enough to demonstrate overfitting as it only exhibits validation curves and no training ones. Also, Figure 3A shows that the performance of 3MP and 50MP are very close --- can we say over-smoothing is an issue here? Additionally, it seems the authors already apply other regularization strategies (dropout, early stopping, etc.) --- aren't they enough to avoid overfitting? \n\n- (W2) Since \"Noisy Nodes\" is introduced as a new simple regularization technique, I believe it is crucial to assess how the proposal compares with alternatives to handle over-smoothing in GNNs, such as PairNorm [1], long-range residual connections [2], and DropEdge [3].\n\n- (W3) The authors provide little support to their claims in Section 4, which are supposed to explain why the method works. For instance, the authors say that \"the GNN learns to go from low probability graphs to high probability graphs\". However, it is hard to guarantee that the noisy graphs lie in low-density regions, especially if we consider high-dimensional spaces.  \n\n- (W4) Is there any aspect of the method that makes it particularly tailored to molecular data or regression tasks? The gains on Arxiv seem to be less significant compared to those in other datasets.\n\n**Additional observations**\n* Noise has also been used to increase the expressive power of GNNs [4]. It should probably be mentioned in the paper. \n* Is there any particular reason for picking base GNNs with virtual nodes in Section 6? \n* It is weird to see tables full of numerical results with 0 std (Tables 1 and 3). \n* Typos: \"..adversarial noise during to input..\" (page 2); \" range of of \" (page 4); \"which report which report\" (page 9). \n* Reference Zonghan Wu et al. is duplicated.\n\n[1] Lingxiao Zhao, Leman Akoglu. PairNorm: Tackling Oversmoothing in GNNs, ICLR, 2020.\n\n[2] Ming Chen et al., Simple and Deep Graph Convolutional Networks. ICML, 2020.\n\n[3] Yu Rong et al. Dropedge: Towards deep graph convolutional networks on node classification. ICLR, 2020.\n\n[4] Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. arXiv preprint, 2020.\n\n", "summary_of_the_paper": "The paper proposes a new regularization method for tackling both over-smoothing and overfitting in GNNs. The main idea is to add node-level noise to input graphs and augment the loss function with a denoising term. The authors run experiments on three benchmarks for 3D molecular property prediction and three datasets (with no 3D features) from Open Graph Benchmark. Overall,  GNS (Graph Network Simulator) and vanilla GCNs are used as base GNNs. The proposed method outperforms the existing approaches. \n", "summary_of_the_review": "Overall, I like the simplicity of the proposal and the fact the paper tackles relevant issues in GNNs. Also, the paper shows good empirical results. However, my main concerns revolve around limited technical novelty, lack of experiments to validate some intuitions behind the method, and comparison against related methods. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vv7sjFAmTKW", "reviewer": "Reviewer_7LGq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrength:\n- The method is pretty simple, but the results are very positive on several benchmarks, which has achieved improvements over several baselines. \n\nWeakness:\n\nThe writing needs to be improved, especially for the method section and the experiment section. I find many important details are missing. For example. How is the noise-enhanced auxiliary loss incorporated into the 3D prediction task? How are the two aspects balanced?\n\nThe claimed relative improvements need to be broken down in Table 1-4. For\nexample, in table 1, there are many different factors that may contribute to\nthe claimed improvements over existing baselines: is the improvement due to\nthe advantage of GNS over baselines, or the changes made to GNS, or the core\nidea of noise injection?\n\nSome important baselines are missing. Two categories of baselines need to be\nadded: (1) There are several existing works on injecting adversarial losses\ninto graph neural networks, which can be also combined with GNS for 3d\nmolecular prediction. How does this proposed method compare with those\nexisting methods? (2) For 3D molecular modeling, there are other more powerful\nequivariant models, such as Tensor Field Network, Se(3)-Transformers, GemNet.\nWithout including those baselines, it is hard to conclude how this method\nadvances the SOTA for 3d molecular property prediction.\n\nIt's better to perform sensitivity analysis to the injected noise. How are the\nnoises chosen for different datasets, and how sensitive is the model to such\nnoise?\n\n", "summary_of_the_paper": "This paper proposes a noise-injected training mechanism for graph neural network. The idea is to perturb the node attributes and add an auxiliary loss for the noise-correction task. The authors. Implementing this strategy for the GNS model And observed improvements on several 3D molecular prediction tasks. \n", "summary_of_the_review": "The proposed idea is intuitive and sound, and the results are positive. However,  important baselines and detailed experiments needs to be added to support the claims and justify the contributions. The writing also needs to be improved.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GpB_8659suZ", "reviewer": "Reviewer_kA11", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Pros:\n\n1. The node noise method is simple and easy-to-use with significant improvement on 3D molecular prediction tasks and non-spatail tasks, which may make it a standard trick in the future.\n2. The abalation study is comprehensive.\n\n* Concerns:\n\n1. I agree that it would be interesting to incorporate the methods into small data regime.\n2. Such an autoencoder structure may guide the learning process in GNNs into some bias. For example, it would be intriguing to see how to use it to analyze whether GNNs can keep focusing on local information after stacking many layers. (This question is a little bit derivating from the original motivation of this paper.)", "summary_of_the_paper": "To overcome oversmoothing in GNNs, this paper proposes node noise with noise-correction training, similar to denoising autoencoder, to make node representation distinguishable. On 3D molecular property prediction tasks, it improves GNS significantly, achieving state-of-the-art. It also improves GCNs on non-spatial tasks.", "summary_of_the_review": "I recommend to accept it for its straightforward intuition, simple methods and strong experiment results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline paper. The most enthusiastic reviewer does not have much confidence in the score. The other reviewers think the paper has some value after the rebuttal, but also feel there is little technical novelty. The proposed applications of the approach are interesting.

After reading the reviews, rebuttal, and the paper, I agree that there is little technical novelty. The idea of adding node-label noise to a GNN to improve GNN expressiveness dates back to (Murphy et al., 2019) and has been also explored by (Dasoulas et al., 2019), (Vignac et al., 2020), (Loukas, 2020) among others [one of which is suggested by a reviewer] (this literature is entirely missing from the paper). The paper has some novelty in proposing a regularization method for tackling the node-level noise by augmenting the loss function with a denoising term. The oversmoothing justification is not properly investigated (whether the proposed solution really solves the issue in practice).

If there is space in the borderline decision boundary, this paper could be a worthwhile inclusion.

Dasoulas, G., Santos, L.D., Scaman, K. and Virmaux, A., 2019. Coloring graph neural networks for node disambiguation. arXiv preprint arXiv:1912.06058.
Loukas, A., 2020. How hard is to distinguish graphs with graph neural networks?. arXiv preprint arXiv:2005.06649.
Vignac, C., Loukas, A. and Frossard, P., 2020. Building powerful and equivariant graph neural networks with structural message-passing. arXiv preprint arXiv:2006.15107.
Murphy, R., Srinivasan, B., Rao, V. and Ribeiro, B., 2019, May. Relational pooling for graph representations. In International Conference on Machine Learning (pp. 4663-4673). PMLR.
Decision: Accept (Poster)