Pre-training via Denoising for Molecular Property Prediction
Authors: Sheheryar Zaidi
Michael Schaarschmidt
James Martens
Hyunjik Kim
Yee Whye Teh
Alvaro Sanchez-Gonzalez
Peter Battaglia
Razvan Pascanu
Jonathan Godwin
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Molecular Property Prediction
Pre-training
Graph Neural Networks
Denoising
Molecules

Abstract: Many important problems involving molecular property prediction from 3D structures have limited data, posing a generalization challenge for neural networks. In this paper, we describe a pre-training technique based on denoising that achieves a new state-of-the-art in molecular property prediction by utilizing large datasets of 3D molecular structures at equilibrium to learn meaningful representations for downstream tasks. Relying on the well-known link between denoising autoencoders and score-matching, we show that the denoising objective corresponds to learning a molecular force field -- arising from approximating the Boltzmann distribution with a mixture of Gaussians -- directly from equilibrium structures. Our experiments demonstrate that using this pre-training objective significantly improves performance on multiple benchmarks, achieving a new state-of-the-art on the majority of targets in the widely used QM9 dataset. Our analysis then provides practical insights into the effects of different factors -- dataset sizes, model size and architecture, and the choice of upstream and downstream datasets -- on pre-training.

Published as a conference paper at ICLR 2023
PRE-TRAINING VIA DENOISING FOR MOLECULAR PROPERTY PREDICTION
Sheheryar f Michael Schaarschmidt"
James Martenst Hyunjik Kim²
Yee Whye Teh Alvaro Sanchez-Gonzalezz
Peter Battaglia* Razvan Pascanu Jonathan Godwin
'University Oxford, ' 'DeepMind
"Equal contribution. Work done during internship DeepMind.
Correspondence to: shehzaide di @deepminds com, mschaarsschn..dtgooo....c.
ABSTRACT
Many important problems involving molecular property prediction from 3D struc- tures have limited data, posing a generalization challenge for neural networks. In this paper, we describe a pre-training technique based on denoising that achieves a new state-of-the-art in molecular property prediction by utilizing large datasets 3D molccular structures at equilibrium to learn meaningful representations for downstream tasks. Relying on the well-known link between denoising autoen- coders and score-matching, we show that the denoising objective corresponds learning molecular force field arising from approximating the Boltzmann distribution with a mixture of Gaussians directly from equilibrium structures. Our experiments demonstrate that using this pre-training objective significantly improves performance on multiple benchmarks, achieving a new state-of-the-art on the majority of targets in the widely used QM9 dataset. Our analysis then provides practical insights into the effects of different factors dataset sizes, model size and architecture, and the choice of upstream and downstream datasets on pre-training.
INTRODUCTION
The success of the best performing neural networks in vision and natural language processing (NLP) relies on pre-training the models on large datasets to learn meaningful features for downstream tasks
(Dai & Le 2015 Simonyan & Zisserman 2014 Devlin et al.) 2018 Brown et al. 2020 Dosovitskiy 2020). For molecular property prediction from 3D structures (a point cloud of atomic nuclei in RR), the problem of how to similarly learn such representations remains open. For example, none of the best models on the widely used QM9 benchmark use any form of pre-training (e.g. Klicpera et al.) 2020a Liu et al. 2022b Schüit al.) [0211] Thölke & De Fabritiis. 2022), in stark contrast with vision and NLP. Effective methods for pre- pre-trrining could have a significant impact on fields such as drug discovery and material science.
al.
In this work, we focus on the problem of how large datasets of 3D molecular structures can be utilized to improve performance on downstream molecular property prediction tasks that also rely 3I) structures as input. We address the question: how can one exploit large datasets like PCQM4Mx2 that contain over million structures, improve performance on datasets such as DES15K that orders of magnitude smaller? Our answer is form of self-supervised pre-training that generates useful representations for downstream prediction tasks, leading to sfate-of-the-ee (SOTA) results.
Inspired by recent advances in noise regularization for graph neural networks (GNNs) (Godwin al. 2022). our pre-training objective is based on denoising the space of structures (and hence self-supervised). Unlike existing pre- pre-training methods, which largely focus on 2D graphs, our approach targets the setting where the downstream task involves 3DD point clouds defining the molecular structure. Relying on the well-known connection between denoising and score-matching (Vincent, 2011 Song & Ermon 2019 Ho et al. 2020), we show that the denoising objective is equivalent to learning particular force field, adding a new interpretation of denoising in the context of molecules and shedding light on how aids representation learning.
Note that PCQM4Mv2 is new version of PCQM4M that now offers 3D structures.

---

Published as a conference paper at ICLR 2023
Pre-training via Denolsing
30
Upstream
E(n)-GNN
Graph Neural
Network
28
PaiNN LeConn-SE(3)
Test MAE (HOMO)
26
to atom
Predict noise
24
SphereNet
DimeNet++
22
Graph Neural Network
HOBD
TDrch-MD-Net (SOTAA)
Downstream
Pre-trained GNS-TAT
Graph Neural Network
Iateraction coergy
14
200
400
600
800
1000
Gradient step (thousands)
Figure I: GNS-TAT pre-trained via denoising on PCQM4Mv2
CCQM4MYY2uttperffmm outperforms prior work on QM9.
The contributions of work are summarized follows:
We investigate a simple and effective method for pre-training via denoising in the space of 3D structures with the aim of improving downstream molecular property prediction from such 3D> structures. Our denoising objective shown to related to learning a specific force field.
Our experiments demonstrate that pre-training via denoising significantly improves performance on multiple challenging datasets that vary in size, nature of task, and molecular composition. This establishes that denoising over structures successfully transfers to molecular property prediction, setting. in particular, a new state-of-the- on 10 out of 12 targets in the widely used QM9 dataset. Figure illlstratee performance on one of the targets in QM9.
We make improvements to a common GNN, in particular showing how to apply Tailored Activa- tion Transformation (TAT) (Zhang et al., 2022 Graph Network Simulators (GNS) (Sanchez, Gonzalez et al. 2020). which is complementary to pre-training and further boosts performance.
We analyze the benefits of pre-training by gaining insights into the effects of dataset size, model size and architecture, and the relationship between the upstream and downstream datasets.
RELATED WORK
Pre-training GNNs. Various recent works have formulated methods for pre-training using graph data (Liu et al. 2021b Hu et al. 2020a Xie et al. 2021 Kipf & [eeling|[000 rather than 3D point clouds of atom nuclei as this paper. Approaches based on contrastive methods rely on learning representations by contrasting different views of the input graph (Sun et al. 2019: Veličković et al. 2019: You et al. 2020 Liu et al. 2021a), or bootstrapping (Thakoor et al. 2021). Autoregressive or
reconstruction- basertuctcoo--bb-dddddeed approaches, such as ours. learn representations by requiring the model to predict
aspects of the input graph (Hu et al. 2020a by Rong et al. 2020 Liu et al. 2019). Most methods in the current literature graphs. The closest work to ours is GraphMVP (Liu et al.) (001a)), where 3D structure is treated as one view of 2D molecule for the purpose of upstream contrastive learning. Their work focuses on downstream tasks that only involve 2D information, while our aim is to improve downstream models for molecular property prediction from 3D structures. After the release of this pre-print, similar ideas have been studied by Jiao et al. 2022) and Liu et al. 2022a)
not designed handle 3D structural information, focusing instead 2D)
Denoising, representation learning and score-matching. Noise has long been known improve generalization in machine learning (Sictsma & Dow 1991; Bishop 1995). Denoising autoencoders have been used effectively learn representations by mapping corrupted inputs to original inputs (Vincent et al. 2008 2010). Specific to GNNs (Battaglia et al. 2018; (Scarselli et al.) 2009 Bronstein ct 2017), randomizing input graph features has been shown to improve performance (Hu et al. 2020a Sato et al. 2021). Applications physical simulation also involve corrupting the state with Gaussian noise (Sanchez- Gonzalezz et al. 2018 2020 Pfaff et al., 2020). Our work builds on Noisy auxiliary task to improve perfor- mance, indicating the effectiveness of denoising for molecular property prediction (ef. Section (..22).
Nodes (Godwin et al. (2022), which incorporates denoising
Denoising is also closely connected to score-matching (Vincent) 2011), which has become popular
2

---

Published as a conference paper at ICLR 2023
2020 Hoogeboom et al. 2022 Xu et al. 2022 Shi et al. 2021). We also rely on this connection to show that denoising structures corresponds to learning force field.
for generative modelling (Song & Ermon 2019:
2020: Ho et al.
Equivariant neural networks for 3D molecular property prediction. Recently, the dominant approach for improving models for molecular property prediction from 3D structures has been through the design of architectures that incorporate -ot--translaiional inductive biases into the model, such that the outputs are invariant translating and rotating the input atomic positions. A simple way to achieve this is to use roto-translation lation invariant features as inputs, such as inter-atomic distances (Schütt et al. 2017, Unke & Meuwly 2019), angles (Kliepera al. 2020b a Shuaibi et al. 2021 Liu al. 2022b). or the principal axes of inertia (Godwin et al. 2022). There is also broad literature on equivariant neural networks, whose intermediate activations transform accordingly with roto-translations of inputs thereby naturally preserving inter-atomic distance and orientation information. Such models can be broadly categorized into those that are specifically designed for molecular property prediction
(Thölke & De Fabritis 2022, Schüit et al. 2021 Batzner al. [021 Anderson et al. 2019 Miller
ctt al.
2020) and gencral-purpose architectures (Satorras et al. 2021] Finzi et al. 2020 Hutchinson
et al.
2021 Thomas al. 2018 Kondor et al., 2018: Brandstetter al. 2021). Our pre-training
technique is architecture-agnostie and we show that it can be applied to enhance performance in both
GNN-based architecture (Sanchez-Gonzale et al. 2020) and a Transformer-based one (Thölke &
De Fabritiis 2022). We conjecture that similar improvements will hold for other models.
3
METHODOLOGY
3.1 PROBLEM SETUP
Molecular property prediction consists of predicting scalar quantities given the structure of one or more molecules input. Each data example is a labelled specified as follows: are provided with a set of atoms S = pi),,,i...... (a|s)- P(s))) }. where a; E (1,.... 118) and p: E TR3 are the atomic number and 3D position respectively of atom in the molecule, alongside label y IR We assume that the model, which takes S as input, is any architecture consisting of backbone, which first processes 5 to build latent representation of it, followed by vertex-level or graph-level "decoder", that returns per-vertex predictions or single prediction for the input respectively.
3.2 PRE-TNAINING VIA DENOISING
Given a dataset of molecular structures, we pre-train the network by denoising the structures, which operates as follows. Let Datructures = {S Sn} denote the upstream dataset of equilibrium structures, and let GNN, denote a graph neural network with parameters 0 which takes S E D struetures
as input and returns per-vertex predictions GNN,(S) (ê1... (é...... Es)) The precise parameterization of the models we consider in this work is described in Section 3.3 and Appendix
Starting with an input molecule S E Sstuuttrrss we perturb by adding i.idd Gaussian noise to its atomic positions p:- That is, we create noisy version of the molecule:
$ {(11 p.),
p(ss)) where p. P: de; and e; Is),
The noise scale a is a tuneable hyperparameter (an interpretation of which is given in Section I.2.1p. We train the model as a denoising autoencoder by minimizing the following loss with respect 0:
P3(3,5) [|
[NNNO(5) (E1 (E1.-... G(S)
The distribution p(5. 5) corresponds to sampling a structure S from Danucures and adding noise to according to Equation (I). Note that the model predicts the noise, not the original coordinates. Next, we motivate denoising as our pre-training objective for molecular modelling.
3.2.1 DENOISING AS LEARNING FORCE FIELD
Datasets in quantum chemistry are typically generated by minimizing expensive-to-compuue inter- atomic forces with methods such density functional theory (DFT) (Parr & Weitao 1994). We speculate that learning this force field would give rise to useful representations for downstream tasks, since molecular properties vary with forces and energy. Therefore, reasonable pre-training objective
3

---

Published as a conference paper at ICLR 2023
would be one that involves learning the force field. Unfortunately, this force field is either unknown or expensive to evaluate, and hence it cannot be used directly for pre-training. An alternative is to approximate the data-generating force field with one that can be cheaply evaluated and use it to learn good representations an approach we outline in this section. Using the well well-known link between denoising autoencoders and score-matching (Vincent, 2011 Song & Ermon 2019 2020), we can show that the denoising objective in Equation (2) is equivalent to learning a particular force field directly from equilibrium structures with some desirable properties. For clarity, in this subsection we condition on and suppress the atom types and molecule size in our notation, specifying a molecular structure by its coordinates x E R3N (with N as the size of the molecule).
From the perspective of statistical physics, a structure x can be treated as random quantity sampled from the Boltzmann distribution Px) xx exp(-F((x)) where F(xx) is the (potential) energy of x. According to Pphysical, low energy structures have a high probability of occurring. Moreover, the per-atom forces are given by Vx Pphysical (x)
F())x, which is referred to as the force field. Our goal is to learn this force field. However both the energy function E' and distribution Pphysical are unknown, and we only have access to a set of equilibrium structures X1, : Xn that locally minimize the energy E. Since xi...., Xn are then local maxima of the distribution Pphysical our main approximation is to replace Pphysical with a mixture of Gaussians centered at the data:
Pphysical (x) 4o (x)
Σ do (x xx),
Xx r² IIN). This approximation captures the fact that Pphysical will have local maxima at the equilibrium structures, vary smoothly with x and is computationally convenient. Learning the force field corresponding to g- (x) now yields a score-matching objective:
where we define 4o [x xx)
E (8)
[|) - Vx lo 1o (x)]
(3)
As shown by Vincent 201 D. and recently applied to generative modelling (Song & Ermon 2019 2020 Ho et al. 2020 Shi et al. 2021 et al. 2022). this objective is equivalent to the denoising - 0(x = x,) to be the empirical distribution and 4o (xx = (x x))0 (x), the objective in Equation (3) is equivalent to:
objective. Specifically, detining qo (x)
27
(x.x)
[|IGNNo(*)
V log (x x)]]" = (x
GNNN(X))
x
x
We notice that the RHS corresponds to the earlier denoising loss in Equation ( (up to a constant factor of 1/a applied to GNN, that can be absorbed into the network). To summarize, denoising equilibrium structures corresponds to learning the force field that arises from approximating the distribution Pphysical with a mixture of Gaussians. Note that we can interpret the noise scale a as being related to the sharpness of Pphysical E around the local maxima X,. We also remark that the equivalence between Equation (3) and the LHS of Equation (4) does not require qo (x x,)
to be a Gaussian distribution (Vincent 2011), and other choices will lead to different denoising objectives, which we leave as future work. See Appendix B for technical caveats and a discussion of the differences between denoising for generative modelling versus learning forces.
3.2.2 NOISY NODES: DENOISING AS AN AUXILIARY LOss
Recently, Godwin et al. (2022) also applied denoising as an auxiliary loss to molecular property prediction, achieving significant improvements a variety of molecular datasets. In particular, their approach, called Noisy Nodes, consisted of augmenting the usual optimization objective for predicting y with auxiliary denoising loss. They suggested two explanations for why Noisy Nodes improves performance. First, the presence of a vertex-level loss discourages oversmoothing (Chen) et al., 2019: Cai & Wang 2020) of vertex/edge features after multiple pesssge-passing layers common problem plaguing GNNs because successful denoising requires diversity amongst vertex features in order to match the diversity in the noise targets E;, Second, they argued that denoising can aid representation learning by encouraging the network to learn aspects of the input distribution.
The empirical success of Noisy Nodes indicates that denoising can indeed result in meaningful repre- sentations. Since Noisy Nodes incorporates denoising only as an auxiliary task, the representation learning benefits of denoising are limited to the downstream dataset on which it is used an auxiliary task. Our approach is to apply denoising as pre-training objective on another large (unlabelled) dataset of structures to learn higher-quality representations, which results in better performance.
4

---

Published as a conference paper at ICLR 2023
QM9
PCOMAN2
10a
Parcentage elemerts
covered PCQM4MV2
80
PCQMAMv2
Friquency
10a
60-
OMB
40-
10
20-
109
Molecular composition (ordered by frequency QM9)
QM9 DES15K OC20
Dataset
Figure 2: Left: Frequency of compositions of molecules appearing in QM9 overlayed with the corresponding frequency PCQM4Mv2. Each bar represents one molecular composition (e.g. one carbon atom, two oxygen atoms). Right: Percentage of clements appearing in QM9, DESISK, OC20 that also appear in PCQM4Mv2.
3.3
GNS AND GNS-TAT
The main two models we consider in this work are Graph Net Simulator (GNS) Sanchez-Gonzalee al. 2020), which is a type GNN, and a perter-performing variant we contribute called GNS- TAT. GNS-TAT makes use of recently published network transformation method called Tailored Activation Transforms (TAT) (Zhang al. 2022), which has been shown to prevent certain degenerate behaviors at initialization in deep PlPs/conveets that are reminiscent of oversmoothing in GNNs (and are also associated with training difficulties). While GNS not by default compatible with the assumptions of TAT, we propose novel GNN initialization scheme called "Edge- Delta" that makes it compatible by initializing to zero the weights that carry "messages" from vertices to edges. This marks the first application of TAT to any applied problem in the literature. See Appendix A for details.
4 EXPERIMENTS
The goal of our experimental evaluation in this section is to answer the following questions. First, does pre-training a neural network via denoising improve performance on the downstream task compared to training from a random initialization? Second, how does the benefit of pre-training depend on the relationship between the upstream and downstream datasets? Our evaluation involves four realistic and challenging molecular datasets, which vary in size, compound compositions (organic or inorganic) and labelling methodology (DFT- or CCSS((D-geeeratee)) as described below.
4.1 DATASETS AND TRAINING SETUP
Datasets. First, the main dataset we use for pre-training is PCQM4My2 (Nakata & Shimazaki 2017), which contains 3.4 million organic molecules, specified by their 3D structures equilibrium The molecules in PCQM4Mv2 only contain one label, however the labels are not used as denoising only requires the structures. The large scale and diversity PCQM4Mv2 makes it well-suited for pre-training via denoising. Second, as a dataset for fine-tuning, we use QM9 (Ramakrishnan et al. 2014), which contains around 130,000 small organic molecules and is widely used as molecular property prediction benchmark (Klicpera et al. 2020a; Fuchs et al. 2020, Satooras al. 2021 Finzi al. 2020 Hutchinson et al.) 2021 Schütt et al., 2021 Thölke & De Fabritis, 2022 Godwin et al. 2022). Each molecule is specified by its structure alongside 12 associated molecular property labels. Third, Open Catalyst 2020 (OC20) (Chanussot* et al.
calculated using DFT The
2021) is recent large benchmark of interacting surfaces and adsorbates relevant to catalyst discovery. OC20 contains various tasks, such as predicting the relaxed state energy from an initial high-energy structure (IS2RE). We explore different combinations of upstream and downstream tasks as described in Section Lastly, DES15K (Donchev et al.] 2021) is a small dataset we use for fine-tuning, which contains around 15,000 dimer geometries (l.e. molecule pairs) with non-covalent molccular
2 An earlier version of this dataset without any 3D structures, called PCQM4M, was used for supervised pre-training (Ying et al. (021), but to our knowledge, this is the first time the 3D structures from v2 have been used and in self-supervised manner.
5

---

Published as a conference paper at ICLR 2023
Table I: Results on QM9 comparing the performance of GNS-TAT Noisy Nodes (NN) with and without pre-training on PCQM4My2 (averaged over three seeds) with other baselines.
Pre-trained CAS
GNS GNS-TAI
Target Unit
SchNet H(n)-GNN DimeNet++ SphereNet PaiNN TorchMID-NET
+ +NN + NN
NN
D 0.033
0.029
0.030
0.027
0.012
0,011
0.025
0.021
0.016
%g 0.235
0.071
0.043
0.047
0.045
0.059
0.052
0.047
0.040
FHOMO
meV
41.0
29.0
24.6
23.6
27.6
20.3
20.4
17.3
14.9
CLUMO
meV
34.0
20.4
18.6
17.5
17.1
14.7
meV
63.0
48.0
32.6
32.3
45.7
36.1
28.6
25.7
R
0.07
0.11
0.33
0.29
0.07
0.033
0.70
0.65
0.44
ZPVE
meV
1.700
1.550
1.210
1.120
1.280
1.840
1.160
1,080
1.018
Up
meV
14.00
11.00
6.32
6.26
5.85
6.15
7.30
6.39
5.76
U
mev
19.00
12.00
6.28
7.33
5,83
6.38
7.57
6.39
5.76
meV
14.00
12.00
6.53
6.40
5,98
6.16
7.43
6.42
5.79
meV 14.00
12.00
7.35
7.62
8.30
7.41
C
Bol 0.033
0.031
0.023
0.022
0.024
0.026
0.025
0.022
0.020
interactions. Each pair is labelled with its interaction energy computed using the gold-standard CCSD(T) method (Bartlett & Musial, (2077). CCSD(T) is usually both more expensive and accurate than DFT, which is used for all aforementioned datasets. See Appendix for further details and discussion about the choice of using DFT-generated structures for pre-training.
Figure 2 (right) shows what percentage of elements appearing in each of QM9, OC20 and DES15K also appear in PCQM4Mv2. Whereas QM9 fully covered by PCQM4Mv2, we observe that DESI5K has less element overlap with PCQM4Mv2 and less than < 30% of clements in OC20 contained in PCQM4Mv2. This is owing to the fact that surface molecules OC20 are inorganic lattices, none of which appear in PCQM4Mv2. This suggests that we expect least transfer from PCQM4Mv2 to OC20. We also compare PCQM4My2 and QM9 in terms of the molecular compositions, i.e. the number of atoms of each element, that appear in each. Due to presence of isomers, both datasets contain multiple molecules with the same composition. For each molecular composition in QM9, Figure 2 (left) shows its frequency in both QM9 and PCQM4Mv2. We observe that most molecular compositions in QM9 also appear in PCQM4Mv2. We also remark that since pre-training is self-super using only unlabelled structures,
contamination not possible-
in fact, PCQM4Mv2 does not have most of the labels in QM9.
Training setup. GNS/GNS- TAT were implemented in JAX (Bradbury al., 2018) using Haiku and Jraph (Hennigan et al. 2020 Godwin et al. (0220) All experiments were averaged over 3 seeds. Detailed hyperparameter and hardware settings can be found Appendices and
4.2 RESULTS ON QM9
We evaluate two variants of our model on QM9 in Table GNS-TAT with Noisy Nodes trained from a random initialization versus pre-trained parameters. Pre-training is done on PCQM4Mv2 via denoising. For best performance on QM9, we found that using atom type masking and prediction during pre-training additionally helped (Hu et al. 2020a). We fine-tune a separate model for each of the 12 targets, as usually done on QM9, using a single pre-trained model. This repeated for three seeds (including pre-training). Following customary practice, hyperparameters. including the noise scale for denoising during pre-training and fine-tuning, are tuned the HOMO target and then kept fixed for all other targets. We first observe that GNS-TAT with Noisy Nodes performs competitively with other models and significantly improves upon GNS with Noisy Nodes, revealing the benefit of the TAT modifications. Utilizing pre- training then further improves performance across all targets, achieving a new state-of-the-ee art compared to prior work for 10 out 12 targets. Interestingly, for the electronic spatial extent target (R²)). found GNS-TAT to perform worse than other models, which may be due to the optimal noise scale being different from that of other targets.
4.3 RESULTS ON OC20
Next, we consider the Open Catalyst 2020 benchmark focusing on the downstream task predicting the relaxed energy from the initial structure (IS2RE). We compared GNS with Noisy Nodes trained

---

Published as a conference paper at ICLR 2023
OC20
DES15K
0.65
Validation energy MAE
No PT or NN
0.60
0.55
PT
PT with AttrMask NN
Model
PT PCQM4MV2 + NN
0.50
NN
NN
PT on OC20 NN
0.45
with denoising + NN
0.40
100
300
500
0.0
0.2
0.4 Test MAE (kcal/mol)
0.6
0.8
Gradient step (thousands)
split) with GNS. See Table a comparison to other models in the literature. Right: Test performance curves for predicting interaction energies of dimer geometries in the DESI5K dataset with GNS-TAT. "PT" and "NN" stand for pre-traue pre-training and Noisy Nodes respectively.
Figure 3: Left: Validation performance curves on the OC20 IS2RE task
from scratch versus using pre-trained parameters. We experimented with two options for pre-training: (1) pre-training via denoising on PCQM4Mv2, and (2) pre-training via denoising on OC20 itself. For the latter, we follow Godwin et al.Js s [002]] approach of letting the denoising target be the relaxed structure, while the perturbed input a random interpolation between the initial and relaxed structures with added Gaussian noise this corresponds to the IS2RS task with additional noise. As shown in Figure 3 (left), pre-training on PCQM4Mv2 offers no benefit for validation performance on IS2RE, however pre-training on OC20 leads to considerably faster convergence but the same final performance. The lack of transfer from PCQM4Mv2 to OC20 is likely due the difference in nature of the two datasets and the small element overlap as discussed in Section [.1 and Figure P (right). On the other hand, faster convergence from using parameters pre-trained that denoising learned meaningful features. Unsurprisingly, the final performance is unchanged since the upstream and downstream datasets are the same this case, so pre- pre-training with denoising is identical to the auxiliary task of applying Noisy Nodes. The performance achieved is also competitive with other models the literature as shown Table 9
OC20 suggests
4.4 RESULTS DES15K
In our experiments so far, all downstream tasks were based on DFT-generater datasets. While DFT calculations are more expensive than using neural networks, they are relatively cheap compared even higher quality methods such as CCSD(T) (Bartlett & Musial 2007). In this section, evaluate how useful pre-training on DFF--generaed structures from PCQ PCQ24My2 is when fine-tuning on the recent DESISK dataset which contains higher quality CCSD(T)-generatte interaction energies. Moreover, unlike QM9, inputs from DESI DES15K are systems of two interacting molecules and the dataset contains only around 15,000 examples, rendering it more challenging. We compare the test performance on DESISK achieved by GNS-TAT with Noisy Nodes when trained from scratch versus using pre- trained parameters from PCQM4Mv2. As a baseline, we also include pre-training on PCQM4Mv2 using 2D-based AttrMask (Hu et al. 2020a) by masking and predicting atomic numbers. Figure 3 (right) shows that using Noisy Nodes significantly improves performance compared to training from scratch, with further improvement resulting from using pre-training via denoising. AttrMask underperforms denoising since it likely does not fully exploit the 3D structural information. Importantly, this shows that pre-training by denoising structures obtained through relatively cheap methods such as DFT can even be beneficial when fine-tuning on more expensive and smaller downstream datasets. See Appendix G.I for similar results another architecture.
5
ANALYSIS
5.1
PRE-TRAINING DIFFERENT ARCHITECTURE
To explore whether pre-training is beneficial beyond GNS/GNS-TAN TAT, applied pre-training via denoising to the TorchMD-NET architecture (Thölke & De Fabritiis 2022). TorchMD-NET is a
transformer-hased architecture whose layers maintain per-atom sealar features x; R. and vector
7

---

Published as a conference paper at ICLR 2023
Test MAE (HOMO)
100
NN
No PT NN
PCOM4MN2 NN
Test MAE (HOMO)
24
22
GNS + NN GNS-TAT + NN
Trajectory energy MAE
0.50
0.45
130M from scratch
PT on OC20- 53M
PT on OC20 11M
80
PT on OC20 130M
0.40
20
PT on OC20 1.28
60
0.35
18
40
166
14
5000
11355
25787
58564
113880
10000
32027
102577
328532
1052213
3378696
0.20
300 400
Training data size (QM9)
Pre-training data size (PCOMMM22
Gradient step (thousands)
Figure 4: Left: Impact of varying the downstream dataset size for the HOMO target in QM9 with GNS-TAT. Middle: Impact of varying the upstream dataset size for the HOMO target in QM9. Right: Validation performance curves on the OC20 S2EF task food_bote split) for different GNS model sizes, "PT" and "NN" stand for pre-training and Noisy Nodes respectively.
features v R3xF, where F is the feature dimension, that are updated in each layer using self- attention mechanism. We implemented denoising by using gated equivariant blocks (Weiler et al.) 2018; Schüitt et al., (02T)) applied to the processed scalar and vector features. The resulting vector features are then used as the noise prediction.
In T we evaluate the effect of adding Noisy Nodes and pre-training to the architecture on the HOMO and LUMO targets in QM9. Pre-training yields boost in performance, allowing the model to achieve SOTA results. Note that the results shown for TorchMD -EET are from runs us-
Table 2: Performance of TDrT with Noisy Nodes and pre-training on PCQM4My2.
Method
6HOMO
FLUMO
TorchMD-NET
22.0 = 0.6
18.7 : 0.4
Noisy Nodes
18.1 + 0.1
15.6 + 0.1
ing Thölke & De Fabritisss [2022] open-source code, which led to slightly worse results than their
+ Pre-training
15.6 = 0.1
13.2 + 0.2
published ones (our pre-training results still outperform their published results).
Our code for
experiments on TorchMD-NET is open source
5.2 VARYING DATASET SIZES
We also investigate how downstream test performance on the HOMO target in QM9 varies as a function of the number of upstream and downstream training examples. First, we compare the performance of GNS-TAT with Noisy Nodes either trained from scratch using pre-trained parameters for different numbers of training examples from QM9; we also include the performance of just GNS- TAT. As shown in Figure (left), pre-training improves the downstream performance for all dataset sizes. The difference in test MAF also grows as the downstream training data reduces. Second, we assess the effect of varying the amount of pre-training data while fixing the downstream dataset size for both GNS and GNS-TAT as shown in Figure y (middle). For both models, we find that downstream performance generally improves as upstream data increases, with saturating performance for GNS-TAT. More upstream data can yield better quality representations.
5.3 VARYING MODEL SIZE
We study the benefit of pre-training as models are scaled up on large downstream datasets. Recall that the S2EF dataset in OC20 contains around 130 million DFT evaluations for catalytic systems, providing three orders of magnitude more training data than QM9. We compare the performance of four GNS models with sizes ranging from 10 million to 1.2 billion parameters scaled up by increasing the hidden layer sizes in the MLPs. Each is pre-trained via denoising using the trajectories provided for the IS2RE/IS2RS tasks as described in Section 4.3 We also compare this to a 130 million parameter variant of GNS trained from scratch. As shown in Figure 4 (right), the pre-trained models continue to benefit from larger model sizes. We also observe that pre-trainig is beneficial, as the model trained from scratch underperforms in comparison: the 130 million parameters model trained from scratch is outperformed by pre-trained model of less than half the size.
GitHub http:p/s//ggiiiiooooooaaaaaaaaaa
w denoi
8

---

Published as a conference paper at ICLR 2023
5.4 PRE-TRAINING IMPROVES FORCE PREDICTION
As shown in Section 3.2.1| denoising structures corresponds to learning an approximate force field directly from equilibrium structures. We explore whether pre-training via denoising would therefore also improve models trained to predict atomic forces. We compare the performance of TorchMD-NET for force predic- tion on the MD17 (aspirin) dataset with and without pre-training on PCQM4Mv2. Table 3 shows that pre-training improves force prediction. We also assess the effect of pre-training on the OC20 dataset for force prediction in Appendix G.3 similarly finding an improvement due to pre-training.
Table 3: Performance of TorchMD-NET for force predic- tion on MD17 (aspirin).
Method
Test MAE
TorchMD-NET
0.268 0.003 Pre-training
0.222 + 0.003
5.5 FREEZING PRE-TRAINED PARAMETERS
Finally, we perform an experiment to assess how useful the features learned by pre-training are if they are not fine-tuned for the downstream task but kept fixed instead. Specifically, on the HOMO target in QM9, we freeze the backbone of the model and fine-tune only the decoder (cf. Appendix A). To using random parameters from
120
Test MAE (HOMO)
100
Decoder training random features
evaluate this, we compare initialization for the model's backbone, which allows us to
80
Decoder training w/ PT features
fine-tuning
60
isolate how useful the pre-trained features are. As described in Appendix A the decoder is simple module involving no message- passing. Figure showss that only training the decoder while keeping the pre-trained parameters fixed results in test MAE of 40 Ve which is worse than fine-tuning the entire model but substantially better than the performance of >100 meV in test MAE resulting from training the decoder when the remaining parameters are randomly initialized. This suggests that the features learned by denoising are more discriminative for downstream prediction than random features. We note that training only the decoder is also substantially faster than training the entire network one batch on single V100 GPU takes 15ms, which is 50x faster than one batch using 16 TPUs for the full network.
40
message-passing.
20
200
400
600
800
1000
Gradient stop (thousands)
Figure 5: Training only the decoder results in significantly better perfor- mance with pre-trained instead of random features with GNS-TAT.
LIMITATIONS & FUTURE WORK
We have shown that pre-training can significantly improve performance for various tasks. One additional advantage of pre-trainee trainrd models is that they be shared in the community, allowing practitioners to fine-tune models on their datasets. However, unlike vision and NLP, molecular networks vary widely and the community has not yet settled a "standard" architecture, making pre-trained weights less reusable. Moreover, the success of pre-training inevitably depends on the relationship between the upstream and downstream datasets. In the context of molecular property prediction, understanding what aspects of the upstream data distribution must match the downstream data distribution for transfer is an important direction for future work. More generally, pre-training models on large datasets incurs computational cost. However, our results show that pre-training for 3D molecular prediction does not require the same scale as large NLP and vision models. We discuss considerations on the use of compute and broader impact in Appendix C
CONCLUSION
We investigated pre-trainne pre-training neural networks by denoising in the space of 3D molecular structures. We showed that denoising in this context is equivalent to learning a force field, motivating its ability to learn useful representations and shedding light successful applications of denoising in other works (oowwin et al. (022). This technique enabled us to utilize existing large datasets of 3D structures for improving performance on various downstream molecular property prediction tasks, setting a new SOTA in some cases such as QM9. More broadly, this bridges the gap between the utility of pre-training in vision/NLP and molecular property prediction from structures. We hope that this approach will be particularly impactful for applications of deep learning to scientific problems.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our theoretical results, complete with assumptions and proofs, are included in Section 3.2.1 and Ap- pendix The code to reproduce our experiments on the TorchMD-NET architecture are open source. We report hardware setup with training times in Appendix and detailed hyperparameter settings in Appendix Datasets are described with their respective licenses in Appendix D
REFERENCES
Brandon M. Anderson, T. Hy, and R. Kondor. Cormorant: Covariant molecular neural networks. In NeurIPS, 2019. 3
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: resnets are the answer, then what is the question? In International Conference Machine Learning, pp. 342-350. PMLR, 2017. [6
Rodney J. Bartlett and Monika Musial. Coupled-cluster theory in quantum chemistry. Rev. Mod. Phys., 79:291-352, Feb 2007. doi: 10.1103/RewModdhys.... 79.291. URL https: //lp:k///inkk aps org brg/doi/10, /t../../..../. 1103/ 1ww111////eeMPpiie
79. 291
21
Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and K. Kavukcuoglu, Inter- action networks for learning about objects, relations and physics. ArXiv, abs/1612.00222,
15
P. Battaglia, Jessica B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, A. Santoro, R. Faulkner, Çaglar Gülççhree, H. Song, A. J. Ballard, Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charlie Nash, Victoria Langston, Chris Dyer, N. Heess, Daan Wierstra, P. Kohli, M. Botvinick, Oriol Vinyals, Y. Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. ArXiv, abs/1806.0122 012611, 2018. 2
Simon Batzner, T. Smidt, I.. Sun, Mailoa, M. Kornbluth, N. Molinari, and Kozinsky. Se(3)- equivariant graph neural networks for data-efficient and accurate interatomic potentials. ArXiv, 104/2201.,3114, 2021.
Charles M. Bishop. Training with noise is equivalent to tikhonov regularization. Neural Computation, 7:108-116, 1995. _
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne. and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http:/ /ithub. com/ coon//ooole//ee 6
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Ge- ometric and physical quantities improve (3) equivariant message passing. arXiv preprint 10.02:21 2021.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.1 2
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vons Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL / neuriss pdf
cc/papper/222/F1110 010///15770006ob 9674449444444 f
10

---

Published as a conference paper at ICLR 2023
Chen Cai and Yusu Wang. A note over-smoothing for graph neural networks. abs/2006. 13318, 2020. URL
CoRR, 16
wttp://gggg........../ 13318
Lowik Chanussot*, Abhishek Das*, Siddharth Goyal*, Thibaut Lavrile, Muhammed Shuaibi*, Morgane Riviere, Kevin Tran, Javier Heras- Deras-Dominno. Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitmick, and Zachary Ulissi. Open catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 2021. doi:
5 21
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over- smoothing problem for graph neural networks from the topologiaal view. CoRR, abs/1909.00 .0221 2019.
org. abs /albs//9999 1909. C3211
16
Andrew M Dai and Quoe V Le. Semi-supervised sequence learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL http:: / http::://///cccoceec/e/... neur ps. cc/
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/18 101118800,4855, 2018. URL
http://arxix http:://aaxxv..og/abboo. abspgg///b//ll//
04805
[
Tien Huu Do, Due Minh Nguyen. Giannis Bekoulis, Adrian Munteanu, and N. Deligiannis. Graph con- volutional neural networks with node transition probability-bazz message passing and dropnode regularization. Expert Syst. Appl., 114711 2021. T6
Alexander G. Donchev, Andrew G. Taube, Elizabeth Decolvenaere, Cory Hargus, Robert T. McGib- bon, Ka-Hei Law, Brent A. Gregersen, Je-Luen Li, Kim Palmo, Karthik Siva, Michael Bergdorf, John I.. Klepeis, and David Shaw. Quantum chemical benchmark databases of gold-standard dimer interaction energies. Scientific Data, 8, 2021. 5
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020. URL https::/ //riiiv//rxvvva org. 2aa.:/2210..11
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson, Generalizing convolutional neural networks for equivariance to lie groups arbitrary continuous data. In International Conference on Machine Learning, pp. 3165 -376. PMLR, 2020. 3
Fuchs, Daniel E. Worrall, Volker Fischer, and M. Welling. Se(3)- transforrmes: 3d roto-translation equivariant attention networks. ArXiv, abs/2006. 10tss/2000.100,,, 2020. 5
Jonathan Godwin Thomas Keck*, Peter Battaglia, Victor Bapst. Thomas Kipf, Yujia Li, Kimberly Stachenfeld, Petar Veličković, and Alvaro Sanchez- Jraph: A library for graph neural networks in jax., 2020. URL http:: http://tww.......oo///.eeee/t/ee
com
jraph 6
Jonathan Godwin, Michael Schaarschmidt, Alexander L Gaunt, Alvaro Sanchez-Gonzale Yulia Rubanova, Petar Veličković, James Kirkpatrick, and Peter Battaglia. Simple GNN regularisation for 3d molecular property prediction and beyond. In International Conference Learning Representations, 2022. URL Utt.t:.p:://ppeeeeiii
net/ oett::/ooruummt.1.lweee.eee..3.
22
Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020.
URL http::///// /htp:pp.////ubb...o//oo/
thub. com http://eepmnn/d/e//eee/ ddd/mmmmaiik
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffussonn probabilistic models. arXiv preprint arxiv:2006.1 /239, 2020.
19
Emiel Hoogehoom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d, 2022. URL WRLLihtts://arxxtt org/
17003
Weihua Hu, Bowen Liu, Joseph Gomes, M. Zitnik, Percy Liang, V. Pande, and J. Leskovec. Strategies
for pre-training graph neural networks. arXiv: Learning, 2020a. 2
11

---

Published as a conference paper at ICLR 2023
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. GPT-GNN: generative pre-training of graph neural networks. CoRR, abs/2006.15437, 2020b. URL https: //axxvv
org/abss 2006. 15437 2
Michael J Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. Lietransformer: Equivariant self-attention for lie groups. In International Conference on Machine Learning, pp. 4533-4543 PMLR, 2021.
Rui Jiao, Jiaqi Han, Wenbing Huang, Yu Rong, and Yang Liu. Energy-motivated equivariant pretraining for 3d molecular graphs. arXiv preprint 2xXi:2207.088244 2022. 2
John Jumper. Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer. Simon A A Kohl, Andrew J Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 596 (7873):583-589 2021.
doi: 10. 10001///41166022--0055050 1--2. 21
Thomas Kipf and Max Welling. Variational graph auto-encoders. ArXiv, abs/1611 1.7308,, 2016. 2
Johannes Klicpera, Shankari Giri, Johannes Margraf, and Stephan Günnemann. Fast and uncertainty- directional message passing for non-equilibrium molecules. CoRR, abs/2011.14115, 2020a. /tLLittps:///rxxw.orgg http:///wwvvvoggaaaaaaaaaaa 14115
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. ArXiv, abs/2003.03 0000003..11222 2020b. 3
Risi Kondor, Hy Truong Son, Horace Pan, Brandon M. Anderson, and Shubhendu Trivedi. Covariant compositional networks for learning graphs. CoRR,
2018. URL http://
arxiv.org/ abs 1101110011111111
Shengchao Liu, Mehmet F. Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised representation for graphs, with applications to molecules. In NeurlPS, 2019. 2
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. CoRR, http://aaxvv.v//
2021a. URL
07728
2
Shengchao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with se(3)-invariant
denoising distance matching. arXiv preprint 22XX::2206..366/0 2022a. 2
Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang. Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. In International Conference on Learning Representations, 2022b. URL
net. Forum? wwwp:amiiidiiiiiiisiissss.sm 20re 9
Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S. Yu. Graph self-supervieee learning: A survey. CoRR, abs/2103.00111, 202 URL https: http:p::////x/a./aa/aaea http://axxii..oo//////10000
00111
2
James Martens, Andy Ballard, Guillaume Desjardins, Grzegorz. Swirszez, Valentin Dalibard, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Rapid training of deep ncural networks without skip connections normalization layers using deep kernel shaping, 2021. BtLhttt:: / //axivv
orq/abs 22t...110.. 0176716666 65
16
Benjamin Kurt Miller, Mario Geiger, Tess E Smidt, and Frank Noé. Relevance of rotationally
equivariant convolutions for predicting molecular properties. arXiv preprint 2008 08461,
2020. 3
Maho Nakata and Tomomi Shimazaki. Pubchemqe project: A large-scale first-principles electronic structure database for data-driven chemistry. Journal of Chemical Information and Modeling. 57 (6):1300- 1308, 2017. doi: 10.102 101102//ccs,aa jeim. 7b00083. URL http::ttps///
1021/
acs. jci 7b00083
PMID: 28481528.
5
20
12

---

Published as a conference paper at ICLR 2023
Robert G. Parr and Yang Weitao.
Theory of Atoms and Molecules. Oxford
www.wwwwwaaa........oooo. University Press, USA, 1994. ISBN 0195092767. Btt.::///ww..aa......co
David A. Patterson, Joseph Gonzalez, Quoc V. Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network
training. CoRR, 0bs/2214.11336 2021. URL https: arwii. wwwp:/i..orr/ooooooo a210 .tt.//111..11.0000010011111
10350
20
T. Pfaff, Meire Fortunato, Alvaro Sanchez--oonzlezz, and P. Battaglia. Learning mesh-based simula- tion with graph networks. ArXiv, abs/2010.03409, 2020. 2
R. Ramakrishnan, Pavlo O. Dral, M. Rupp, and O. A. von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014. 5 21
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. The truly deep graph convolutional networks for node classification. CoRR, has/1907.10903 2019. URI. htt.ttt://arxii..rrrg
abs/1907 10903
16
Yu Rong. Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-super graph transformer on large-scale molecular data, 2020. URL hotl:https:/aaat //axxiv
org/abs/2001 2008 02835 2
Alvaro Sanchez-Gonzalez, N. Heess, Jost Tobias Springenberg, J. Merel, Martin A. Riedmiller, R. Hadsell, and P. Battaglia. Graph networks as learnable physics engines for inference and control.
ArXiv, ab.0/206 2018.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In Hal Daumé III and Aarti
Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8459 8468. PMLR, 13-18 Jul 2020. URL
http: htttt::/pprccediii ngs ngs .at.s.m... wwt.ssmwr.....r.... www.presssvll wwt.://1e/aacngggoaaaeee az2zaalle22a html
15
R. Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. In SDM, 2021.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks, 2021.
B 5
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2009. doi: 10.1 (109/TNN.0008.00000
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoe Sauceda Felix, Stefan Chmiela, A. Tkatchenko, and K. Müller. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. In NIPS, 2017. 3
Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning. pp. 9377-9388. PMIR, 2021.
g
Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In International Conference on Machine Learning, 2021. 3
Muhammed Shuaibi, Adcesh Kolluru, Abhishek Das, Aditya Grover, Anuroop Sriram, Zachary Ulissi, and C Lawrence Zitnick. Rotation invariant graph neural networks using spin convolutions. arXiv preprint arXiv:2206.000
2021. 3
J. Sietsma and Robert J. F. Dow. Creating artificial neural networks that generalize. Neural Networks, 4:67-79, 1991.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2014. URL /tl/nttess//arttt.orr//aaa/////p 1556
13

---

Published as a conference paper at ICLR 2023
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribu- tion. Curran Associates Inc., Red Hook, NY, USA, 2019. 3 19 20
Yang Song and Stefano Ermon. Improved techniques for training score- score-based generative models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN
19
Fan- Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. In Interna- tional Conference on Learning Representations, 2019.
Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Velickovic, and Michal Valko. Bootstrapped representation learning on graphs. CoRR, abs/2102.06514, 2021. URL https: / /tt::///axiivvvv otg/ abs/ /2tas./2222.co 06514
Philipp Thölke and Gianni De Fabritüs.. Torchmd-net: Equivariant transformers for neural network based molecular potentials. arXiv preprint arxiv:2202.025441 2022.
Nathaniel Thomas, Tess Smidt, Steven M. Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equuivriinn neural networks for 3d point clouds. CoRR, abs/ 802.08219, 2018. Ut./sttpp///arxivcccc///ss/////p
Oliver T. Unke and Markus Meuwly. Physnet: A neural network for predicting energies, forces, dipole moments, and partial charges. Journal of Chemical Theory and Computation, (56));377-33) May 2019. ISSN 1549-9626. doi: 10.1021/acs,jcc:::00011. URL http::t//// / doi http:/xxxdoooorg////// p021/acs.o jctc. 9b00131
3
Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep Graph Infomax. International Conference on Learning Representations, 2019. URL httptss::///peeeeeaeiteee
net forum? wwwrmmidddrki9i..a.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computa- tion, 23(7):1661-1674 2011. doi: 10.1162/NECO___00 1a100___01111 00142. 2
18
Pascal Vincent, H. Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.
Extracting and
composing robust features with denoising autoencoders. ICML. '08, 2008. 2
Pascal Vincent, H. Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. J. Mach. Learn. Res., 11 11:3311-3400 2010. 2
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. NeurIPS, 2018.
Lechao Xiao, Yasaman Bahri, Jascha Sohl- Sohk-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolu- tional neural networks. In International Conference on Machine Learning, pp. 5393-5402, 201 8.
16
Yaochen Xie, Zhao Xu, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. CoRR, 105/2102.10757, 2021. URL URL htt.ttttt///xxii...o.
2102.10755 2
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffuuion model for molecular conformation generation. In International Conference on Learning Representations, 2022.
Chaoqi Yang, Ruijie Wang, Shuochao Yao, Shengzhong Liu, and Tarek Abdelzaher. Revisiting" over-smoothing" in deep gens. arXiv preprint arXiv:2003. 13663, 2020. 16
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform bad for graph representation? In NeurIPS, 2021.
14

---

Published as a conference paper at ICLR 2023
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5812 5823. Curran Associates, Inc., 2020. URL 2020/F11e/3fe2333388 htt.://eee....aaaaae2222223333a3
weeerrpss.. wtt.rip...ccaaapee capppper/r///// pdf
2
Guodong Zhang, Aleksandar Botev, and James Martens. Deep learning without shortcuts: Shaping the kernel with tailored rectifiers. In International Conference on Learning Representations, 2022. URL htttps://ppsnreieeeeee
ww.nee/fffe forrmm? deformmiddtt00 id=tok7XNTiR
16
L. Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. ArXiv, 0tss/199..120 2020. [6
Kuangqi Zhou, Yanfei Dong, Wee Sun Lee, Bryan Hooi, Huan Xu, and Jiashi Feng. Effective training strategies for deep graph neural networks. CoRR, abs/2006.07107, 2020. httptt
7/arxiv. wrrxiv.rrg/abs/2000..8 07107
16
A ARCHITECTURAL DETAILS
A.1 STANDARD GNS
As our base model architecture we chose Graph Net Simulator (GNS) (Sanchez-Gonzalez et al. 2020). which consists of an ENCODER which constructs graph representation from the input S, a PROCESSOR of repeated message passing blocks that update the latent graph representation. and a DECODER which produces predictions. Our implementation follows Godwin et al.Js [2022) modifications to enable molecular and graph-level property predictions which has been shown to achieve strong results across different molecular prediction tasks without relying on problem-specific features.
In the ENCODER, represent the set of atoms S (I), (a2, P2)..... (d), (as) P(s))
directed graph G (V, where V = V2.. V2...., Visi and E ferjhis are the sets of "featurized" vertices and edges, respectively. Edges C.2 E are constructed whenever the distance between the i-th and j-th atoms is less than the connectivity radius Rew (given in Appendix F). in which case we connect V, and V, with a directed edge e, from to j that a featurization of the displacement vector Рj Pa- Mcanwhile, for the i-th atom, V, is given by learnable vector embedding of the atomic number a..
The PROCESSOR consists of L message-passing steps that produce intermediate graphs GI...., Gr. (with the same connectivity structure as the initial one). Each of these steps computes the sum of a shorteut connection from the previous graph, and the application of Interaction Network (Battaglia et al. 2016). Interaction Networks first update each edge feature by applying an "edge update function" to a combination of the existing feature and the features of the two connected vertices. They then update each vertex feature by applying "vertex update function" to combination of the existing feature and the (new) edge features of incoming edges. In GNS, edge update functions are 3 hidden layer fully-connected MLPs, using "shifted softplus" (ssp(z) = log(0.5e"* + 0.5)) activation function, applied to the concatenation the relevant edge and vertex features, followed by a layer normalization layer. Vertex update functions are similar, but are applied to the concatenation of the relevant vertex feature and sum over relevant edge features.
In our implementation of GNS we applied the same PROCESSOR in sequence three times (with shared parameters), with the output of each being decoded to produce a prediction and corresponding loss value. The loss for the whole model is then given by the average of these. (Test-time predictions meanwhile computed using only the output of the final PROCESSOR.)
The DECODER is responsible for computing graph-level and vertex-level predictions from the output of each PROCESSOR. Vertex-level predictions, such as noise as described in Section 3.2 are decoded using MLP applied to each vertex feature. Graph-level predictions (e.g. energies) are produced by applying an MLP to each vertex feature, aggregating the result over vertices (via a sum), and then applying another MI.P to the result.
15

---

Published as a conference paper at ICLR 2023
A.2
GNS WITH TAILORED ACTIVATION TRANSFORMATION (GNS-TAT)
[+4-1)
(f+1)
0.9
v1 0.92
Layer
3 hiddlen layer MLP Tailored ReLU
hidden layer w/ shifted soñtlls
Initialize to zero
Layer norm
Concatenate
Concatenate
e
1,3
Figure 6: Diagram showing the edge update for a single step f of the PROCESSOR. Leftt: Edge update for GNS. Right: Edge update for GNS-TAT (with modifications shown in red).
Tailored Activation Transformation (TAT) Zhang et al., 2022) is method for initializing and transforming neural networks to make them easier to trai,, and is based on a similar method called Deep Kernel Shaping (DKS) (Martens al. 2021). TAT controls the propagation of "q values", which are initialization-tiin approximations to dimension-normmalize squared norms of the network's ' layer-wise activation vectors, and "c values", which are cosine similarities between such vectors (for different inputs). In other words, q values approximate [|=(z)]|² (z))), where 2(x) denotes a layer's output as a function of the network's input æ, and e values approximate 2(x) ([]])((((((()))))))) = II), where x' is another possible network input. In standard deep networks, constant value Coc. E 00, 1). that "geometric information" lost, which leads to training difficulties (Martens et al. 2021). DKS/TAT prevents this convergence through combination of careful weight initialization, and transformations to the network's activation functions and sum/average layers.
e values will converge
Oversmoothing (Chen et al. 2019 Cai Wang 2020 Rong et al. 2019 Zhou et al. 2020 Yang
al., 2020 Zhao & Akoglu 2020 Do et al. 202 | is phenomenon observed in GNN architectures
where vertex/edge features all converge to approximately the same value with depth, and is associated with training difficulties. It is reminiscent of how, when C.o 1, feature vectors will converge with depth to a constant input-independent vector in standard deep networks. It therefore seems plausible that applying TAT GNNs may help with the oversmoothing problem and thus improve training performance.
Unfortunately, the GNS architecture violates two key assumptions of TAT. Firstly, the sums over edge features (performed in the vertex update functions) violate the assumption that all sum operations must be between the outputs of linear layers with independently sampled initial weights. Secondly, GNS networks have multiple inputs for which information needs to be independently preserved and propagated to the output, while DKS/TAT assumes a single input (or multiple inputs whose representations evolve independently in the network).
To address these issues we introduce a new initialization scheme called "Edge-Delta", which initializes to zero the weights that multiply incoming vertex features in the edge update functions (and treats these weights as absent for the purpose of computing the initial weight variance). This approach is inspired by the use of the "Delta initialization" (Balduzzi al. 2017; Xiao et al. 2018) for convolutional networks in DKS/TAT, which initializes filter weights of the non-coe non-central locations to zero, thus allowing geometric information, in the form of values, propagate independently for each location the feature map. When using the Edge-Det Delta initialization, edge features propagate independently of each other (and of vertex features), through what is essentially standard deep
16

---

Published as a conference paper at ICLR 2023
(++1)
((+1)
2/
0.9
0.92
Layer
hidden layer
Tailored ReLU
3 hidden layer
MLP
shifted softplus
Concatenate
Concatenate
(c)
(t)
0;
U3
(+++1)
[f++1]
(+++)
(1++1)
(++1)
[t (t++1)
e a,
c2
eb,
c,
Figure 7: Diagram showing the vertex update for single step of the PROCESSOR. Left: Vertex
update for GNS. Right: Vertex update for GNS-TAT (with modifications shown in red).
residual network (with edge update functions acting the residual branches), which will refer to as the "edge network".
Given the of Edge-Delas Delta activation functions with TAT's transformed Leaky-Rel.U activation functions (or "Tailored ReLUs")), which we compute with TAT's y) parameter
then apply TAT to GNS follows First, replace GNS's
0.8, and its "subnetwork maximizing function" defined on the edge n We also replace each sum involving shortcut connections with weighted sums, whose weights are 0.9 and v1 0.9% for the shortcut and non-shortcut branches respectively. We retain the of layer normalization layers in the edge/vertex update functions, but move them to before the first fully- fully-connected layer, as this seems to give the best performance. As required by TAT, we use standard Gaussian fan-in initialization for the weights, and zero initialization for the biases, with Edge-Delte used only for the first linear layer of the edge update functions. Finally, we replace the sum used to aggregate vertex features the DECODER with an average. See Figures and 7 for illustration of these changes.
We experimented with an analogous "Vertex-Delta" initialization, which initializes to zero weights in the vertex update functions that multiply summed edge features, but found that Edge-Delta gave the best results. This might be because the edge features, which encode distances between vertices (and are best preserved with the Edge-Delta approach), generally much more informative than the vertex features in molecular property prediction tasks. We also ran informal ablation studies, and found that each of our changes to the original GNS model contributed to improved results, with the use of Edge-Dell DDelt and weighted shortcut sums being especially important.
GNS vs. GNS-TAT on OC20. Our preliminary experiments used to determine appropriate choices for the use of Eddge-Delta/Ve SiettaVVetexx-eleee initialization, weighting shortcut sums and TAT hyperpa- rameters such as 7 were conducted on QM9. This led to parameterization of GNS-TAT which gave improved performance on QM9/DES15K as shown earlier in Sections 4.2 and 4.4 However, we kept
'Note that Edge- Delta initialization is compatible with TAT, since for the purposes of g/e value propagation, zero-initialized connections in the network can be treated absent.
'For the purposes of computing the subnetwork maximizing function we ignore the rest of the network and just consider the edge network. While the layer normalization layer (which we move before the MLP) technically depends on the vertex features, this dependency can he ignored as long as the q values of these features is (which will be true given the complete of changes make to the GNS architecture).
17

---

Published as a conference paper at ICLR 2023
the original parameterization for GNS from Godwin et al. (2022) OC20, the model size was different from QM9/DESS5K (see Appendix and re-tuning hyperparameters on OC20 would be
computationally expensive. Using the same TAT settings QM9 on OC20 results in approximately the same performance as the original GNS model as shown Table 4
Table 4: GNS vs. GNS-TAY energy prediction MAE on OC20. TAT hyperparameters were tuned with experiments on QM9 and then kept fixed for OC20.
Model
Validation Dataset
GNS
GNS + NN
GNS-TAT'S + NN
ID
0.5233
0,4196
0.4199
OOD Adsorbate
0.6295
0.4900
0.5013
OOD Catalyst
0,5202
0,4316
0,4319
OOD Both
0,5617
0,4282
0.4373
B DENOISING AS LEARNING FORCE FIELD
We specify molecular structure as x (x(1), x (N) RNN,, where x()) E R is the coordinate of atom i. Let E(x) denote the total (potential) energy of x, such that E(x are the forces on the atoms. As discussed in Section 3.2.1 learning the force field, i.e. the mapping x a reasonable pre-training objective. Furthermore, learning the force field can be viewed as score- matching if we define the distribution Pphysical (x) exp(-E(x)) and observe that the score of Pphysical is the force field: Vx log Pphysical (x) -V
FxF(x),
However, technical caveat is that Pphysical is an improper probability density, because it cannot be normalized due to the translation invariance of E. Writing the translation of a structure as x + t == (x(1) + tt....,,' ((N) + t) where t E R3 is a constant vector, we have F(x + t) = F(x). This implies that the normalizing constant faax Pphysical dx diverges to infinity. To remedy this,
can restrict ourselves to the (3N - -3) -immennsioa subspace V == {x E RNN Σ, x = 0) S TR
consisting of the mean-centered structures, over which Pphysical can be defined normalizable distribution.
Proceeding similarly as Section 3.2.1 let x....., X, E V be set of mean-centered equilibrium structures. For any x E V, we now approximate
n n
Pphysicul (x) = qo (x)
To (x x;),
where the Gaussian distributions 9-($x (xc x;) are defined on as:
1 x, xx(?3
(X x,)
(2 (3N (3N - 31/2 exp
2g
X
For convenience, we have expressed structures as vectors in the ambient space R3N however they are restricted to lie in the smaller space V. Note that the normalizing constant accounts for the fact that is (3.V- 3)-dimensional As before, we define (x(x) = Σ 6(x = x,) to be the empirical distribution and qo (x, x) qo (x). The score-matce motte-natching objective is given by:
J1 ( Egg (x) [[]GNN(X)
Vx log gr (x)g)(()]]
(5)
where the expectation now over V. As shown by Vincent (011). minimizing the objective above is equivalent to the minimizing the following objective:
(z) Eyg (x)) [[GNNoX)) Vx log x)]??
This is recognized as a denoising objective, because Vg lo
9o (x x) = (x- x)-x) A practical implication of this analysis is that the noise (x x)/o² V should be mean-centered, which is intuitive since it is impossible to predict translational component in the noise.
We include proof of the equivalence between Equations and (6) for completeness:
18

---

Published as a conference paper at ICLR 2023
Proposition 1 Vincent 2011 ) The minimization objectives (0) and J2(0) equivalent.
Proof. We first observe:
J1 (0) y (8)
[|NNN(
2Eg (8) I(GNNO (x), Vx log (x))l + +01
(2) g. (x)
[[)NN(())] g (xxx) (X), x log to (x x))] + C2,
where C1. C2 are constants independent of A. Therefore, it suffices to show that the middle terms on the RHS are equal. Since expectations over 4o (x) and go (X, x) are restricted to S RR we apply change of basis to write them as integrals against the (3.V - 3) ddimensional Lebesgue measure. Pick an orthonormal basis {UI...., 33N-3) C R3N for V and let Pv [Ui...., PaN-3] R (3 be the projection matrix, so z Pyx expresses a mean-centered structure x in terms of the coordinates of the chosen basis for V. Noting that Py has orthonormal columns and that it yields bijection between V and R3N- 3 we calculate:
(x)(x) [(GNNo( Vx log 4o (x))
90 (PNV)) (PV log do (Pvž)) dž Vgo dž
RaN-3
La (1(V22 (NN6((P22).
RaN
2o (Pr2)
/ JgAN-
(GNNo(PV2), Vyo (Pv2)) dē
GNNo(PV2), 1
V (Pvž 2 xi) dē
JaN
i=1
(NV2,
(Pvž V 4o (Pvz xi) dž
JaNN3
lo (Pvž x;) (GNN6 (Pv2), Vlogo. xx)) dž
-a n
(x.x) ((), Vx 1o x))]
B.1 DIFFERENCES BETWEEN DENOISING FOR GENERATIVE MODELLING VS. LEARNING FORCES
Denoising has recently seen widespread use means for generative score-based modelling (e.g. Song & Ermon 2019 2020 Ho et al. 2020) due to the equivalence between denoising and score- matching (Vincent (011). In this section, elaborate the differences between the equivalence of denoising and score-matching for generative modelling versus the equivalence of denoising and force learning. Both of these equivalences rely the result of Vincent (201T) although with different assumptions and different aims in practice.
As background, generative score-based modelling consists of modelling target distribution p(x) by learning an approximation of its score function V log p(x) using neural network given a training set of samples X1, x,, ~ p(x). Using the learned approximation of the score function, new samples can be generated using score-bases MCMC techniques, such as Langevin MCMC. Learning the score function corresponds the optimization objective
min Ep(x) [])xo(x)= -----iil Vlogp(x))1*]
Although the expectation pxx) can be approximated using the samples x,, the true score V log p(x) is unknown, rendering the objective function intractable as is. Denoising score-matching (Vincent, 2011) approaches this by replacing p(x) with approximation that leads tractable
19

---

Published as a conference paper at ICLR 2023
objective. We can replace p(x) with noisy approximation (x) f fo (x: x)p(x) dx where 1-(x x) == xx T) is the noising distribution. The advantage of doing so is that score- matching with qo (x) is equivalent to the following tractable denoising objective by Proposition
min g (xxx))(x))
[|100(*)
V log xx)]]]
However, in order for this approximation be accurate, it is crucial that the noise level is as small as possible, ideally zero in which case the approximation is exact. As shown by [Snn & Ermon (011). choosing a value for close to zero leads to unreliable learned estimates of the score away from the regions of p(x) with high probability density. Therefore, they proposed multi-scale denoising where the model simultaneously learns to approximate the scores of (o; for 1,... L, where > 02 > ... > JL is sequence of increasing noise levels such that dg is close to zero and Or R p. In practice, this then leads to multi-scate denoising.
In the context of learning forces, we first observe that the score function of the Boltzmann distribution p(x) = Pphysical(X] (x) x (x)) is equal to the forces V log Ppinysical (x) = (x)). Therefore,
learning the forces corresponds to the following score-matching objective:
min (pppes(())) 0
[1xPo(x) - V log Pphysical (x))s (x)
Once again, the score function V log Pptysscaa (x) is unknown and must be approximated. However, un- like previously, now have access these examples are local maxima of the distribution Pphyscal (x) rather than samples from as in the case before. Let go (x) denote mixture of Gaussians with noise scale a centered at X1....,.... X,,, In or- der to capture the fact that Pphysical xx) has local maxima at Xj,..., Xn. we can approximate ( with (x). This distribution g. (X) can also be written as (x) = J qo (x x) dx, where (x) = > 8(x = x;), hence we can apply Proposition) lggain and reduce the optimization objective to denoising as explained in Section 3.2.1
of equilibrium, energy-minimizee
structures X1,-... ........ Xai
In contrast with before, in this case, a desirable noise scale a is not zero, since x X, are local maxima of Pphysical and, intuitively, the approximation should contain some non-zero mass around each local maxima. As result, we do not use multi-scale denoising in practice. It also worth noting that in contrast with generative modelling, our aim is to learn the score function itself (i.e. forces) rather than producing new samples by using the learned score function with MCMC approaches.
C BROADER IMPACT
Who may benefit from this work? Molecular property prediction works towards a range of applications in materials design, chemistry, and drug discovery. Wider use pre-trained models may accelerate progress in a similar manner to how pre-trained language or image models have enabled practitioners avoid training on large datasets from scratch. Pre-training via denoising simple to implement and can be immediately adopted to improve performance on a wide range of molecular property prediction tasks. As research converges on more standardized architectures, we expect shared pre-trained weights will become more common across the community.
Potential negative impact and ethical considerations. Pre-training models on large structure datasets incurs additional computational cost when compared to training potentially smaller model with less capacity from scratch. Environmental mitigation should taken into account when pre- training large models (Patterson et al. 2021). However, the computational cost of pre-training can and should be offset by sharing pre-trained embeddings when possible. Moreover, in our ablations of upstream dataset sizes for GNS-TAT, observed that training
subset of PCQM4Mv2 was sufficient for strong downstream performance. In future work, we plan to investigate how smaller subsets with sufficient diversity can be used to minimize computational requirements, e.g. by requiring fewer gradient steps.
D DATASETS
PCQM4Mv2. The main dataset we use for pre-training is PCQM4Mv2 (Nakata & Shimazaki 2017) (license: CC BY 4.0), which contains 3,378,606 organic molecules, specified by their 3D structures
20

---

Published as a conference paper at ICLR 2023
at equilibrium (atom types and coordinates) calculated using DFT. Molecules in PCQM4Mv2 have around 30 atoms on average and vary in terms of their composition, with the dataset containing 22 unique elements in total. The molecules in PCQM4Mv2 only contain one label, unlike e.g. QM9, which contains 12 labels per molecule, however we do not use these labels as denoising only requires the structures.
QM9. QM9 is a dataset (Ramakrishnan et al. 2014) (license: CCBY 4.0) with approximately 130,000 small organic molecules containing up nine heavy C, N, O, atoms, specified by their structures. Each molecule has 12 different labels corresponding to different molecular properties, such as highest occupied molecular orbital (HOMO) energy and internal energy. which we use for fine-tuning. Following prior work, we randomly split the dataset into 1 114k examples for training, 10k examples for validation and 10k examples for testing.
OC20. Open Catalyst 2020 (Chanussot* et al. (021) (OC20, license: CC Attribution 4.0) is a recent large benchmark containing trajectories of interacting surfaces and adsorbates that are relevant to catalyst discovery and optimization. This dataset contains three tasks: predicting the relaxed state energy from the initial structure (IS2RE), predicting the relaxed structure from the initial structure (IS2RS) and predicting the energy and forces given the structure at any point in the trajectory (S2EF). For IS2RE and IS2RS, there are 460,000 training examples, where each data point a trajectory of a surface-adsorbate molecule pair starting with high-energy initial structure that is relaxed towards low-energy, equilibrium structure. For S2EF, there are 113 million examples of mnnnequillbrium) structures with their associated energies and per-atom forces. We evaluate the models the provided validation sets, some of which include out-of-distribttion data relative to the training data, as detailed in Chanussot x et al. (2021).
DES15K. DES (Donchev et al. 2021 (license: CCO 1.0) is small dataset containing around 15,000 interacting molecule pairs, specifically dimer geometries with non-covalent molecular interac- tions. Each pair is labelled with the associated interaction energy computed using the coupled-cluster method with single, double, and perturbative triple excitations (CCSD(T)) (Bartlett & Musial 2007). which is widely regarded as the gold-standard method in electronic structure theory. We randomly split the dataset into 13k training examples, 1K validation examples and 1k testing examples.
Usage of DFT-generated structures for pre-training. The structures in PCQM4Mv2 are obtained using DFT calculations, which in principle could have also been used to generate labels for molecular properties in DFT- DFF--generaedd downstream datasets, such as QM9. However, there are multiple reasons why denoising remains a desirable pre-training objective in such settings. First, although there is a computational cost for generating datasets such as PCQM4Mv2, it is openly available and part of our aim is to understand how to leverage such datasets for tasks on other datasets (analogous to how ImageNet is expensive to build, but once it is available, it is important to understand how it can improve downstream performance on other datasets). Second, even if PCQM4Mv2 contained all the labels in QM9, pre-training via denoising structures allows one to pre-train a single, label-agnostic model which can be individually fine-tuned on any of the targets in QM9. This is substantially cheaper than per- per-target pre- mraitrainnng and the resulting pre-trained model is also re-useable for differing needs and downstream tasks.
In other settings where the downstream dataset generated using more expensive methods than DFT, pre-training via denoising DFT-relaxed structures can also be helpful and has clear benefit, as shown by our experiment on the CCSDD(T)-eeeraatee dataset DES15K where denoising on PCQM4Mv2 improves performance for downstream task involving a "higher" level of theory such CCSD(T). Generally, we emphasize that the methodology of denoising structures can be applied to any dataset of structures (regardless of whether they are computed using DFT or not). We hope that denoising will be useful in the future for learning representations by pre-training on structures obtained through other methods such as experimental data (in which case labeling may be expensive) and databases generated by other models such as AlphaFold (where only structures are available) (Jumper et al. 2021
21

---

Published as a conference paper at ICLR 2023
E EXPERIMENT SETUP AND COMPUTE RESOURCES
Below, we list details on our experiment setup and hardware resources used.
GNS & GNS-TAT. GNS-TAT training for QM9, PCQM4Mv2 and DESI5K was done cluster of 16 TPU v3 devices and evaluation on a single V100 device. GNS training for OC20 was done on 8 TPU v4 devices, with the exception of the 1.2 billion parameters variant of the model, which was trained on 64 TPU v4 devices. Pre-training on PCQM4Mv2 was executed for 3 10 gradient updates (approximately 1.5 days of training). Fine-tuning experiments were run until convergence for QM9 (10% gradient updates taking approximately 2 days) and DES (10% gradient updates taking approximately 4 hours) and stopped after 5 10 gradient updates on OC20 (2.5 days) to minimize hardware use (the larger models keep benefiting from additional gradient updates).
TorchMD-NET. We implemented denoising for TorThMD-NET on top Thölke & De Fabritiis [2022] open-source code / Models were trained on QM9 using data parallelism over two NVIDIA RTX 2080Ti GPUs. Pre-training on PCQM-Mv2 was done using three GPUs accommodate larger molecules while keeping the batch size approximately the same QM9. All hyperparameters except the learning rate schedule were kept fixed at the defaults. Pre-training took roughly 24 hours, whereas fine-tuning took around 16 hours.
Hyperparameter optimization. We note that effective pre-training via denoising requires sweeping noise values, as well as loss co-efficients for denoising and atom type recovery. For GNS/GNS- TAT, we relied on the hyperparameters published by Godwin et al. (2022) but determined new noise values for pre-training and fine-tuning by tuning over the set of values {0.005, 0.01, 0.02, 0.05, 0.1) for each of PCQM4Mv2 and QM9 (on the HOMO energy target). We used the same values for DESI5K without modification. We also ran similar number of experiments to determine cosine cycle parameters for learning rates.
Availlabl GitHub at: htttte///gthhttt.ccm////a/m//r/r
22

---

Published as a conference paper at ICLR 2023
F HYPERPARAMETERS
We report the main hyperparameters used for GNS and GNS-TAT below.
Table 5: GNS-TAT hyperparameters for pre-training on PCQM4Mv2.
Parameter
Value description
Gradient steps
3 105
Optimizer
Adam with warm up and 1-cycle cosine decay schedule
81
0.9
32
0.95
Warm up steps
10a
Warm up start learning rate
10-5
Warm up max learning rate
10-4
Cosine min learning rate
10-7
Cosine cycle length
5 10
Loss type
Mean squared error
Batch size
Dynamic to max edge/vertex/graph count
Max vertices in batch
256
Max edges in batch
9216
Max graphs in batch Distance featurization
8
Bessel first kind
Max edges per vertex Rewt
20
10
MLP number of layers
3
MLP hidden sizes
1024
Activation
Tailored ReLU (with negative slope chosen using TAT)
message passing layers
10
Block iterations
3
Vertex/edge latent vector sizes
512
Decoder aggregation
Mean
Position noise
Gaussian (μ = 0, a 0.02)
Parameter update
Exponentially moving average (EMA) smoothing
EMA decay
0,9999
Position loss coefficient
1.0
Atom type mask probability
0.75
Atom type loss coefficient
4.0
23

---

Published as a conference paper at ICLR 2023
Table 6: GNS-TAT hyperparameters for fine-tuning QM9 and DES15K.
Parameter
Value or description
Gradient steps
10
QM9 10 DESI5K
Optimizer
Adam with warm up and I-cycle cosine decay schedule
81
0.9
82
0.95
Warm up steps
10*
Warm up start learning rate
10-5
Warm up max learning rate
10-1
Cosine min learning rate
3 10-7
Cosine cycle length
10
QM9 10 DESISK
Loss type
Mean squared error
Batch size
Dynamic to edge/vertex/graph count
Max vertices in batch
256
Max edges in batch
3072
Max graphs in batch Distance featurization
8
Bessel first kind
Max edges per vertex Reu
20
10
MLP number layers
3
MLP hidden sizes
1024
Activation
Tailored ReLU (with negative slope chosen using TAT)
message passing layers
10
Block iterations
3
Vertex/edge latent vector sizes
512
Decoder aggregation
Mean
Position noise
Gaussian (μ 0, 0.05)
Parameter update
Exponentially moving average (EMA) smoothing
EMA decay
0.9999
Position loss coefficient
0.01
Atom type mask probability
0.0
Atom type loss coefficient
0.0
24

---

Published as a conference paper at ICLR 2023
Table 7: GNS hyperparameters for OC20.
Parameter
Value or description
Gradient steps
5 10
Optimizer
Adam with warm up and l-cycle cosine decay schedule
$1
0.9
82
0.95
Warm up steps
5 10
Warm up start learning rate
10-5
Warm up max learning rate
10-1
Cosine min learning rate
5 10-6
Cosine cycle length
5 106
Loss type
Mean squared error
Batch size
Dynamic to max edge/vertex/graph count
Max vertices in batch
1024
Max edges in batch
12800
Max graphs in batch Distance featurization
10
Gaussian (μ 0, o 0.5)
Max edges per vertex Rew
20
6
MLP number layers
3
MLP hidden sizes
1024
Activation
shifted softplus
message passing layers
5
Block iterations
5
Vertex/edge latent vector sizes
512
Decoder aggregation
Sum
Position noise
Gaussian (μ 0, 0.2)
Parameter update
Exponentially moving average (EMA) smoothing
EMA decay
0.9999
Position loss coefficient
1.0
Atom type mask probability
0.0
Atom type loss coefficient
0.0
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 PERFORMANCE OF TORCHMD-NET ON DES15K
In addition to the experiments involving GNS-TAT on DESI5K in Section 4.4 we also consider the performance of TorchMD-NET on DESI5K with and without pre-training on PCQM4Mv2. As shown Table 8 TorchMD-NET outperforms GNS-TAT when trained from scratch, and pre-training then yields a further boost in performance with GNS-TAT.
Table Performance of TorchMD-NET with and without pre-training for interaction energy predic- tion on DESI
Model
Test MAE (kcal/mol)
TorchMD-NET
0.721
Pre-training on PCQM4Mv2
0.406
25

---

Published as a conference paper at ICLR 2023
G.2 COMPARISON OF OC20 IS2RE PRE-TRAINING PERFORMANCE WITH OTHER ARCHITECTURES
Table compares the performance of models with various other architectures proposed in prior work. GNS yields SOTA performance on each of the four validation sets for the direct IS2RE task. As discussed in Section 4.3 and shown in Figure 3 (left), the model pre-trained on OC20 itself achieves SOTA performance with faster convergence than all other GNS variants. Note that since we pre-train on OC20 itself, the pre-trained model performs equally well convergence as the model trained from scratch with noisy nodes (ef. Section 4.3)
Table 9: Comparison of different variants of GNS with other baseline architectures on IS2RE prediction for OC20.
Validation MAE for IS2RE
Model
ID
OOD Adsorbate OOD Catalyst
OOD Both
Average
DimeNet ++
0.5636
0.7127
0,5612
0.6492
0.6217
GemNet
0,5561
0.7342
0.5659
0.6964
0,6382
SphereNet
0,5632
0.6682
0,5590
0.6190
0.6024
SEGNN
0.5310
0.6432
0.5341
0.5777
0.5715
GNS
0.5233
0.6295
0.5202
0.5617
0.5587
GNS NN
0,4196
0.4900
0,4316
0.4282
0.4424
GNS NN (PT on PCQM4Mv2)
0.4135
0.4856
0.4245
0.4245
0.4370
GNS NN (PT on OC20)
0.4164
0.4836
0,4267
0.4237
0.4376
G.3 PRE-TRAINING VIA DENOISING FOR FORCE PREDICTION ON OC20
Recall that in Section 5.4 we explored whether pre-training also improves force prediction models, given the link between denoising and learning forces as described in Section 3.2.1 In this section, consider a second experiment for force models. The OC20 dataset contains a force prediction task (S2EF), where model is trained to predict point-wise energy and forces (each point being a single DFT evaluation during a relaxation trajectory) from given 3D structure. Tables [0] and [1 show the performance of GNS when trained from scratch vs. pre-trained via denoising on equilibrium structures in OC20, as described in Section 4 We show two metrics for measuring force prediction performance on each of the four validation datasets: mean absolute error (lower is better) and cosine similarity (higher is better). We observe that the pre-trained model improves upon the model trained from scratch for both metrics and all four validation datasets, with improvements up to 15%.
Table 10: Force prediction on OC20 by MAE (lower is better).
Model
Validation Dataset
GNS
Pre-trained GNS
ID
0,0332
0.0282
OOD Adsorbate
0,0366
0.0314
OOD Catalyst
0.0360
0.0335
OOD Both
0.0406
0.0382
Table Force prediction on OC20 by cosine similarity (higher better).
Model
Validation Dataset
GNS
Pre-trained GNS
ID
0.4845
0,5517
OOD Adsorbate
0,4730
0.5414
OOD Catalyst
0,4553
0.4983
OOD Both
0.4417
0.4849
26


Reviews:
[{"review_id": "drHb56SpHB", "reviewer": "Reviewer_tMrH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates (1) the denoising objective for pretraining and (2) tailored activation transformation (TAT) for 3D graph neural networks. The authors show how their denoising objective is equivalent to learning molecular force field based on interpretation of denoising objective as a score function. The proposed approach demonstrates a notable improvement over several tasks. ", "summary_of_the_review": "I recommend acceptance for this work given the following strengths:\n- significance of the problem being solved \n- the proposed methodology being simple and easy to implement \n- well-executed experiments demonstrating benefit of the proposed approach over various settings", "strength_and_weaknesses": "Strength 1: The problem of pretraining 3D molecular representation using a small number of labels seems very significant. \n\nStrength 2: The proposed denoising objective is quite simple and works well across varying datasets.\n\nStrength 3: Tailored activation transformation is also useful trick for improving 3D GNN performance in general.\n\nWeakness 1: The experiment does not consider applying the denoising objective for the recently proposed 3D graph neural network architectures like GemNet, SphereNet, or SEGNN. It is unclear whether if the proposed approach is useful for the recently proposed architectures too.\n\nWeakness 2: The interpretation of denoising objective as a score matching objective is not very novel.\n\nWeakness 3: The experiments do not compare with the recently proposed non-3D molecular representation learning frameworks. While attribute masking is considered as a baseline, there exist more recently proposed works, like GraphMVP or 3D infomax, which considered 3D information during training. \n\nWeakness 4: While I appreciate how the authors provide a detailed description of their hyperparameter in Appendix E and F, the description could be made more clear for the future works to build on. For example, in Appendix E, it is unclear what the authors mean by tuning on “approximately” 5 values (hyperparameters) on the QM9 dataset. What do the authors mean by approximate? Also, is the validation set used for tuning the hyperparameters? I hope the authors clearly describe this since future works will likely follow similar protocols.\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JWMw9R91HuM", "reviewer": "Reviewer_YJNn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a pretraining method to encode 3D small molecules into atom-level encodings which can be useful for downstream prediction tasks. The method is essentially to start with the 3D coordinates of a molecule, add random Gaussian noise to the atom coordinates, and train a 3D GNN to predict the added noise. The authors connect this method to denoising score matching, which has been proposed previously in the context of diffusion models. The authors also offer an interpretation of the method as a way to approximate the local force field of the atoms, which the GNN is attempting to predict. The paper demonstrates that pretraining using this method (along with a few other tricks like Noisy Nodes (NN) and Tailored Activation Transformation (TAT)) can be used in conjunction with pretraining to achieve better predictive performance on a variety of downstream tasks.", "summary_of_the_review": "This paper presents a novel method of pretraining for small molecules, which has some nice theoretical connections to denoising score matching, and has a direct connection with statistical mechanics. The authors show several compelling analyses which demonstrate the effectiveness of pretraining in a variety of situations. The improvements over current SOTA are perhaps modest, but present. The authors also explore the effects of pretraining combined with other techniques such as Noisy Nodes, and an application of Tailored Activation Transformation to GNNs. The analyses are limited in that there are very few direct comparisons of their GNNs with pretraining to current SOTA models, certain analyses focus on different GNNs for unknown/unjustified reasons, and the task which pretraining should theoretically benefit the most does not seem to enjoy these benefits as significantly as one would hope. Regardless, I believe the authors have shown that their method of pretraining is functional and works well, and their numerous analyses on their method help give the reader intuition on how and why pretraining, Noisy Nodes, and Tailored Activation Transformation can be helpful for molecular prediction tasks.", "strength_and_weaknesses": "Overall, the paper demonstrates a solid contribution. It provides several compelling analyses which show how the proposed method can generate embeddings which assist in several downstream tasks. The authors compare their method to many existing architectures, and even show that pretraining can assist another architecture (which is compatible with their proposed method). The paper also analyzes a few design choices (e.g. model size and dataset size) to show the effect of these choices on the model’s performance.\n\nThere are a few areas which I feel are weaker:\n### Limited comparisons with other models from prior work\nThe comparisons with prior work that are shown in the paper are good to see (i.e. Table 1 and Table 8), as they directly compare GNS and/or GNS-TAT with the performance of other models. However, Table 1 only shows the comparison with other models on a single benchmark, QM9. The improvements brought on by GNS-TAT + PT + NN are certainly there, but not ubiquitous and not always huge. The comparisons in Table 8 notably do not include models such as TorchMD-NET. All other analyses in the paper are effectively only comparing GNS/GNS-TAT with itself (when PT/NN are added), or when other models like TorchMD-NET are pretrained using this method. This suggests that although the improvements in performance offered by this paper are there, they may be limited to certain datasets/benchmarks, and are not necessarily as strong. Since one of the main claims made by the paper is that PT/NN/TAT help performance, it is important to directly compare the performance of GNS/GNS-TAT (with PT/NN) with all other models (especially the strongest ones like TorchMD-NET) on more than just one benchmark. Of course, showing that TorchMD-NET performs better with pretraining (which is already done in a limited capacity) is also helpful to expand upon.\n### There are several models being analyzed\nMost analyses seem to use either GNS or GNS-TAT, and many of the analyses add either PT or NN (or both). Unfortunately, some analyses use GNS-TAT and others use GNS (with different flavors of adding PT or NN). This can be a little confusing for the reader, and also casts a bit of doubt on the robustness of the method. It is very possible that some benchmarks/tasks do better with TAT and others do not (or with or without NN, etc.), and if so, this should be stated explicitly as a potential caveat (i.e. TAT or NN may be better suited for some tasks and may be worse for others).\n### Force-field prediction is expected to be significantly better\nThe connection between the denoising objective and Boltzmann-distribution force fields is a good insight offered, and it certainly suggests that the model should yield large improvements in force-field prediction. However, the main text only offers one analysis on this, which is that pretraining on TorchMD-NET seems to improve the MAE of the predicted force field for a single molecule, aspirin. Appendix G offers some more results in improving force-field predictions comparing GNS with and without pretraining. The paper would be stronger if these results were included in the main text, and it would be even better if the authors could show significant improvements in force-field prediction with GNS (or GNS-TAT) with pretraining, similar to Table 1.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4K6JRLiXZN", "reviewer": "Reviewer_8vAk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a simple and efficient pretraining strategy, which predicts the added noise into 3D positions. The paper further shows the connection between denoising and molecular force field learning. Experiments demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "Overall, I think this is a good paper, and recommend accepting it.", "strength_and_weaknesses": "Strengths:\n- The proposed method is simple and effective\n- The analysis of the connection between denoising and force field is insightful\n- The experiment is comprehensive, and the results are impressive.\n\nWeaknesses & Questions:\n- Downstream applications. There are many molecules that do not have DFT 3D conformations, and thus cannot use the proposed 3D denoising task. Therefore, the proposed method seems cannot be used in many traditional molecular property prediction tasks, like MoleculeNet.  \n- Some experiment settings are unclear, for example, the data split setting for QM9.\n- In Sec.5.1, Only TorchMD-Net model is tested. Did you try some transformer-based architectures, like Graphormer?\n- will you open-source the model/code?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%