ATOM3D: Tasks On Molecules in Three Dimensions
Authors: Raphael John Lamarre Townshend
Martin Vogele
Patricia Suriana
Alex Derry
Alex Powers
Yianni Laloudakis
Sidhika Balachandar
Brandon M Anderson
Stephan Eismann
Risi Kondor
Russ Altman
Ron O. Dror
Venue:  (2021)
Keywords: machine learning
structural biology
biomolecules

Abstract: While a variety of methods have been developed for predicting molecular properties, deep learning networks that operate directly on three-dimensional molecular structure have recently demonstrated particular promise. In this work we present ATOM3D, a collection of both novel and existing datasets spanning several key classes of biomolecules, to systematically assess such learning methods. We develop three-dimensional molecular learning networks for each of these tasks, finding that they consistently improve performance relative to one- and two-dimensional methods. The specific choice of architecture proves to be critical for performance, with three-dimensional convolutional networks excelling at tasks involving complex geometries, while graph networks perform well on systems requiring detailed positional information. Furthermore, equivariant networks show significant promise but are currently unable to scale. Our results indicate many molecular problems stand to gain from three-dimensional molecular learning. All code and datasets are available at github.com/xxxxxxx/xxxxxx.

Under review as a conference paper at ICLR 2021
ATOM3D: TASKS ON MOLECULES IN THREE DIMENSIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
While variety of methods have been developed for predicting molecular prop- erties, deep learning networks that operate directly on three-dimensional molec- ular structure have recently demonstrated particular promise. In this work we present ATOM3D, a collection of both novel and existing datasets spanning sev- eral key classes of biomolecules, to systematically assess such learning meth- ods. We develop three-dimensional molecular learning networks for each of these tasks, finding that they consistently improve performance relative to one- and two- dimensional methods. The specific choice of architecture proves to be critical for performance, with three- direeddimensooal convolutional networks excelling at tasks involving complex geometries, while graph networks perform well on systems re- quiring detailed positional information. Furthermore, equivariant networks show significant promise but are currently unable to scale. Our results indicate many molecular problems stand to gain from threc-dimensional molecular learning. All code and datasets are available at figthu.... com/ XXXXXXXXXXXXXXXXXXXxXXxXXXXXXXXX XXXXXX
INTRODUCTION
A molecule's three-dimensional (3D) shape is critical to understanding its physical mechanisms of action, and can be used answer number of questions relating to drug discovery, molecular design, and fundamental biology. A molecule's 's atoms often adopt specific 3D configurations that minimize its free energy, and by representing these 3D positions- the atomistic geometry- can model this 3D shape in ways that would not be possible with 1D or 2D representations such as linear sequences or chemical bond graphs (Table T). However, existing works that examine diverse molecular tasks, such as MoleculeNet (Wu et al. 2018) or TAPE (Rao et al. (2019), focus on these lower dimensional representations. In this work, we demonstrate the benefit yielded by learning on 3D atomistic geometry and promote the development of 3D molecular learning by providing collection of datasets leveraging this representation.
Furthermore, argue that the atom should be considered "machine learning datatype" in its own right, deserving focused study much like images in computer vision or text in natural language processing. All molecules, including proteins, small molecule compounds, and nucleic acids, can be homogeneously represented as atoms in 3D space. These atoms can only belong to a fixed class of element types (e.g. carbon, nitrogen, oxygen), and are all governed by the same underlying laws of physics, leading to important rotational, translational, and permutational symmetries. These systems also contain higher-level patterns that are poorly characterized, creating a ripe opportunity for learning them from data: though certain basic components are well understood (e.g. amino acids, nucleic acids, functional groups), many others can not easily be defined. These patterns are in turn composed in hierarchy that itself is only partially elucidated.
While deep learning methods such as graph neural networks (GNNs) and convolutional neural net- works (CNNs) seem especially well suited to atomistic geometry, to date there has been no system- atic evaluation of such methods on molecular tasks. Additionally, despite the growing number of 3D structures available in databases such as the Protein Data Bank (PDB) (Berman et al., 2000), they require significant processing before they are useful for machine learning tasks. Inspired by the success of accessible databases such ImageNet (Jia Deng al. 2009) and SQuAD (Rajpurkar jet al.] 2016) in sparking progress in their respective fields, we create and curate benchmark datasets for atomistic tasks, process them into simple and standardized format, systematically benchmark

---

Under review as a conference paper at ICLR 2021
Table 1: Representation choice for molecules. Adding in 3D information consistently improves per- formance. The depicted ID representations are the amino acid sequence and SMILES (Weininger, 1988 for proteins and small molecules, respectively.
Examples
Structure Level
Dimension
Representation
Proteins
Small Molecules
primary
1D
linear sequence
KVKALPDA
CC(O)CC(C)OO
secondary
2D
chemical bond graph
oo
ooooo
0ogoogoogo8
tertiary
3D
atomistic geometry
3D molecular learning methods, and present a set of best practices for other machine learning re- searchers interested in entering the field of 3D molecular learning. We develop new methods for several datasets and reveal a number of insights related to 3D molecular learning, including the con- sistent improvements yielded by using atomistic geometry, the lack of single dominant method, and the presence of several tasks that can be improved through 3D molecular learning.
2 RELATED WORK
Three dimensional molecular data have long been pursued as an attractive source of information in molecular learning and chemoinformatics, but until recently have achieved underwhelming results relative to ID and 2D representations (Swamidass et al.) 20055 Azencott et al. 2007). However, due increases in data availability and methodological advances, machine learning methods based on 3D molecular structure have begun to demonstrate significant impact in the last couple of years on specific tasks such as protein structure prediction (Senior al., (020), equilibrium state sampling (Noé et al. 2019), and drug design (Zhavoronkov et al. 2019). While there have been some broader assessments groups of related biological tasks, these have focused on on either ID (Rao et al.) 2019) 2D (Wu et al. (2018) representations. By focusing instead atomistic geometry, we can consistently improve performance and address disparate problems involving any combination of small molecules, proteins, and nucleic acids through unified lens.
Graph neural networks (GNNs) have grown to be a major area of study, providing a natural way of learning from data with complex spatial structure. Many GNN implementations have been moti- vated by applications to atomic systems, including molecular fingerprinting (Duvenaud et al.) 2015). property prediction (Schütt al. 2017 Gilmer al. 2017 Liu et al., 2019), protein interface pre- diction (Fout et al. 2017). and protein design (Ingraham et al. (2019). Instead of encoding points in Euclidean space, GNNs encode their pairwise connectivity, capturing structured representation of atomistic data.
Three--iimesioonll CNNs (3DCNNs) have also become popular way capture these complex 3D geometries. They have been applied to a number of biomolecular applications such as protein interface prediction (Townshend et al. 2019), protein model quality assessment (Pagès al. 2019 Derevyanko et al. 2018). protein sequence design (Anand et al.] 2020), and structure-based drug discovery WWallach et al. 2015 Torng & Altman| 2017: Ragoza al. [017]] Jiiénez et al., 2018). These 3DCNNs can encode translational and permutational s symmetries, but incur significant com- putational expense and cannot capture rotational symmetries without data augmentation.
In attempt to address many of the problems of representing atomistic geometries, equivariant neu- ral networks (ENNs) have emerged as a new class of methods for learning from molecular systems. These networks are built such that geometric transformations of their inputs lead to well-defined transformations of their outputs. This setup leads to the neurons of the network learning rules that resemble physical interactions. Tensor field networks (Thomas al. 2018) and Cormorant (Kondor)
2

---

I23 is the position space and Under review as a conference paper at ICLR 2021
2018
Anderson et al. 2019) have applied these principles to atomic systems and begun to demon- strate promise on extended systems (Eismann et al., 22000; Weiler et al. 20188. However, in general, these methods have not been applied to larger-scale molecular tasks.
3D MOLECULAR LEARNING
We define 3D molecular learning as the set of tasks where the input space is atoms in three di- mensions. We write this space as AN where = x
E= (C, N, ...... is the element space.
We select 3D molecular learning tasks from structural biophysics and medicinal chemistry that span variety of molecule types and address range of important problems. Multiple of these datasets are novel, while others are extracted from existing sources (Table 2). We provide all datasets a standardized format that requires specialized libraries. Alongside these datasets, present responding best practices, including splitting and filtering criteria, to minimize data leakage concerns and ensure generalizability and reproducibility. Taken together. hope these efforts will lower the barrier to entry for machine learning researchers interested in developing methods for 3D molecular learning and encourage rapid progress in the field. Detailed descriptions of the preparation of each dataset can be found in Appendix C.I
Table 2: Tasks included in ATOM3D dataset, along with schematic representation of their inputs. P indicates protein, SM indicates small molecule, R indicates RNA. Lines indicate interaction and the smaller square within proteins indicates an individual amino acid. New datasets are in bold.
Name (Task Code)
Schematic
Objective
Source
Small Molecule Properties (SMP)
QM9
SM
Properties
Ruddigkeit al.
2012
Protein Interface Prediction (PIP)
Amino Acid Interaction
DIPS
Townshend al.
2019
P1
P2
DB5 (Vreven et al.
2013
Residue Identity (RES)
Amino Acid Identity
New, created from PDB (Berman et al. 2000)
P1
a P2
Mutation Stability Prediction (MSP)
Effect of Mutation
New, created from SKEMPI Jaankuskaiit et al. 2019)
P1
E P2
Ligand Binding Affinity (LBA)
Binding Strength
PDBBind Wang et al. 2004
SM
P
SM
Ligand Efficacy Prediction (LEP)
Drug Efficacy
New, created from PDB (Berman et 2000)
vs.
P
SM
Protein Structure Ranking (PSR)
CASP-QA Kryshtafovych et al. 2019
P
Ranking
RNA Structure Ranking (RSR)
PARZAR2-Puzzles Watkins & Das 2019
R
Ranking
3

---

Under review as a conference paper at ICLR 2021
3.1 SMALL MOLECULE PROPERTIES (SMP)
Impact- - Predicting physico-chemice properties of small molecules is a common task in medicinal chemistry and materials design. Quantum chemical calculations can save expensive experiments but are themselves costly and cannot cover the huge chemical space spanned by candidate molecules. Dataset The QM9 dataset (Ruddigkeit et al., 2012; Ramakrishnan al. (014) contains structures and energetic, electronic, and thermodynamic properties for 134,000 stable small organic molecules, obtained from quantum-chemical calculations.
Metrics We predict the molecular properties from the ground-state structure. Split We split molecules randomly.
3.2 PROTEIN INTERFACE PREDICTION (PIP)
Impact- Proteins interact with each other in many scenarios- for example, antibody proteins rec- ognize diseases by binding antigens. critical problem in understanding these interactions is to identify which amino acids of two given proteins will interact upon binding.
for
Dataset For training, we use the Database of Interacting Protein Structures (DIPS), comprehen- sive dataset of protein complexes mined from the PDB (Townshend et al. (2019). We predict on the Docking Benchmark (Vreven et al. 2013), smaller gold standard dataset. Metrics We predict if two amino acids will come into contact when their respective proteins bind.
Split - We split protein complexes by sequence identity at 30%.
3.3
RESIDUE IDENTITY (RES)
Impact- - Understanding the structural role of individual amino acids is important for engineering new proteins. We understand this role by predicting the propensity for different amino acids at a given protein site based on the surrounding structural environment (Torng & Altman 2011).
Dataset We generate novel dataset consisting of atomic environments extracted from non- redundant structures in the PDB.
Metrics We formulate this classification task where we predict the identity of the amino acid in the center of the environment based on all other atoms. Split- We split residue environments by protein topology class.
3.4 MUTATION STABILITY PREDICTION (MSP)
Impact Identifying mutations that stabilize a protein's interactions is key task in designing new proteins. Experimental techniques for probing these are labor-intensive (Antikainen & Martin 2005
Lefevre et al.) 1997), motivating the development of efficient computational methods. Dataset- We derive novel dataset by collecting single-point mutations from the SKEMPI database Jankauskaite et al. 2019) and model each mutation into the structure to produce mutated structures.
Metrics We formulate this as a binary classification task where predict whether the stability of the complex increases as a result of the mutation.
Split We split protein complexes by sequence identity 30%.
3.5 LIGAND BINDING AFFINITY (LBA)
Impact- Most therapeutic drugs and many molecules critical for biological signaling take the form of small molecules. Predicting the strength of the protein-smal) molecule interaction is a challenging but erucial task for drug discovery applications.
Dataset We use the PDBBind database (Wang et al.) 2004 Liu et al. 2015), curated database containing protein-ligand complexes from the PDB and their corresponding binding strengths. Metrics - We predict yk = log(K), where K is the binding affinity in Molar units.
Split We split protein- -igead---iaaddd complexes by protein sequence identity at 30%.
3.6 LIGAND EFFICACY PREDICTION (LEP)
Impact- - Many proteins switch on or off their function by changing shape. Predicting which shape drug will favor is thus important task in drug design.
4

---

Under review as a conference paper at ICLR 2021
Dataset We develop novel dataset by curating proteins from several families with both "active" and "inactive" state structures, and model in 527 small molecules with known activating or inacti- vating function using the program Glide (Friesner al. 2004).
Metrics - We formulate this as a binary classification task where we predict whether or not a molecule bound to the structures will be activator of the protein's function not. Split - We split complex pairs by protein.
3.7 PROTEIN STRUCTURE RANKING (PSR)
Impact- Proteins are one of the primary workhorses of the cell, and knowing their structure is often critical to understanding (and engineering) their function.
Dataset The Critical Assessment of Structure Prediction (CASP) (Kryshtafovych et al. 2019 is
blind international competition for predicting protein structure.
Metries- We formulate this as a regression task, where we predict the global distance test (GDT. TS)
from the true structure for each of the predicted structures submitted in the last 18 years of CASP. Split We split structures temporally by competition year.
3.8 RNA STRUCTURE RANKING (RSR)
Impact- - Similar to proteins, RNA plays major functional roles (e.g., gene regulation) and can adopt well-defined 3D shapes. Yet the problem is data-poor, with only a few hundred known structures. Dataset Candidate models generated by FARFAR2 (Watkins & Das 2019) for the first 21 released RNA Puzzle challenges (Cruz et al. 2012). blind structure prediction competition for RNA. Metrics We predict the root-mean- noo--mea--sqaree deviation (RMSD) from the ground truth structure. Split- We split structures temporally by competition year.
4 EXPERIMENTAL SETUP
To assess the benefits of 3D molecular learning, we use combination of existing and novel 3D molecular learning methods, and implement number of robust baselines. Our 3D) molecular learn- ing methods belong to one of each of the major classes of deep learning algorithms that have been applied to atomistic systems: graph networks, ariant networks. Here we describe the core networks and the novel extensions needed to adapt them to certain datasets. See Appendix C.2 for task-specific details and hyperparameters.
dimeeniiinii convolutional networks, and equiv-
4.1 CORE NETWORKS
For GNNs, we represent molecular systems as graphs in which each node is an atom. Edges are defined between all atoms separated by less than 4.5 Å, and weighted by the distance between the atoms. Node features are one-hot-encoded by atom type. Our core model uses five layers of graph convolutions, each followed by batch normalization and ReLU activation, sum pooling layer, and two fully-connected layers with dropout.
For 3DCNNs, we represent our data as cube of fixed size (different per task due the different molecular sizes) in 3D space that is discretized into voxels with resolution of 1 Á to form grid. Each voxel is associated with a one-hot-encoded vector which denotes the presence or absence of each atom type. Our core model consists of four 3D-convolutional layers, each followed by max- pooling, dropout, and ReLU activation, and two fully-connected layers.
For ENNs, we use SE(3)-equivariant networks that represent each atom of structure by its position as absolute coordinates 3D space with one-hot-encoded atom type as features. No rotational augmentation is needed due the rotational symmetry of the network. The core of all architectures in this work a network of four layers of covariant neurons that use the Clebsch- Gorcdan transform nonlincarity, described and implemented in Anderson et al. (2019).
4.2 STAMESE ARCHITECTURES
For tasks involving comparing two sets of atoms sampled from the same distributions, we also de- velop new architectures that Siamese in nature. Specifically, the PIP dataset involves predicting
5

---

Under review as a conference paper at ICLR 2021
symmetric interaction between two proteins, while the MSP and LEP datasets involve symmetric comparison between two interactions. Taking inspiration from Townshend et al. (2019)'s 's use of a Siamese 3DCNN network for the PIP dataset, we replicate that architecture for our PIP, MSP, and LEP datasets, and develop new Siamese GNN and Siamese ENN networks. Specifically, we train pair of core networks with tied weights, ensuring symmetric treatment of both items of the pair. We then combine the final learned embeddings from both core networks to output final prediction. Beyond the novelty of weight-tying atom-level GNNs and ENNs, our knowledge this is also the first use of weight-tying across SE(3) -equivarnaa networks.
4.3 AMINO ACID OUTPUTS
Certain tasks involve making a prediction specific amino acid (PIP, RES, and MSP: see Table 2) yet GNNs and ENNs typically rely on summing over all node embeddings to compute a final graph embedding, making it difficult isolate this amino acid. To remedy this, after our convolutional layers we implement the novel procedure of extracting the embedding of only the Ca atom of the amino acid in question, thereby allowing our GNNs and ENNs isolate it.
5 RESULTS
To assess the utility of 3D molecular learning, we evaluate our methods on the ATOM3D datasets and compare performance to state-of-the-ar methods using ID or 2D representations. We stress that in many cases, 3D molecular learning methods have never been applied to the proposed tasks, and that several of the tasks are novel. In the following sections, we describe the results of our benchmarking and some key insights that can be derived from them. We also aggregate these results along with additional metrics and standard deviations over three replicates in Appendix
5.1 3D REPRESENTATION CONSISTENTLY IMPROVE PERFORMANCE
Our evaluation of 3D methods on the tasks in ATOM3D reveals that incorporating atomistic ge- ometry leads to consistently superior performance compared to 1D and 2D methods. For small molecules, state-of tf-the-rr methods do not use ID representations, so we focus instead on compar- ing to representations at the 2D level, i.e. the chemical bond graph. This is the approach taken by the 2D GNN introduced by TTsubaki et al. (2019) or the N-gram graph method by Liu et al. (2019), which both obtain similar results (Table 3) on the omall-molecule-only dataset SMP. When add 3D distance, done for our GNN model, we improve performance across all targets in SMP (Table 3)
For tasks involving biopolymers (proteins and RNA), state-of-the-art methods do not use 2D rep- resentations, primarily because most of the chemical bond graph can be easily re-derived from the 1D representation, i.e. the linear sequence that makes up the biopolymer. We thus compare to rep- resentations at the ID level (Table [7 For MSP and RES, both new datasets, we evaluate against Rao et al. (2019)'s 's TAPE model, transformer architecture that operates on protein sequence and is statc-of-the- version of BIPSPI Mauchez-Garce
amongst ID methods for many tasks. For PIP, we compare to the sequence-only
al., 2018), state-of-the-art boosted decision tree method for protein interaction prediction. We find that 3D methods outperform these ID methods on all biopolymer-only datasets (PIP, RES, MSP).
For tasks involving both biopolymers and small molecules, we compare DeepDTA (özüürk et al. 2018). This network uses a ID representation via a IDCNN for both the biopolymer and small molecules. For LBA, we additionally compare to DeepAffinity (Karimi et al. (2019) which uses pairs of ligand SMILES string and novel representation of structurally-annotea Using 3D representation for both ligand and protein instead leads to improved performance for the joint pootein-small molecule datasets (LBA and LEP, see T5)
protein sequences.
The biopolymer structure ranking tasks (PSR and RSR) are inherently 3D in nature, they involve evaluating the correctness of different 3D shapes taken on by the same biopolymer. Thus, critically, ID or 2D representation would not be able to differentiate between these different shapes since the linear sequence and chemical bond graph would remain the same. We therefore compare to ofth--oftthe-rtt 3D methods as shown in Table G

---

Under review as a conference paper at ICLR 2021
Table 3: Small molecule results. Metric is mean absolute enror (MAE).
3D
Non-3D
Task
Target
3DCNN
GNN
ENN
Tsubaki (0119)
et al. 2019)
JA [D]
0.572
0.068
0.046
0.496
0.520
SMP
Egap [eV]
0.589
0.091
0.065
0.154
0.184
U [eV]
1.615
0.070
0.023
0.182
0,218
Table 4: Biopolymer results. AUROC is the area under the receiver operating characteristic curve. Asterisks (*) indicate that the exact training data differed (though splitting criteria were the same).
3D
Non-3D
Task
Metric
3DCNN
GNN
ENN
Sanchez-Gareia al. 2018
PIP
AUROC
0.844
"0.669
0.841
Rao 2019)
RES
accuracy
0.451
0.082
*00072
*033
MSP
AUROC
0.520
0.637
0.678
0.554
Table 5: Joint small molccule/bipolyymer results. Rs is Spearman correlation. Rp Pearson correlation, AUROC area under the receiver operating character- istic curve, and RMSD is root-mean squaree deviation. Asterisks (*) indicate that the exact training data differed (though splitting criteria were the same).
3D
Non-3D
Task
Metric
3DCNN
GNN
ENN
Öztürk al. 2018)
Karimi et al. 2019
LBA
RMSD
1.520
1.936
$1,429
1.565
1.893
glob. Rp
0.558
0.581
*0.541
0.573
0.415
glob. Rss
0.556
0.647
*0.532
0.574
0.426
LEP
AUROC
0.824
0.678
0.569
0.696
Table 6: Structure ranking results. Rs is Spearman correlation, Rp is Pearson correlation. Mean measures the correlation for structures corresponding to the same biorolymer, whereas global measures the correlation across all biopolymers.
Task
Metric
3DCNN
GNN
SotA
PSR
mean Rs
0.177
0.327
0.432
Pagès al.
2019
glob. Rs
0.837
0.716
0.796
Pages
2019
RSR
R.s
0.414
0.195
0.173
Alford et al.
2017
glob. As
0.656
0.309
0.304
AAlfrd et al.
2017

---

Under review as a conference paper at ICLR 2021
More generally, we find that learning methods that leverage the 3D geometry of molecules hold state-of-the-art on all tasks on our benchmark (Appendix()) D)
5.2 MANY 3D MOLECULAR LEARNING PROBLEMS REMAIN UNDEREXPLORED
As demonstrated the previous section, formulating molecular problem through the lens of 3D molecular learning can lead to significantly improved performance. However, many important prob- lems have not been studied within this framework, leaving significant room for further improvement. This opens up a ripe field of research with much low-hand hanging fruit. One prominent example we plore here is RNA structure ranking, where the state-of of--hh--trtt method uses Rosetta (Alford et al.] 2017). hand-designed potential energy function. When we instead apply our 3DCNN method that learns directly from the 3D atomistic geometry, we see dramatic increases in performance (Table
In similar vein, the ligand efficacy prediction task find that the 3DCNN method outperforms Glide (Friesner et al. 2004). statc-of- offt---rrtt--ttttt scoring function for protein-small molecule docking. The 3DCNN achieves an AUROC of 0.824, compared to Glide's 0.770.
We also find room for improvement in domains where 3D molecular learning is already being em- ployed. Protein structure ranking one such area, and we see that the 3DCNN model is competitive with the -fate-of-the--ee art deep learning method by Pagès et al. (2019) (Table surassing it in terms of absolute assessment of correctness (i.e., comparing 3D candidates from different biopolymers) though not in terms of relative assessment (i.e., comparing 3D candidates from the same biopoly-
mer).
Overall, these results demonstrate the potential of 3D molecular learning to address wide range of problems involving molecular structure, and we anticipate that continued development of such models on less well-studied tasks will aid progress in biomedical research.
5.3 DIFFERENT TASKS REQUIRE DIFFERENT ARCHITECTURES
While atomistic methods consistently outperform their non-3D counterparts and provide a system- atic way of representing molecular data, our results also provide evidence that architecture selection plays critical role in performance. For tasks primarily focused on small molecules (SMP and LBA), we see superior performance from the particle-based methods (GNN and ENN) than from the vol- umetric 3DCNN. Small molecules are already intuitively represented as graph-like structures, with nodes (atoms) and edges (bonds), and the quantities computed in these small molecule tasks depend less complex 3D geometry and more on the exact position of each atom relative to its neighbors. Unlike particle-based methods, 3DCNNs must approximate these positions, and while increasing spatial resolution increases precision, it also leads cubic scaling of complexity.
On the other hand, for larger molecules and complex 3D geometries that are critical to tasks like PSR, RSR, LEP, and RES, we see that the 3DCNN method outperforms GNNs. Here, the 3DCNN's ability to directly represent differences between patterns in 3D space, as opposed to trying to reconstruct them through pairwise distances, is likely what allows it to perform well. In these tasks, the exact position and relational information between atoms is less important than their overall conformation.
Finally, equivariant networks show promise but suffer from scalability issues. One motivation for the development of equivariant networks is that they fill a "happy medium" where they can both represent atom positions precisely and capture complex geometries. On some of the tasks where we could test the ENN (LBA, SMP), we often observed close to tate-of-thhear performance, even on a reduced training set (LBA). Unfortunately, current implementations do not yet scale to most of our tasks due to the compute- and memory-intensive nature of the Clebsch-Gordan products used to maintain rotational equivariance. For some tasks, the performance was severely limited by only training on fraction of the data (< 1% for RES) portion of the entire atomic structure (LEP), and for others we could not apply the ENN at all. These limitations point to the need for further architectural innovations before their performance can be demonstrated extended systems.
8

---

Under review as a conference paper at ICLR 2021
CONCLUSION
In this work we present a vision of 3D atom-level data as a new "machine learning datatype" deserv- ing focused study. Atomistic data shares several underlying symmetries, contains poorly understood higher-level patterns, and can be used to address many high-impact but unsolved problems.
We create several benchmark datasets and compare the performance of different types of 3D molec- ular learning models across these tasks. Many of these architectures were developed specifically for the tasks in question, such as the Siamese ENN and GNN models used for paired tasks (PIP, MSP, and L.P)). For tasks that can be formulated in lower dimensions, we demonstrate that 3D molecular learning yields consistent gains in performance over 1D and 2D methods. We also show that selec- tion of an appropriate architecture is critical for optimal performance on given task; depending on the structure of the underlying data, a 3DCNN, GNN, or ENN may be most appropriate. As equiv- ariant networks continue to improve in efficiency and stability, we expect these become more and more viable due to their close modeling of physical laws.
While ATOM3D establishes first set of benchmark datasets, there many other open in biomedical research and molecular science that are ripe for 3D molecular learning, especially as structural data becomes readily available. Such tasks include virtual screening and pose prediction of small molecule drugs, or the prediction of conformational ensembles instead of static structures. The use of multiple 3D conformations per molecule represents an especially promising direction, as they would more faithfully reproduce the entire set of states given molecule could adopt. As such, envision expanding the ATOM3D framework beyond the tasks described here.
Through this work, we hope to lower the entry barrier for machine learning practitioners, encourage the development of machine learning algorithms focused on 3D atomistic data, and promote novel paradigm within the fields of structural biology and medicinal chemistry.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rebecca F. Alford, Andrew Leaver-Fay, Jeliazko R. Jeliazkov, Matthew J. O'Meara, Frank P. Di- Maio, Hahnbeom Park, Maxim V. Shapovalov, P. Douglas Renfrew, Vikram K. Mulligan, Kalli Kappel, Jason W. Labonte, Michael S. Pacella, Richard Bonneau, Philip Bradley, Roland L. Dun- brack, Rhiju Das, David Baker, Brian Kuhlman, Tanja Kortemme, and Jeffrey J. Gray. The Rosetta All-Atom Energy Function for Macromolecular Modeling and Design. Journal of Chemical The- ory and Computation, 13(6):3031-333 -004-00888818 2017. ISSN 15499626. doi: 10. 1a..02//acs..cttcc0000000.
S F Altschul, W Gish, W Miller, E W Myers, and D J Lipman. Basic local alignment search tool. Journal of Molecular Biology, 215(3)403-10, 1990. ISSN 0022-2836. doi:
URLhttp:: /ttt:://ww... www.eneeeeeetttcooooo.. wttp://rweet.cccccc./eeeeoee
atticle/s11/0002288333333
Namrata Anand, Raphael R Eguchi, Alexander Denr, Russ B Altman, and Po-Ssu Huang. Protein sequence design with a learned potential. January 2020.
Brandon Anderson, Truong-Son Hy, and Risi Kondor. Cormorant: Covariant Molecular Neural Networks. 2019. URL http::ttpaarx:t http://aaaxxv.ooaaa///eease 04015,
Antonina Andreeva, Dave Howorth, Cyrus Chothia, Eugene Kulesha, and Alexey G. Murzin. SCOP2 prototype: A new approach to protein structure mining. Nucleic Acids Research, 42 (D1):310-33) 314, 2014. ISSN 03051048. doi: 10. 10.1093/nar/gk1122...
Nina M Antikainen and Stephen Martin. Altering protein specificity: techniques and applications. Bioorganic & medicinal chemistry, 13(8):2701-2116 2005.
Chloé-Agathe Azencott, Alexandre Ksikes, S Joshua Swamidass, Jonathan Chen, Liva Ralaivola, and Pierre Baldi. One-to four-dimensional kernels for virtual screening and the prediction of physical, chemical, and biological properties. Journal of chemical information and modeling, 47 (3):965-974, 2007.
Pedro Ballester and John O Mitchell. A machine learning approach to predicting protein-ligand binding affinity with applications to molecular docking. Bioinformatics, 26(9):1169-1175 May 2010.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Πlya N Shindyalov, and Philip Bourne. The protein data bank. Nucleic acids research, 28(1): 235-242, 2000.
Wouter Boomsma and Jes Frellsen. Spherical convolutions and their application in molecular mod- elling. In I Guyon, U V Luxburg. S Bengio, H Wallach, R Fergus, S Vishwanathan, and R Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3433-3443, Curran Asso- ciates, Inc., 2017.
Sidhartha Chaudhury, Sergey Lyskov, and Jeffrey Gray. Pyrosetta: script-based interface for im- plementing molecular modeling algorithms using rosetta. Bioinformatics, 26(5))689,691, 2010.
Tianqi Chen and Carlos Guestrin. Xghoost: scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Min- ing, KDD '16, pp. 785-794, New York, NY, USA, 2016. Association for Computing Machin- ery. ISBN 9781450342322. doi: 00.1145/2999672.29377777 URL https:::t//iiccooe/ 1145/2939672.29388
Jianlin Cheng, Arlo Randall, Michael Sweredoski, and Pierre Baldi. Scratch: protein structure and structural feature prediction server. Nucleic acids research, 33:W72-6,, 08 2005. doi: 10.1093/ nar/gki396.
José Almeida Cruz, Marc-Frédérick Blanchet, Michal Boniecki, Janusz M Bujnicki, Shi-Jie Chen, Song Cao, Rhiju Das, Feng Ding, Nikolay V Dokholyan, Samuel Coulbourn Flores. et al. Rna- puzzles: a casp-like evaluation of ma diree-dimensional structure prediction. Rna, 18(4):610-625, 2012.
10

---

Under review as a conference paper at ICLR 2021
Natalie L Dawson, Tony E Lewis, Sayoni Das, Jonathan G Lees, David Lee, Paul Ashford, Chris- tine Orengo, and lan Sillitoe. Cath: an expanded resource to predict protein function through structure and sequence. Nucleic acids research, 45(D1):D29--2295, 2017.
Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. Deep convolu- tional networks for quality assessment of protein folds. Bioinformatics, 34(23))4046-4053, 2018.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzin and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances in neural information processing systems. pp. 2224-2232. 2015.
Stephan Eismann, Raphael J L Townshend, Nathaniel Thomas, Milind Jagota, Bowen Jing, and Ron Dror. Hierarchical, rotation- equtiiin-quuvariantt neural networks to predict the structure of protein complexes. arXiv, 2020.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur. Protein interface prediction using graph convolutional networks. In Guyon, U V Luxburg, S Bengio, H Wallach, R Fergus, S Vish- wanathan, and R Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6530 6539. Curran Associates, Inc., 2017.
Richard A Friesner, Jay L Banks, Robert B Murphy, Thomas A Halgren, Jasna Klicic, Daniel Mainz, Matthew Repasky, Eric H Knoll, Mee Shelley, Jason K Perry, al. Glide: new approach for rapid, accurate docking and scoring. 1. method and assessment of docking accuracy. Journal of medicinal chemistry, 47(7):1739-17493 2004.
M. V. Shapovalov G. G. Krivov and R. L. Dunbrack. Improved prediction of protein side-chain conformations with scwrl4. Proteins, 2009. URL php/]
fecc. hew.aabbaack..ccc..ee/////.....
Joffrey Gabel, Jérémy Desaphy, and Didier Rognan. Beware of machine learning-based scoring
functions-on the danger developing black boxes. J. Chem. Inf. Model., 54(10))2807-2815, October 2014.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry, 2017.
Jie Hou, Renzhi Cao, and Jianlin Cheng. Deep convolutional neural networks for predicting the quality of single protein structural models. bioRxiv, 2019. doi: 10. 1011/90000.. URL https:
wwww www..biorxie
John Ingraham, Vikas K Garg. Regina Barzilay, and Tommi Jaakkola. Generative models for Graph- Based protein design. March 2019.
Justina Jankauskaité, Brian Jiménez-Garcin Justas Dapkūnas, Juan Fernández- and Iain H Moal. Skempi 2.0: an updated benchmark of changes in protein-protein protein binding energy, kinetics and thermodynamics upon mutation. Bioinformatics, 35(3):62-4463 2019.
Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierar- chical image database. pp. 248-255, 2009. doi: 10. (09/ccpr..200955266.22
José Jiménez, Miha Škalič, Gerard Martinez-Rosell, and Gianni De Fabritii.. KDEEP: Protein- Ligand absolute binding affinity prediction via 3D-Convolutional neural networks. Chem. Inf. Model., 58(2):287-22) February 2018.
Robbie Joosten, Tim Beek, Elmar Krieger, Maarten Hekkelman, Rob Hooft, Reinhard Schneider, Chris Sander, and Gert Vriend. A series of pdh related databases for everyday needs. Nucleic acids research, 39:D411-9, 11 2010. doi: 10. 10.1093/nar/gkq/ 105.
Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: Pattern recog- nition of hydrogen- bonded and geometrical features. Biopolymers, 22(12):57772263, 1983. ISSN 1097-0282. doi: 10.10022bip.36022221.. URL http:://dx www..ooooo.....ooooo/
bip. 360221211

---

Under review as a conference paper at ICLR 2021
Mostafa Karimi, Di Wu, Zhangyang Wang, and Yang Shen. Deepaffinity: interpretable deep learn- ing of compound-prost affinity through unified recurrent and convolutional neural networks. Bioinformatics, (1118)))329-333333 2019.
Thomas N Kipf and Max Welling. Supi-super classification with graph convolutional net- works. arXiv preprint arXiv: 1609.029077 2016.
Risi Kondor. N-body networks: covariant hierarchical neural network architecture for learning atomic potentials, 2018.
Christian Kramer and Peter Gedeck. Leave-cluster-out cross-validation is appropriate for scoring functions derived from diverse protein data sets. J. Chem. Inf. Model., 500 1):1961-169, Novem- ber 2010.
Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Critical assessment methods of protein structure prediction (casp)-round xiii. Proteins: Structure, Function, and Bioinformatics, 87(12):1011-1022 2019.
Andrew Leaver-Fay, Michael Tyka, Steven Lewis, Oliver Lange, James Thompson, Ron Jacak, Kris- tian Kaufman, P. Renfrew, Colin Smith, Will Sheffler, Ian Davis, Seth Cooper, Adrien Treuille, Daniel Mandell, Florian Richter, Yih-En Ban, Sarel Fleishman, Jacob Corn, David Kim, and Philip Bradley. Rosetta3: an object-oriented software suite for the simulation and design of macromolecules. Methods in enzymology, 487:545-74, 01 2011.
Fabrice Lefèvre, Marie-Hélène Rémy, and Jean-Michel Masson. Alanine-stretch scanning muta- genesis: simple and efficient method probe protein structure and function. Nucleic acids research, 25(2):4474448 1997.
Yan Li, Zhihai Liu, Jie Li, Li Han, Jie Liu, Zhixiong Zhao, and Renxiao Wang. Comparative assessment scoring functions on an updated benchmark: compilation of the test set. J. Chem. Inf. Model., 54(6):1700-1166 June 2014.
Yang Li and Jianyi Yang. Structural and sequence similarity makes significant impact on machine- learning-based scoring functions for cootein-ligan interactions. Journal of Chemical Information and Modeling, 57( 1007-1012, 2017.
Shengchao Liu, Mehmet F Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised repre- sentation for graphs, with applications to molecules. In H. Wallach, H. Larochelle, A. Beygelz- Fox, and Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8464 8476. Curran Associates, Inc., 2019.
imer, F. d' (Alché-Buc,
Zhihai Liu, Yan Li, Li Han, Jie Li, Jie Liu, Zhixiong Zhao, Wei Nie, Yuchen Liu, and Renxiao Wang. PDB-wide collection of binding data: current status of the PDBbind database. Bioinformatics, 31 (3)4055442, February 2015.
Christophe Magnan and Pierre Baldi. Sspro/acepro 5: Almost perfect prediction protein sec- ondary structure and relative solvent accessibility using profiles, machine learning, and structural similarity. Bioinformatics (Oxford, England), 30:2592 97, 05 2014. doi: 10. oo.10///iooiformmaiic/// btu352.
Simon Mitternacht, FreeSASA: open source library for solvent accessible surface area caleu- lations. F F1000Research, 5(189), 2016. ISSN 1759796X. doi: 10. 2668//1000rresarcch 7.311.1
Frank Noé, Simon Olsson, Jonas Köhler, and Hao Wu. Boltzmann generators: Sampling cquilibrium states of many-body systems with deep learning. Science, (6457), 2019. ISSN 0036-8075. doi: 00.1126/science.aav 147. URL htttss::/sccieeee sc www.scccinnnaanm.. org content/
365/ /457//4a77waa/
Hakime Öztürk, Elif Ozkirimli, and Arzucan Özgür. DeepDTA: Deep Drug-Target Binding Affinity Prediction. pp. 1-17, 2018. / wwwprrwwww....ve.ee.oo.e abs /10//1801111 101 93
//arxiv///xxv...
Guillaume Pagès, Benoit Charmettant, and Sergei Grudinin. Protein model quality assessment using 3d oriented convolutional neural networks. Bioinformatics, 35(18):3313-3319 2019.
12

---

Under review as a conference paper at ICLR 2021
Matthew Ragoza, Joshua Hochuli, Elisa Idrobo, Jocelyn Sunseri, and David Ryan Koes. Protein- Ligand scoring with convolutional neural networks. J. Chem. Inf. Model., 57(4):942 957, April 2017.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuad: 100,000+ questions for machine comprehension text. EMNLP 2016- Conference on Empirical Methods Natural Language Processing, Proceedings, (i):2333-2339) 2016.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp. and O. Anatole Von Lilienfeld. Quan- tum chemistry structures and properties of I 134 kilo molecules. Scientific Data, 1:1-7, 2014. ISSN 20524463. doi: 1...0//sdaaa.204.2222
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue Buce E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 32, pp. 9689- 9701. Curran Associates, Inc., 2019. URL http::/
nips cc/paper/ 9163 eva evaluating protein transfer Learning with tape.
Lars Ruddigkeit, Ruud van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 Billion Organic Small Molecules in the Chemical Universe Database GDB-17. Journal of Chemical Information and Modeling, (2(11):2864-222 2875, nov 2012. ISSN 1549-9596. doi: 10.
1021/ci300415d. URL https:: /wi. org 1021 /tt////c130411100
Ruben Sanchez-Garre Garcia, C O S Sorzano, J M Carazo, and Joan Segura. method for the prediction of partner-specific protein-protein interfaces. Bioinformat-
BIPSPI:
ics, (5(14):343-353, 2018. ISSN 1367-4803. doi: 10. 10.199//hointommatisshhyy//
URL 1t..1///..oinfommaaii///////////
advance art.nneeerticcl/ooil..
10.1093/
Soumya Sanyal, Ivan Anishchenko, Anirudh Dagar, David Baker, and Partha Talukdar. Proteingen: Protein model quality assessment using graph convolutional networks. bioRxiv, 2020. doi: 10. 1101/2020.04 00..0000......000000 06.22266. URL http::://wwwwwbe.. wwwpxxi....orooooooooooooooooo wttp:///oooeee//rrr///00000
04/07/2020 04. 06. 028266
Schrödinger, LLC. The PyMOL molecular graphics system, version 1.8. November 2015.
Kristof T. Schütt, Pieter-Jan Kindermans, Huziel E. Sauceda, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions, 2017.
Andrew W Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Žídek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, pp. 1-5, 2020.
Minyi Su, Qifan Yang, Yu Du, Guoqin Feng. Zhihai Liu, Yan Li, and Renxiao Wang. Comparative assessment of scoring functions: The CASF-2016 update. J. Chem. Inf. Model., 59(2):895-913, February 2019.
S Joshua Swamidass, Jonathan Chen, Jocelyne Bruand, Peter Phung, Liva Ralaivola, and Pierre Baldi. Kemels for small molecules and the prediction of mutagenicity, toxicity and anti-cancer activity. Bioinformatics, 21(suppl 1):1359-i368. 2005.
Nathaniel Thomas, Tess Smidt, Steven M. Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equiivrriant neural networks for 3d point clouds. CoRR, abs/
2018. URL http::://axx...crr/aaaa..a 08211
Wen Tomg and Russ B Altman. 3D deep convolutional neural networks for amino acid environment similarity analysis. BMC Bioinformatics,
June 2017.
Raphael Townshend, Rishi Bedi, Patricia Suriana, and Ron Dror. End-to-end learning on 3d protein
structure for interface prediction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bur Buch E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 15642- 15651, 2019.
13

---

Under review as a conference paper at ICLR 2021
Masashi Tsubaki, Kentaro Tomii, and Jun Sese. crmpound-protelde interaction prediction with end- to-end learning of neural networks for graphs and sequences. Bioinformatics, 35(2):309-315 2019. ISSN 14602059, doi: 10. 10.1003/boinfoomattt//br55533
Karolis Uziela, David Menéndez Hurtado, Nanjiang Shu, Bjöm Wallner, and Arne Elofsson. ProQ3d: improved model quality assessments using deep learning. Bioinformatics, pp. btw819, jan 2017. doi: h0.100//boonformaicssvv////
URL attps///ee..org/100.10
2ebioinformaiics2222..
Thom Vreven, lain H. Moal, Anna Vangone, Brian G. Pierce, Panagiotis L. Kastritis, Mieczys- law Torchala, Raphael Chaleil, Brian Jiménez-García, Paul A. Bates, Juan Fernandez-Recin Alexandre MJJ.. Bonvin, and Zhiping Weng. Updates to the integrated protein prottnn inter- action benchmarks: Docking benchmark version 5 and affinity benchmark version 2. Jour- nal of Molecular Biology, 427(19):3031 3041, 2015. ISSN 0022-2836. doi: https://doi. org/10. h10p:0000000.....000000000000000 Aaticcee/s1/000022885500000
URL
www.wsciee.. wwt.://ciieeeedeeeeee......ooooe comp. cww.cooocccceooooo ence/
Izhar Wallach, Michael Dzamba, and Abraham Heifets. Atomnet: A deep convolutional neural network for bioactivity prediction in structure-based drug discovery, 2015.
Renxiao Wang. Xueliang Fang, Yipin Lu, and Shaomeng Wang. The pdbbind database: collection of binding affinities for proteinligand complexes with known three-dimensional structures. Journal
of Medicinal Chemistry, (7(12))2977-2980) 2004. doi: 10. 10.102//j0330580. URL https:
//doi. www.oorr/.........
//doi.org/0..1111 1 021 /0000000000000001
PMID: 15163179.
Andrew M Watkins and Rhiju Das. Farfar2: Improved de novo rosetta prediction of complex global ma folds. bioRxiv, pp. 764449, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D steerable
CNNs: Learning rotationally equivariant features in volumetric data. July 2018.
David Weininger. SMILES, Chemical Language and Information System: 1: Introduction to Methodology and Encoding Rules. Journal of Chemical Information and Computer Sciences, 28 (1):31-36, 1988. ISSN 00952338. doi: 10. 10110//00005:
Zhengin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learn- ing. Chem. Sci., 9:513-530, 2018. doi: 10. 10.1039/C7SCU22644, URL http:://d doi org/ 10. 70.1039/C7Sc026644
Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins, 57:702-10, 09 2007. doi: 10.1002/prot.20244. URL https://
zhanglab. ccmb med. umi edu/ TM- score/
Alex Zhavoronkov, Yan A. Ivanenkov, Alex Aliper, Mark S. Veselov, Vladimir A. Aladinskiy, Anas- tasiya V. Aladinskaya, Victor A. Terentiev, Daniil A. Polykovskiy, Maksim D. Kuznetsov, Arip Asadulaev, Yury Volkov, Artem Zholus, Rim R. Shayakhmetov, Alexander Zhebrak, Lidiya Minaeva, Bogdan A. Zagribelnyy, Lennart H. Lee, Richard Soll, David Madge, Li Xing, Tao Guo, and Alán Aspuru-Guzik. Deep learning enables rapid identification of potent DDR1 ki- nase inhibitors. Nature Biotechnology, doi /ttp://tpp.//aa...o..oo/////////
10400 2019. ISSN 15461696. doi: 10.1038/
x415877011--224-x.
David Zilian and Christoph A Sotriffer. SFCscore(RF); a random forest-based scoring function for improved affinity prediction of protein-ligand complexes. J. Chem. Inf. Model., 1328)19923, August 2013.
14


Reviews:
[{"review_id": "j8S72pGxZY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is concerned with 3D molecule learning. They propose a collection of existing and new datasets (curated from existing datasets). They show that a lot of existing tasks can do well when 3D structure is considered. \n\nThe main contribution of the paper is in curating the datasets into a well defined framework with consistent splits and evaluation metrics. This would allow the community members to easily benchmark their approaches against a variety of tasks. Also, the tasks encompass a range of modalities including RNA, Proteins and Small molecules. Some tasks like Ligand Binding Affinity (LBA) and Ligand Efficacy Prediction (LEP) requires modeling representations of both proteins and small molecules. I believe this benchmark dataset will be useful to the community.\n\nI have some complaints with the experimental evaluations. For SMPs, a variety of unsupervised feature extractions methods have been proposed, which can be applied on 1D (SMILES representations) as well as graphs. For example see [1] and [2]. In QM9, N-Gram XGB (See table 2 from [2]), performs very well (top 1 performance on 9 out of 12 tasks). Did adding 3D information improve on this result? It is not clear. \n\nSimilarly, for binding affinity prediction, the authors compare with DeepDTA. However, DeepAffinity[3] has shown promising results just using 1D representations. Are 3D models better than this? What happens if we use the same models on PDBBind data?\n\n[1]  Rong et al: GROVER: Self-supervised Message Passing Transformer on Large-scale Molecular Data\n https://arxiv.org/pdf/2007.02835.pdf\n[2] Liu et al: N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules\n[3] Karimi et al: DeepAffinity: Interpretable Deep Learning of Compound-Protein Affinity through Unified Recurrent and Convolutional Neural Networks\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "gJQffi9YHPF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors introduce a repository of datasets for several atomistic learning tasks. These datasets are processed into a simple and standardized format. A systematic benchmark with atomistic learning methods is presented, showcasing the value of using 3D atom-level data instead of 1D or 2D features. The authors argue that these datasets will serve as a stepping stone for machine learning researchers interested in developing methods for atomistic learning and rapidly advance this field. The paper also presents the best practices for each of the tasks, as well as the splitting and filtering criteria to ensure generalizability and reproducibility.\n\nStrong points of the paper\n- The authors present a systematic evaluation of atomistic learning across multiple tasks and show that 3D data consistently yields better performance than 1D and 2D methods.\n- The experiments are thorough and the literature for every task has been covered extensively. The performance evaluation for each task is fair, respecting the corresponding metrics.\n- Novel molecular tasks and their corresponding datasets are introduced. This is a strong point for this paper. The availability of standard curated datasets helps to advance the field by making it easier to develop and compare new methods for these tasks.\n\nWeak points of the paper\n- The idea of using atomistic learning or at least 3D derived features have already been implemented or at least contemplated in many of the presented tasks (Gilmer et al [1], Wu et al [2], Townshend et al. [3]). In fact, these methods are often SOTA in their fields. Because of this, I feel that there is not much novelty with atomistic learning. \n- If the authors are selecting tasks where there are SOTA 3D methods, it would be also interesting to evaluate the performance against these methods (Townshend et al. [3] for instance). Or at least explain how their 3D approach is different.\n- I feel that the selection of multiple tasks is limiting the authors in the amount of information that they can fit in the actual paper. As it is now, the paper relies heavily on the Appendix as important information is described there and not in the main paper.\n\nMy overall recommendation is a weak accept. This is because the methods used and the idea of atomistic learning are not novel. The novelty of the paper lies in the introduction of the curated datasets and the novel tasks defined by the authors. By creating a standardized set of prediction tasks and associated data sets, the authors have presented a resource that may help the community to compare 3D atomistic methods quickly and fairly. However, I personally feel that this work could be a better fit for a more biologically inclined venue.\n\n\nQuestions for the authors\n- For the MSP task, could you explain why the problem was framed as a binary classification problem? There seems to be a fair amount of papers tackling the non-binary problem (Montanucci et al. [4], Cao et al. [5], Rodrigues et al. [6] to name a few).\n- Is PPI the appropriate name for this task? Townshend et al., which as stated in the paper is a motivation for the approach, frames the problem as Protein Interface Prediction. The same goes for the baseline (Sanchez-Garcia et al. [7]). Although related, the name PPI could generate confusion as it spans a whole different literature. \n\nSuggestions\n- I feel that it would be interesting to have a more detailed discussion on why the atom-level data improves performance on each task.\n- If what the authors want is to show that 3D features outperform 2D or 1D for these molecular tasks, then a systematic evaluation of the same methods but with different types of features would be more significant. For instance 3D CNNs vs 2D CNNs (where possible).\n \nReferences\n[1] Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2017). Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212.\n[2] Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., ... & Pande, V. (2018). MoleculeNet: a benchmark for molecular machine learning. Chemical science, 9(2), 513-530.\n[3] Townshend, R., Bedi, R., Suriana, P., & Dror, R. (2019). End-to-end learning on 3d protein structure for interface prediction. In Advances in Neural Information Processing Systems (pp. 15642-15651).\n[4] Montanucci, L., Capriotti, E., Frank, Y., Ben-Tal, N., & Fariselli, P. (2019). DDGun: an untrained method for the prediction of protein stability changes upon single and multiple point variations. BMC bioinformatics, 20(14), 335.\n[5] Cao, H., Wang, J., He, L., Qi, Y., & Zhang, J. Z. (2019). DeepDDG: predicting the stability change of protein point mutations using neural networks. Journal of chemical information and modeling, 59(4), 1508-1514.\n[6] Rodrigues, C. H., Pires, D. E., & Ascher, D. B. (2018). DynaMut: predicting the impact of mutations on protein conformation, flexibility and stability. Nucleic acids research, 46(W1), W350-W355.\n[7] Sanchez-Garcia, R., Sorzano, C. O. S., Carazo, J. M., & Segura, J. (2019). BIPSPI: a method for the prediction of partner-specific protein–protein interfaces. Bioinformatics, 35(3), 470-477.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9tx-1rgzM4", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a large benchmark of machine learning tasks for molecules represented by the 3D coordinates of their atoms. The benchmark is a combination of existing data sets and newly created ones, and covers a variety of applications and tasks, from small molecules to RNA or protein structures, and including classification, regression and ranking tasks. In addition, three deep-learning algorithms are implemented and evaluated on these benchmarks, and compared to state-of-the-art methods that do not use 3D information, and empirically demonstrate the benefit of incorporating 3D information in the networks.\n\nThe vast majority of machine learning methods that have been developed for molecules use either 1D or 2D information. Therefore, this resource and the empirical demonstration that using 3D information can improve performance can be very valuable to the community. However, I have a number of concerns about this paper:\n\n1. I am assuming this has not been incorporated to the paper for anonymization reasons, but could the authors confirm that they are indeed planning to make both the data sets and the code used to produce the results (in particular, the three proposed neural networks architecture) available? This is obviously essential to the paper and I would feel more comfortable accepting a version of the paper that includes this information (possibly with URLs withdrawn if there is a concern about maintaining the review process blind).\n\n2. The paper does not discuss the nature of the 3D information further than \"By representing a molecule's atoms and their 3D positions\". However, molecules do not have a fixed 3D structure, but rather multiple conformations driven in particular by rotatable bonds. Determining the multiple possible conformations of drug-like molecules is still an ongoing research topic (see for example the review of Hawkins (2017)), not to mention the determination of the 3D structure of proteins, which is indeed the topic of one of the data sets provided. What information is provided in the different data sets (a single conformation? multiple conformations?) and how does this affect both algorithms (if several conformations are used) and prediction performance? \n\n3. I would really refrain from using \"atomistic learning\" to describe what the community has been referring to as \"learning from 3D molecular representations\" for decades. \n\nActually, the abstract (and, more generally, the paper) reads as if neural networks were the only kind of machine learning algorithms that could be applied to molecules and that very little work has been done in the past to incorporate 3D information in chemoinformatics. While it is true that most current techniques rely mostly on 2D (for small molecules) or 1D (for large molecules) representations, it is not for lack of trying to incorporate 3D information, but because 1) this information was either lacking or incomplete (in the sense that a single conformation gives somewhat limited information; for example, you may have the crystal structure of a protein, but that doesn't inform you directly on the pose of its pocket when binding a specific small molecule) and 2) earlier attempts at making use of 3D information have often found that it did not improve performance (see Swamidass et al. (2005) or Azencott et al. (2007)), either because of the aforementioned incompleteness or because the methods were not up to par. The framing of the paper ignores decades of work in chemoinformatics, in particular (but not limited to) around kernel methods. I am listing a few examples of such papers below, not because I think they should all be included in this paper, but because in my opinion the paper would benefit from considering this body of work.\n\nIn addition, although some authors have already used \"atomistic machine learning\" in the context of chemoinformatics (see Schütt et al. (2018)), the term \"atomistic learning\" is already often used in opposition to \"holistic learning\" in education.\n\n4. Finally, in Section 4, the paper could benefit from stating very explicitly what is novel and what is not novel in the three proposed 3D architectures.\n\nAxen, Seth D., et al. \"A simple representation of three-dimensional molecular structure.\" Journal of medicinal chemistry 60.17 (2017): 7393-7409.\n\nAzencott, C.-A., et al. \"One-to four-dimensional kernels for virtual screening and the prediction of physical, chemical, and biological properties.\" Journal of chemical information and modeling 47.3 (2007): 965-974.\n\nGaüzere, B., Brun, L., and Villemin D,. \"Two new graphs kernels in chemoinformatics.\" Pattern Recognition Letters 33.15 (2012): 2038-2047.\n\nHawkins, Paul C. D. \"Conformation generation: the state of the art.\" Journal of Chemical Information and Modeling 57.8 (2017): 1747-1756.\n\nMahé, P., et al. \"Graph kernels for molecular structure− activity relationship analysis with support vector machines.\" Journal of chemical information and modeling 45.4 (2005): 939-951.\n\nMohr, J. A., Jain, B. J., and Obermayer, K. \"Molecule kernels: a descriptor-and alignment-free quantitative structure–activity relationship approach.\" Journal of chemical information and modeling 48.9 (2008): 1868-1881.\n\nNettles, J. H. et al. Bridging chemical and biological space: “target fishing” using 2D and 3D molecular descriptors. J. Medicinal Chem. 49, 6802–6810 (2006).\n\nRhodes, N., Clark, D. E. & Willett, P. Similarity searching in databases of flexible 3d structures using autocorrelation vectors derived from smoothed bounded distance matrices. J. Chem. Info. Mod. 46, 615–619 (2006).\n\nSchütt, K. T. et al. SchNet - a deep learning architecture for molecules and materials. The Journal of Chemical Physics 148(24), 241722 (2018)\n\nSwamidass, S. J., et al. \"Kernels for small molecules and the prediction of mutagenicity, toxicity and anti-cancer activity.\" Bioinformatics 21.suppl_1 (2005): i359-i368.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: For many problems such as ligand-protein binding, quantitative structure activity prediction (QSAR), predicting protein function from structure, etc., the 3D geometry of the molecules is of great importance.  One way to represent this is simply to assign locations to all atoms in 3-dimensional space.  If using graph convolutional kernels or other relational representations such that aligning molecules is not necessary, these approaches with 3D geometry can be efficient and far more effective than 1D or 2D representations.  The contribution of the paper is to make this point and to produce a resource with this kind of 3D data.  Such a resource would be of high value.  Nevertheless, reviewers feel provision of such a resource is perhaps not a major contribution to the ICLR and ML communities.  There is a sense that more innovative and substantial contribution would come from addressing also the challenge that 3D geometry can changes and that there may be multiple low-energy conformations of biomolecules that should be considered.  The authors contend that unlike ligands which are small and may have many low-energy conformations, large biomolecules have a much more constrained conformational space.

This meta-reviewer is sympathetic to the authors' point and appreciates the importance of the resource.  Nevertheless, even large biomolecules often have some portions of flexible conformation and high 3D structure variation that should be considered.  And indeed addressing the kind of multiple instance problem that arises by considering multiple conformations of large molecules or of ligands binding to large molecules would certainly require and likely yield bigger ICLR/ML innovations.  In the end the paper contributes a useful resource but does not excite the reviewers substantially enough, without those extensions or others, for a recommendation of acceptance at this time.
Decision: Reject