Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materials
Authors: Shengchao Liu
weitao Du
Yanjing Li
Zhuoxinran Li
Zhiling Zheng
Chenru Duan
Zhi-Ming Ma
Omar M. Yaghi
Anima Anandkumar
Christian Borgs
Jennifer T Chayes
Hongyu Guo
Jian Tang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: physics
geometric representation
equivariant
group theory
pretraining
molecule
chemistry
protein
biology
crystal
crystalline material
TL;DR: We provide a new aspect in categorizing the physics-informed symmetric deep leaning models, and a benchmark on molecule, protein, and crystalline material tasks.

Abstract: Artificial intelligence for scientific discovery has recently generated significant interest within the machine learning and scientific communities, particularly in the domains of chemistry, biology, and material discovery. For these scientific problems, molecules serve as the fundamental building blocks, and machine learning has emerged as a highly effective and powerful tool for modeling their geometric structures. Nevertheless, due to the rapidly evolving process of the field and the knowledge gap between science ({\eg}, physics,  chemistry, \& biology) and machine learning communities, a benchmarking study on geometrical representation for such data has not been conducted. To address such an issue, in this paper, we first provide a unified view of the current symmetry-informed geometric methods, classifying them into three main categories: invariance, equivariance with spherical frame basis, and equivariance with vector frame basis. Then we propose a platform, coined Geom3D, which enables benchmarking the effectiveness of geometric strategies. Geom3D contains 16 advanced symmetry-informed geometric representation models and 14 geometric pretraining methods over 52 diverse tasks, including small molecules, proteins, and crystalline materials. We hope that Geom3D can, on the one hand, eliminate barriers for machine learning researchers interested in exploring scientific problems; and, on the other hand, provide valuable guidance for researchers in computational chemistry, structural biology, and materials science, aiding in the informed selection of representation techniques for specific applications. The source code is available on \href{https://github.com/chao1224/Geom3D}{the GitHub repository}.

Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materials
Shengchao Liu Weitao Du', Yanjing Li*, Zhuoxinran Li³, Zhiling Zheng", Chenru Duan',
Zhiming Ma", Omar Yaghi', Anima Anandkumar", Christian Borgs',
Jennifer Chayes", Hongyu Guo,, Jian Tang 210.11
Miila Quéhce Artificial Intelligence Institute Université de Montréal
University of Chinese Academy of Sciences 'Carnegie Mellon University
UUiiversity of Toronto 'University California, Berkeley "Massachusetts Institute of Technology
Califfornia Institute of Technology Natiionl Research Council Canada
"HEC Montréal
"CIFAR AI Chair
# Correspondencc:: shengchao. liu @iw@:luuu@@u @umonreall.co jian.tang hec.ca
Abstract
Artificial intelligence for scientific discovery has recently generated significant interest within the machine learning and scientific communities, particularly in the domains of chemistry, biology, and material discovery. For these scientific prob- lems, molecules serve as the fundamental building blocks, and machine learning has emerged highly effective and powerful tool for modeling their geometric structures. Nevertheless, due to the rapidly evolving process the field and the knowledge gap between science (e.g.. physics, chemistry, & biology) and machine learning communities, a benchmarking study on geometrical representation for such data has not been conducted. To address such issue, in this paper, we first provide a unified view of the current cymmetry-informen geometric methods, classifying them into three main categories: invariance, equivariance with spherical frame basis, and equivariance with vector frame basis. Then we propose a platform, coined Geom3D, which enables benchmarking the effectiveness of geometric strate- gies. Geom3D contains 16 advanced symmetry-informed geometric representation models and 14 geometric pretraining methods over 52 diverse tasks, including small molecules, proteins, and crystalline materials. We hope that Geom3D can, on the one hand, eliminate barriers for machine learning researchers interested in exploring scientific problems; and, on the other hand, provide valuable guid- ance for researchers in computational chemistry, structural biology, and materials science, aiding in the informed selection of representation techniques for specific applications. The source code available on the GitHub repository.
Introduction
Artificial intelligence (AI) for molecule discovery has recently seen many developments, including small molecular property prediction [13, 17, 23, 38, 66, 78, 101, 103, 104, 119, 130, 132, 135], small molecule design and optimization [6, 57, 85, 137], small molccule reaction and retrosynthesis [40, 111. 116], protein property prediction [27, 141], protein folding and inverse folding [48, 64, 92], protein design [15, 41, 46, 88, 91], and crystalline material design [33, 125, 128]. One of the most fundamental building blocks for these tasks is the geometric structure of molecules. Exploring effective methods for robust representation learning to leverage such geometric information fully remains open challenge that interests both machine learning (ML) and science researchers.
To this end, symmetry-informee geometric representation [1] has emerged as a promising approach. By leveraging physical principles (i.e., group theory for depicting symmetric particles) into spatial
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Pretraining
BE Equivariant Force Prediction
Delonet
Piitriiing Mithod:
p00 Mi-ttppidd GauBSLARR GesSaL SENANCE Gecdia, viaicc, GecSaLOOM
PCOOWIN
pD 30 SOL Mi-sspodd Grapet/vt. 30 wrsttta Volecu udeE
A
Dataset
Small Moleccllea
Feature Extraction
Model
Target Task
Secall Molrcalee:
Ditaar Saraeiir
Jraxdann Modal:
Malecake:
VD1T, IND17,
Proteins
Softie, Strehret Sphasiied Gentii
Protere::
Angll Cxrndion
Spherical Baoks Eqahariast Model
Pariod:
Bor
Crystallie
Frana Boals Consanction
VVutor Rowes Book Squkeritat Model
S0S5amren, hegul Mlggn
Sreal Malecatie bee and Pestiles
Crysca Boa Matoridl::
Figure 1:
Pipeline for Geom3D, including dataset preprocessing, feature extraction, geometric preetraining and representation, and target tasks. We additionally demonstrate the SE(3)-qqivvarian force prediction task.
representation, they facilitate more robust representation of small molecules, proteins, and crystalline materials. Nevertheless, pursuing geometric learning research still challenging due to its evolving nature and the knowledge gap between science (e.g., physics) and machine learning communities. These factors contribute to a substantial barrier for machine learning researchers to investigate scientific problems and hinder efforts reproduce results consistently. To overcome this, we introduce Geom3l,, a benchmarking of the geometric representation with four advantages, as follows.
(1) A unified and novel aspect in understand- ing symmetry-informet geometric models. The molecule geometry needs to satisfy certain physical constraints regarding the 3D Euclidean space. For instance, the molecules' force needs to be equivariant to translation and rotation (see -Equiiequie in Fig. 1). In this work, we classify the geometric methods into three categories: invariant model, SE(3)--quivariant model with spherical frame basis and vector frame basis. The invariant models only consider features that are constant w.r.t. the SE(3) group, while the two families of equivariant models can be further unified using the frame basis to cap- ture equivariant symmetry. An illustration of three categories is in Fig. 2. Building equivari- ant models on the frame basis provides novel and unified view of understanding geometric models and paves the way for intriguing more ML researchers to explore scientific problems.
Insurient Represantatio
Pesture Extracion
Dalance
Representation
Driedral Angle
(b) Sphtrral From: Bas Equivariant Representziion
Representation
(c) Vestor Frame Bass Equivariani Representaton
Representation
Three categories of geometric modules. (a) Invariant models only consider type-0 fcatures. Equivari- ant models use either (b) spherical harmonies frames or (c) vector frames by projecting the coordinate vectors.
Figure 2:
(2) A unified platform for various scientific domains. There exist multiple platforms and tools for molecule discovery, but they are (1) mainly focusing on molecule's 2D graph representation [77, 102, 145]; (2) using 3D geometry with customized data structures APIs [3, 105]; or (3) covering only few geometric models [76]. Thus, it is necessary to have a platform benchmarking the geometric models, especially for researchers interested in solving scientific problems. In this work, we propose Geom3D. geometric modeling framework based on PyTorc Geometric (PyG) [31], one of the most widely-used platforms for graph representation learning. Geom3D benchmarks 16 geometric models on solving 52 scientific tasks, and these tasks include the three most fundamental molecule types: small molecules, proteins, and crystalline materials. Each of them requires distinct domain -spccficc preprocessing steps, e.g., crystalline materials molecules possess periodic structures and thus need a particular periodic data augmentation. By leveraging such a unified framework, Geom3D serves as comprehensive benchmarking tool, facilitating effective and consistent analysis components to interpret the existing geometric representation functions in a fair and convenient comparison setting.
(3) A framework for wider range of ML tasks. The geometric models in Geom3D can serve as a building block for exploring extensive ML tasks, including but not limited studying the molccule dy- namic simulation and scrutinizing the transfer learning effect on molecule geometry. For example, pre-
In what follows, we may use "molecule" to refer to "small molecule" for brevity.

---

H
Potential Energy Surface
2D Topology
3D Geometry
Glabal Mininun
(a) An example of small molecule structure.
(b) An illustration of the potential energy surface.
wmu
(c) An example of protein sequence and structure.
(d) An example of crystalline material.
Figure 3: Fig. 3(a) illustrates 2D topology and 3D conformation for molecule Glycine. Fig. 3(c) displays the 3D structure of protein. Fig. 3(d) shows simple cubic crystal the element Po. Fig. 3(b) demo of PES.
training is important strategy to quickly transfer knowledge target tasks, and recent works explore geometric pretraining on 3D conformations (including supervised and self-supervised) [59, 80, 136] and
pretraining 2D topology and 3D geometry [30, 79, 86). Other transfer learning venues include multi-task learning [82, 84] and out-of-distrriution or domain adaptation [58, 133, 134), yet no geometry information has been utilized. All of these directions promising for future ex- ploration, and Geom3D serves as an auxiliary tool to accomplish them. For example, as will be shown Sec. 4, we leverage Geom3D to effectively evaluate 14 pretraining methods with benchmarks.
(4) A framework for exploring data preprocessing and optimization tricks. When comparing different symmetry-informt geometric models, we find that in addition to the model architecture, there are two important factors affecting the performance: the data preprocessing (e.g., energy and force rescaling and shift) and optimization methods (e.g., learning rate, learning rate schedule, number of epochs, random seeds). In this work, we explore the effect of four preprocessing tricks and around 2-10 optimization hyperparameters for each model and task. In general, observe that each model may benefit differently in different tasks regarding the preprocessing and optimization tricks. However, data normalization is found help improve performance hugely in most cases. We believe that Geom3D is an effective tool for exploring and understanding various engineering tricks.
2 Data Structures for Geometric Data
Small molecule 3D conformation. Molecules sets of points in the 3D Euclidean space, and they move in a dynamic motion, as known as the potential energy surface (PES). The region with the lowest energy corresponds to the most stable state for molecules, and molecules at these positions called conformations, as illustrated in Fig. 3(b). For notation, we mark each 3D molecular graph as g (X, R), where X and R are for the atom types and positions, respectively.
Crystalline material with periodic structure. The crystalline materials or extended chemical structures possess a characteristic known as periodicity: their atomic or molecular arrangement repeats in a predictable and consistent pattern across all three spatial dimensions. This is the key aspect that differentiates them from small molecules. In Fig. 3(d), show original unit cell (marked in green) that can repeatedly compose the crystal structure along the lattice. To model such a periodic structure, unit cell, we shift it along the lattice three dimensions and connect edges within a cutoff value (hyperparameter). For more details on the two augmentation variants, please check Appendix A.
adopt the data augmentation from CGCNN [129]: for each original
Protein with backbone structure. Protein structures can be classified into four primary levels, and the primary structure represents the linear arrangement of amino acids, and each amino acid is molecule consisting of atoms. Geometric methods mainly focus on the tertiary structure, i.e. the 3D geometry of each atom, encompassing the complete organization of single protein. However,

---

(i Alon Type
Distance
[c) Angle
infograph
(eI Gec85L Represntadde
dlddddii Geerwity
100 Goodit Destristive
Cametry
Na-3
Caaeery
Invoramerr Eop satwi Eepreeertion
Podta
Negette
Recivati
Figure 4: Pipelines for seven single-modal geometric pretraining methods. (a-c) conduct self seef-prediction. (d) maximizes the MI between nodes and graphs. (c-g) are GeoSSL, maximizing the MI between views gi and ga.
atom-level modeling for proteins is consuming due to the large volume of atoms and the GPU memory limit. One solution is modeling each amino acid's backbone structure. The backbone structure of each amino acid is N - C. C, and the C. is bonded to the side chain. 20 common types of side chains corresponding to amino acids, as illustrated in Fig. 3(c). Thus, modeling the backbone structure can balance the computational efficiency and the key geometric information.
Symmetry-Informed Geometric Representation
3.1 Group Symmetry and Equivariance
Symmetry means the object remains invariant after certain transformations [127]. and is everywhere on Earth, such as in animals, plants, and molecules. Formally, the set of all symmetric transformations satisfies the axioms of a group. Therefore, the group theory and its representation theory common tools to depict such physical symmetry. Group is a set G equipped with group product x satisfying:
(1) Be E G, a e = e a, Va E G;; (2) = aa x =-=- e;
(8) a x (b x c) = a x b c. (1)
Group representation is mapping from the group G the group of linear transformations of vector space X with dimension d (see [138] for more rigorous definition):
px(+): G Bax
s.t.
p(e) = A px(a)px(b) = px (a x b), Va, ba G.
(2)
During modeling, the space can be the input 3D Euclidean space, the equivariant vector space in the intermediate layers, or the output force space. This enables the definition of equivariance below.
Equivariance is the property for the geometric modeling function f Y as:
f(x((a)=) (a))(m)) Va G
(3)
As displayed in Fig. 1, for molecule geometric modeling, the property should be rotation-equivariant and monsllatioo-euivariann (i.e., -uuiivaaqia) More concretely, px(a) and py (a) are the SE(3) group representations on the input (e.g., atom coordinates) and output space (e.g., force space), respectively. 3E(3)-equivarian modeling in Eq. (3) essentially saying that the designed deep learning model ff is modeling the whole transformation trajectory on the molecule conformations, and the output the transformed ý accordingly. Further, we want to highlight that, in addition to the network architecture or representation function, the input features can also be represented as an equivariant feature mapping from the 3D mesh to Rd [11], where d depends on input data, e.g., d == (for atom type dimension) + 3 (for atom coordinate dimension) on small molecules. Such features are called steerable features in [5, 11] when only considering the subgroup -quuvaaeure
Invariance is a special type of equivariance. defined as:
f(ex(a)x) f(x), Va G.
(4)
with Py (a) as the identity Va E G. The group representation helps define the equivariance condition for f to follow. Then, the question boils down to how to design such an equivariant f. In the following, we will discuss geometric modelings from a novel and unified perspective using the frame. In the next sections, we will provide a novel and unified aspect of understanding the advanced geometric representation and pretraining methods using the frame basis (details in Appendix H).

---

3.2 Invariant Geometric Representation Learning
One simple way of achieving SE(3) group symmetry is invariant modeling. It means the geometric model only considers the type-0 features [1 [112], i.e., features that are invariant with respect to rotation and translation. Existing works have been adopting the invariant features for modeling, including pairwise distance (SchNet [109)), bond angles (DimeNet [68]), and torsion angles (SphereNet [89] and GemNet [67]). Note that the torsion angles are angles between two planes defined by pairwise bonds.
3.3 Equivariant Geometric Representation Learning
Invariant modeling only captures the type-O features. However, equivariant modeling of higher-order particles may bring in extra expressiveness. For example, the elementary particles in high energy physics [98] inherit higher order symmetries in the sense of SO(3) representation theory, which makes the equívariant modeling necessary. Such higher-order particles include type-1 features like coordinates and forces in molecular conformation. There are many approaches to design such -qqiivaquiaariant model satisfying Eq. (3). There are two main venues, as will be discussed below.
Spherical Frame Basis. This research line utilizes the irreducible representations [37] for building SO(3)-equivariant representations, and the first work is TFN [112]. Its main idea is to project the 3D Euclidean coordinates into the spherical harmonics space, which transforms equivariantly according the irreducible representations of SO(3), and the translation-equivariant can be trivially guaranteed using the relative coordinates. Following this, there have been variants combining it with the attention module (Equiformer [73]) with more expressive network architectures (SEGNN [4], Allegro [95]).
Vector Frame Basis. An alternative philosophy of equivariant modeling utilizes the vector (in physics) frame basis. It constructs three vectors bases, serving reference frame to help locate the vectors in each corresponding local environment. Works along this line for molecule discovery include DeePMD [140] for dynamies simulation, 3D-EMGP [59] and MoleculeSDE [79] for geomet- ric pretraining, and ClofNet [20] for conformation generation. For macromolecules like protein, the equivariant vector frame has been used for protein design (StructTrans [53]) and protein folding (Al- phaFold2 [64]). We also want to highlight that, from a mathematical perspective, equivariance and in- variance can be transformed each other by the scalarization technique. Please check [49] for details.
The spherical frame basis can easily extended to higher-order particles, yet may suffer from the high computational cost. On the other hand, the vector frame basis is specifically designed for the 3D point clouds; thus, is more efficient but cannot generalize to higher-order particles. Mcanwhile, we would like to acknowledge other equivariant modeling paradigms, including using orbital features [99] and elevating 3D Euclidean space to SE(3) group [32, 52]. Please check Appendix for details.
3.4 Geometric Pretraining
Recent studies have started to explore single-modal of geometric pretraining on molecules. The GeoSSL. paper [80] covers a wide range of geometric pretraining algorithms. The type prediction, distance prediction, and angle prediction predict the masked atom type, pairwise distance, and bond angle, respectively. The 3D InfoGraph predicts whether the node- and graph-level 3D representation are for the same molecule. GeoSSL is novel geometric pretraining paradigm that maximizes the mutual information (MI) between the original conformation gi and augmented conformation g2. where g2 is obtained by adding small perturbations to g1. RR, InfoNCE, and EBM-NCE optimize the objective in the latent representation space, either generative or contrastive. GeoSSL-DDM [80, 136] optimizes the same objective function using denoising score matching. 3D-EMGP [60] has the same strategy and utilizes an equivariant module to denoise the 3D noise directly. Another research line is the multi-modal of topological and geometric pretraining. GraphMVP [86] first proposes one contrastive objective (EBM-NCE) and one generative objective (VRR) to optimize the MI between the 2D topologies and 3D geometries in the representation space. 3D InfoMax [1 [114] is special case of GraphMVP, with the contrastive part only. MoleculeSDE [79] extends GraphMVP by introducing two SDE models for solving the 2D and 3D reconstruction. We illustrate these algorithms in Figs. 4 and 8.
3.5 Discussion: Reflection-antisymmeenn in Geometric Learning
Till now, we have discussed the -F(3)-equivariance, i.e., the translation and rotation equivariance. As highlighted in the recent work [61, 79], the molecules needlessly satisfy the reflection reflcction--qqiivrrann equivarratt,,,
5

---

Table 1: Results of 26 models on 12 quantum mechanics prediction tasks in QM9, with 110K for training. 10K for validation, and 11K for testing. The task unit is specified, and the evaluation the mean absolute error (MAE).
Festurization
Model
13
Sanaan 1
Eu 1 mocl
#
C..
G
H[
R²º 03
11
Un mnV
ZPVE
incl
mcV
D
m
meV
mcV
meV
MLP
2.231
0.526
0.919
2158.66
2358.22 23
2340.61
2314.77
ID
3.801
165.72
183.04
0.534
3729.94
3705. 75
3678.25
XGB
2.748
199.71
139.88
165.43
1,062
2563.93
2804.27
82.959
2786.28
2769.29
180.989
ID SMILES
CNN
BERT
0.364
0.313
165.22
117.50
124.65
84.93
114.81
58.88
0.566
0.446
0.173
0.176
156.
170.01
170.59
183.43
20.403
18.002
166.18
183.84
169.89
188.60
10.070
13.410
ID SELFIES
CNN
BERT
0.345
0.348
157. .14
123.11
115.51
91.15
113.00
90,50
0.161
0.499
0.168
0.203
136.45
168. 30
187.50
146.56
20,080
19.125
204.93
143.00
195.98
140,01
10.149
17.328
OCN
1.338
145.82
96.21
0.526
1198.
1291.57
37.585
1281.03
1303.39
85.
ENN-S28
1.401
270.59
0.577
1487
34.00
1800.
1521.32
51.236
GraphSAGE
1.601
131.45
93.3
0.402
0.544
1473-42
1617-73
38.112
1553.01
1565.65
GAT
1.132
135.90
90.70
VRL
D.405
D22001
911 282
991-31
26-583
1161-29 22
592.67
2D Graph
GIN
D-MPNN
0.568
1.165
175.82
118.42
90.66
85.01
110.74
86.20
0.579
0:441
0.241
0:691
8/8.24 21
-23.18
1090 36
450. 39
24
35.110
24.916
1498.23
470.01
1361.18
445.91
108.331
29.201
PNA
0.681
148.88
58.72
97.3
0.361
0.409
664.98
$92, 744
23.855
616.70
694,92
57.217
Griphommer
2,836
79.27
$4.24
$2.4
D:330
01090
2066 72X
25-6601
151. 158
2229 XM
2525.51
144,595
AWARE
0.297
144.91
133.89
98. 98.96
0.602
0.129
89.62
91.17
22.180
93.59
95.73
5.275
OraphGPSS
0.209
75.98
$4.75
$4.53
0.258
0.089
$28.50
693.19
12:188
296.00
411.16
49,888
SchNet
0.060
44.13
27.64
22.55
0.028
0.031
14.19
14.05
0.133
13.93
13.27
1.749
0.044
36.22
20.01
16.66
0.028
0.022
7.45
6.14
0.323
6.33
7.18
L.118
Strn)-Taans
0.137
56.52
0,050
0.063
70.70
1.747
68.02
5.478
0.062
30.08
0.030
10:01
0.089
1.519
PaiNN
42.73
1.119
3D Graphs
GemNer-T
0.041
17.85
0.021
0.023
0.271
5.88
1.232
SphereNett
0.347
38.93
21 45
18
B.077
01025
8.16
13.68
082888
6.77
7.43
1,295
SEGNN
0.344
33.61
17.66
17.01
B101
B106
11.40 60
12.45
0.404
11.29
12.20
1,590
Allegro
0.997
102.44
61.86
63.17
0.176
0.032
12.00 00
41.96
1.977
44.61
44.13
2.919
NegulP
0.066
61.94
42.00
31 64
0.036
0.028
22. 08
23.36
0.415
23.23
23.02
1,899
Equilormer
0.U51
33.46
17.93
16.85
0.015
0.023
14.49
14.60
0.433
14.88
13.78
2,342
but instead, they should be reflection-nntiyymmeen [79]. One classic example is that the energy of small molecules is reflection-aniismmmett
in binding system. Each of the two equivariant categories discussed Sec. 3.3 can solve this problem easily. The spherical frame basis can achieve this by adding the reflection into the Wigner-D matrix [4], and the vector frame basis can accomplish this using the cross-product during frame construction [79].
4 Geometric Datasets and Benchmarks
In Sec. 3, we introduce a novel aspect for understanding symmetry-informed geometric models. In this section, we discuss utilizing Geom3D framework for benchmarking 16 geometric models over 52 tasks, For the detailed dataset acquisitions and task specifications (e.g., dataset size, splitting, and task unit). please check Appendix B. Geom3D also covers 7 1D models and 10 2D graph neural networks (GNNs) and benchmarks the 14 pretraining algorithms to learn a robust geometric representation. Additionally, we want to highlight Geom3D enables exploration of important data preprocessing and optimization tricks for performance improvement, as will be introduced next.
4.1 Small Molecules: QM9
QM9 [100] is dataset consisting of 134K molecules, each with up to 9 heavy atoms. It includes 12 tasks that are related to the quantum properties. For example, UO and U298 are the internal energies at temperatures of OK and 298. 15K, respectively. On the QM9 dataset, we can easily get the ID descrip- tors (Fingerprints/FPPs [106], SMILES [126], SELFIES [70]), 2D topology, and 3D conformation. This enables to build models on each of them respectively: (1) We benchmark 7 models on ID de- scriptors, including multi-layer perception (MLP), random forest (RF), XGBoost (SGB), convolution neural networks (CNN), and BERT [18]. (2) We benchmark 10 2D GNN models on the molecu- lar topology, including GCN [23, 66], ENN-S2S [38], GraphSAGE [43], GAT [119], GIN [130], D-MPNN [132], PNA [13], Graphormer [135], AWARE [17], GraphGPS [101]. (3) We benchmark
11 3D geometric models on the molecular conformation, including SchNet [109], DimeNet++ [68],
SE(3)- Trans [35], EGNN [108], PaiNN [1]0], GemNet-T [67], SphereNet [89]. SEGNN [4], Alle- gro [95], NequlP [3], Equiformer [73]. The evaluation metric is the mean absolute error.
The results of these 28 models are in Table 1, and two important insights are observed: (1) There is no one universally best geometric model, yet DimeNet++, PaiNN, GemNet, and Equiformer perform well in most tasks. However, PaiNN takes less than 20 GPU hours, and the other three models take up to GPU days per task. (2) The geometric conformation is important for quantum property prediction. The performance of 3D models is better than all the ID and 2D models by orders of magnitudes.

---

Table 2: Results energy (krel and force ( keal prediction tasks in MD17 and rMD17 (w/o normalization), A
the metric the mean absolute error (MAE). The data split and complete results are Appendices
I.
Drane
Appice
Trauees
Druud:
Nuadeiiee
Baago
04455
14403
0212
0154
0.2-
1.55
0220
0.24
2.415
01122
6318
1.22
1385
1344
1.45
1.44
3430
1445
1217
1.48
1.505
6521
1746
010
135
1.50
1258
112
1.50
130
07325
1023
6371
6850
335
3316
0331
012
115
019
122
83%
SREWN
RecF
1350
1.38
8.50
2086
Neualt
0.34
003
0386
155
122
Allegoo
Bagg
548
141
4151
5243
4.50
1.48
1.96
2399
21785
3812
4.90
2145
0.56
614
0.34
9.90
6.66
83301
0770
000
I03M
NO1T
INDT
Telk Auplin
Tie Chanol
Taak Mairakonrdee
Talk Notrhanee
Taxn Sabkyts
Taas Talees
Figure Ablation study on the effect data normalization. Here are visualizations on performance differences on 6 tasks and datasets, with MAE(force pred w/o normalization) MAF(force pred w/ normalization).
4.2 Small Molecules: MD17 and rMD17
MD17 [8] is dataset of molecular dynamics simulation. It has 8 tasks corresponding to eight organic molecules, and each task includes the molecule positions along the PES (see Fig. 3(b)). The goal is to predict each atom's 's energy and interatomic forces for each molecule's position. We follow the literature [68, 89, 109, 110] of using subtasks, 1K for training and 1K for validation, while the test set (from 48K to 991K) is much larger. However, the MD17 dataset contains non non- nog-ngglii numerical noises [9], and it is corrected by the revised MD17 (rMD17) dataset [10]. 100K structures were randomly chosen for each task/molecule in MD17, and the single-point force and energy calculations were performed for each structure using the PBE/def2-SVP level of theory. The calculations were conducted with tight SCF convergence and a dense DFT integration grid, significantly minimizing the computational noises.
The results on MD17 and rMD17 are in Table 2. We select 12 tasks for illustration, and more comprehensive results be found in Appendix I. We can observe that, in general, PaiNN, GemNet and Equiformer perform well on MD17 and rMD17 tasks. We also report ablation study on data normalization. NequlP [3] and Allegro [95] introduce normalization trick: multiplying the predicted energy with the mean of fround-truth force (reproduced results in Appendix J). We plot the performance gap, MAE(w/o normalization) MAE(w/ normalization), Fig. 5, and observe most of the gaps are positive, meaning that adding data normalization can lead to generally better performance.
4.3 Small Molecules: COLL
The COLL dataset [36] comprises energy and force data for 140K random snapshots obtained from molecular dynamics simulations of molecular colli- sions. These simulations were conducted using the semiempirical GFN2-xTB method. To obtain the data, DFT caleulations were performed utilizing the revPBE functional and def2-TZVP basis set, which also incorporated D3 dispersion corrections. The task is to predict the energy and force for each atom the molecule, and we consider 10 advanced ge- ometric models for benchmarking. The results are in Table 3, and GemNet, SphereNet, and Equiformer reach more optimal performance.
Table 3: Results on energy and force prediction in COLL. 120k training, 10k for val, 9.48k for test. The metric is the mean absolute error (MAE).
Model
Energy (cV)
Force (cV/A)
SchNet
0.178
0.130
DimeNet++
0.036
0.049
EGNN
1.808
0.234
PaiNN
0.030
0.052
Gem-Ne-T
0.017
0.028
SphereNet
0.032
0.047
SEGNN
7.085
0.642
NequlP
0.120
0.113
Allegro
0.161
0.130
Equiformer
0.036
0.030

---

Table 4: Results on 2 binding affinity prediction tasks. We select three evaluation metrics for LBA: the root mean squared error (RMSD). the Pearson correlation (ft,) and the Spearman correlation (Rt)). LEP is binary classification task, and we use the area under the curve receiver operating characteristies (ROC) and precision-rees recall (PR) for evaluation. We run cross- validation with 5 seeds, and the mean and std are reported.
LBA
LEP
Model
RMSD
Rc
ROC
PR ~
SchNet
1,521 + 0.02
0,474 + 0.01
0,452 + 0.01
0,450 + 0.03
0.379 + 0.03
DimeNNtt+
1.672 + 0.09
0,550 = 0.01
0.556 = 0.01
0,590 = 0.06
0.496 = 0.05
EGNN
1.494 + 0.04
0,503 + 0.04
0,483 + 0.05
0.657 + 0.05
0.559 + 0.05
PaiNN
0.583 0.02
0.580
0.02
G
0.659 1 0.05
0,506 1 0.05
SplsereNet
1.581
0.529
0.523 0.04
SEGNN
1.416 + 0.03
0,366 + 0.02
0,550 + 0.02
0,574 = 0.03
0.485 = 0.03
NequiP
1.006 + 0.02
0.537 0.01
0.520 + 0.01
0.538 0.12
0.481 + 0.07
Allegro
1.567 1. 0.02
0.547 0.00
0.534 1. 0.00
0.627 1 0.04
0.525 1. 0.03
Equilormer
1,392 + 0.03
0,598 0.02
0,578 + 0.02
0.618 0.06
0,510 + 0.05
Table 5: Results on 10 protein tasks from six datasets: ECSingle, ECMultiple, Fold (Fold, Sup.. Fam.), GO (MF, BP, CC), MSP, and PSR. The evaluation metries are Accuracy (ACC, %), Fmaz (definition Appendix B), ACC, Fmoz, receiver operating characteristics (ROC), and Spearman's 8 p. respectivcly.
Fold
GO
ECSingle
ECMultiple
MSP
PSR
Fold
Sup.
Fam.
MF
ACC
Froua
ACC 1
ACC f
ACC 1
Fivcz 1
Foncx
Frous
ROC1
Global
Mean
IBConv
45.0
69.7
98.9
GVP-GNN
65.5
0.712
34.8
52.7
95.0
0.476
0.312
0.389
0.574
0.744
0.302
GearNet
78.8
0.799
29.1
43.1
95.9
0.477
0.283
0.373
ProNet
86.4
0.823
52.7
70.3
0.559
0.367
0.414
0.634
0
0,462
CDConv
86.9
0.862
60.0
79.9
99.5
0.649
0.435
0.450
0.717
0.817
0.500
4.4 Small Molecules & Proteins Binding: LBA & LEP
The binding affinity measures the strength of the binding interaction between small molecule (ligand) to the target protein. In Geom3D, we consider modeling both the ligands and proteins with their 3D structures. During binding, cavity in protein can potentially possess suitable properties for binding a small molecule, and it is called a pocket [113]. Due to the large volume of protein, Geom3D follows existing works [118] by only taking the binding pocket instead of the whole protein structure. Specifically, Geom3D models up to 600 atoms for each ligand and protein pair. For the benchmarking, we consider two binding affinity tasks. (1) The first task is ligand binding affinity (LBA) [123]. It is gathered from [124], and the task is to predict the binding affinity strength between a ligand and protein pocket. (2) The second task ligand efficacy prediction (LEP) [34]. The input is a ligand and both the active and inactive conformers of protein, and the goal is to classify whether or not the ligand can activate the protein's 's function. The results on two binding tasks are in Table 4, and we can observe that PaiNN, SEGNN, and Equiformer are generally outstanding on the two tasks.
4.5 Proteins: ECSingle, ECMultiple, Fold, GO, MSP, and PSR
ECSingle a classification task [45] that classifies 37K proteins into 384 four-level Enzyme Com- mission (EC) types. This task aims to recognize the fundamental role of proteins as bio-catalysts or enzymes, which are essential facilitating biological reactions. The EC numbering system [63] serves as a comprehensive numerical classification scheme, systematically organizing the varied functionalities of enzymes and providing a structured approach to understanding their biological roles.
ECMultiple is a multi-lahel classification task proposed by Gligorijevic et al. [39], where 19K proteins are associated with 538 distinct EC categories, including both three-level and four-Ievel types and single protein can be concurrently labeled with several three-level or four-level EC numbers.
Fold is a task classifying 16K proteins into 1,195 fold patterns [47, 74J. It is an important biological task in predicting the 3D structures from 1D amino acid sequences. We further consider three testsets (Fold, Superfamily, and Family) based on the sequence and structure similarity [94].
GO (Gene Ontology) is a dataset [39] with 36K proteins for GO term classification, where the GO term provides a consistent description of gene product attributes across species and databases [12]. Concretely, each protein contains up to three types of GO terms, corresponding to three types of clas- sification tasks: (1) Molecular Function (MF) has 489 classes; (2) Biological Process (BP) has 1,943
8

---

Table 6: Results on the tasks from MatBench and I task from QMOF (with optimal DA). The data split and task unit are in Appendix B, and the metric is the mean absolute (MAE).
MatBench
QMOF
Model
Per. Euen 1 18,928
Dielectric
|
Toyno
logun
Easto |
Phonons |
Band Gap 106,113
Exom J 132.752
Band Gap
4,764
10,987
10,987
636
1,265
20,425
SchNet
0.040
0.334
0.081
0.060
65.201
42.586
0.327
0.026
0.236
Dime Net++
0.037
0.357
0.081
0.058
68.685
38.319
0.208
0.025
0.234
EGNN
0.038
0.331
0.087
0.064
78.015
74.846
0.211
0.026
0,256
PaiNN
0.038
0.317
0.080
0.053
67.752
44,602
0.022
0.190
0.207
GemNet-T
0.042
0.325
0.088
0.061
68.425
48.986
0.186
0.026
0.207
SphereNet
0.043
0.388
0.087
0,061
72.987
36,300
0.217
0.029
0.251
SEGNN
0,046
0.360
0.087
0.059
65.052
43.638
0.330
0.047
0.330
Equiformer 0.046
0.280
0.087
0.057
62.977
37,381
0.202
0.027
0.234
classes; and (3) Cellular Component (CC) has 320 classes. Notice that each protein be associated with multiple GO terms in each GO term type, thus all three tasks are multi-label classifications.
MSP & PSR two protein tasks from collection of benchmark datasets for machine learning in structural biology [118]. MSP (Mutation Stability Prediction) aims to predict whether the stability of protein increases after mutation. The dataset is mutation dataset containing 4K proteins. It is constructed by incorporating single-point mutations given in the SKEMPI database [56]. PSR (Protein Structure Ranking) is regression task based on the Critical Assessment of Structure Prediction (CASP) [71]. In CASP, protein structure is predicted and quality score, the global distance test (GDT_ TS), is calculated between the predicted structure and experimentally determined structure. This task aims to predict this score for 44K proteins.
The results of 5 models are in Table 5. CDConv [29] outperforms other models by large margin on almost all 10 tasks, while ProNet [122] performs second well in general. and reaches the best result on the PSR task with global p metric. Notice that certain entries in the table are temporarily left blank due to memory constraints encountered. More detailed dataset specifications are in Appendix B.
4.6 Crystalline Materials: MatBench and QMOF
MatBench [21] explicitly created to evaluate the performance of machine learning models in predicting properties of inorganic bulk materials covering mechanical, electronic, and thermodynamic material properties [21]. Here we consider 8 regression tasks with crystal structures, including predicting the formation energy (Perovskites, E, exfoliation energies (Eexto), band gap, shear and bulk modulus (logı0G and I)) etc. Please check Appendix B for more details.
Quantum MOF (QMOF) [107] is dataset of over 20K metal-organic frameworks (MOFs) and predict the band gap, the energy gap between the valence band and the conduction band. The results of 8 geometric models on 8 MatBench tasks and QMOF task are in Table and we can observe that the performance of all the models is very close, while DemeNett+, PaiNN, GemNet-T T, and Equiformer are slightly better.
coordination polymers derived from DFT. The task
We also conduct ablation study on periodic data aug- mentation crystal materi-
0836
9820
als. We note that there are two data augmentation (DA) methods: gathered and ex- panded. Gathered DA means that we shift the original unit cell along three dimensions, and the translated unit cells will have the same node in- dices as the original unit cell, i.e., multi-edee graph. How- ever, expanded DA will assume the translated unit cells have different node indices from the original Appendix A). We conduct an ablation study on the effect
-0123
-0323
-2213
Task Diclectrio
Tosk kosy(K
Taok Egdoo
EGNN
PaNh
-2078
-4237
Exprrs
Figure 6: Ablation study on the performance gap with data augmentation (DA): MAB(expanded DA) MAE(gathered DA).
unit cell. (A visual demonstration of these two DAs, and we plot MAE(expanded DA) MAE(gathered DA) on six tasks in Fig. 6. It reveals that for most of the models (except EGNN), using gathered DA can lead to consistently better performance, and thus it is preferred. For more qualitative analysis, please check Appendix J.

---

Table 7: QM9 downstream results after pretraining, and the backbone model is SchNet. We take 110K for training. 10K for validation, and 11K for testing. The evaluation metric is the mean absolute error (MAE).
Pretraining
a
37 4
EnCMo &
ELUM 4
C+ 4
GL
+.
Uo 4
ZPVE
(random init)
0.060
44.13
0.028
0.031
13.27
1.749
Supervised
0.062
40.31
25.57
21.69
0.030
0.030
14.36
14.68
0.308
15.21
16.13
1.638
Type Prediction
0.073
45.38
28.76
24.83
0.036
0.032
16.66
16.28
0.275
15.56
14.66
2.094
Distance Prediction
0.065
45.87
0.033
15.81
15.07
1.837
Angle Prediction
0,066
48.45
29.
24.40
0.034
0.031
14.13
13.77
0.214
13,50
13.47
1.861
3D InfoGraph
0.062
45. 96.96
29.29
24.60
0.028
0.030
13.93
13.97
0.133
13.55
13.47
1.644
GeoSSL-RR
0.060
43.71
27.71
22.84
0.028
0.031
14.54
13.70
0.122
13.81
13.75
1.694
GeoSSL-IntoNCE
0.061
44.38
27.67
22.85
0.027
0.030
13.38
13.36
0.116
13.05
13.00
1.643
GeOSSL-EBM-NCE
0.057
43.75
27.05
22.75
0.028
0,030
12.87
12.65
0.123
13.44
12.64
1.652
3D InfoMax
0.057
42.09
25.90
21.60
0.028
0,030
13.73
13.62
0.141
13.81
13.30
1.670
GraphMVP
0.056
41.99
1.609
GeoSSL-DDM-II IL
0,058
42.64
26.32
21.87
0.028
0.030
12.61
12.81
0.173
12.45
12.12
1.696
GeoSSL-DDM
0.056
42.29
25.61
21.88
0.027
0.029
11.54
0.168
11.06
10.96
1660
MoleculeSDB (VE)
0.056
41.84
25.79
21.63
0.027
0.029
11.47
10.71
0.233
11,04
10.95
1.444
MoleculeSDE (VP)
0.054
41.77
25.74
21.41
0.026
0.028
13.07
12.05
0.151
12.54
12.04
L,587
4.7 Geometric Pretraining on Small Molecules
We run 14 pretraining algorithms, including one supervised pretraining: the pretraining dataset (e.g., PCQM4Mv2 [51]) possess the energy or energy gap label for each conformation, which can be naturally adopted for pretraining. The benchmark results of using SchNet as the backbone model pretrained on PCQM4Mv2 and fine-tuning on QM9 tasks are in Table 7. We observe that MoleculeSDE and GeoSSL-DDM utilizing the geometric denoising diffusion models outperform other pretraining methods in most cases. On the other hand, supervised pretraining (pretrained on energy gap V reaches outstanding performance on V downstream task, yet the generalization other tasks is modest. Please check Appendix for more pretraining results with different backbone models.
5 Conclusion and Future Directions
Geom3D provides a unified view on the -Euuivaqui models, together with the implementations. Indeed these can serve as the building blocks to various tasks, such as geometric pretraining (as displayed in Sec. 4.7) and the conformation generation (ClofNet [20], MoleculeSDE [79]), paving the way for building more foundational models and solving more challenging tasks.
Limitations on models and tasks. Geom3D includes 10 topological models, 16 geometric models, 14 geometric pretraining methods, and 52 diverse tasks. We would also like to acknowledge there exist many more tasks (e.g., Atom3D [118], Molecule3D [131], OC20 [7]) and more geometric models (e.g., OrbNet [99]. MACE [2], Uni-Mol [144], and Lie Transformmer [52]). The continual updating may necessitate the collective efforts of our entire community, exemplifying our collaborative endeavors.
Foundation model as future exploration. Recently, there have been certain explorations on building the foundation models for molecule discovery, especially by incorporating textual data the molecule's 's functionalities [25, 26, 83, 87, 88, 115, 139, 143]. However, existing works mainly focus on the ID sequence or 2D topology, while the 3D geometric structure molecules is rarely considered. We believe that Geom3D can offer essential support for future explorations along this direction.
Reproducibility and Tutorials
The codes of Geom3D have been released on this GitHub repository. Both the raw and preprocessed datasets have been released on this HuggingFace link. The checkpoints of all models have been released on this HuggingFace link. We further added four tutorials on using Geom3D on customized data, energy prediction, force prediction, and geometric pretraining. These tutorials can sufficiently demonstrate how users can inject new methods into Geom3D platform, showcasing its potential as a fundamental building block for tackling wide range of machine learning tasks.
10

---

Acknowledgement
The authors would like thank Zichao Rong. Chengpeng Wang, Jiarui Lu, Farzaneh Heidari, Zuobai Zhang, Limei Wang, and Hanchen Wang for their helpful discussions. This project is supported by the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronics Co., Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bird Gift Fund, and National Research Council of Canada (NRC) Collaborative R&D Project. This project was also partially funded by IVADO Fundamental Research Project grant PRE-2019-35831397277
References
[1] Kenneth Atz, Francesca Grisoni, Gishert Schneider. Geometric deep learning on molecular represen- tations. Nature Machine Intelligence, 3(12):1023-1032, 2021.
[2] Ilyes Batatia, Simon Batzner, Dávid Péter Kovács, Albert Musaelian, Gregor N. C. Simm, Ralf Drautz, Christoph Ortner, Boris Kozinsky, and Gábor Csányi The design space of e(3)-equivarian atom-centered interatomic potentials, 2022. 10
[3] Simon Batzner, Albert Musaclian, Lixin Sun, Mario Geiger, Jonathan Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E(3)-cquivariant graph neural networks for data- efficient and accurate interatomic potentials. Nature communications, 13(1))1-11, 2022. 2, 6, 7, 34, 36, 45, 51
[4] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling, Geometric and physical quantities improve e(3) equivariant message passing. arXiv preprint arXiv:21 2021. 5. 34, 51
[5] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104. 2021. 4, 32
[6] Nathan Brown, Marco Fiscato, Marwin HSS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3): 1096-11008. 2019.
[7] Lowik Chanussot Abhishek Das Siddharth Goyal Thibaut Lavril*, Muhammed Shuaibi Morgane Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open catalyst 2020 (oc20) datasct and community challenges. ACS Catalysis, 2021. 10
[8] Stefan Chmicla, Alexandre Tkatchenko, Huziel E. Sauceda, Igor Poltavsky, Kristof Schütt, and Klaus- Robert Müller. Machine learning of accurate energy-comserving molecular force fields. Science advances, 2017. 7, 23
[9] Anders Christensen and O. Anatole von Lilienfeld. Revised md17 dataset. Materials Cloud Archive, 2020. 7
[10] Anders Christensen and O Anatole von Lilienfeld. On the role of gradients for machine learning of molecular energies and forces. arXiv.org, 2020. 7, 23
[11] Taco Cohen and Max Welling. Steerable cnns. arXiv preprint arXiv:1612.08-990 2016. 4
[12] Gene Ontology Consortium. The gene ontology (go) database and informatics resource. Nucleic acids research, 32(suppl_I):2258___p2222 1):2258-D261, 2004. 8, 26
[13] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbour- hood aggregation for graph nets. Advances Neural Information Processing Systems, 0ee113320-17777 2020. 1, 6, 21
[14] Jose M Dana, Aleksandras Gutmanas, Nidhi Tyagi, Guoying Qi, Claire O' D'Dnovvn, Maria Martin, and Sameer Velankar. Sifts: updated structure integration with function, taxonomy and sequences resource allows 40-fold increase in coverage of structure- annotations for proteins. Nucleic acids research,
47(D1):D482-DeL DD899, 2019. 24
[15] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning based protein sequence design using proteinmpnn. Science, 378(6615):49 56, 2022.
11

---

[16] Pierre-Paul De Breuck, Matthew L Evans, and Gian-Marco Rignanese. Robust model benchmarking and bias-imbalance in data-driven materials science: case study on modnet. Journal of Physica: Condensed Matter, 33(40)4404002, 2021. 27
[17] Mehmet F Demirel, Shengchao Liu, Siddhant Garg, Zhenmei Shi, and Yingyu Liang. Attentive walk- aggregating graph neural networks. TMLR, 2022. 1, 6, 21
[18] Jacob Devlin, Ming-Wos Wei Chang, Kenton Loe, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018. 6
[19] Weitao Du, Yuanqi Du, Limei Wang. Dieqiao Feng, Guifeng Wang, Shuiwang Ji, Carla Gomes, and Zhi-Ming Ma. A new perspective building efficient and expressive 3d equivariant graph neural networks, 2023. 29, 32, 33,37
[20] Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie-Yan Liu. Se (3) equivariant graph neural networks with complete local frames. In International Conference on Muchine Learning, pages 5583-5608. PMLR, 2022. 5, 10, 29, 32,
[21] Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materials property prediction methods: the mathench test and automatminer reference algorithm. arXiv.org, 2020. 9, 51
[22] Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain, Benchmarking matcrials prop- crty prediction methods: the mathench test set and automatminer reference algorithm. npj Computational Muterials, 6(1):138, 2020. 27, 48
[23] David Duvenaud, Dougal Maclaurin, Jorge Hirzel, Alán Aspun--Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. arXiv preprint arXiv: 150p:0999.9222 2015. 1, 6, 21
Rafael Gómmez-ombarellit Timothy
[24] Nadav Dym and Haggai Maron. On the universality of rotation equivariant point cloud networks. arXiv preprint arXiv:2010.0220 2020. 33
[25] Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, and Heng Ji. Translation between molecules and natural language. arXiv preprint arXiv:2204. 18817 2022. 10
[26] Carl Edwards, ChengXiang Zhai, and Heng Ji. Text2mol: Cross-modal molecule retrieval with natural language queries. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 595-607, 607, 2021.
[27] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Wang Yu, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language of lifes code through self- self-supervised deep learning and high performance computing. IEFE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2021.
[28] Thomas Engel and Johann Gasteiger. Applied chemoinformatics: achievements and future opportunities. John Wilcy & Sons, 2018. 50
[29] Hehe Fan. Zhangyang Wang, Yi Yang, and Mohan Kankanhalli. pentinuous-discree convolution for geometry-scquence modeling in proteins. In The Eleventh International Conference on Learning Representations, 2023. 9, 31,1, 34, 50,50, 51
[30] Xiaomin Fang, Lihang Liu, Jicqiong Lei, Donglong He, Shanzhuo Zhang, Jingho Zhou, Fan Wang, Hua Wu, and Haiteng Wang. Chemrl-gem: Geometry enhanced molccular representation lcarning for proporty prediction. arXiv preprint 1rXiv:2106.061
2021. 3
[31] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019. 2
[32] Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. International Conference on Machine Learning, pages -176. PMLR, 2020. 5, 32
[33] Daniel Flam-Shephin Shepherd and Alán Aspuru-Guze Guzik. Language models can generate molecules, materi- als. and protein binding sites directly in three dimensions as xyz, cif, and pdb files. arXiv preprint 055000008.. 2023. 3.
12

---

[34] Richard A Friesner. Jay L Banks, Robert B Murphy, Thomas A Halgren, Jasna J Klicic, Daniel T Mainz, Matthew P Repasky, Eric 1 Knoll, Mee Shelley, Jason K Perry, al. Glide: new approach for rapid, accurate docking and scoring. 1. method and assessment docking accuracy. Journal of medicinal chemistry, 4737):1739-1 1749, 2004. 8, 24
[35] Fahian Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling. teans-rrasss 3d roto- translation equivariant attention networks. arXiv preprint arXiv:2006. 10503, 2020. 6, 34, 36, 51
[36] Johannes Gasteiger, Shankari Giri, Johannes Margraf, and Stephan Günnemann. Fast and uncertainty-
aware directional message passing for non-equilibrium molecules. 2020. 7, 24
[37] Mario Geiger and Tess Smidt. c3nn: Fuclidean neural networks. arXiv preprint arXiv:220..0000000 2022.
5, 30, 33, 51
[38] Justin Gilmer, Samuel S Schocnholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference machine learning, pages 1263 1272. PMLR, 2017. 1, 6, 21, 30
[39] Vladimir Gligorijevié, P Douglas Renfrew, Tomasz Kosciolek, Julia Kochler Leman, Daniel Berenberg, Tommi Vatanen, Chris Chandler, Bryn Taylor. Ian M Fisk, Hera Vlamakis, al. Structure-based protein function prediction using graph convolutional networks. Nature communications, 12(1))3168, 2021. 8, 25
[40] Sai Krishna Gottipati, Boris Sattarov, Sufeng Niu, Yashaswi Pathak, Haoran Wei, Shengchao Liu, Simon Blackhurn, Karam Thomas, Connor Coley, Jian Tang, et al. Learning to navigate the synthetically accessible chemical space using reinforcement learning. In International Conference Machine Learning, pages 3668-3679. PMI.R, 2020.
[41] Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance- Lafrance-Vanasse. Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson. Protein design with guided discrete diffusion, arXiv preprint arXiv:2305. 2023.
[42] Thomas A Halgren. Merck molecular force field. basis, form, scope, parameterization, and performance of mmff94. Journal of computational chemistry, 17(5-6):40--51, 1996. 50
[43] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017. 6, 21
[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 770-778, 2016.
36
[45] Pedro Hermosilla, Marco Schäfer, Matěj Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora Kozliková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3d protein structures. 2020. 8, 24, 34, 51
[46] Brian Hie, Varun Shanker. Duo Xu, Theodora UJ Bruun, Payton A Weidenbacher. Shaogeng Tang, Wesley Wu, John E Pak, and Peter S Kim. Efficient evolution of human antibodies from general protein language models. Nature Biotechnology, 2023.
[47] Jie Hou, Badri Adhikari, and Jianlin Cheng. Deepsf: deep convolutional neural network for mapping protein sequences to folds. Bioinformatics, 34, 2018. 8, 25, 26
[48] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.
[49] Elton P Hsu. Stochastic analysis manifolds. Number 38. American Mathematical Soc., 2002. 5, 29
[50] Qian-Nan Hu, Hui Zhu, Xiaobing Li, Manman Zhang, Zhe Deng, Xiaoyan Yang, and Zixin Deng. Assignment of numbers enzymatic reactions with reaction difference fingerprints. PloS one, 7(12)e52901 -e00622,00--2901,,., e52901, 2012. 24
[51] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec, Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint trXiv:2005.00633 2020. 10
[52] Michael J Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee whye Teh, and Hyunjik Kim. Lietransformer: Equivariant self attentton for lie groups In International Conference on Machine Learning, pages 4533 4543, PMLR, 2021. 5, 10, 32
13

---

[53] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances neural information processing xystems, 32, 2019. 31
[54] Clemens Isert, Kenneth Atz, and Gisbert Schneider. Structure-basse drug design with geometric deep learning. Current Opinion in Structural Biology, 79:102548, 2023.
[55] Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materials, 111):01 1002, 2013.
27, 51
[56] Justina Jankauskaité, Brian Jiménez-García, Justas Dapkūnas, Juan Fernández-Rccio, and lain H Moal. Skempi 2.0: an updated benchmark of changes in namics upon mutation. Hioinformatics, 35(3):662-469 2019. 9, 26
hinding energy, kinetics and thermody-
[57] Jan H. Jensen. A graph-basch genetic algorithm and generative model/monte carlo tree search for the exploration chemical space. Chemical science, 10(12):3567 3572. 2019.
[58] Yuanfeng Lu Zhang, Jiaxiang Wu, Bingzhe Wu, Long-Kai Huang, Tingyang Xu, Yu Rong, Lanqing Li, Jie Ren, Ding Xue, et al. Drugood: -ut-tf-distributtin (ood) dataset curator and benchmark for ai-aided drug discovery-a focus on affinity prediction problems with noise annotations. arXiv preprint arXiv:2201. 006377777.7 2022. 3
[59] Rui Jiao, Jiaqi Han. Wenbing Huang, Yu Rong. and Yang Liu. 3d equivariant molecular graph pretraining. arXiv preprint 220.:2207700824 2022. 3,5, 34, 52
[60] Rui Jiao, Jiaqi Han, Wenbing Huang, Yu Rong. and Yang Liu. Energy-motivated equivariant pretraining for 3d molecular graphs. arXiv preprint arXiv:2207.08824. 2022. 5, 39
[61] Bowen Jing. Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffusion for molecular conformer generation. arXiv preprint arXiv:2206.0177 2022.
[62] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael L Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. 2020. 34, 51
[63] J.M. Enzyme nomenclature: prepared by edwin c. webb, academic press. 1992. #34.00 (xiii 862 pages) isbn 0 12 227165 3. Trends in biochemical sciences (Amsterdam. Regular ed.), 18, 1993. 8
[64] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídck, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583 5899, 2021. 1,5
[65] Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks, 2019. 33
[66] Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint 1609v 00000,,099..... 2016. 1, 6, 21
[67] Johannes Klicpera, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. In Conference on Neural Information Processing Systems (NeurIPS), 2021. 5, 6,
34, 31
[68] Johannes Klicpera, Shankari Giri, Johannes T Margraf, and Stephan Günnemann. Fast and uncertainty- aware directional message passing for non-equilibrumm molecules. arXiv preprint arXiv:201 141115, 2020. 5, 6, 7, 23, 51
[69] Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123, 2020. 35
[70] Mario Krenn, Florian H"ase, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru Guzui. Self- referencing embedded strings (selfies): A 100% robust molecular string representation. Machine Learning: Science and Technology, 1(4):045024, 2020. 6, 21
[71] Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Critical assessment of methods of protein structure prediction (casp)-round xiii. Proteins: Structure, Function, and Bioinformatics, 87(12):101
2019. 9, 26
[72] Greg Landrum et al. RDKit: A software suite for cheminformatics, computational chemistry, and predictive modeling. 2013. 50
14

---

[73] Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint arXiv:2206.11 1990, 2022. 5, 6, 33, 34, 51
[74] Chen Lin, Ying Zou, Ji Qin, Xiangrong Liu, Yi Jiang, Caihuan Ke, and Quan Zou, Hierarchical classification of protein folds using novel ensemble classifier. PloS one, 8(2)=56499,, 2013. 8
[75] Chen Lin, Ying Zou, Ji Qin, Xiangrong Liu, Yi Jiang, Caihuan Ke, and Quan Zou. Hicrarchical classification of protein folds using a novel ensemble classifier. PloS one, 80)x:56499-<55499, 2013. 25
[76] Meng Liu, Youzhi Luo, Limei Wang, Yaochen Xie, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang. Yi Liu, et al. Dig: A tumkey library for diving into graph deep learning research. The Journal of Machine Learning Research, 22(1):10873-100 10881, 2021. 2
[77] Meng Liu. Youzhi Luo, Limei Wang, Yaochen Xie, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang. Yi Liu, Keqiang Yan, Haoran Liu, Cong Fu, Bora M Oztekin, Xuan Zhang, and Shuiwang Ji. DIG: A turnkey library for diving into graph deep learning research. Journal of Machine Learning Research, 22(240):1-9, 2021.
[78] Shengehao Liu, Mehmet F Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised representa-
tion for graphs, with applications to molecules. NeurlPS, 32, 2019. 1, 21
[79] Shengchao Liu, Weitao Du, Zhiming Ma. Hongyu Guo, and Jian Tang. A group symmetric stochastic differential equation model for molecule multi-modal pretraining. In International Conference on Machine Learning, 2023. 3, 5, 6, 10, 31. 34, 39. 52
[80] Shengchao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with se (3)-invariant denoising distance matching. arXiv preprint arXiv:2206. / /3602, 2022. 3, 39
[81] Shengehao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with SE(3)-invariant denoising distance matching. In ICLR, 2023. 34, 52
[82] Shengchao Liu, Yingyu Liang, and Anthony Gitter. Loss- task weighting to reduce negative transfer in multi-task learning. In Proceedings the AAAI conference on artificial intelligence, volume 33, pages 9977-9978, 2019. 3
[83] Shengchao Liu, Weili Nie. Chengpeng Wang. Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang. Chaowei Xiao, and Anima Anandkumar. Multi-modal molecule structure-text model for text-based retrieval and editing. arXiv preprint 12v: 10789, 2022. 10
[84] Shengchao Liu, Meng Qu, Zuobai Zhang, Huiyu Cai, and Jian Tang. Structured multi-task learning for molecular property prediction. In International Conference on Artificial Intelligence and Statistics, pages 8906 89206899..0000 PMLR, 2022.
[85] Shengchao Liu, Chengpeng Wang, Weili Nie. Hanchen Wang, Jiarui Lu, Bolei Zhou, and Jian Tang. Grapheg: Unsupervised discoverr of steerable factors in graphs, 2023.
[86] Shengchao Liu, Hanchen Wang. Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang, Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728, 2021. 3, 5, 34, 39, 52
[87] Shengchao Liu, Jiongxiao Wang, Yijin Yang, Chengpeng Wang, Ling Liu, Hongyu Guo, and Chaowei Xiao. Chatept-powered conversational drug editing using retrieval and domain feedback. arXiv preprint arXiv:2305.180% 2023. 10
[88] Shengchao Liu, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Anthony Gitter, Chaowei Xiao, Jian Tang. Hongyu Guo, and Anima Anandkumar. A text-guided protein design framework. arXiv preprint 2XXi:2302.046/1 1, 2023. 1, 10
[89] Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint 0000v:2102.05013, 2021. 5, 6, 7, 23, 34, 35, 51
[90] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608. 2016. 23
[91] Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint 203449200440349, 2020. 1
15

---

[92] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. bioRxiv, 2021.
[93] Jan Mostowski and Joanna Pietraszewicz. Quantum versus classical angular momentum. 2019. 33
[94] Alexey Murzin, Steven E. Brenner, Tim Hubbard, and Cyrus Chothia. Scop: of proteins database for the investigation of sequences and structures. Journal of molecular biology. 247 247(4):536-540, 1995. 8, 25
structural classification
[95] Albert Musaelian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. arXiv preprint w2x.:2204.002455 2022. 34, 36, 45, 48, 51
[96]
Emmy Noether and M. A. Tavel. Invariant variation problems. 2005. 28
[97] Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L Chevrier, Kristin A Persson, and Gerbrand Ceder. Python materials genomics (pymatgen); A robust, open-source python library for materials analysis. Computational Materials Science, 68:314-319, 2013. 22
[98] Donald Perkins. Introduction to high energy physics. CAMBRIDGE university press, 2000.
[99] Zhuoran Qiao, Matthew Welborn, Animashree Anandkumar, Frederick Manby, and Thomas Miller. Orbnet: Deep learning for quantum chemistry using symmetry-adapted atomic-orbital features. The Journal of chemical physics, 153(12), 2020. 5, 10, 32
[100] Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014. 23
[101] Ladislav Rampášck, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for General, Powerful, Scalable Graph Transformer. Advances in Neural Information Processing Systems, 35, 2022. 1, 21
[102] Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O'Reilly Media, 2019. https://wwx amazon. com/ www.--iiieeeeeeee-eeeeee-e-ee
Deep-Learning-Liffe
[103] Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John F. Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. bioRxiv, 2021.
[104] Roshan M Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure learners. bioRxir, 2020. 1
[105] Patrick Reiser, Andre Eberhard, and Pascal Friederich. Graph neural networks in tensorflow-keras with raggedtensor representation (kgenn). Software Impacts, page 100095, 2021. 48,
[106] David Rogers and Mathew Hahn. Extended-connectivig fingerprints. Journal of chemical information
and modeling, 50(5):742-75) 754, 2010. 6, 21
[107] Andrew S Rosen, Shaelyn M Iyer, Debmalya Ray, Zhenpeng Yao, Alan Aspuru Guzik, Laura Gagliardi, Justin M Notestein. and Randall Q Snurr. Machine learning the caantum-chemica properties of metal- organic frameworks for accelerated materials discovery. Matter, (553:1578-1997 2021. 9,27
[108] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. arXiv preprint arXiv:2102.09844, 2021. 6, 34, 36, 51
[109] Kristof T Schütt, Huziel E Sauceda, P-J Kindermans, Alexandre Tkatchenko, and K-R Miller. Schnet-a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241722, 2018. 5, 6, 7, 23, 34, 35, 51
[1 [0] Kristof T Schütt, Oliver T Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial propertics and molecular spectra. arXiv preprint 15X10:2100.001/50 2021. 23, 33, 34, 51
[111] Chence Shi, Minkai Xu, Hongyu Guo, Ming Zhang, and Jian Tang. A graph to graphs framework for retrosynthesis prediction. In International Conference on Machine Learning, pages 8818 8818-8827. PMLR, 2020.
16

---

[112] Tess Smidt, Nathaniel Thomas, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and auuuiaaion-equivariann neural networks for 3d point clouds. arXiv preprint arXiv: 1et..0222.99999 2018. 5, 30, 35,
[113] Antonia Stank, Daria B Kokh, Jonathan C Fuller, and Rebecca C Wade. Protein binding pocket dynamics. Accounts of chemical research, 49(5):809-45 815, 2016. 8,
[114] Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günne- mann, and Pietro Lió. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pages
PMLR, 2022. 5, 34, 39, 52
[115] Bing Su, Dazhao Du. Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji-Rong Wen. A molecular multimodal foundation model associating molecule graphs with natural language. arXiv preprint urXiv:2209.05480 2022. 10
[116] Ruoxi Sun, Hanjun Dai, Li Li, Steven Kearnes, and Bo Dai. Energy- based view of retrosynthesis. arXiv preprint 2vXiv:2007.13437 2020. 1
[117] Ruoxi Sun, Hanjun Dai, and Adams Wei Yu. Rethinking of graph pretraining on molecular representation. 2022. 42
[118] Raphael JL Townshend, Martin Vögele, Patricia Suriana, Alexander Derry, Alexander Powers. Yianni Laloudakis, Sidhika Balachandar. Brandon Anderson, Stephan Eismann, Risi Kondor, et al. Atom3d: Tasks on molecules in three dimensions. arXiv preprint 20Xx:20022.04355, 2020. 8, 9, 10, 24,
[119] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.1090) 2017. 1,
[120] Soledad Villar, David W. Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. Scalars are universal: Equivariant machine learning, structured like classical physics. 2021. 33
[121] Hanchen Wang, Jean Kaddour, Shengchao Liu, Jian Tang, Matt Kusner, Joan Lasenby, and Qi Liu. Evaluating self-supervised learning for molecular graph embeddings. arXiv preprint arXiv:2206.0806
2022. 42
[122] Limei Wang, Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning hierarchical protein representa- tions via complete 3d graph networks. 2022. 9, 34, 35, 51
[123] Renxiao Wang, Xucliang Fang, Yipin Lu, and Shaomeng Wang. The puhbind database: Collection of binding affinities for protein- ligand complexes with known three-dimensional structures. Journal of medicinal chemistry, 47(1))2977-2980 2004. 8, 24
[124] Renxiao Wang, Xueliang Fang, Yipin Lu, Chan Yie Yang, and Shaomeng Wang. The puhbind database: methodologies and updates. Journal of medicinal chemistry, 48(12):111-4113 2005. 8, 24
[125] Shiyu Wang, Xiaojie Guo, and Liang Zhao. Deep generative model for periodic graphs. arXiv preprint arXiv:2201.11932 2022. 1
[126] David Weininger. Smiles, a chemical language and information system. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1 1):31366, 1988. 6, 21
[127] Hermann Weyl. Symmetry, volume 47. Princeton University Press, 2015. 4
[128] Tian Xie, Xiang Fu, Octavian-Eugen Ganca, Regina Barzilay, and Tommi Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. arXiv preprint arXiv:21 10.06 2021. I
[129] Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and
interpretable prediction of material properties. Physical review letters. 155311),, 2018. 3, 22
[130] Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanic Jegelka. How powerful are graph neural networks?
arXiv preprint arXiv: 18100088,..00866 2018. 1, 6, 21
[131] Zhao Xu, Youzhi Luo, Xuan Zhang. Xinyi Xu. Yaochen Xie, Meng Liu, Kaleb Andrew Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3d: A benchmark for predicting 3d geometries from molecular graphs, 2021. 10
[132] Kevin Yang. Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 59(8))3370-333 2019. 1, 6, 21, 35
17

---

[133] Huaxiu Yao, Ying Wei, Long- Kai Huang, Ding Xue, Junzhou Huang, and Zhenhui Jessie Li. Functionally regionalized knowledge transfer for low-resource drug discovery. Advances in Neural Information Processing Systems, 34:8256-8228 8268, 2021.
[134] Huaxiu Yao. Xinyu Yang. Xinyi Pan, Shengehao Liu, Pang Wei Koh, and Chelsea Finn. Leveraging domain relations for domain generalization. arXiv preprint arXiv:2302.02600 2023.
[135] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng. Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? Advances Neural Information Processing Systems, 34:28877 28888, 2021. 1, 6, 21
[136] Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee whye Teh, Alvaro Sanchez- Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206. 00133, 2022. 3, 5, 34, 39,
[137] Chengxi Zang and Fei Wang. Moflow: an invertible flow model for generating molecular graphs. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 617-626, 2020.
[138] Anthony Zec. Group theory in a nutshell for physicists, volume 17. Princeton University Press, 2016. 4, 28
[139] Zheni Zeng. Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridging molecule struc- ture and biomedical text with comprehension comparable human professionals. Nature communications, 13(1):862, 2022. 10
[140] Linfeng Zhang. Jiequn Han, Han Wang. Roberto Car, and EJPRL Weinan. Deep potential molecu- lar dynamics: a scalable model with the accuracy quantum mechanies. Physical review letters. 120(14):14001,, 2018.
[141] Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Jiazhang Lian, Qiang Zhang, and Huajun Chen. Ontoprotcin: Protein pretraining with gene ontology embedding, arXiv preprint 22X1v2200.11144 2022.
[142] Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelic Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. 2022. 34, 35, 51
[143] Haiteng Zhao, Shengchao Liu, Chang Ma. Hannan Xu, Jie Fu, Zhi-Hong Deng, Lingpeng Kong, and Qi Liu. Gimlet: A unified graph-text model for instruction- based molecule zero-shot learning. biokxiv, pages 2023-05, 2023. 10
[144] Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. 2023. 10
[145] Zhaocheng Zhu, Chence Shi, Zuobai Zhang, Shengehao Liu, Minghao Xu, Xinyu Yuan, Yangtian Zhang, Junkun Chen, Huiyu Cai, Jiarui Lu, et al. Torchdrug: A powerful and flexible machine learning platform for drug discovery. arXiv preprint arXi:22202.088 2022. 2, 51
18


Reviews:
[{"review_id": "xDKfEKgzD5", "reviewer": "Reviewer_8p61", "summary": "", "questions": "", "limitations": "See weakness.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The presented framework lays a solid foundation for further research on ML tasks, particularly in the field of deep learning.\n2. The pipeline's natural and intuitive design encompasses dataset preprocessing, feature extraction, geometric pretraining, representation, and target tasks.\n3. The experiments conducted cover a diverse array of fields and applications, enriching numerous scientific research projects.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "NsHPe2OdW7", "reviewer": "Reviewer_m8T2", "summary": "", "questions": "", "limitations": "Yes, the authors have acknowledged the limitations of this study and plan to address them in future versions of this paper.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. provide a novel overview perspective on existing geometric models as symmetry-informed and classify them into three categories\n2. comprehensive over the area of molecular geometric representation learning, take all small molecule, protein and material into consideration\n3. extensive experiments with 16 geometric models and 14 geometric pretraining methods across 46 different tasks. The inclusion of ablation studies on training tricks further enhances the paper's credibility, providing robust evidence to support the findings and insights.\n4. build a well-organized and easy-to-use platform with methods, datasets, evaluation, and training tricks, serving as a valuable tool for future researchers. This practical aspect facilitates the replication of experiments and encourages further exploration and development in the field.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "zdCjyHrZz8", "reviewer": "Reviewer_pq7Y", "summary": "", "questions": "", "limitations": "The limitations of the work are adequately addressed.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The main strength of the work lies in the great variety of models and datasets that the authors have brought under a common framework. It takes a substantial amount of effort to unify such a large number of disparate architectures, and having these available for the research community to test for their own applications and datasets should be immensely helpful.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Oo66opj4sE", "reviewer": "Reviewer_XV1e", "summary": "", "questions": "", "limitations": "It is not clear how new methods can be fairly compared with those methods tried in this manuscript. Many other existing methods were not included in this benchmarking study. It is not clear how to expand the comparison to them.\n\nNo negative societal impact detected.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "In the past few years, many geometric representation learning models have been reported. However, it is hard to compare them as their experimental results were not always comparable. This study attempts to solve this problem. The community needs such a work.\nShow that it is important to consider data preprocessing and optimization tricks for specific datasets/tasks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "S03uAqBAfg", "reviewer": "Reviewer_1hwb", "summary": "", "questions": "", "limitations": "To the best of my abilities, I think the authors discuss limitations. This is, however, a bit outside my area of expertise, so I may be missing something.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The framework gives useful insights into the various heuristics that are employed within the studied models, which helps us better understand what makes a model useful. This is a valuable contribution.\n* It is useful to have a framework to compare different geometric models in a consistent manner.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper constitutes a substantive contribution to molecular geometric representation learning in AI. The authors offer an exhaustive survey, benchmark, and the Geom3D platform, which integrates numerous public datasets, cutting-edge methods, and evaluative metrics. This consolidated model facilitates deep learning on 3D systems relevant to chemistry, materials science, and biology, and its utility extends across diverse scientific disciplines.

The paper introduces a new viewpoint on symmetry-informed geometric models, categorized into three types, and performs robust experiments with 16 geometric models and 14 geometric pretraining methods across 46 tasks. Inclusion of ablation studies on various training techniques provides essential insights into their performance impacts. The findings of some guidance for method selection for specific applications.

The paper's integration of multiple methods and datasets into a common framework will be useful to method developers and end-users. Given its potential benefits to the wider research community and society, the paper is recommended for acceptance.
Decision: Accept (Poster)