SaVeNet: A Scalable Vector Network for Enhanced Molecular Representation Learning
Authors: Sarp Aykent
Tian Xia
Venue: NeurIPS 2023 poster (2023)
Keywords: geometric deep learning
molecule property prediction
geometric representation learning
TL;DR: We introduce a scalable message-passing framework for molecular representation learning.

Abstract: Geometric representation learning of molecules is challenging yet essential for applications in multiple domains. Despite the impressive breakthroughs made by geometric deep learning in various molecular representation learning tasks, effectively capturing complicated geometric features across spatial dimensions is still underexplored due to the significant difficulties in modeling efficient geometric representations and learning the inherent correlation in 3D structural modeling. These include computational inefficiency, underutilization of vectorial embeddings, and limited generalizability to integrate various geometric properties. To address the raised concerns, we introduce an efficient and effective framework, Scalable Vector Network (SaVeNet), designed to accommodate a range of geometric requirements without depending on costly embeddings. In addition, the proposed framework scales effectively with introduced direction noise. Theoretically, we analyze the desired properties (i.e., invariance and equivariant) and framework efficiency of the SaVeNet. Empirically, we conduct a comprehensive series of experiments to evaluate the efficiency and expressiveness of the proposed model. Our efficiency-focused experiments underscore the model's empirical superiority over existing methods. Experimental results on synthetic and real-world datasets demonstrate the expressiveness of our model, which achieves state-of-the-art performance across various tasks within molecular representation learning.

Sa A Scalable Vector Network for Enhanced
Molecular Representation Learning
Sarp Aykent Auburn University sarp@auburn. edu
Tian Xia Auburn University tian0auburn edu
Abstract
Geometric representation learning of molecules is challenging yet essential for applications in multiple domains. Despite the impressive breakthroughs made by geometric deep learning in various molecular representation learning tasks, effectively capturing complicated geometric features across spatial dimensions is still underexplored due to the significant difficulties in modeling efficient geometric representations and learning the inherent correlation in 3D structural modeling. These include computational inefficiency, underutilization of vectorial embeddings, and limited generalizability integrate various geometric properties. To address the raised concerns, we introduce an efficient and effective framework, Scalable Vector Network (SAVeNet), designed to accommodate range of geometric requirements without depending on costly embeddings. In addition, the proposed framework scales effectively with introduced direction noise. Theoretically, we analyze the desired properties (i.e., invariance and equivariant) and framework efficiency of the SAVENET. Empirically, we conduct comprehensive series of experiments to evaluate the efficiency and expressiveness of the proposed model, which achieves state-of-the-art performance across various tasks within molecular representation learning.
Introduction
The field of geometric deep learning (GDL) has seen rapid expansion in recent years, thanks to the successful application of Graph Neural Networks (GNNs) for modeling graph structures [4]. The
ability to learn complex geometric representations has driven significant breakthroughs across various disciplines and has proven to be particularly beneficial in diverse areas such social science [24]. physics [14], and neuroscience [6]. Within the realm of molecular representation learning, the use of message-pas pawpagg--pssinnn-baeee
GNNs has demonstrated remarkable outcomes, especially in understanding the 3D structures of molecules.
Although geometric deep learning has recently seen promising results in various applications in molecular representation learning, its full potential in the field is still untapped. To begin with, notable research gap persists within this field: the trade-off between expressiveness and efficiency in GNNs. Many studies often resort to the use of multi-hop neighbors [9, 17, 21] or complex embeddings [19] in their pursuit to augment the expressiveness of gvessage-passingg based GNNs [9, 29, 33]. While these approaches can enhance the expressiveness and predictive power of the model, they invariably compromise scalability and efficiency due to their computationally intensive nature. As these networks grow more expressive, they demand more computational resources. impeding their ability to scale to larger or more complex datasets. Addressing the challenge of efficiently handling molecular structures to sufficiently capture the complete, multi-level structural information during learning without compromising the computational efficiency of the network is of critical importance. Our approach circumvents these issues by avoiding using multi-hop GNNs and
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

expensive embeddings. Novel approaches for initializing and processing vector-typed embeddings have been designed to work with vector representations with the goal of achieving a balance between maintaining numerical stability, facilitating faster convergence, and enhancing the model's l ability to generalize to new datasets [25,33] Utilizing one-hop vector embeddings work with our proposed SAVENET significantly enhances the network's s efficiency, demonstrating their effective integration in this context. Figure provides a visual illustration of the advantages of our proposed SAVENET, which compares our method with state-of-the-ar methods on the QM9 dataset. The values on the x-axis show the latency of the model, and the values on the y-axis show the standard mean absolute error (std. MAE) of the models' performance. Because lower is better for both latency and std. MAE, the best model in both efficiency and expressiveness is the one that is closest to the (0,0) point. In terms of performance, our base model, denoted as SAVENET -B, surpasses the existing state-of-the-art method Equiformer [19], while concurrently enhancing training duration efficiency by factor of 14.6. This substantial improvement underscores the efficacy of our model in achieving superior results with increased efficiency.
Furthermore, although several prior works [2, 3, 12, 18, 19, 25, 27, 29, 30] proposed equivariant neural networks that handle scalar and vector representations together for molecular representation learning, one primary problem with these existing works is that vector- valued representations show limited performance or scalability and can- equally contribute towards performance improvements in the network [12, 18, 29]. It remains challenging scale the models to improve the expressiveness of the model further. We propose an efficient framework to tackle this challenge. Our model's scala- bility is exemplified by its capacity stack N layers, effectively serving as an encoder for learning the latent geometric represen- tations, thereby augmenting the model's expressiveness. These superior scaling capabilities enhance the model's expressiveness and effec- tively distinguish our work from existing research, which often struggles to scale efficiently [9, 19]. The scalability of our approach is not merely theoretical but is empirically demonstrated Section 5.1. Figure| illustrates one example of the scaled larger version of our model: SAVENET -L, which exhibits enhanced capability in contrast to our base model and existing baselines. This advancement substantiates the prospective capacity of our model to scale effectively, thereby indicating potential for superior performance in future implementations.
1.0
* PaNN
DimeNet++
ComENer
0.9
SphereNet
EQGAT
std. MAE
ET
0.7-
SaVeNet-B
Equiformar
* SaVeNet-L
0.6
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Latency (ms)
Figure 1: Comparison of our proposed SAVENET and the state-of-the art methods: latency vs. std. MAE on QM9 dataset.
Our paper's 's main contributions are: (i) Proposed an efficient and effective wessage-passing framework coupled with several decoder mechanisms for leveraging the geometric representations within the 3D molecular graphs. (ii) Novel approaches for initializing and processing vector-typed embeddings have been designed to work with vector representations to ensure numerical stability, speed up model convergence, and improve the model's ability to adapt to new datasets. (iii) A thorough evaluation of the expressiveness, efficiency, and scalability our models compared with existing benchmarks, as well as detailed ablation studies, underscore the advantages of our proposed SAVENET.
2 Related work
Equivariant Graph Neural Networks The properties of graph representations can change with transformations such as translation, rotation, and permutation. By design, GNNs inherently permuta- tion equivalence [15]. While it is possible to introduce diverse transfornations to the network through data augmentation, this approach can lead to computational inefficiencies.
The concepts of invariance and equivariance have been long acknowledged as critical to the success of various tasks [2, 13, 35, 36]. Recent research in molecular representation learning has predominantly addressed this challenge with equivariant network architectures. Within the realm of molecules,

---

numerous equivariant models have been proposed for tasks such as chemical property prediction [16], protein structure prediction [13], and energy prediction [17].
Molecular Property Prediction Over recent years, there have been considerable advancements in molecular property prediction [9, 16, 19, 28]. 'hese developments have been primarily driven by the adoption of graph models for molecular representation, coupled with the application of kernel methods for structural learning. Such kernels operate based on geometric features, such as interatomic distances, which are encoded using radial basis or Bessel basis functions. Several other studies employ spherical embeddings to model paasaagg [3, 9, 17, 19, 21, 33). Dimenet++ [9], GemNet [17] and SphereNet [21] depend on a multi-hop message- m to model angular properties, which introduces computational complexity that curtails scalability with respect to the model size of these approaches. These advanced representations have resulted in more precise property predictions, including but not limited to dipole moment, HOMO-LUMO gap, energy, and force. However, the precision of these predictions is accompanied by increased computational complexity due to their dependence on quadratic or even cubic operations. Recent works have strived to address this issue, proposing innovative techniques that improve computational efficiency with minimal or no detriment to prediction accuracy [21, 33]. These novel techniques continue to operate on higher-order features during feature generation, underscoring the ongoing challenge of balancing computational efficiency with predictive precision. Both ET [30] and EQGAT [18] incorporate attention mechanisms into vector representations. However, the complex interactions in these works often limit the scalability and efficiency of the networks. More recently, networks such as SEGNN [3] and Equiformer [19] have utilized Clebsch-Gordar tensor products to achieve high performance. However, these tensor products are empirically found to be significantly slower than the previously discussed methods, posing challenge when applying such methods to large, real-world datasets.
Our model utilizes 1-hop geometric representations, offering a balance of expressivity and com- putational efficiency. This design choice circumvents the need for complex and potentially lossy conversions often associated with higher-order representations. By harnessing the rich information in 1-hop representations, we maintain data integrity and enhance model performance without substantial computational overhead.
3 Preliminaries
Notations. We formulate the atomic structure graph representation, where each input molecule is represented as a graph و (V, V is the set of N nodes in the graph, each node representing an atom in the molecule, and S V x V is the set of edges that connects the nodes. The average degree of node, denoted by k. represents the average number of edges per node. The geometric information of a node for each v E V is represented by the coordinates c = (I ys. 2,) in the Cartesian coordinate system. The scalar representations generally include properties like relative distance r, while the vector representations V include direction vectors d. We use @ to denote the element- wise (Hadamard) product and || ||2 to denote the row-wise 2 norm. The cross-product between two vectors is denoted as x.
+ Y is said to be equivariant to a transformation if applying the transformation to the input of the function is the same as applying the transformation to the output of the function. Formally, for transformation T, a function f is equivariant if for ((T())). This means that the function preserves the structure of the
Invariance and Equivariance. A function f: X-
all x X: (()()) transformation. On the other hand, a function is invariant to a transformation if its output does not change when the transformation is applied its input. Formally, for a transformation T', a function f is invariant if for all x E X: f(T(z)) = f(x). This implies that the function is invariant to the transformation; the function's 's output remains unchanged regardless of the applied transformation.
Euclidian Transformations. In this paper, we frequently use the special orthogonal group SO(3), comprising all 3x3 rotation matrices R that satisfy R R I and det(R) = 1, and the special Euclidean group S F33,, consisting of all 4 x 4 special Fuclidean transformation matrices that can R t
be written as T 0
where R 5O(3), E R is a translation vector, and 0 is a row vector
of zeros. These matrices satisfy T- and det(T) = 1. SFF(3 forms a manifold, the product of SO(3) and 3 allowing operations combining rotations and translations in 3D space.
3

---

Scalable Vector Network
Our goal design message-passing neural network that aware of the geometric characteristies of the molecular graph, with consideration of expressiveness and efficiency, and preserve these characteristics beneficial for downstream tasks. We first introduce the representations used in SAVENET and how we handle these geometric representations. Then we describe the overall mussage-passing framework to model the geometric properties in the graph and show how integrate our proposed framework into the representation learning process on downstream tasks.
4.1 Efficient Representations for 3D graphs
The initial representations fed into the network are instrumental in achieving complete geometric representations [21, 33J. It is crucial to efficiently represent geometric graphs without the loss of their properties. Our framework operates by employing embeddings from one-hop neighbors exclusively, Invariant properties are encoded as scalar values, which enable the network to maintain invariance without resorting to computationally expensive operations. For equivariant properties, we extract the inherent directional information from the input graphs and encode these features into vector representations. The directional information between nodes i and j is represented with Bi and defined as follows:
B = d.j
(()))) F.j = - -c
(a (c)
o.j o5= diy Eig
R
(I)
Téj
Here, represents the average of and x denotes the cross-product operation. With the geometric representation defined, the efficacy of our scalar and vector representations is further demonstrated in the following lemma:
Lemma 1. Consider a known geometric graph with at least one node, and assume that each node in this graph has least one connection. Let new node j be added to this graph, stipulating that j is connected to at least one existing node. Then, the position P3 of node j can be determined in constant time using the (Pj. dij) properties.
The proof for Lemma is provided in the Appendix A. Lemma can be further generalized, requiring no positional information by translating the po reference point to the origin using to translation vector. In light of this, we propose the following theorem, which demonstrates the SAVENET's ability to reconstruct the rotation- -quivvrinquivariant geometric structure of input graph:
Theorem 1. Assuming the input graph is strongly connected, the input space of SAVENET, defined as (r, d), is capable of reconstructing the rotation- caaiionn-quvvriiint geometric structure of the input graph. The reconstructed graph maintains the relative spatial configuration of nodes and edges in the input graph, the translation transformation required for exact alignment with the original graph.
The proof for Theorem 1 is provided in Appendix B. The vec- tor representations are an effective way model equivariant networks and are widely used networks [11, 12, 25, 29] on different challenging tasks. With the proof of Lemma 1 and Theorem 1, we present provably efficient and lossless ge- ometric encoding for the 3D graphs without reliance on the multi-hop neighborhoods. Despite the vector representation offering effective encodings, several unresolved challenges hin- der its successful application towards enhancing the model's 's expressiveness and scalability. In response to this, we propose the incorporation of direction noise initialization and vector activation class within our SAVENET framework, specifically designed to address this issue.
VEN(F))
Figure 2: Direction noise.
Direction Noise. Vector representations are typically initialized
using dataset-specific features [12, 25], such as direction vectors between sequential neighbors in protein's s amino acid sequence. However, in the absence of such data, existing works [18, 29] often default to initializing these vectors null, allowing the network to learn these vector representations intrinsically. Such initialization can unnecessarily consume computational resources, particularly in the network's 's initial layers. In this work, as illustrated in Figure 2, we propose a novel strategy of

---

introducing directional noise utilizing the spherical coordinate system as an initialization measure for the vector representations generated based on the node type. This approach is designed mitigate numerical instability issues and empower the network to explore the latent space we propose through direction noises. We follow the following convention to generate a direction vector for a given node type F:
VEN(F:) -
((c))(0))) - {7(in Og cos Φb), (sin((0)) sin(())),,((co(((() 0, E lR di ER]
where 0g and o; are learnable parameters for node type and y is control parameter governing the magnitude of the direction embeddings. This approach effectively generates direction vectors using the spherical coordinate system. The value of is associated with the progress of the training, similar learning rate schedulers. The value of y is adjusted throughout training. following a decay schedule that ensures it reaches zero before training concludes. This methodology offers an effective means for creating direction vectors using a spherical coordinate system while allowing for the adaptive control of the direction embeddings" magnitude throughout training.
Vector Activations. Applying similar activation functions used in scalars on vector representations, where the function operates on each element of the vector independently. can break the model's equivariance. This because these functions are not designed to handle the geometric transformations that the vectors might undergo. Consequently, SAVENET introduces vector activation function in which the activation is computed based on the vector's L norm. For vector representation V, the vector activation function is written as follows:
VA(V) (V @ aI(VI +) V E (
where o, is a sealar activation function. To further regulate the activation, the vector activation function incorporates bias term b. where b is the same shape as the representation dimension C. Our experimentation found that vector activation functions significantly enhance the numerical stability during training, particularly with larger-scale models.
4.2 SAVENET for 3D Graphs
This section describes the overall architecture of SAVENET in detail. The overall SAVENET frame- work is -quivaaqui message-passing neural network incorporating the vector initialization and activation for scalable molecular representation learning. It based on an encoder-decoder structure comprising multiple stacked encoder and decoder layers. The encoder layers learn the latent representation of the geometric features, and the decoder layers decode the learned latent representations for downstream tasks on either invariant or equivariant targets.
Encoder. The encoder in SAVENET follows the message- passing paradigm and incorporates the above-defined geometric representations. The main two components of the encoder are inter-atomic interactions followed by atom-wise blocks.
Vector Updates. Vector updates are employed to compute channel-wise interactions in vector representations. By conducting identical operations across spatial dimensions, vector updates ensure that the equivariance property maintained. More specifically, given a vector representation V e RC3 the vector update applics a learnable weight matrix W e RCxC", where and C" denote the dimensions of the input and output representations, respectively. Therefore, the output vector V is computed as V WV.
Inter-atomic interactions. Our model encapsulates inter-atomic interactions by integrating both equivariant direction vectors and invariant distance filters, which are encoded using radial basis functions. We have designed the scalar interaction path to interface with the vector interactions in manner that preserves the equivariance property, achieved by appropriately scaling the magnitudes of the vectors. The overall formulation of the inter-atomic interactions can be expressed as follows:
I TA(s, V,r, (ss, Vi) ES e(sj, Vj, Bgy (s/, Vi)
S 8; E Co(s)) Vir Bkg))
V = V/++ Ejex. eu(oi, Vi, rig By)
5

---

In this equation, Mú represents the set of neighbors of node i, and e is the interaction function, denotes the equivariant vectors. To accommodate a variety of equivariant properties, we fuse these vectors with vector updates VA. The interaction function is formalized as follows:
(s Vg, reys B,y) o,(ss) no (ray)
(5)
(8 V3, rig. (.z) = (1ag)) o oa(s;) @ ma(ris) + V, o ( (ri))
where e denotes sequentially stacked perceptron layers and denotes distance encoding. Specifically, @b serves as vector update layer as it operates on vector imputs. The distance encoding calculated using the formula ni(Ta)) = or (r(y)) w(ri))), where x(-) denotes a basis function and w-() stands for a cutoff function.
Atom-wise blocks. Following the execution of the interaction block, the workflow proceeds to the atom-wise blocks, which responsible for computing interactions between invariant and equivariant representations, as well as performing channel-wise updates. The formal expression of this process is defined
AW V) = = {0m (I (V)Il (VA(oeu(V) @ (V)
(6)
where (fm,e) denotes multi-layer perceptron (MLP) and 6mn. denotes vector updates. Here, the vector representations interact with scalar representations via the L norm of updated vector representations. Moreover, the scalar representations scale the vector representations after applying channel-wise interactions &.. This process presents bi-directional communication between different representations, thus enhancing the model's ss ability to learn and extract meaningful information from data.
Decoder. The framework utilizes decoder variations to accommodate various geometric requirements of given task. By taking the latent representation pair (se. Ve) from the output of the encoder, the decoder can effectively leverage these representations to maximize task performance. For invariant largets where the tasks do not depend global rotations, transformations, and permutation, the decoder takes encoded Se representations and processes them with MLP layers. For equivariant targets where tasks sensitive global rotations. the decoder takes the scalar and vector represen- tations and processes them with stacked AW blocks. The final vector representations are scaled with scalar representations generate the predictions. If the task is graph level, final representations aggregated with global sum pooling.
Equivariance and Invariance. Addressing the strategic initialization of the model and the design of operations that uphold invariant and equivariant properties is equally important. SAVENET uphold such properties, which can be stated more formally as follows:
Theorem 2. The equivariant representation of SAVENET is equivariant to any given rotation matrix in R E S3(3). The invariant representation of SAVENET is invariant to any given transformation matrix in SE(3).
The formal proof of Theorem 2 is provided in Appendix C, demonstrating the validity of these assertions and the robustness of the model under various transformations.
5 Experiments
This section evaluates our proposed SAVENET on three tasks in geometric representation learning over both syn- thetic and real-world datasets. We im- plement the model with Py PoTch [26]. All experiments are conducted on an NVIDIA 3090 GPU with 24 GB mem- ory. All models use the Adam op- timizer [22] for the optimization. De- tailed descriptions of the datasets are compared in Table 1, which includes the average number of nodes, the average number of edges, and the total number
Table 1: Dataset details. The number edges are com-
puted with radius graph where d( = 5.0
Datasets
QM9
N-Body
Molecule3D
Avg. # of Nodes
18.02
20
29.11
Max # of Nodes
29
20
137
Avg. # of Edges
280.72
380
553.00
Graphs
130,831
7,000
3,899,647
# of Tasks
12
4
Splits
84:8:8
42:29:29
6:2:2
Invariant
X
Equivariant
/

---

Table Performance comparisons on QM9. denotes using different data partitions.
Task
A
FHOMO meV
ELUMO meV
M
C,
G
I
It²
Uo meV
ZPVE meV
std.
log
Units
45
meVV
D
cal
meV
meV
me
%
NMP
.022
69
43
38
.030
040
19
17
.180
20
20
1.50
1.78
-5.08
SchNet
.235
63
41
34
.0330
.330
14
14
.073
19
14
1.70
1.76
-5.17
Commorant
.885
61
34
38
.038
026
20
21
-961
21
22
2.03
2.14
-4.75
LicCony!
.884
49
30
25
.032
038
22
24
.800
19
19
2.28
1.35
-4.99
PhysNet
.061
42.5
32.9
24.7
.0529
.2800
9.4
8.42
.765
8.34
8.15
1.39
1.37
-5.35
DimeNet
.047
34.8
27.8
19.7
.029
.0249
8.98
8.11
.331
7.89
8.02
1.29
1.05
-5.57
DimeNet++
.044
32.6
24.6
19.5
.0297
.2300
7.56
6.53
.331
6.28
6.32
1.21
0.98
-5.67
TEN
.23
58
40
38
064
.011
SBC3)-TH.
.142
53
35
33
.051
054
EGNN
071
48
29
25
.029
031
12
12
.106
12
11
1.55
1.23
-5,43
PaiNN
.05
45.7
27.6
20.4
.0120
024
7.35
5.98
.066
5.83
5.85
1.28
1.01
-5.5
ET
.059
36.1
20.3
17.5
.011
026
7.62
6.16
.033
6.38
6.15
1.84
0.84
-5.90
SphereNet
.046
31.1
22.8
18.9
.0245
0215
7.78
6.33
.268
6.36
6.26
1.12
0.91
-5.73
ComENet
045
32.4
23.1
19.8
.0245
.0200
7.98
6.86
.259
6.82
6.69
1.20
0.93
-5.69
SEGNN1
.060
42
24
21
.023
.031
15
16
.660
13
15
1.62
1.08
-5.27
EQGAT
053
32
20
16
.011
.024
23
24
.382
25
25
2.00
0.86
-5.28
Equiformer
.06
30
15.4
14.7
.0117
.0230
7.63
6.63
.251
6.74
6.59
1.26
0.70
-5882
SAVENET-B
.099
24.8
18.4
16.3
.0033
.0227
6.64
5.43
.058
5.48
5.43
1.18
0.69
-6.04
SAVENET-I.
.035
22.7
16.6
15.1
.0085
.0210
6.10
4.83
.049
4.74
4.83
1.10
0.63
-6.14
of graphs. The three datasets include both invariant and equivariant targets for comprehensive eval- uation of our proposed model. Further dataset details are provided in Appendix D. The rationale behind our design decisions and hyperparameters, as well as their specific values, are documented in Appendix F for reference.
QM9 dataset was used evaluate the performance and efficiency of the models across twelve tasks for invariant target predictions. We present two distinct configurations of the SAVENET model, distinguished by their respective sizes: The SAVENET-B denotes the base variant, while the SAVENET-L. is the larger model. We report various baseline models, including NMP [10], Schnet [28]. Cormorant [1], LieConv [7]. PhysNet [32], Dimenet [16], Dimenet++ [9], TFN [31], SE(3) Transformer [8], EGNN [27]. PaiNN [29]. ET [30], SphereNet [21]. ComENet [33], SEGNN [3],
QM9:
EQGAT [18], and Equiformer [19].
N-Body: To assess the effectiveness of SAVENET with equivariant tasks, we conducted experiments
on the extended N-Body dataset with harder targets [5]. The proposed SAVENET is compared with
GNN, Tensor Field Networks [3 1. SE(3) Transformer [8], Radial Fields, PaiNN [29], EGNN [27], ClofNet [5], GCPNet [25]. Most of the baseline results adopted from [5, 25J. The results for
PaiNN are not reported in previous work [25] due to numerical stability. Notably, we incorporated
vector activation functions into PaiNN's interaction layers address this issue, allowing to include
PaiNN in the performance comparison.
Molecule3D: The Molecule3D dataset, possessing over 29 the quantity of graphs as the QM9 dataset and an approximate 1.6x and 1.9x increase in the average number of nodes and edges respectively, is employed evaluate the scalability of the model with respect to dataset size. For the Molecule3D dataset, we adopted the baselines as per [33]. J. As only the gap target has been reported in preceding works, utilized the hyperparameters provided in [33] for evaluating additional targets. The details of the hyperparameters are described in Appendix F.1 The comparative baseline models for the Molecule3D include GiN-Virtual [33], Dimenet++ [9], Spherenet [21]. and ComENet [33].
Evaluation Metrics. A set of metrics are used to measure the performance of the models. For the QM9 and Molecule3D datasets, each task is assessed using the Mean Absolute Error (MAE). For the N-Body dataset, the performance is measured by the Mean Squared Error (MSE) the future position predictions. Given that each dataset encompasses multiple tasks, we employ two additional aggregate measures: standardized (std.) and logarithmic (log) metrics, to assess overall performance. As the error be disproportionately influenced by a few outliers, report the logarithmic error to prevent these outliers from dominating the evaluation.
Efficiency Analysis. To critically assess the efficiency of the models, we employ a set of metrics including training speed, model complexity, and memory consumption. All the measurements are conducted utilizing hyperparameters from the respective model authors, and the reference imple-
7

---

Table Efficiency comparison of the proposed SAVENET with the baselines on QM9.
Model
std.
log MAE
Batch Sizet
Memory GB +
Latency
Samples/s
FLOPs
Param.
MACs
Metric
MAE +
ms +
t
G
M
G
DimeNet++
0.98
-5.67
357
4.82
38.9
1646
30.15
1.89
15.08
ComENet
0.93
-5.69
1174
0.96
26.4
2422
14.36
3.81
7.18
SphereNet
0.91
-5.73
238
6.65
51.6
1240
31.80
1.89
15.90
EQGAT
0.86
-5.28
980
1.74
53.3
1200
17.00
0.93
8.49
Equiformer 0.70
-5.83
94
14.04
337.3
190
3.53
SAVENET-B 0.69
-6.04
1660
0.85
23.1
2767
11.60
1.37
5.76
SAVENET-L 0.63
-6.14
580
2.48
50.6
1237
55.36
7.72
27.51
mentations are based on [18-20, 33]. To ensure consistent comparison, the experimental runs are executed on the same configurations utilizing an RTX 3090 GPU with 24 GB of graphics memory. The details the efficiency analysis procedure are described in Appendix and the theoretical time complexity is analyzed in Appendix E.
Training Speed. To gauge the training speed of each model, we recorded the time consumed by single forward/backward pass, expressed milliseconds. Alongside this latency, we computed the number of samples that could be processed per second, employing similar measurement techniques.
Model Complexity. The complexity of the models was evaluated by considering two key parameters: floating point operations (FLOPs) and multiply-accumut were computed utilizing the DeepSpeed library [23], with consistent batch size.
operations (MACs). These metrics
Memory Consumption. The memory consumption of each model was evaluated by assessing the amount of memory utilized across multiple batches. After processing each batch, the cache was cleared to ensure realistic appraisal of memory usage. However, this practice of cache clearing significantly impedes the evaluation process, rendering a full dataset evaluation impractical.
5.1 Results
Accuracy results on QM9. Table 2 listed the experimental results of the QM9 dataset. SAVENET- demonstrated superior performance on seven out of the twelve targets, surpassing all baseline models on these measures. While SAVENET-B did not yield the highest performance for certain бномO, éLUMO, C., R², and ZPVE- competitive results, indicating its robustness across diverse molecular properties. SAVENET-L. further improved on SAVENET-B's performance, securing the best results on nine out of the twelve targets, with the exceptions of EHOMO. ELUMO. and R² Furthermore. in terms of the overall performance measures, SAVENET-L outperformed all other models, exhibiting the lowest standardized MAE and logarithmic MAE, reflecting its superior accuracy across all target properties. These results highlight effectiveness of our proposed models, particularly SAVENET-L, in accurately predicting a wide range of molecular properties in the QM9 dataset.
noteworthy that still delivered
Table 4: Performance comparisons on N-Body dataset.
Method
ES(5)
ES(20)
G+ES(20)
L+ES(20)
std.MSE
logMSE
GNN
0.0131
0.0720
0.0721
0.0908
0.0151
-4,8866
TFN
0.0236
0.0794
0.0845
0.1243
0.0192
-4.0978
SE(3)-Tr.
0.0329
0.1349
0.1000
0.1438
0.0252
-3,0037
Radial Field
0.0207
0.0377
0.0399
0.0779
0.0110
-4.2122
PaiNN
0.0158
0.0997'
0.1029
0.1356
0.0835
-2.5329
ET
0.1653
0.1788
0.2122
0.2989
0.0535
-2.5587
EGNN
0.0079
0.0128
0.0118
0.0368
0.0044
-5.2241
ClofNet
0.0065
0.0073
0.0072
0.0251
0.0029
-6.0324
GCPNET
0.0070
0.0071
0.0073
0.0173
0.0024
-6.1104
SAVENET
0.0062
0.0063
0.0082
0.0123
0.0020
-6.2268

---

Efficiency results on QM9. The results of efficiency experiments are presented in Table 3. We select the top five models based on their std. MAE performance for the efficiency comparison. Regarding efficiency, SAVENET-B stands out with its superior results, processing the highest number of samples per second and the lowest memory consumption. This demonstrates an impressive balance between performance and resource utilization. Additionally, SAVENET-B shows the lowest latency and number of FLOPs and MACs. The latency of SAVENET-B is 14.5x lower compared to Equiformer, the previously most performant model with the lowest std. and log MAE. Even with significant reduction in compute time, SAVENET-B surpasses the Equiformer in performance, illustrating efficiency advancements of SAVENET without performance compromise.
SAVENET-L manages to maintain a competitive efficiency despite its larger size. Its memory consumption is 2.6x and 5.6x times smaller compared to high- terh-peerorming models such as SphereNet and Equiformer, respectively. Notably, SAVENET-L. processes 1237 samples per second, which comparable to the rates of SphereNet and EqGAT. Compared to Equiformer, the baseline model with the highest accuracy, SAVENET-L. exhibits approximately 6x lower latency. This suggests that SAVENET-L, even with its larger size, can maintain high performance without sacrificing speed.
Results on N-Body dataset. Table 4 presents the results of the N-Body systems. SAVENET outperformed all baseline methods on the ES(5), ES(20), and L+ES(20) tasks, whilst demonstrating competitive performance on the G+ES(20) task. Remarkably, SAVENET achieved the lowest average MSE and logarithmic MSE across all four tasks, illustrating its superior prediction precision for equivariant targets. These results emphasize the capability of our proposed model accurately predict the future positions of particles in N-Body systems, even under challenging conditions where particles are under the influence of the gravity and force fields.
Table 5: Performance comparisons on Molecule3D dataset.
Symmetry Task
Invariant As
Equivariant j7
/
HOMO
LUMO
std. MAE
logMAR
GIN-Virtual
0.0882
0.0692
0.0632
0.1036
0.0592
-2.6677
1.5233
SchNet
0.0532
0.0275
0.0265
0.0428
0.0263
-3.6633
1,2082
DimeNet++
0.0293
0.0240
0.0190
0.0306
0.0188
4.0139
1.2014
SphereNet
0.0288
0.0239
0.0183
0.0301
0.0184
-40337
1.1836
ComENet
0.0345
0.0288
0.0252
0.0326
0.0220
-3.8403
1.3521
SAVENET-B
0.0183
0.0190
0.0173
0.0290
0.0156
-4,2257
0.0108
SAVENET-L
0.0136
0.0159
0.0143
0.0239
0.0128
-4,4088
0.0090
Accuracy results on Molecule3D. Our proposed models, SAVENET-B and SAVENET-L, out- performed several baseline models on the Molecule3D dataset. In terms of relative performance, SAVENET-B outperforms all tasks compared to other established models. SAVENET-B showed improved performance by approximately 37% the μ task and 92% the jł task compared to the next best model, SphereNet. Our S SAVENET-L model, further improved the performance of SAVENET-B. For the μ4 task, SAVENET-L showed an improvement of approximately 26% over SAVENET-B; for the jī task, it showed an improvement of about 17%. Regarding aggregate measures, SAVENET-L also outperformed all other models, showing relative improvement of around 18% in the std. MAE metric and 5% in the log MAE metric compared to SAVENET-B, These results demon- strate the superior performance and scalability of our proposed models, especially SAVENET-L, on the Molecule3D dataset.
Efficiency results on Molecule3D. The results of the efficiency evaluations the Molecule3D dataset are illustrated in Table 6. Impressively, SAVENET-B continues to dominate with respect to efficiency, manifesting the highest sample processing rate and minimal memory consumption. This shows an exemplary trade-off between accuracy and resource management.
'The results are reported after adding vector activation functions. These results were initially unavailable due numerical instability.

---

Table 6: Efficiency comparison of the proposed SAVENET with the baselines on Molecule3D.
Model
std.
log
Batch Sizet
Memory
Latency
Samples/s
FLOPs
Param.
MACs
Metric
MAE +
MAE +
GB +
ms 4
t
G
M 1
G G
DimeNetPP
0,0188
-4,0199
62
12.14
84.2
704
70.53
1.89
35.27
SphereNet
0.0184
-4.0327
43
17.32
134.6
448
92.56
1.90
46.28
ComENet
0.0220
-3.8403
277
3.37
40.7
1536
47.25
7.36
23.63
SAVENET-B
0.0156
-4.2257
329
2.84
27.8
2304
15.33
1.05
7.63
SAVENET-L.
0.0128
-4.4408
159
5.37
51.9
1216
38.93
3.01
19.37
5.2
Ablation Study
Table 7 presents the results of our ab- lation study on proposed SAVENET model, where we investigate the impacts of Direction Noise (DN), Vector Activa- tions (VA), and varying the number of lay- ers (L) on the QM9 dataset. In Table 7, report standard Mean Absolute Error (std. MAE), the percentage change std. MAE, and the logarithmic Mean Absolute Error (logMAE), respectively. The importance DN is illustrated in the first two rows, where the std. MAE is reduced by 2.5%. Moreover, scaling the model with DA from six to eight layers yields 7.8% reduction in error. Introducing VA into an eight-layer model further improves the performance by 2.8%. Lastly, expanding the model to 12 layers offers substantial performance improvement of 8.7%.
Table Ablation study of SAVENET
Model
L
std.
Astd.
log
SAVENET DN VA
6
0.79
-5.85
SAVENET VA
6
0.77
-2.5%
-5.95
SAVENET VA
8
0.71
-10.1%
-5.98
SAVENET
8
0.69
-12.7%
-6.04
SAVENET
12 0.63
-20.3%
-6.14
6 Conclusion, Limitations and Future Works
This paper has focused on learning geometric representations of molecular structures. We introduced Sa VeNet, scalable pessage-passing neural network designed for learning geometric representations from 3D graphs. Furthermore, we implemented a unique approach integrate geometric features, capturing complex geometric relationships within 3D structures without resorting to computationally complex operations. The performance of this framework was demonstrated on both synthetic and real-world datasets. For the latter, Sa VeNetet was evaluated across various core tasks in molecular representation learning. When compared with state-of-the-arr methods, the effectiveness and ef- ficiency of our proposed architecture were clearly validated. Our comprehensive examination of network efficiency and our comprehension of the trade-offss between expressiveness and efficiency will provide foundation for future advancements in molecular representation learning. However, like many studies in this area, our work does have limitations. The primary constraint, shared with the baseline model discussed in this paper, is the dependency known geometric structures of molecules at their equilibrium state. In practice, obtaining accurate 3D confirmations can be costly. An in- triguing direction for future research is explore end-to-end framework that employs sequential representations molecules, such as SMILES strings [34], to generate their 3D geometry. where the predicted 3D geometry could then be used model molecular properties. Another promising avenue is to investigate the performance of larger variations of SaVeNet. Furthermore, assessing the impacts of efficiency improvements larger models, potentially achieved through shared parameters in various configurations, is worthwhile endeavor. We anticipate that our findings will pave the way for future investigations in this domain, potentially leading even more efficient and effective modeling techniques.
10

---

References
[1] Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32, 2019.
[2] Sarp Aykent and Tian Xia. GBPNet: Universal Geometric Representation Learning on Protein Structures, In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery Data Mining, KDD '22, Washington, DC, USA, 2022. Association for Computing Machinery.
[3] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling. Geometric and physical quantities improve e(3) equivariant message passing, In International Conference on Learning Representations, 2022. URL /ttpss//opeeneraat www.weww.eeefooeine///oo id= 1dd__xwr@@BBe1.
[4] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34 (4):18-42. 2017.
[5] Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie- Yan Liu. SE(3) equivariant graph neural networks with complete local frames. In Kama- lika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 Proceedings of Machine Learning Research, pages 5583-5608, PMLR, 2022. URL https: ttttps://pree //ppoc////.cccciiga....m mlr wwwppeeeesleeee heeess/vv12/ee wwwp/du22e..eeeeeeeeaeaaaaeaaaae html.
[6] Timothy W Dunn, Jesse D Marshall, Kyle S Severson, Diego E Aldarondo, David GC Hilde- brand, Selmaan N Chettih, William L Wang. Amanda Gellis, David E Carlson, Dmitriy Aronov, al. Geometric deep learning enables 3d kinematic profiling across species and environments. Nature methods, 1865):544-77,, 2021.
[7] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference Machine Learning, pages 3165-3176. PMLR, 2020.
[8] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Transfrrrssssrmmrs: In Advances in Neu-
3D Koto-Translation Equivariant Attention Networks. ral Information Processing Systems, volume 33, pages 1970-1981. Curran As- httpsseerrr.p..ppeeeeeeeeeeeeeee
sociates, Inc. URL //ppce/eingg./.....oo neurips. wttriiss.ccpaappee pptpp/app//////aaa/// 00P33aafeeeebb7a1333776226688883 Abstract. html.
[9] Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and uncertainty-awwre directional message passing for non-cquilibrium molecules. In Machine Learningfoo for Molecules Workshop, NeurlPS, 2020.
[10] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning. pages 1263-1272. PMLR, 2017.
[11] Bowen Jing. Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. International Conference on Learning Representations, 2021. URL htttps://oppereeeeee wtwparvvviieeeee/orrr.... id=1YLJDvSx6J4.
[12] Bowen Jing, Stephan Fismann, Pratham N Soni, and Ron O Dror. Equivariant graph neural networks for 3d macromolecular structure. arXiv preprint arXiv:2106.d0 106.08333, 2021b.
[13] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool Russ Bates, Augustin Židek. Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589, 2021.
[14] Yashar Kiarashinejad, Mohammadreza Zandehshahvar, Sajjad Abdollahramezani, Omid Hem- matyar, Reza Pourabolghasem, and Ali Adibi. Knowledge discovery in nanophotonics using geometric deep learning. Advanced Intelligent Systems, 2(2): 2020.

---

[15] Thomas N. Kipf and Max Welling. Semi-supervine classification with graph convolutional networks. International Conference on Learning Representations (ICLR), 2017.
[16] Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations, 2020.
[17] Johannes Klicpera, Florian Becker, and Stephan Günnemann. Gemnet:
Universal directional graph neural networks for molecules. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https:
//openreiiew net
[18] Tuan Le, Frank Noé, and Djork-Amé Clevert. Equivariant graph attention networks for molecular property prediction. arXiv preprint urXiv:2202.0989) 2022.
[19] Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In International Conference on Learning Representations, 2023. URL attps://openreviet: www.errreeennnnt/oommnnm.n net//frummiddd K
[20] Meng Liu, Youzhi Luo, Limei Wang, Yaochen Xie, Hao Yuan, Shurui Gui, Haiyang Yu, Zhao Xu, Jingtun Zhang, Yi Liu, Keqiang Yan, Haoran Liu, Cong Fu, Bora M Oztekin, Xuan Zhang, and Shuiwang Ji. DIG: A turnkey library for diving into graph deep learning research. Journal of Machine Learning Research, 22(240):1-9, 2021. URL http:// jmlrr wwwplllorgg/pppppe pappps/pprs/2 21-0343. html.
[21] Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. In International Conference on Learning Representations, 2022. URL https: /atp:://pppeeeeeeeeee wwtpeeewwwwweeeeeeeeeeeeeeeeeeee idggsn
[22] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://opeareviev.neettt id-Bkg6RiCqy7.
[23] Microsoft. Deepspeed: A deep learning optimization library for pytorch. https://githin
com//nicrosooft/Deppppeee 2020.
[24] Federico Monti, Fabrizio Frasca, Davide Eynard, Damon Mannion, and Michael M Bron- stein. Fake news detection on social media using geometric deep learning. arXiv preprint arXiv:1902.06673, 2019.
[25] Alex Morehead and Jianlin Cheng. Geometry-complete perceptron networks for 3d molecular graphs. AAAI Workshop on Deep Learning on Gruphs: Methods and Applications, 2023.
[26] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[27] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) Equivariant Graph Neural Networks. In Proceedings of the 38th International Conference on Machine Learning, pages 9323 9332. PMLR. URL
mlr www.mm..r..sss.. ptress/v13//satorreerrrr
html.
[28] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Rober Müller. Schnet: A continuous- convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
[29] Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pages 9377-9388. PMLR, 2021.
[30] Philipp Thölke and Gianni De Fabritiis. Equivariant transformers for neural network based molecular potentials. In International Conference on Learning Representations, 2022.
[31] Nathaniel Thomas, Tess Smidt, Steven Kearne,, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equiiariint neural networks for 3D point 08219.
clouds. URL http: // /arxivv org/
12

---

[32] Oliver T Unke and Markus Meuwly. Physnet: A neural network for predicting energies, forces, dipole moments, and partial charges. Journal of chemical theory and computation, 15(6): 3678 -698- 2019.
[33] Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. ComENet: Towards complete and efficient message passing for 3d molecular graphs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https / //opnreveeeeeeeee net/ orunmiidmmC2MeeeWS..
[34] David Weininger. Smiles, a chemical language and information system. 1. introduction methodology and encoding rules. Journal of Chemical Information and Computer Sci- ences, 28( 1):36,,, 1988. doi: 10. 10.1021/:1005570005 URL atttps://ooi 2tg/10.102/. ci00057a005.
[35] Tian Xia and Wei-Shinn Ku. Geometric graph representation learning on protein structure prediction. In Proceedings of the 27th ACM SICKDD Conference on Knowledge Discovery Data Mining, KDD '21, page 1873-1883, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 1450383325. doi: 10. 10.1145/3445548.466177 3467333 URL https: 3467323.
//doi.org/10
[36] Tian Xia, Sarp Aykent, and Wei-Shimi Ku. Extrinsic-intrinsi representation learning framework for drug discovery. In Proceedings of the 2023 SIAM International Conference Data Mining (SDM), pages 523-531. SIAM, 2023.
[37] Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3d: benchmark for predicting 3d geometries from molecular graphs. arXiv preprint arXiv:2110.01717, 2021.
13

---

Appendix
A Proof of Lemma 1
Lemma I: Consider a known geometric graph with at least one node, and assume that each node in this graph has least one connection. Let new node j be added to this graph, stipulating that j is connected to at least one existing node. Then, the position Pj of node j can be determined in constant time using the (Po, Taj.
dzz) properties.
Proof of Lemma 1: In our context, we dealing with geometric graph whose node properties known priori. When a new node j is appended to the graph, such that is connected to at least one node, we can select any node E .N from the neighbors of j to establish the position сy of node j.
Proceeding by contradiction, assume that multiple feasible positions Py could satisfy the conditions ||. - | Fij and dij for given position pt. In multi-dimensional spaces, including 3D, there exist multiple positions satisfying the distance constraint ||c; c,ll = raj- Specifically, any point lying on sphere in 3D space, with as the center and riy as the radius, could satisfy this condition. Yet, there is only one such position that can satisfy the direction vector di from e to cj. This fact contradicts our assumption, and hence concludes the proof.
B Proof of Theorem 1
Assuming the input graph is strongly connected, the input space of SAVENET defined (r, d), is capable of reconstructing the rotation-equivariatt geometric structure of the input graph. The reconstructed graph maintains the relative spatial configuration of nodes and edges in the input graph, the translation transformation is required for exact alignment with the original graph.
Theorem 1:
Proof of Theorem 1: By the definition of a strongly connected graph, for any given node i, there exists a path II to every other node j. Hence, we can arbitrarily choose any node from the graph as reference for reconstructing the ootation-equivarte select node for this purpose. Since e; is arbitrary, we can conveniently set it at the origin without loss of generality.
geometric structure of the input graph. Let's
Utilizing Lemma 1, we can identify the relative position of every other node j E V by traversing each edge on the path IT: from to j. In essence, the relative position of node j with respect to node i can be computed by the accumulation of relative positions along the path II,.
Assuming, by way of contradiction, that multiple relative positions of node j with respect to node i exist. However, this is counter Lemma 1, which affirms that for given G2 Fij. rij.dy. there exists unique location. This principle can be extrapolated to any number of nodes in the graph. Further, as we can ascertain the position of node after it's added the reference graph, we can similarly determine the position of any subsequent node k relative to V U {i}.
Thus, we conclude that the input graph's 's rotation-equivariant geometric structure can be reconstructed using (d, r) properties, thereby completing our proof.
C Proof of Theorem 2
Theorem 2: The equivariant representation of SAVENET is equivariant any given rotation matrix in R E 8O(3). The invariant representation of SAVENET is invariant to any given transformation matrix in SE(3).
Proof of Theorem 2: Our proof unfolds in two stages: first, we discuss the symmetrical properties of SAVENET in the input space. Second, extend these properties to the entire network.
Inputs. The input SAVENET consists of direction vectors, denoted 3. It's clear that these vectors, representing direction, maintain their properties under any global rotation. Formally, if (g) represents the function that extracts 3 from input graph G, we have that RG) fo(G)) for all
14

---

R E SO(3). Moreover, any noise introduced during initialization decays over time until reaches zero before the end of the training phase. Therefore, having no effect on the inference process.
The invariant input vectors of SAVENET, which represent atomic numbers and interatomic distances, clearly invariant under any transformation in S3)) because these properties do not depend on the geometrical information of the system. Hence, atomic numbers remain constant under global rotations and translations. Similarly, the distances between atoms, being local properties, remain unchanged under global transformations.
Network Propagation: Having established the symmetries at the imput level, we extend our analysis to the entire network of SAVENET. Two critical components are Inter-atomic Interactions and Atom-wise Blocks, which shall now scrutinize.
Inter-atomic interactions. The interaction function e can be dissected to analyze its operations on the equivariant and invariant branches. Here, Φb. 6a. and de denote the vector update layer, the linear transformation, and the multiplication of e representation with the linear transformation of sj. respectively. In the equivariant branch, operations can be expressed as:
Eo Vj, Tay. B.g) = (b(g)) @ oa(s)) @ na(ris) + V, o (s,)) & (rig)
(7)
Each component in the above expression preserves symmetry properties. More specifically, the first term @6 (B,y) preserves equivariance because the same transformation is applied across spatial dimensions. The second term, (s,), is linear transformation applied to an invariant, and therefore, remains invariant under any transformation in T. The third term, na(ray), which multiplies a basis function and a cutoff function for the distance rij. does not change under transformations in T. Thus, is also invariant under T.The fourth term, (sy)),, involves a multiplication operation of representation with a linear transformation of sj. hence, it produces equivariant vector since the linear transformation does not affect the spatial dimensions. Finally, the term 77 (Tay)) represents an invariant scalar, remaining unchanged under transformation.
The first part (B1g) @ (a(3;) @ ma(riy) involves the Hadamard product of an equivariant vector (o,) E RC33 with invariant scalar values (da(s;): na(ri)) RC). Hadamard product operates coment-wise and does not alter the spatial dimensions, hence the resulting vector preserves the equivariant properties. The second part V, O (n,))) @ 7) (riz) involves the Hadamard product of an equivariant vector (V, E Rx33 with an invariant scalar representations. Again, the resulting vector is equivariant since the same values are multiplied across spatial dimensions. Finally, the addition operation in 6, combines two equivariant vectors. As the addition of equivariant vectors preserves equivariance, the result will be equivariant under any rotation matrix R E SO(3). This concludes our examination of the equivariance of the interaction function e, under any rotation R E 5O(3). Similarly, the operations in the invariant update branch can be captured as:
es (8), Uj, Tij, B.g)) = (s,)) O 17s (Taj)
(8)
where o. denotes a linear transformation, and Π)s represents a transformation applied using basis and cutoff functions. Both these functions are invariant under any transformation in T'.
The updated and V representations, determined for each adjacent node, subsequently aggre- gated through summation. Given that summation inherently upholds symmetry characteristics, the procedures employed in the Inter-atonte interactions layer maintain the equivariant nature of V for all rotations R in (3)). as well as the invariant nature of s under any transformation T in SE33).
Atom-wise Blocks. Assuming that the inputs to this block are invariant for scalar representations and equivariant for vector representations, we can extend our proof to the operations within this block. The operations in the invariant update branch can be encapsulated as follows:
&n (Il (V) (V]
In this equation, de., denotes the vector update function which is, as previously established, equivari- ant under any rotation matrix Rt The norm operation || applied to V is invariant, as the magnitude of the vector does not change with rotation. Consequently, the composition |l (V)ll is invariant to global rotations. Furthermore, applying the linear transformation @. to the concatenation of two invariants, in this case ((cc(()) and s. remains invariant under any transformation in SE(3). The operations in the equivariant branch of the Atom-wise blocks can be described as:
VA (V) @ do ol (V)I
(10)
15

---

This equation is similar to the one describing the invariant update branch, with a difference lying in the linear transformation o, that uses different learnable parameters. Hence, the second term is invariant. The first term, @nu (V), is equivariant as it is vector update function applied to V. The Hadamard product of this equivariant representation with the invariant term preserves its equivariance under any rotation R E SO(3). In conclusion, the Atom-wise blocks preserve the symmetrical properties for both (s, V) representations. The proof for Theorem 2 is thus complete.
D Datasets
The three datasets used in this study evaluate proposed SAVENET three tasks in geometric representation learning over both synthetic and real-world datasets. In addition to their diversity in their real-world implications, but also in their requisite output specifications. In other words, they demand the prediction of both invariant and equivariant targets, each necessitating different computational considerations. This multi-faceted evaluation strategy, which includes varying input data types and output requirements, provides a comprehensive view of the model's capabilities, and furthers the understanding of its strengths and potential areas for refinement. By analysing the model's performance across these distinct scenarios, we aim to exhibit SAVENET 's adaptability and potential as a valuable tool in advancing the field of geometric representation learning.
cases, these tasks demonstrate rich diversity not just in their practical
QM9. Widely adopted small molecule property prediction dataset. It consists of 131,831 stable molecules with nine heavy atoms. We utilize the established split from [16], a standard in subsequent works [9, 21, 33]. QM9 contains twelve tasks that involve various aspects of molecular properties. The multi-task nature of the dataset demonstrates the effectiveness of the evaluated models.
Newtonian many-body system. Newtonian many-body system is a synthetic dataset that is generated based on particle simulations. The task is to predict the future positions of the particles based on the initial location and velocity. The performance was measured with the mean square error (MSE) metric between predicted and ground truth. Recently the existing work by [27] extended by increasing the particle size and introducing various foree fields. By introducing the force fields, the existing task became more difficult. The initial work evaluated the models' performance based on the future position on 5-body (ES(5)). More challenging tasks to predict future locations of the 20-body (ES(20)) task is introduced in [5]. Furthermore, complex conditions integrated with gravity field (G+ES(20) and Lorentz-like force field (L+ES(20)). The performances of the models were evaluated under challenging conditions. The
of our work SAVENET were chosen based on the best-performing hyper-parameters on ES(5) based on the best validation score.
Molecule3D. This dataset comprises 3,899,647 molecules, making it approximately 29.5 times larger than the QM9 dataset, providing significant platform to assess the scalability of our model. Predefined splits are utilized, based on random sampling as detailed in [37]. Though Molecule3D has fewer targets than QM9, with only four targets including dipole moment vector (ī²), energies of the highest occupied and lowest unoccupied molecular orbitals (GHOMo. fLUMO), and orbital energy gap (Ac), it offers an essential testbed for model evaluation. The dipole magnitude (f)) can be easily derived from f7, highlighting the significance of equivariance in models due to the influence of global rotations on μž. For fair comparison, have reported standardized MAE for invariant properties.
E Theoretical complexity analysis
In the following section, we present a de- tailed theoretical complexity analysis of our proposed model and baselines, which summarized in a comprehensive table for understanding the model's 's computa- tional efficiency. In the table, we provide a breakdown of the computational com-
Table 8: Complexity analysis.
Complexity
Dimenet++
SphereNet
ComENer SAVENET
Initialization
n
n'
n²
Eneoder
Ln Land
Lnnaaa Laud
L Land
Decoder
land?
Overall
a²d
x?d?
plexities associated with the main components of the model's architecture. This includes complexities during the graph embedding phase, the encoder, and the decoder phase. By presenting this theoretical complexity analysis, we aim to provide an in-depth understanding of the operational demands of our model and highlight its computational ad advantages.
16

---

Implementation details
The reference implementation can be found in http:///gghhbbbcmm////hicicuuuus
F.1 Hyperparameters
The hyperparameter search space of SAVENET-B on QM9 and Molecule3D is shown in Table The final hyperparameters chosen based the hyperparameters with the best performance the validation set of each dataset. The hyperparameters of S SAVENET-L is determined based on the best hyperparameters found for SAVENET-B the validation Then the hidden dimensions and the number of layers are scaled on SAVENET-L. On the QM9 dataset, SAVENET-L. has two times more hidden dimensions and decoder hidden dimensions compared SAVENET-B. SAVENET-L utilized 256 hidden dimensions compared to 128 dimensions, and SAVENET-L has 1.5 more layers increasing from eight layers twelve layers. On the other hand for the Molecule3D dataset, hidden dimensions are increased 1.5 times from 128 to 192 and the number of layers increased to eight layers from six layers. The hyperparameter search space of SAVENET on N-Body shown in Table 10.
Table 9: Hyperparameter search space of SAVENET-B on QM9 and Molecule3D.
Hyperparameters
QM9
Molecule3D
Epochs
1000
300
Batch size
32, 64, 128
64, 128, 256
Number layers
Hidden dimensions
64, 128
128, 192
Decoder dimensions
128, 256
256, 384
Number of RBF
12, 16, 24
12, 16, 24
Base Learning Rate
le-3, 5e-4, 2e-4
1e-3, 5e-4, 2e-4
Learning rate decay Learning rate patience
0.6, 0.8, 0.9
0.6, 0.8, 0.9
10, 15, 20
10, 15, 20 0.0, 0.1, 0.25
Vector dropout
0.0, 0.1, 0.25
Direction noise decay
150, 250
150, 250
Table 10: Hyperparameter search space of SAVENET-B on N-Body,
Hyperparameters
N-Body
Epochs
500
Number of layers
Hidden dimensions
128
Decoder dimensions
128, 256
Number of RBF
12, 16, 24
Base Learning Rate
le-3, 5e-4, 2e-4
Learning rate decay Learning rate patience
0.6, 0.8, 0.9
10, 15, 20
Vector dropout Direction noise decay
0.0, 0. 0.25
150, 250
G Efficiency evaluation
In this section, we conduct rigorous evaluation of our proposed model, employing set of care- fully devised efficiency metrics. These metrics are instrumental in providing a comprehensive understanding of the model's operational efficacy and practical viability.
The key metrics used include:
Training Speed: This metric is crucial in assessing the efficiency of the model from computational perspective. gauges the rate at which the model processes data adjusts parameters during train- ing, which is particularly pertinent in large-scale applications. For the purpose of deriving accurate
17

---

results, each model underwent a warm-up phase of 5,000 steps before we timed the forward/backward passes across the entire training dataset. The training dataset was deliberately left unshuffled to maintain consistent batches, thereby facilitating efficient and comparable measurements. The final result was then determined by calculating the average time spent on these passes.
Model Complexity: A model's complexity often directly correlates with its performance. However, balance between complexity and computational efficiency is desirable. Hence, we consider this metric to quantify the intricacy our model's architecture and algorithms, which also informs us about its interpretability and potential for scaling. Mirroring the approach adopted during the training speed experiment, we maintained uniform batches throughout the entire training dataset, and the resultant metrics were averaged upon the conclusion of an epoch. The metries were computed with consistent batch size of 64 to accommodate memory-demanding models.
Memory Consumption:
The amount of memory consumed during the model's operation significant factor, especially for deployment in resource-constrained environments. This metric provides insights into the model's memory requirements, both during the training phase. Since evaluating the complete dataset evaluation was impractical, strategy. To accommodate the impact various sample sizes on memory usage, we constructed worst-case scenario, selecting batches containing the largest samples in given dataset. An efficient strategy for identifying these substantial samples involved considering the number of nodes in each sample. Consequently,
employed an effective measurement
extracted the n largest samples from the dataset, where is equivalent to the batch size. The memory consumption was then repeatedly measured for this batch, and the results averaged to yield the final value.
18


Reviews:
[{"review_id": "k9a4194OAW", "reviewer": "Reviewer_6Z9X", "summary": "The authors propose an efficient and scalable equivariant GNN (SaVeNet) for 3D molecular conformations. The architecture follows an encoder-decoder style framework, where the encoder is composed of \"Inter-atomic Interactions\" and \"Atom-wise blocks\" learning mechanisms. Several other modeling augmentations are proposed such as 1-hop molecule embeddings in the feature space, non-null vector initializations (\"Direction Noise\"), and norm-aware vector activation functions (\"Vector Activations\"). ", "questions": "N/A", "limitations": "Yes -- the discussion in Section 5 was appreciated and meaningful.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The thrust of this paper is clear: designing GNNs with SO(3)/SE(3) equivariance can introduce complexity into the model. This slows down inference-time latency, and may inhibit scaling the model to larger sizes (i.e. a RNN with parameter capacity equal to a Transformer may have similar performance, but Transformers scale significantly better).  Accordingly, the authors develop an equivariant GNN that uses only one-hop neighborhood vector embeddings (i.e. cheap & scales), but (assuming a connected graph -- which is the case for molecules) this feature-space representation is sufficient to reconstruct the entire molecule. The empirical results align with the theory, and state-of-the-art results are attainted on a variety of 3D benchmarks.\n\nTo summarize, I believe the paper (a) tackles a relevant problem (b) proposes a relevant solution and (c) demonstrates theoretically and empirically that solution works. Therefore, I recommend acceptance.\n\nI'd also like to note the punctilious empirical evaluation---efficiency analyses can be especially difficult, but it's my opinion that the authors perform a fair comparison among methods to constitute their empirical findings. ", "weaknesses": "I am not an expert on equivariant GNNs and am not familiar with much of the related work. I am also unable to recognize any method/evaluation-specific \"red flags\" which may appear and defer to the expertise of the other reviewers. \n\nOne facet that stood out; however, is that the writing at times feels long. It could be significantly shortened in many areas. For instance, the abstract is 19 lines and could be made more \"pointed\". I would also suggest creating a \"Figure 1\" that is a visual depiction of the SaVeNet architecture to supplement the material in Section 3.2 on learning mechanisms. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6Z9X,\n\nWe would like to express our gratitude for taking the time to review our manuscript and providing detailed and constructive feedback. We appreciate your positive reception of our work and carefully considered each of your points and would like to address your comments as follows:\n\n**Strengths:** We're grateful that you recognized the relevance and significance of the problem we tackled, as well as our solution's novelty and empirical strength. Your acknowledgment of our empirical evaluation further motivates us to maintain high standards in our research endeavors.\n\n**Weaknesses:**\n\n1. **Length and Clarity:** We acknowledge your concern regarding the length of certain sections. In light of your feedback, we are committed to revising the manuscript to make the content more concise, especially in places like the abstract, to enhance readability and clarity.\n    \n2. **Visual Depiction of SaVeNet:** We take your suggestion regarding creating a  figure visually representing the SaVeNet architecture. We believe that such a depiction will undoubtedly aid readers in grasping the nuances of our model more intuitively. You can find the architecture illustration attached to the uploaded PDF Figure 2. We will ensure to include this visual aid in our revised submission.\n\n**Limitations:** Thank you for noting the value of the discussion in Section 5. We always strive for transparency in our work and believe that addressing limitations is crucial for both the integrity of research and future work in the area.\n\nOnce again, we thank you for your positive feedback, constructive criticism, and thoughtful suggestions. We're excited about the potential of our work in advancing the field, and your feedback plays an integral role in refining our contribution.\n"}, {"review_id": "lj8azB4rAr", "reviewer": "Reviewer_T8Du", "summary": "This paper proposes a framework called SaVeNet for geometric representation learning of molecules. The paper includes theoretical analysis and empirical experiments to demonstrate the superiority of SaVeNet over existing methods in terms of efficiency and expressiveness.", "questions": "1. I’m wondering if the direction noise and vector activation also applicable to other equivariant models?\n\n2. Can you add more equivariant baselines like PaiNN to Figure 1 for efficiency comparison? \n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "1. This paper proposes an efficient geometric encoding, a novel directional noising strategy in the spherical coordinate system, and a novel vector activation function.\n\n2. The experimental results show that SaVeNet achieves very good performance over multiple invariant and equivariant molecular tasks. \n\n3. As an equivariant model, the efficiency of SaVeNet is pretty good.\n", "weaknesses": "It will help readers to better understand if authors can add a figure of the proposed SaVeNet’s architecture, including the illustration of components described in Sec 3.1 and 3.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer T8Du,\n\nFirstly, we'd like to extend our gratitude for your detailed review and the constructive feedback provided on our manuscript. We would like to address your questions as follows:\n\n> Applicability of direction noise and vector activation to other equivariant models\n\nWe appreciate your thought-provoking query about the potential adaptability of our proposed direction noise and vector activation to other equivariant models. At its core, the novelty of our approach in integrating direction noise and vector activation is designed with scalability and robustness in mind. This adaptability, as you rightly inferred, opens the door to improving the performance of other existing models.\n\nFor instance, take the case of PaiNN. Prior works, such as [1], couldn't report PaiNN's results due to issues related to numerical stability. In response, we took the initiative to enhance PaiNN by incorporating our proposed vector activation functions into its interaction layers. This intervention not only resolved the aforementioned stability concern but also empowered us to compare PaiNN's performance with our model, as demonstrated in our performance comparison in Table 4.\n\nWhile our methods indeed present a promising avenue for refining other models, we acknowledge that custom tailoring might be essential. Achieving optimal results could require model-specific adjustments that respect the original architecture's nuances and intent.\n\nOur proposed framework, with its innovative features, serves as a template for upcoming research. While the direct adaptation of our innovations—namely the direction noise and vector activation—on other models was beyond this study's purview, we are confident that our findings lay the groundwork for subsequent explorations. We envisage a landscape where researchers, inspired by our findings, dive deeper into harnessing these concepts, eventually unearthing even more potent modeling techniques.\n\n> Addition of more equivariant baselines to Figure 1\n\nThank you for your constructive suggestion regarding the expansion of equivariant baselines in our efficiency comparison. We concur that broadening the range of baselines would offer a richer, more comprehensive insight into our analysis. While our initial evaluations were anchored around top-performing models, we appreciate the significance of juxtaposing our approach with well-established benchmarks, especially those that hold high regard in the community, like PaiNN.\n\nIn response, we've undertaken supplementary efficiency tests, **incorporating two additional equivariant baselines: PaiNN and ET [2]**. These enhancements have been reflected in Figure 1 of the uploaded PDF, and we ensure their inclusion in the subsequent version of our manuscript.\n\nUpon analysis, an intriguing observation emerges from Figure 1: PaiNN, despite boasting the best latency among all contenders, unfortunately, falls short in expressiveness. This reiterates a pivotal point we emphasize in our work — the delicate balance between expressiveness and efficiency. We staunchly believe that **efficiency should not be pursued in isolation but should harmoniously coexist with effectiveness**.\n\nAgain, we appreciate your feedback and hope that our enhanced comparison provides a more holistic view of our contribution in the context of the current landscape.\n\n\n> SaVeNet architecture illustration\n\nWe thank the reviewer for the suggestion. We agree with you and believe this illustration can provide readers with a clearer mental model of our architecture, amplifying their comprehension and facilitating deeper engagements with our work. The illustration of the overall architecture including the components in Sec 3.1 and 3.2 is provided in the uploaded PDF, titled Figure 2. \n \n**Conclusion:**  \nIn conclusion, we sincerely thank you for the time and effort you invested in understanding our work and proposing suggestions. We remain open to further feedback and are committed to making all necessary improvements to serve the scientific community better.\n\n[1] A. Morehead and J. Cheng, ‘Geometry-Complete Perceptron Networks for 3D Molecular Graphs’, AAAI Workshop on Deep Learning on Graphs: Methods and Applications, 2023.\n\n[2] P. Thölke and G. D. Fabritiis, “Equivariant Transformers for Neural Network based Molecular Potentials,” presented at the International Conference on Learning Representations, Oct. 2021. "}, {"review_id": "19r3rMQtWm", "reviewer": "Reviewer_QLtV", "summary": "This paper proposes an SE(3)-equivariant model called SaVeNet, designed to accommodate various geometric requirements. The proposed framework can effectively scale with the introduction of directional noise. Theoretical analysis and empirical results on several datasets are provided to validate the efficiency and expressiveness of SaVeNet.", "questions": "* The metrics std. MAE and log MAE lack detailed descriptions. How did the authors calculate these metrics?\n* The proposed direction noise module's SE(3)-equivariance is unclear, as there is no provided theoretical proof.", "limitations": "The authors well clarify the limitations and potential negative societal impact of their work in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The proposed method achieves outstanding performance across several synthetic and real-world datasets.\n* SaVeNet exhibits better efficiency than baseline methods.", "weaknesses": "* Many claims about the proposed method and related works lack supportive evidence. For example,\n    * Line 40, \"Novel approaches for initializing ... achieving a balance between maintaining numerical stability, facilitating faster convergence, and enhancing the model’s ability to generalize to new datasets.\" This sentence states that the proposed approach can maintain numerical stability and facilitate faster convergence, but there is a lack of convincing empirical ablation studies or theoretical evidence supporting these claims.\n    * Line 58-63, \"one primary problem...limited performance or limited performance or scalability and cannot equally contribute towards performance improvements in the network.\" These sentences make informative claims about the limitations of existing methods, but the authors do not provide detailed discussions or analyses to support these points, weakening the motivation for SaVeNet.\n    * Line 63-68, the authors propose to tackle the scalability challenge by stacking multiple encoder layers, which is a widely adopted operation in equivariant models. However, it is unclear why other baseline models (especially those vector-based models such as PaiNN) cannot improve scalability using the same operations.\n* The methodology section (Section 3) is poorly organized and written. \n    * Some notations are not well defined. E.g., $IA$, $s_i$ and $e$ in Equation 4 are not defined. $h_j$ in Equation 5 is not defined.\n    * Some equations are difficult to understand. E.g.,  the procedure of updating vector $V'_i$ from Equation 4 is unclear. It seems that some details of the equation are missing, making it difficult to integrate Equation 5 and Equation 4.\n* The overall novelty of SaVeNet is limited. Some techniques in the framework (e.g., the scalar/vector-based equivariant operations) have been studied in related works such as PaiNN, and ET. The newly proposed techniques, direction noise and vector activations, account for the main contribution of this paper. However, the ablation results in Table 6 show that removing these two modules only results in a slight performance drop, which cannot validate their importance. Accordingly, the technical contribution of SaVeNet is limited. Considering that many details in the methodology section are unclear, all comments are made based on the current version.\n* There are some grammatical mistakes. \n    * Line 47, `model's` should be `model`\n    * Line 187, $R^{Cx3}$ should be $R^{C \\times 3}$\n\nIn summary, while the proposed method demonstrates exceptional performance across various downstream tasks and exhibits promising scalability, the limited technical novelty and issues with the presentation render the current version unsuitable for acceptance in NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QLtV,\n\nThank you for taking the time to review our manuscript and for your invaluable feedback. We acknowledge the importance of clarity in our presentation, and we address your concerns as follows.\n\n> The metrics std. MAE and log MAE\n\nThe standardized MAE is derived by normalizing the model's MAE with the standard deviation of a target, and the average across targets is taken. Formally:\n$$\\text{std. MAE}(D, x, y) = \\frac{1}{T} \\sum_{t}^T \\Big(\\frac{1}{D} \\sum_{i}^{D}\\frac{|x_{t,i}-y_{t,i}|}{\\sigma(y_{t})} \\Big)\n$$\nHere, $D$ is dataset size, $T$ is the number of targets, and $x_{t,i}$ and $y_{t,i}$ represent the model's prediction and the ground truth for target $t$ of sample $i$, respectively.\nWe also introduce the log MAE metric:\n$$\n\\text{log MAE}(D, x, y) = \\frac{1}{T} \\sum_{t}^T \\log\\Big(\\frac{1}{D} \\sum_{i}^{D}\\frac{|x_{t,i}-y_{t,i}|}{\\sigma(y_{t})} \\Big)\n$$\nThe log MAE smoothens the scores to avoid errors being dominated by a few difficult targets such as $\\epsilon_{homo}$. \n\n\n> Symmetrical properties of the direction noise module\n\nWe appreciate your query about the SE(3)-equivariance of our direction noise module. Briefly, our direction noise is constructed based on node types, introducing SO(3) invariance. This allows for latent space traversal typically inaccessible for equivariant models. The model retains its equivariance during inference as the noise decays over training. Its introduction heightens the learning challenge, aiding in model scaling and countering issues like over-smoothing [2]. Therefore, the module rests on the principle of SO(3) invariance and is bolstered by practical results.\nIn addition, we conducted an ablation study with three distinct equivariant initialization techniques. Please refer to the reviewer FRcS on the experiment using `alternative vector initialization techniques`.\n___\n> Clarification on the introduction section\n- **Line 40: Ablation Study and Support to Generalize New Datasets:** \nSaVeNet was designed to handle large-scale real-world data without sacrificing representation quality. Beyond mere model depth, it's tailored for large molecules and datasets. As evidenced in our evaluation on Molecule3D, SaVeNet accelerates training and inference, improving task performance by up to 125 times. This is crucial in today's ML landscape with growing data scales. Our focus is on maintaining model effectiveness as data scales increase. Referring to Table 6, we conducted an ablation study examining the effects of DN, VA and layer variations on the QM9 dataset.\n- **Line 58-63 - Limitations of Existing Methods**: We provided a detailed comparison with the related methods in lines 223-245, highlighting specific limitations of cited works, and more explicitly connecting them with the motivation for SaVeNet. We are happy to discuss the novelty of our work during the reviewer-author discussion period.\n- **Line 63-68: Scalability through Multiple Encoder Layers:** While stacking multiple encoder layers is a general approach for enhancing model scalability, not all models benefit equally from this operation. Our reference to [1] underscores this very point – the challenges in numerical stability faced by PaiNN and other existing methods. As demonstrated in Table 4, PaiNN exhibited convergence issues for complex tasks, such as ES(20), G+ES(20), and L+ES(20) [1]. This suggests that merely stacking layers might not necessarily lead to improved scalability for all models. **This lack of convergence was not a mere reflection of the model's depth but of inherent limitations in its architecture when tasked with these complex objectives.** \n\n>  **Clarification on Notations:**\n\n- $\\text{IA}$: This denotes the interatomic interactions within the molecule.\n- $s_i$: Represents the scalar representations associated with node $i$ in our graph.\n- $e$: denotes the message functions for scalar and vector tuples, specifically formulated as $e = (e_{s}, e_{v})$.\n- $h_j$: This notation was intended to delineate the linearly transformed version of scalar representations, $s$. \n2. **Revised Equation for Enhanced Clarity**:\n    Following your feedback, we reevaluated our notations in Section 3 and implemented amendments to further the clarity and coherence of our representation. Additionally, we observed that the $h_j$ operation can be subsumed since $\\phi_s$, $\\phi_d$, and $\\phi_v$ inherently encapsulate this linear transformation. This realization enabled us to simplify our representation further. Our revised equations, integrating these clarifications, are articulated as follows:\n$$\\begin{eqnarray} \ne_s(s_j, V_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij}) &=& \\phi_s(s_j) \\odot \\eta\\_s(r\\_{ij}) \\\\\\\\\ne_v(s_j, V_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij}) &=& \\text{VA}\\big(\\phi\\_b(\\vec{\\beta}\\_{ij}) \\odot \\phi_d(s\\_j) \\odot \\eta_d(r\\_{ij})  + V_{j} \\odot \\phi\\_v(s\\_j) \\odot \\eta_r(r\\_{ij})\\big)\n\\end{eqnarray}$$\n\tEq. 4 and 5:\nThe disconnect between Equation 4 and 5, is clarified by expanding the definitions. We split Equation 4 into two parts for clarity. \n$$\\begin{split}\n\\text{IA}(s, V, r, \\vec{\\beta})  &=  (s\\_i, V\\_i) + \\sum\\nolimits\\_{j\\in\\mathcal{N}\\_i}{e(s\\_j, V\\_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij})} = (s'\\_i, V'\\_i) \\\\\\\\\ns'\\_i &= s\\_i + \\sum\\nolimits\\_{j\\in\\mathcal{N}\\_i}{e\\_{s}(s\\_j, V\\_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij})} \\\\\\\\\nV'\\_i &= V\\_i + \\sum\\nolimits\\_{j\\in\\mathcal{N}\\_i}{e\\_{v}(s\\_j, V\\_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij})}\\end{split}$$\n\n> grammatical mistakes \n\nThank you, this has been corrected.\n\nWe thank the reviewer for the detailed analysis of the proposed method. **We revised the paper and incorporated all the feedback in the related sections.** We believe that the revised version represents the proposed method effectively. \n\n[1] A. Morehead and J. Cheng, \"Geometry-Complete Perceptron Networks for 3D Molecular Graphs\"\n\n[2] D. Chen, Y. Lin, W. Li, P. Li, J. Zhou, and X. Sun, “Measuring and Relieving the Over-Smoothing Problem for Graph Neural Networks from the Topological View”\n"}, {"review_id": "lQUZV8c0aK", "reviewer": "Reviewer_jP4K", "summary": "This paper proposes an effective and efficient equivariant graph neural network for geometric learning on molecules. The model encodes 3D graphs with node types and coordinates and outputs scalar and vector representations. The message passing process is purely scalar-based, which enjoys more efficiency than baselines using irreducible representations and high-order tensor objects . A vector initialization based on spherical coordinate system is utilized for better numerical stability. Experiments on QM9, N-body and molecule3D demonstrate the superiority of the proposed model over the baselines.", "questions": "1. What is the $h_j$ in Eq.5?\n2. What do $n, l_e, l_d$ represent in the theoretical complexity analysis?\n3. Are the coordinates fixed during message passing?\n4. Why can the directional initialization improve numerical stability? What is the reason for using spherical coordinate system as initialization? What is the difference if we use random vectors as initialization and anneal their norm to zero during training?\n5. I think the computational scalability of GNNs is mainly restricted by the fully connected message passing which has quadratic complexity with respect to the number of nodes. Therefore in large graphs (e.g. proteins), people often use k-nearest neighbors to reduce the complexity to be linear to the number of nodes. However, the expressiveness of scalar-based equivariant GNNs can only be ensured in nearly fully connected message passing [1]. So can you show that the proposed models can maintain its superiority over the baselines when adapted to larger graphs like proteins, where edges might have to be much more sparser than in fully connected graphs. Or, under the current experiment settings, will the superiority be maintained if the edges get sparser (e.g. use k-nearest neighbors for message passing and decrease the K or decreasing the cutoff in the radial graph)?\n6. Will the scalability holds when keeping increasing the hidden size and the number of layers?\n\n[1] Scalars are universal: Equivariant machine learning, structured like classical physics", "limitations": "When scaling to larger graphs (e.g. proteins), the edges may have to be sparser (e.g. K-nearest neighbors) and the scalar-based message passing potentially will have worse expressiveness.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed scalar-based message passing scheme enjoys theoretically better efficiency and empirically better performance compared to existing baselines.\n2. The experiments are thorough, including three popular benchmarks and a variety of baselines.\n3. The empirical analysis on efficiency is comprehensive, covering  time for forward/backward pass, FLOPs, MACs and memory consumption.", "weaknesses": "1. Several points claimed in the contribution are not well supported:\n   - The authors state that the vector initialization can ensure numerical stability and speed up model convergence. However, no theoretical analysis on why it can ensure numerical stability is provided, and neither experimental evidence nor theoretical proof of speeding up convergence is provided.\n   - The scaled version of the baselines should also be compared with SAVENET-L to show that the proposed model benefits more from the scaling (i.e. Has better scalability). Moreover, it is not shown whether the scalability of the proposed model can hold when keeping increasing the model size.\n2. The presentation of the model structure can be optimized (e.g. add an overview figure).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer jP4K,\n\nFirstly, we extend our gratitude for the careful review and insightful feedback on our manuscript. We address each comment in detail to offer a clearer understanding of our work.\n\n> Numerical stability and model convergence\n\n1. **Relevance of Vector Initialization**: Recent works such as SchNet, DimeNet, SphereNet, and ComENet have established that enhancing the richness of features, can significantly boost performance in molecular tasks. Initializing with vectorial features, as opposed to empty or sparse values, provides the network with a richer context from the outset. This better informed start allows the model to begin its learning process from a more advantageous position.\n2. **Stability and Convergence**: PaiNN's challenges with the N-Body dataset highlight instability issues seen in some models. Rather than simply failing to converge, these models can have error spikes during training due to numerical instability. This often arises in networks using vector representations, especially when merging vector and scalar values, like in architectures such as GVP, PaiNN, ET, and EQGAT. If these networks perform vector-scalar combinations too early, before directional data is introduced, they can become unstable, as observed with PaiNN. This is particularly problematic when the sum of neighboring vectors is close to zero, leading to gradient explosions. Hence, initiating robust representations is pivotal, especially when early vector-scalar operations are involved.\n\n\n> Scaling the baselines\n\nTo address this question, we conducted experiments on the QM9 dataset to scale high-performing baselines to investigate their scalability compared to SaVeNet-L.\n\n**1. Baseline Selection**:\nWe intentionally chose to scale both vectorial and spherical representation models to provide a broad-based comparison. \n- **ET**, which excels in performance with aggregated std. MAE and log MAE metrics, representing vectorial models.\n- **ComENet** was chosen for its efficiency advancements over SphereNet, representing spherical models.\n\n**2. Metrics and Findings from the Comparison**:\n\nFrom our experiments below, a few key insights emerge:\n\n|Model|std.|log|Batch|Memory|Latency|Samples/s|FLOPs|Param.|MACs|\n|---|---|---|---|---|---|---|---|---|---|\n|ET|0.84|-5.90|418|3.66|49.5|1280|42.98|6.86|21.40|\n|ET-L|0.84|-5.96|287|5.33|74.9|832|63.36|10.00|31.55|\n|ComENet|0.93|-5.69|1174|0.96|26.4|2368|14.36|3.81|7.18|\n|ComENet-L|0.97|-5.74|748|4.45|35.7|1728|39.50|11.30|19.75|\n\nObserving the comparison, it's evident that SaVeNet not only demonstrates superior performance with the base model but also capitalizes on its design to scale effectively, as reflected in SaVeNet-L's results. \n\n> Illustration of the model structure\n\nAs suggested, we uploaded PDF Figure 2 for an illustration.\n\n> What is $h_j$ in Eq.5?\n\nThe $h_j$ denotes the latent representation of a node $j$ computed with a linear transformation without bias to scalar representations $s$ . The linear transformation is applied to distinguish between a source node $j$ a destination node $i$. However, recognizing that the transformation functions $\\phi_s$ and $\\phi_d$ can encapsulate this linear transformation, we decided to simplify our equation by omitting $h_j$. The revised version of the equations are as follows:\n$$\\begin{eqnarray} \ne_s(s_j, V_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij}) &=& \\phi_s(s_j) \\odot \\eta\\_s(r\\_{ij}) \\\\\\\\\ne_v(s_j, V_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij}) &=& \\text{VA}\\big(\\phi\\_b(\\vec{\\beta}\\_{ij}) \\odot \\phi_d(s\\_j) \\odot \\eta_d(r\\_{ij})  + V_{j} \\odot \\phi\\_v(s\\_j) \\odot \\eta_r(r\\_{ij})\\big)\n\\end{eqnarray}$$\n\n> What do $n$, $l_e$, $l_d$ represent in the theoretical complexity analysis?\n \n- $n$: denotes the number of nodes.\n- $l_e$: denotes the number of encoder layers in the model. \n- $l_d$: denotes the number of decoder layers. \n\n> Are the coordinates fixed during message passing?\n\nIn our method, both invariant and equivariant features are extracted from the input graph's coordinates. While these features, including distance, direction information $\\vec{\\beta}$ (with components {$\\vec{d}\\_{ij},\\vec{t}\\_{ij},\\vec{o}\\_{ij}$}), and node types are utilized during message-passing, the input graph's coordinates and their representations remain unchanged throughout the process.\n\n> Clarifications on directional noise\n\nWe appreciate your in-depth examination of our directional initialization approach. Let's unpack the underlying motivations and empirical results:\n\n1. **Spherical Coordinate System for Initialization**: Using the spherical coordinate system reduces parameters. By initializing with angles $\\phi$ and $\\theta$, we bypass the need to set three distinct values for x, y, and z, ensuring efficiency and minimizing redundancy.\n2. **Random Vectors vs. Spherical Initialization**: Raw vectors require normalization to be purposeful. In high dimensions, adjusting a single axis might have a negligible impact, potentially introducing noise.\n3. **Additional experiment**: As suggested, testing our approach against random initialization without the assistance of a spherical coordinate system yielded:\n\n| Method   | Homo   | Gap    |\n|----------|--------|--------|\n| Random   | 0.0223 | 0.0342 |\n| DN (Ours)      | 0.0190 | 0.0290 |\n\nClearly, performance drops without spherical-based initialization, showing that handling raw noises is less effective than adjusting angular values.\n\n> Experiments with large graphs, sparser graphs, and scale SaVeNet further:\n\nWe appreciate the reviewer's experimental recommendations. Given the rebuttal's constraints, detailed descriptions and results of these experiments can be found in the general response sections: Experiments 1, 2, and 3.\n\nWe believe our responses elucidate the specific design choices and methodologies in our paper. We hope that our clarifications address your concerns adequately.\n"}, {"review_id": "EvoCri0lXc", "reviewer": "Reviewer_FRcS", "summary": "This paper introduces a novel molecule representation network that enhances the learning capacity and scalability through the integration of innovative initialization techniques and activation functions for vector features. The conducted experiments validate the network's proficiency in three distinct molecule representation tasks, demonstrating its capability to effectively learn from data while maintaining scalability.", "questions": "Regarding the direction noise, it would be helpful to clarify whether theta and phi are deterministic functions of the node type. If they are, applying a rotation to the graph would not change the node type, thereby maintaining the invariance instead of the equivariance of VEN(F_i). So why can VEN(F_i) be an equivariant vector initialization?  Further explanations and intuitions could be added to address this point and enhance understanding.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- A new equivariant network capable of learning scalar and vector features.\n- Superior performance and scalability compared to previous methods.\n", "weaknesses": "- The presentation of the method can be improved. The symbols and equations employed are unclear, such as the presence of a comma in Eq. 4 and the excessive use of unnecessary new symbols in Eq. 5. Furthermore, the substitution of the symbol for multiplication with the letter 'x' in line 187 could be clarified. Additionally, it is suggested that the authors consider using a figure to illustrate the message passing of scalar and vector features, similar to previous methods like DimeNet. \n\n- The core difference between the proposed network and previous networks for vector features, such as PaiNN, remains unclear. While it appears that the primary distinction lies in the different processing of features, it would be beneficial to highlight and discuss this core difference in greater detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer FRcS,\n\nThank you for your thorough assessment of our work. We value your feedback and would like to address your concerns and suggestions as follows:\n\n> Presentation and Equation Clarifications\n\n1. **Eq. 4 Ambiguity**: We've refined Eq. 4 to clarify the scalar-vector tuple representation.\n$$\\begin{split} \ne_s(s_j, V_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij}) =& \\phi_s(s_j) \\odot \\eta\\_s(r\\_{ij}) \\\\\\\\\ne_v(s_j, V_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij}) =& \\text{VA}\\big(\\phi\\_b(\\vec{\\beta}\\_{ij}) \\odot \\phi_d(s\\_j) \\odot \\eta_d(r\\_{ij})  + V_{j} \\odot \\phi\\_v(s\\_j) \\odot \\eta_r(r\\_{ij})\\big)\\end{split}$$\n2. **Concerns about Eq. 5**: We concur that minimizing the use of extraneous symbols is crucial for clarity. Hence, we've revisited Eq. 5 and streamlined the notation. We've also ensured that the scalar and vector interaction pathways are defined with precision, offering coherence with the revised Eq. 5.\n$$\\begin{split}\n\\text{IA}(s, V, r, \\vec{\\beta})  &=  (s\\_i, V\\_i) + \\sum\\nolimits\\_{j\\in\\mathcal{N}\\_i}{e(s\\_j, V\\_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij})} = (s'\\_i, V'\\_i) \\\\\\\\\ns'\\_i &= s\\_i + \\sum\\nolimits\\_{j\\in\\mathcal{N}\\_i}{e\\_{s}(s\\_j, V\\_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij})} \\\\\\\\\nV'\\_i &= V\\_i + \\sum\\nolimits\\_{j\\in\\mathcal{N}\\_i}{e\\_{v}(s\\_j, V\\_j, {r}\\_{ij}, \\vec{\\beta}\\_{ij})}\\end{split}$$\n3. **Symbol for Multiplication in Line 187**: Thank you, this has been corrected as '$\\times$'.\n4. **Illustrative Figure for Message Passing**: We have incorporated an illustrative figure in uploaded PDF Figure 2.\n\n> The difference between SaVeNet and previous networks\n  \nWe'd like to elucidate on the core differences as follows:\n- **Embedding:** Our approach to feature embedding is fundamentally novel. Rather than relying on fewer 3D features, our goal is to craft embeddings that relay lossless information for the 3D molecular graph and resonate with SaVeNet's design. To illustrate, while several existing methods, including Spherenet, ComENet, PaiNN, ET, and EQGAT, tap into diverse features like distance, torsion angle, and directional vectors, our unique feature sets, detailed in Section 3.1, set us on a distinct trajectory right from inception. Furthermore, we've provided both theoretical (Theorem 1) and empirical validation, underpinning the efficacy of our representations in determining the 3D structure.\n- **Initialization:** Contrary to conventional practice in existing works, where vector initializations are often nullified, making the network intrinsically derive these representations, our method sidesteps this potential computational pitfall. Our initialization scheme is tailored to be both efficient and resource-conscious. \n- **Architectural Innovations in Message Passing**: SaVeNet introduces a unique message-passing paradigm, elaborated in Sections 3.1 and 3.2. We've appended Figure 2 as suggested. Our design incorporates our proposed vector activation techniques, which not only delineates our model from predecessors but also fortifies its reliability and scalability. To provide perspective, while models like PaiNN were conceived primarily for molecular property prediction, their inherent design may not extend seamlessly to more diverse tasks. A case in point is PaiNN's convergence issues on certain demanding targets [1]. Our proposed vector activation function not only alleviated PaiNN's limitations but also enriched its capabilities, as showcased in Table 4's performance comparison.\n- **Comparative Performance Analysis**:Our evaluations underscore SaVeNet's superiority, which was evident when models like PaiNN and ET were benchmarked against N-Body tasks. The results clearly attest to the innovations and the efficacy SaVeNet brings to computational molecular science.\n\nIn summary, from the initial motivations, through feature design and architecture, to real-world application performances, SaVeNet has been meticulously crafted to push the boundaries of what's possible in the domain, while addressing the limitations of its predecessors. \n\n> Regarding the direction noise\n\nLet us provide clarity on the concerns raised:\n1. **Initialization Concerns with \\($\\theta\\$) and \\($\\phi$\\)**:\n    - Both are intrinsically linked to node types. This design choice is deliberate, taking into account the potential noise from simulations, e.g., DFT. By associating noise with atomic numbers, our model achieves robustness against variances in node coordinates.\n\n2. **Alternative Vector Initialization Techniques**:\n    - **Direction to the Center of Mass (MCM)**:\n      $$\\text{MCM}(i, G) = \\sum_{j \\in \\mathcal{G}}{\\frac{c_{j}}{N}} - c_{i}$$\n      Demonstrably equivariant under rotations as $$\\text{MCM}(i, RG) = \\sum_{j \\in \\mathcal{G}}{\\frac{Rc_{j}}{N}} - Rc_{i}$$.\n    \n    - **Direction to Neighborhood Center of Mass (NCM)**:\n      $$\\text{NCM}(i, G) = \\sum_{j \\in \\mathcal{N_{i}}}{\\frac{c_{j}}{|\\mathcal{N_{i}}|}} - c_{i}$$\n      Provides a localized node representation.\n    \n    - **Direction to the Nearest Node (NN)**:\n      $$\\text{NN}(i, G) = {c_{j} - c_{i} \\quad | \\quad c_{j} = \\text{argmin}_{k\\in \\mathcal{N}} \\left( ||c_k - c_i|| \\right) }$$\n      Recognizes the invariance of the closest node under rotations.\n\n3. **Empirical Results on Molecule3D**:\n| Method   | Homo   | Gap    |\n|----------|--------|--------|\n| MCM      | 0.0213 | 0.0321 |\n| MCM + DN | 0.0208 | 0.0313 |\n| NCN      | 0.0213 | 0.0327 |\n| NCN + DN | 0.0199 | 0.0301 |\n| NN       | 0.0211 | 0.0321 |\n| NN + DN  | 0.0205 | 0.0318 |\n| DN       | 0.0190 | 0.0290 |\n\nThis table elucidates the relevance of introducing directional noise. Across initialization techniques, the integration of DN consistently elevates model performance. \n\nWe're grateful for your insightful comments and hope that these amendments will address your concerns effectively.\n\n[1] A. Morehead and J. Cheng, ‘Geometry-Complete Perceptron Networks for 3D Molecular Graphs’, _AAAI Workshop on Deep Learning on Graphs: Methods and Applications_, 2023.\n\n\n\n\n"}]

Author rebuttal:
Dear Reviewers,

Firstly, we'd like to extend our sincere gratitude for your diligent review of our work and your invaluable feedback. Based on your insights and suggestions, we have undertaken significant efforts to improve our manuscript, making it both more comprehensive and accessible to the wider community.

**Enhanced Clarity & Methodological Details**:

1. **Refined Equations**: We have meticulously revised our equations, ensuring greater clarity and eliminating any ambiguity. This exercise was geared towards providing a clear analytical exposition of our methodology and aiding the reader's comprehension.
2. **Improved Manuscript Clarity**: To provide a holistic overview of our proposed SaVeNet model, we've incorporated a detailed framework illustration, showcasing individual components. This can be referenced in Figure 2 of the uploaded PDF. This graphical representation complements our textual explanations, creating a more immersive understanding of SaVeNet's operation.
3. **Expanded Comparisons**: We value the importance of a comprehensive comparative study. In line with this, we've included additional baseline models in Figure 1, offering readers a broader perspective of SaVeNet's standing in the current research landscape.
4. **Additional Experiments**: In our efforts to thorough evaluation the SaVeNet, as suggested by reviewer jP4K, we tackled three additional experiments. These experiments aimed to validate SaVeNet's capability in various scenarios and provide a comprehensive view of its performance in graph-based neural network applications.

**Experiment 1: Large graphs**

We test the scalability of our model to larger graphs, such as protein graphs. For this purpose, we conducted the experiments on a protein-ligand complex dataset known as LBA [1] to compare with the baseline models.

| Method                                              | RMSE  | Pearson | Spearman |
|-----------------------------------------------------|-------|---------|----------|
| Atom3D-3DCNN [1]            | **1.416** | .550   | _.553_    |
| ProNet-All-Atom [2]                                     | 1.463 | _.551_   | .551    |
| SaVeNet                                               | _1.438_ | **.572**   | **.559**    |
   

SaVeNet displayed a remarkable RMSE of 1.438, only narrowly being surpassed by Atom3D-3DCNN, highlighting its efficacy even in larger sparse graphs. Furthermore, SaVeNet emerged as the leader in both correlation metrics. This not only demonstrates its robustness but also its adaptability across various metrics.


**Experiment 2: Experiments on sparse graphs**

We explored SaVeNet's performance on sparser graphs using the Molecule3D dataset, which contains larger molecular structures. The baseline models primarily worked with radius graphs beyond a radius of $6$. Reducing this radius to $3$ led to a model, SaVeNet3-B, operating on graphs with edge numbers decreasing from **553** to **259**.

Reducing the radius presents challenges. For example, one-hop neighbors can become multi-hop neighbors, complicating data access between nodes. To address this, we adapted SaVeNet3-B with 160 (SaVeNet3-B+) and 192 (SaVeNet3-B++) hidden dimensions to handle multi-hop information flow better.

| Model        | std. MAE | log MAE | Batch Size | Memory  | Latency (ms) | Samples/s |
|--------------|---------|---------|------------|---------|--------------|-----------|
| SaVeNet3-B   | .0205  | -3.949 | 824        | 1.29    | 17.3         | 3648      |
| SaVeNet3-B+  | .0175  | -4.112 | 642        | 1.62    | 19.0         | 3328      |
| SaVeNet3-B++ | .0147  | -4.283 | 518        | 1.93    | 19.9         | 3200      |
| SaVeNet-B    | .0156  | -4.226 | 329        | 2.84    | 27.8         | 2304      |

SaVeNet3-B shows strong performance compared to the baselines, even with a smaller receptive field. SaVeNet3-B+ and SaVeNet3-B++ both outperform the baselines, with SaVeNet3-B++ achieving the best results. A reduced receptive field offers enhanced efficiency, particularly in memory and latency, highlighting SaVeNet's scalability and resilience. We thank the reviewer for the insightful prompt.

 **Experiment 3: scalability even larger SaVeNet**

We extended SaVeNet's architecture to SaVeNet-XL, adopting the same principles used for the previous scale, SaVeNet-L. We conduct this experiment on the larger dataset: the Molecule3D dataset. For SaVeNet-XL, we applied similar scaling principles used for SaVeNet-L, which is 256 hidden dimensions across 10 layers.

| Method      | $\mu$   | Homo   | Lumo   | Gap    |
|-------------|--------|--------|--------|--------|
| SaVeNet-B   | 0.0183 | 0.0190 | 0.0173 | 0.0290 |
| SaVeNet-L   | 0.0136 | 0.0159 | 0.0143 | 0.0239 |
| SaVeNet-XL  | 0.0112 | 0.0136 | 0.0129 | 0.0209 |

Notably, there is a consistent improvement across all metrics when transitioning from SaVeNet-B to SaVeNet-L and then to SaVeNet-XL. 

**Concluding Remarks**:

Our revisions, experiments, and methodological refinements were driven by a commitment to academic rigor and dedication to the community's betterment. We believe these improvements will address concerns and illuminate the significant contributions of our work. Our hope is that the revised manuscript provides a clearer and more insightful understanding of SaVeNet's potential. 

Given the character constraints of the rebuttal, we've endeavored to address all concerns succinctly. If any queries remain or further clarification is required, we welcome the opportunity to respond further. Thank you once again for your time and invaluable feedback.

Sincerely,

Authors

[1] R. J. L. Townshend _et al._, “ATOM3D: Tasks on Molecules in Three Dimensions,” 

[2] L. Wang, H. Liu, Y. Liu, J. Kurtin, and S. Ji, “Learning Hierarchical Protein Representations via Complete 3D Graph Networks,”

Meta-review: This paper studies 3D molecular conformations and proposes an equivariant architecture called SaVeNet. The proposed architecture is novel, efficient, and achieve good performance on several invariant and equivariant molecular tasks. Overall, all reviewers appreciate the contribution and thus I would be happy to accept the paper. However, I would suggest the authors to carefully revise the camera-ready paper to take into account the reviewers' suggestions.
Decision: Accept (poster)