GotenNet: Rethinking Efficient 3D Equivariant Graph Neural Networks
Authors: Sarp Aykent
Tian Xia
Venue: ICLR 2025 Poster (2025)
Keywords: graph neural networks
computational physics
3D graphs
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: GotenNet: An efficient framework that uses high-degree steerable features to model complex 3D molecular structures while maintaining E(3) equivariance.

Abstract: Understanding complex three-dimensional (3D) structures of graphs is essential for accurately modeling various properties, yet many existing approaches struggle with fully capturing the intricate spatial relationships and symmetries inherent in such systems, especially in large-scale, dynamic molecular datasets. These methods often must balance trade-offs between expressiveness and computational efficiency, limiting their scalability. To address this gap, we propose a novel Geometric Tensor Network (GotenNet) that effectively models the geometric intricacies of 3D graphs while ensuring strict equivariance under the Euclidean group E(3). Our approach directly tackles the expressiveness-efficiency trade-off by leveraging effective geometric tensor representations without relying on irreducible representations or Clebsch-Gordan transforms, thereby reducing computational overhead. We introduce a unified structural embedding, incorporating geometry-aware tensor attention and hierarchical tensor refinement that iteratively updates edge representations through inner product operations on high-degree steerable features, allowing for flexible and efficient representations for various tasks. We evaluated models on QM9, rMD17, MD22, and Molecule3D datasets, where the proposed model consistently outperforms state-of-the-art methods in both scalar and high-degree property predictions, demonstrating exceptional robustness across diverse datasets, and establishes GotenNet as a versatile and scalable framework for 3D equivariant Graph Neural Networks.

Published as a conference paper at ICLR 2025
GOTENNET: RETHINKING EFFICIENT 3D EQUIVARIANT GRAPH NEURAL NETWORKS
Sarp Aykent' and Tian X *Comcast Al Technologies, Microsooft
ABSTRACT
Understanding complex three- drre-dimensiooal (3D) structures of graphs is essential for accurately modeling various properties, yet many existing approaches struggle with fully capturing the intricate spatial relationships and symmetries inherent in such systems, especially in large-scale, dynamic molecular datasets. These methods often must balance trade-offss between expressiveness and computational efficiency, limiting their scalability. To address this gap, we propose a novel Geometric Tensor Network (GotenNet) that effectively models the geometric intricacies of 3D graphs while ensuring strict equivariance under the Euclidean group E(3). Our approach directly tackles the expressiveness-effiiinne trade-off by leveraging effective geometric tensor representations without relying on irreducible representations or Clebsch-Gordee overhead. We introduce a unified structural embedding, incorporating geometry- aware tensor attention and hierarchical tensor refinement that iteratively updates edge representations through inner product operations on high-degree steerable features, allowing for flexible and efficient representations for various tasks. We evaluated models on QM9, rMD17, MD22, and Molecule3D datasets, where the proposed model consistently outperforms state-of-the methods both scalar and high-degree property predictions, demonstrating exceptional robustness across diverse datasets, and establishes GotenNet as a versatile and scalable framework for 3D equivariant Graph Neural Networks.
transforms, thereby reducing computational
INTRODUCTION
Gumthee LSSTWN
Centrmme
Accurately modeling 3D molecular systems is increasingly
0.90
Sevelled
crucial in areas such as drug discovery (Chen et al., 2020;
0.84
Jing et al., 2021; Nguyen al., 2021; Huang et al., 2020;
Aykent & Xia, 2022; Yang et al., 2022), materials sci- ence (Reiser et al., 2022; Pablo-García
Std. MAE
0.78
al., 2023; Polat et al., 2025), and structural biology (Zhang et al., 2021; Xia & Ku, 2021: Zhang al., 2022). These tasks require precise understanding of the spatial configurations and sym- metries inherent in molecular structures, as these factors are fundamental to predicting molecular properties. While predicting scalar molecular properties, such as energy and stability, is challenging, predicting molecular forces is par- ticularly difficult due to the vector nature of forces and their dependence on local geometric environments (Klicpera et al., 2021; Liao & Smidt, 2023; Wang et al., 2024; 2023b; Du et al., 2023). Traditional graph neural networks (GNNs), while effective for general graph-structured data, face diffi- culties in handling the geometric and topological complexi- ties of 3D molecular systems, where achieving equivariance remains a significant challenge (Thomas al., 2018: Sator- ras et al., 2021; Jing et al., 2021; Aykent & Xia, 2023).
0.72
0.66
0.60
B
0.54
.6.2
5.6
log MAE
Figure 1: Comparison of GotenNet and baseline models on the QM9 dataset. The æ-axis shows the logarithmic MAE across all targets, while the y-axis shows the standardized MAE. Lower values on both axes indicate better per- formance. Points marked as S, B, and L represent small, base, and large varia- tions of the GotenNet, respectively.
'Sarp Aykent and Tian Xia are co-first authors. Correspond to aykent rprggma com

---

Published as a conference paper at ICLR 2025
Recent advances in equivariant neural networks have led two distinct approaches (Han et al., 2024): scalarization-base models and high-degree steerable models. Scalarization- based models (Satorras et al., 2021; Zhang& & Zhao, 2021; Schütt et al., 2021; Thölke & De Fabritiis, 2022; Aykent & Xia, 2023; Du et al., 2023) operate by projecting 3D geometric information into scalar features before reconstruction, offering computational efficieney and scalability for large-scale applications. However, this projection process may limit expressiveness in capturing complex geometric patterns, particularly in scenarios requiring precise understanding of spatial relationships and symmetries. On the other hand, high-degree steerable models (Batzner et al.. 2022; Batatia et al., 2022b; Musaelian et al., 2023; Batatia et al.. 2022a; Qiao et al., 2022; Liao & Smidt, 2023; Liao et al., 2024) achieve impressive performance through irreducible representations (irreps) and Clebsch-Gordan (CG) transforms, enabling direct manipulation of geometrie features in higher-resolution representation spaces. Despite their theoretical rigor and strong performance, these models incur significant computational overhead due to their reliance on complex tensor operations (Cen et al., 2024; Liao & Smidt, 2023; Liao et al., 2024). This fundamental dichotomy between computational efficiency and geometric expressiveness presents a critical challenge in the field: how to achieve both qualities while maintaining strong performance across diverse molecular properties.
Fundamental breakthroughs in theoretical understanding have revealed promising directions for addressing this challenge. Rather than relying explicit CG coefficients, recent advances have illuminated how inner product operations can effectively capture similar geometric relationships (Cen al., 2024) while being computationally more tractable. This insight suggests the potential for more efficient architectures that maintain the expressiveness of high-degree representations without the computational burden of explicit CG transforms. However, translating this theoretical understanding into practical architectures remains challenging it requires not only a novel formulation of geometric operations but also careful consideration of how to maintain numerical stability and computational efficiency at scale. The challenge is evident in existing models' inability to bridge the gap between scalarization- -based and high-degree steerable approaches while maintaining practical applicability. Most current architectures (Han et al., 2024; Wang et al., 2024; Liao & Smidt, 2023; Liao et al., 2024) either compromise on expressiveness for efficiency or forfeit computational tractability for geometric accuracy, leaving clear divide between models optimized for scalar property prediction and those designed for force field calculations, with few achieving strong performance in both domains.
To address these challenges, we propose novel framework, the Geometric Tensor Network (Goten- Net). Our approach focuses on addressing the trade-off between expressiveness and efficiency. First, we introduce spheriial-scalaaizzttoon model with an efficient representation and embedding strategy designed specifically with geometric tensors, eliminating the need for irreps and CG transforms, thereby reducing computational complexity without sacrificing the expressiveness required for mod- cling intricate 3D structures. Second, we present geometry-awane tensor attention and hierarchical tensor refinement mechanisms. These mechanisms enhance transformer-basee -baed refining edge representations through high-degree steerable features, enabling the architectures by mech- anism to leverage refined geometric relationships in determining node interactions. This refinement process enriches the attention weights with granular geometric information, allowing more precise modeling of spatial relationships in molecular structures. These innovations allow the model to represent molecular properties across multiple scales, adapting to both broad patterns and fine- tne-grained molecular details. As shown in Figure 1, our model consistently outperforms the baselines on the QM9 dataset, excelling in both standard MAE and log MAF metries. This highlights GotenNet' ability to maintain accuracy for large-scale properties while ensuring precision for smaller-scale ones, resulting in strong overall performance across diverse molecular properties.
Through rigorous evaluations on benchmark Jatasets-QM9 Molecule3D, rMD17, and MD22-our approach consistently outperforms state-of-the-ar methods, even in its smallest configuration, estab- lishing GotenNet as a versatile and scalable framework for future developments in 3D equivariant graph neural networks. The demonstrated robustness in predicting both scalar and higher-degree tensor properties highlights its broad potential for applications in fields such as drug discovery. materials science, and molecular dynamics simulations.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
The field of machine learning for molecular representation learning has seen significant advancements in recent years (Gasteiger et al., 2020b; Liu et al., 2022; Gasteiger et al., 2020a; Wang et al., 2022: Liao & Smidt, 2023; Liao et al., 2024), especially for the development of graph neural networks (GNNs) to predict quantum mechanical properties and simulate molecular dynamics. These approaches can be broadly categorized into two main groups: invariant GNNs and equivariant GNNs.
2.1 INVARIANT GNNS
Invariant GNNs focus on extracting rotation and translation invariant features from molecular graphs (Schütt et al., 2017; Xie & Grossman, 2018; Unke & Meuwly, 2019; Gasteiger et al., 2020b;a; Klicpera et al., 2021; Liu et al., 2022: Wang et al., 2022). DimeNet (Gasteiger et al., 2020b) intro- duced the concept of directional message passing, embedding messages between atoms instead of atoms themselves. This approach allowed the incorporation of angular information while maintaining rotational equivariance. GemNet (Kliepera et al., 2021) extended this idea by incorporating dihedral angles, and SphereNet (Liu et al., 2022) efficiently integrated torsion information in the message passing scheme. ComENet ( (Wang et al., 2022) built upon these approaches, introducing a novel message passing scheme that operates within I-hop neighborhoods and achieves global and local completeness in incorporating 3D information.
2.2 EQUIVARIANT GNNS
Equivariant GNNs, on the other hand, directly model rotational equivariance and translational invariance in their architectures (Thomas et al., 2018; Kondor et al., 2018: Schütt et al., 2021; Jing et al., 2021; Thölke& & De Fabritüis, 2022; Aykent & Xia, 2022; Le et al., 2022; Du et al., 2022; 2023; Aykent & Xia, 2023; Wang al., 2023b; 2024). Current equivariant GNNs can be divided into two primary approaches based on their feature processing strategies (Han et al., 2024): scalarization-bant models and high-degree steerable models. ecalarization-base models focus deriving invariant scalar features from 3D coordinates and then reconstruct directional information for equivariant updates (Han al., 2024; Satorras al., 2021). Several models have successfully implemented this strategy: PaiNN (Schütt et al., 2021) incorporated both scalar and vectorial features in its message passing framework. LEFTNet (Du et al., 2023) developed local frame-based representations with structural encodings, and TorchMD-NET (Thölke & De Fabritiis, 2022) introduced an equivariant transformer architecture that processes scalar and vector features separately.
High-degree steerable models leverage high-degres representations and CG tensor products for molecular modeling (Batzner et al., 2022; Batatia et al., 2022b; Qiao et al., 2022; Batatia al., 2022a; Musaelian et al., 2023; Liao & Smidt, 2023; Liao et al., 2024). SE(3)-Transformes (Fuchs et al., 2020) pioneered this direction by introducing attention mechanisms with high-degree steerable features, through computational limitations arose from tensor product operations. Subsequent works like NequlP (Batzner et al., 2022) and MACE (Batatia et al., 2022b) introduced approaches using CG coefficients for equivariance, while Allegro (Musaelian et al., 2023) introduced local equivariant architecture using iterated tensor products. Equiformer (Liao & Smidt, 2023) introduced (E)) equivariance into Transformers through depth-wise tensor products and MLP-based attention mechanisms. Building on this work, EquiformerV2 (Liao et al., 2024) enhanced computational efficiency by incorporating eSCN (Passaro & Zitnick, 2023), although its Fibonacci grid sampling approach incurs an O(L²) computational overhead in achieving euaasiiquivariinee. While these methods show impressive performance, their computational overhead remains significant.
Recent approaches have explored alternative methods to capture geometric information without relying on tensor products and CG coefficients, offering computationally efficient solutions. ViSNet (Wang et al., 2024) linked inner products and Legendre polynomials via vector-scalar interactive message passing, through focusing on the first-degree steerable features. SO3KRATES (Frank et al., 2024) demonstrated that certain applications of CG coefficients are equivalent to inner products of high-degree steerable features, achieving notable performance in property prediction tasks through their equivariant transformer architecture. HEGNN (Cen et al., 2024) further developed these concepts by introducing a scalarization approach using inner products to incorporate high-degree steerable features. This approach proved capable of capturing complete angular information between edge pairs
3

---

Published as a conference paper at ICLR 2025
z
h
X
X
X
x
Eij
Structure
LN
LN
Wxc
HIR
Embedding
W
W
Agg(-)
GATA
W
(+)+)
Ali
7+(-)
Feed Forward
(c) HTR
softmaxx --)
GATA
s-lit(-)
X (-11
Asn
A
P
+
W.o
#0
Forward
++)-)
(+))
W
LN
W
Decoder
Ah:
AX."
(a) Framework
(b) GATA
(d) Node Scalar Initialization
Figure Architecture of GotenNet. The overall framework (a) includes an embedding, interaction module, and a decoder; (b) shows the geometry-awane tensor attention (GATA); (c) illustrates the hierarchical tensor refinement (HTR); and (d) presents the node scalar feature initialization component, which is part of the structure embedding module. In the figure, + denotes addition, denotes dot product, denotes aggregation, (:..) denotes concatenation, denotes element-wiss (Hadamard) product with broadcasting when tensor shapes are different, LN denotes layer normalization, denotes the radial basis functions, and y denotes differentiable functions such as MI.Ps
and demonstrating enhanced model robustness dynamics tasks. Our work advances this direction by introducing geometry-aware tensor attention, which employs a concise formulation of inner product operations combined with hierarchical refinement mechanisms. GotenNet represents a significant advancement in bridging the critical gap between scalarization-base models. The proposed spherical-scalarizan real-world molecular property prediction and force field calculations across diverse datasets.
and high-degree steerable model, GotenNet, achieves superior performance in
GOTENNET
In this section, we present the key components of GotenNet. Our network maintains and updates two types of node features (invariant and steerable) and edge features (invariant) throughout its operations. We first introduce requiring irreps or CG transforms, significantly reducing computational complexity. The network then processes these representations through three main mechanisms: (1) degree-wise attention-based message passing layer (Sec. 3.3) that updates both invariant and steerable features while preserving equivariance, (2) high degheeegree edge refinement layer (Sec. 3.4) that updates edge features across degrees with inner products of steerable features, and (3) node-wise feed-forward refinement layer (Sec. 3.5) that further processes both types of node features. This architecture enables accurate and scalable predictions while maintaining geometric consistency throughout the network.
efficient initialization scheme (Sec. 3.2) that embeds geometric tensors without
3.1 EQUIVARIANT GEOMETRIC TENSOR REPRESENTATIONS
In our model, distinguish between edge scalar features and edge tensor representations, employing spherical harmonics to initialize the latter. The edge tensor representation =:( is initialized based on the relative positions p, and p. of nodes i and j. capturing spatial information from rank 0 to Lmax- Specifically, Fg = (0 f()) ((lmm)), }. where each +() represents I-degree spherical harmonic functions. The components of F.g follow hierarchical structure of increasing geometric complexity.
4

---

Published as a conference paper at ICLR 2025
(00) |p: Pill captures the scalar distance between nodes, providing At the most basic level, Fij" rotation and translation invariant information. The first-degree component [/) -o;) encodes directional information, introducing rotational equivariance. For / 2 2, each ( comprises (2l 1) 1) functions derived from spherical harmonics of degree I, where the degree determines the transformation behavior under rotations, and the parity of determines the behavior under inversion. These functions are chosen to capture complex spatial relationships and rotational symmetries inherent in molecular structures. Leveraging the inherent normalization property of spherical harmonics, each ((() for 1 2 1 is naturally normalized, ensuring consistent scaling across different representations.
We denote the geometric node tensors into two types of features: scalar features h E Rd invariant under transformations, and high-degree steerable features X (2+ whose transformations depend on their degree where dne denotes node embedding dimension. We denote steerable features with tilde, 7, where feature of degree is represented by 1 + 2/ components. These representations are initialized and updated through message passing phases using the edge tensor representation
Tij and edge scalar features tij as input. The notation X without specified degree refers to the collection of features with degrees from Lmax- This initialization strategy enables our model to effectively capture, process, and propagate complex structural information.
Our initialization and feature design ensure equivariance throughout the network. geometric tensor field maps 3D points to tensor quantities that transform equivariantly under geometric transformations, combining both invariant scalars and stecrable features. Each layer of GotenNet processes these tensor fields through equivariant operations while preserving E3) transformations, with the final layer producing either equivariant geometric features or invariant representations as required by the task. This composition of equivariant operations ensures that the entire network maintains equivariance, with complete proofs provided in the Appendices A, B, and C.
3.2
UNIFIED STRUCTURAL EMBEDDING: INTEGRATING CONTENT AND GEOMETRY
Our approach introduces a unified structure embedding that captures intrinsic atomic properties and relational information through an integrated node-edge interaction mechanism. By employing dual representation strategy, we incorporate local geometric structure through node-edge interaction. This allows the model to simultaneously process both semantic and geometric information, enabling efficient message passing for both nodes and edges.
Node Scalar Feature Initialization. Node scalar features obtained through two-step process involving message passing and representation updates. Node information aggregated as:
Σ Abb o (1((F(9)) W ((F!)))
m, =
JEN(()
where E R1 denotes the one-hot encoding of the atomic number, denotes element- wise product, and Anhr E RR. Z|xd is a learnable embedding matrix for neighbor atoms with maximum atomic number |]. The basis functions ((F!())) encode the distance between nodes and which are then
projected through Wndp- A cutoff function ((F(")) is applied to modulate the influence of distant neighbors. It is important to note that the operation applies standard broadcasting rules when the shapes of the two operands are not identical. Specifically, the tensors have mismatched shapes, the tensor with fewer dimensions or smaller size in dimension is automatically expanded to match the shape of the other tensor before the element wise multiplication. For example, Equation (1),
(0) R.. Here, of ((F is broadcast along consider the shapes: ((p(1(()) W ndp) E Ie and ((r()
(0)
the first dimension to conform to the shape of (pp(+() W edp) before the element-wise product is computed. Consequently, the aggregated node information, m,, produced with shape Idd...
The initial node scalar feature is defined as:
,, = a (LN((z:Am..))) Wma)) W
Here, A.a E xdne is learnable embedding matrix for node atoms, a denotes a non- con-linear activa- tion function, and (1.)) denotes concatenation operation. The concatenated node atom embedding and
5

---

Published as a conference paper at ICLR 2025
aggregated neighbor information undergo series of transformations: node representation projections (Ward. W mwr),, and layer normalization (LN).
Edge Scalar Feature Initialization. To effectively model interactions, edge scalar features computed by combining node features with distance-based edge attributes:
init
(hi + hg.ani)
(p(()()),
W crp
Edge attributes by E Red are processed through projection matrix W epp enabling the integration of node-level features and spatial relationships. This formulation captures complex interactions between nodes while maintaining equivariance under molecular transformations.
High-degree Steerable Feature Initialization. The High-degres steerable features X initialized during initial interaction layer with the following formulation:
(o Lmmax
ini M
= split seaty + (tig,ini Wrs.inn) o Y (m) (((F()) dne)):
x Lini
(1) Gnii o 1.
=!!))
JEN(()
where sea.; is self-attention with geometric encoding, Wrs,ini E [Rdeaxdue is a learnable weight matrix, 7/- RRde + IR. Lmax Xdne is a differentiable function, and the split function decomposes the input tensor into dimennimni segments. These segments are used different coefficients for each (-degree stecrable features. + denotes pprmuttaiooninvvann aggregation function.
3.3 GEOMETRY-AWARE TENSOR ATTENTION
We introduce a novel module called which enhances the attention mechanism in graph neural networks by incorporating spatial information. GATA captures the geometric relationships between nodes to improve attention-dii driven message passing.
The GATA module combines self-attention with geometric encoding to generate rich node interaction representations. We compute the query (q), key (k), and value (v) representations:
q: = h, Wg. k = h, Wx. vy = To (hy).
(5)
where W q> WE Ridue Kehe are learnable weight matrices, and 7. Rd - RS is differentiable function (e.g.. MLP). The S variable introduced to generate diffeerent coefficients for each degree of stecrable features and defined as + x Lmax. The attention coefficients ag between nodes and j using the dot product of the query vector qi and geometry-infuss key vector, which is obtained via an element-wise product of kj and a transformed edge embedding:
exp(arr) sea.j ExEN()) exp(a<)) Uy where æij qi (ke o "T
(6)
Here, σg denotes non-linear activation function, and Wre Rdee is learnable weight matrix that transforms the edge scalar features. To incorporate spatial and directional information, augment the attention mechanism with geometric encoding. The GATA operation combines self-attention with geometric features and is then split into components:
d.ll) Liaas t.(l) y split(sea., + (4i, Wrs) Ya (h₂) ((F!))) dne): ogg Loij Jiz1 (oij
where Wrs IR (S
learnable weight matrix, 7/ .Rdn. + S.da. is a differentiable function, and the split function decomposes the input tensor into dne dimennsooaa segments. We define Ah; and AX the residues, which are calculated by:
Ah; =
(of)):
A
(o d.00)
1.0 +
(8)
JEN())
jEN(()
Here, each degree [1, Iemax] contributes its component of steerable features weighted by d,( (,)))
their respective coefficients of and O, Finally, representations are updated using residues with:
h; h; Ah, - AX
By infusing geometric information into the attention mechanism, GATA allows the model to better capture spatial dependencies and fine- mne-grained node interactions, leading to improved performance in molecular property predictions, as demonstrated in Section 4.

---

Published as a conference paper at ICLR 2025
3.4 HIERARCHICAL TENSOR REFINEMENT
The Hierarchical Tensor Refinement (HTR) component processes graph-structured data through multi-scale analysis and layer-wise refinement. High-degree steerable features projected query and key representations using degree-specifte S0(3)-equivarian linear transformation (Deng et al., 2021: Du al., 2023; Wang et al., 2024) as shown Equation (10):
EQ; X (0) Wegs
(1)
EK (1)
w(() ok?
for ( E f1,...,Lmas J
(10)
where W uq. w Rdud are tensor query and key projection matrices, respectively. Here, W og is a shared projection matrix across degrees, while Will) is degree-specific. To preserve equivariance, uniform weights are applied across spatial dimensions. These projections aggregate angular and magnitude information between nodes across tensor degrees, defined as:
Wij
Agg/=1 bimax
(11)
where Way Rd represents the aggregated similarity between nodes and j. and Agg denotes an denotes the aggregation over high-degree steerable features. The aggregated information refines edge representations through residual connection:
bij 4ij Alij, Ali The (wss)) 7e(4)))
(12)
where Yw RRd Rdea and Tt Rd.e Rdes are differentiable functions. some cases, dapd set larger than ded for richer intermediate representations.
3.5 EQUIVARIANT FEED-FORWARD (EQFF) NETWORKS
The EQFF blocks, employed after GATA, facilitates efficient channel-wise interaction while maintain- ing equivariance. By design, the module separates scalar and high-degree steerable features, allowing for specialized processing of each feature type before combining them with non-linear mappings:
(Q x) = ( (h+mm). (x(1) ()) + (m2 X (..))
(13)
where m. m2 splity (Tm (1 Woul 12, ft).
Here 7/m denote differentiable functions such as MLPs, W tm denotes learnable weight matrices, (:-)) denotes concatenation, and ||2 denotes L2 norm. The EQFF module operates on the tensor representations while separating the scalar and high-degree steerable features and combining them through element wise operations. The use of "'m enables the model to learn complex non-linear mappings, enhancing its expressiveness (Ramachandran et al., 2017; 1; Elfwing et al., 2018).
4 EXPERIMENTS
In this section, we compare the performance of GotenNet with other state-of-the-art methods. Experi- ments were conducted with an NVIDIA A100 GPU with 80GB video memory, 512GB RAM, and an AMD EPYC 7713P CPU. We evaluated the models on QM9, rMD17, MD22, and Molecule3D datasets. The best results are bolded and the second best are underlined. Additional details on hyperparameters and scalability, as well as additional experiments, can be found in the Appendix E.
4.1
QM9 DATASET
Dataset. The proposed method is evaluated against comprehensive set of baselines using the QM9 dataset (Ruddigken) et al., 2012: Ramakrishnan et al., 2014). These baselines include Cormorant (Anderson et al., 2019), ClofNet (Du et al., 2022), NMP (Gilmer et al., 2017). EGNN (Satorras et al., 2021), I). SEGNN (Brandstetter et al., 2022), PaiNN (Schütt et al., 2021), DimeNett+ (Gasteiger et al., 2020a). ComENet (Wang et al.. 2022), SphereNet (Liu et al., 2022). LEFTNet (Du et al., 2023), EQGAT (Le et al., 2022), ET (Thölke & De Fabritiis, 2022), HDGNN (An et al., 2024), Geoformer (Wang et al., 2023a), Equiformer (Liao & Smidt, 2023), SaVeNet (Aykent & Xia, 2023),
7

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparisons on QM9 dataset. denotes using different data partitions.
Task
A meV
ELUMO meV
C₁ maa
G
Rt
U
Up meV
ZPVE meV
std. %
log
Units
mag
meV
mD
meV
meV
mej
meV
DimeNet++
44
32.6
24.6
19.5
29.7
23
7.56
6.53
331
6.28
6.32
1.21
0.98
-5.67
SphereNet
31.1
22.8
18.9
24.5
22
7.78
6.33
268
6.36
6.26
1.12
0.91
-5.73
LEFTNet
40
18
23
6
6
1.33
0.91
-582
EQGAT
32
24
24
25
2.00
-5.28
36.1
20.3
17.5
26
6.16
33
6.15
1.84
0.84
-5.90
HDGNN
32
18
16
23
10
8.12
8.34
1.21
0.80
-5.64
Gooformer
33.8
18.4
15.4
22
6.13
4.39
28
4.43
1.28
0.75
-6.12
Equiformer
30
15.4
14.7
23
7.63
6.63
251
6.74
6.59
1.26
0.70
-5.82
SaVeNet-B
24.8
18.4
16.3
23
6.64
5.43
58
5.48
5.43
1.18
6.04
Equiformerva
29.0
14.4
23
7.57
6.22
6.49
6.17
1.47
-5.87
GotenNets
21.2
16.9
13.9
20
5,50
3.70
29
3.67
3.71
1.09
-6.29
GotenNet,
20.5
15.2
13.0
19
5.19
3.44
27
3.49
3.43
1.09
0.56
-6.35
GotenNet,
28
19.8
13.4
12.2
19
4.98
3.30
24
3.41
3.37
1.08
0.52
-6.41
SquiformerV2 (Liao et al., 2024), Transformer-M (Luo et al., 2022), GeoSSL-DDMM (Liu et al., 2023),
3D-EMGP (Jiao et al., 2023), Coord (Zaidi et al., 2023), Frad (Ni et al., 2024a), SliDe (Ni et al., 2024b) and DenoiseVAE (Liu et al., 2025). Table I presents only the ten baseline methods with the lowest std. MAES, while the complete comparison is provided Appendix Table 9.
Model Performance and Size Scaling Analysis. We evaluate three model variants small (S), base (B), and large (L) - to analyze both performance and scaling behavior, with detailed specifications in Appendix E. As shown in Table 1, even smallest variant GotenNet, outperforms baseline methods on nine out of twelve targets while surpassing baselines on std. MAE and log MAF. GotenNet demonstrates further improvements, achieving best performance eleven targets and significantly improving aggregated metrics, reducing standard MAE by over 16% and log MAE by 3% compared the best baseline results. The largest variant GotenNet,, achieves state-of-the-art performance across all metrics, although the relative improvement decreases compared to GotenNetg, which suggests that dataset size may become limiting factor for larger models. To investigate scaling to larger datasets, we conduct experiments in Section 4.2 on the Molecule3D dataset, which contains more than million molecules an order of magnitude larger than QM9. These results establish GotenNet as the new state-of-the-art while revealing important insights about model scaling behavior.
Table 2: Performance comparisons on Molecule3D dataset.
178
KauParee
Split
Random
Scaffold
Gatoriii
Task
CHOMO
CLUMO
Ae
std.
log
Ac
Labonor
VittViltual
.0882
.0692
.0632
.1036
.0592
-2.87
.2371
SchNet
0032
0075
.0265
.0428
.063
-3.66
.1511
DimeNete+
0293
02:00
0190
0306
.0188
-4,01
.1214
SphereNet
0288
0239
.0301
-4.03
ComENer
0345
.0888
0252
.326
.0200
-3.84
.1773
PaiNN
.0196
.0263
.0197
.0307
.0182
-4.08
.1208
ET
0223
.0199
.0194
.0303
.0170
-4.13
.1822
LEFTNet
.0151
.0183
.0157
.0275
.0145
-4,32
.1117
VaVeNet-L
0136
01.59
0143
02.39
0128
1082
Figure 3: Comparison of training latency the models with respect to node count
Geoformer
0202
.135
GotenNet
.1002
the Molecule3D dataset.
4.2 MOLECULE3D DATASET
Dataset. We further evaluate our model on the Molecule3D dataset (Xu et al., 2021). This dataset contains over 29x more graphs than QM9, with approximately 1.6x and 1.9x increases the average number of nodes and edges per graph, providing an ideal benchmark for both model performance and computational scaling. We compare against diverse baseline models, including GIN- Virtual (Wang al., 2022). SchNet (Schütt et al., 2017), DimeNet++ (Gasteiger et al., 2020a), SphereNet (Liu et al., 2022). ComENet (Wang et al., 2022), PaiNN (Schütt et al., 2021), ET (Thölke & De Fabritiis, 2022), LEFTNet (Du al., 2023), Sa VeNe (Aykent & Xia, 2023), and Geoformer (Wang et al., 2023a).
8

---

Published as a conference paper at ICLR 2025
Table 3: Comprehensive comparison of various molecular modeling methods on MD22 dataset. The results are reported in MAE of energy (kcal/mol) and forces (kcal/mol/A) denoted as E and respectively. |G] denotes the size of the graphs in terms of node count.
Molocule
KDDML
ET
Allogin
MACE
EquiSormer
ViSNet
QuisNa
E-LSRM
V.LRMM
503KRATES
GuantNiy
GutesNNia,
Tatapepiide
F
0.9902
0.7968
0.1121
0.1879
0.1068
0.3019
0.0K20
0.0876
0.0828
0.0804
0.0776
0.0972
0,0640
0
0.0770
0,0887
0.0902
0.246
IL8SW
0.0719
0J9567
DELA
0
0.7474
1.3117
0.1209
0.1208
0.1153
0.0322
0.0546
0.1317
0.0066
0.1788
0.1526
0.0668
0.1200
0.0515
0.0878
0.0634
0.883
0.0695
0.379
0.242
0.0996
0,0640
0J0575
0.9421
Stucyyus
4.0497
016744
0.1393
0.1921
0.2485
0.0971
0.1286
0.0576
0.1404
0.0335
0.1283
0.0809
0.2300
0,0543
0.1252
0,0632
0.0767
0.1055
0.442
0.435
0,0711
0.0512
0
QJN427
AT-AT
E
F
0.7255
0.6911
0.1120
0,2006
0.1428
0,0652
0.1055
0.0992
0.0960
0.1309
0.1070
0.1688
0.14m0
0.0682
0.3007
0.0881
0.0722
0.0781
0.178
0.216
0,0632
0.0441
0JN478
0J1444
AT-AT-00-00
118
E
F
0.7028
1.38K5
0,3259
0.2072
0. 0,12
013935
0.1353
0.1578
0.1252
0.1510
0.1563
0.1925
0.1273
0.3400
0.1335
0.1065
0.1063
0.1135
0.332
0.345
0.0624
0.0964
GUMMSS
0J0744
Buckyball
0.5188
0.5624
0.3432
0.3032
catcher
0.3318
0.0887
0.1114
0.1335
0.3091
0.3025
0.237
0
0.0789
Deuble-walled
4.0122
1,4732
2.2097
1.1945
L.1399
1,3300
0.9993
0.6641
nanotube
0.5231
1.0011
0.3428
0.2747
0,3959
0.2473
0.3464
0.1888
Model Performance on Large-Scale Data. As shown in Table 2, GotenNet maintains its superior performance even on this larger Molecule3D dataset, achieving the lowest errors across all tasks, including 14, EHоMO, ELUMO, and Ae. Notably, GotenNet surpasses the previous best model, VeNet I., by a significant margin of 24% in jr and more than 32% HHOMO- The best log error of -4.65 in the random split further demonstrates the model's robustness on larger datasets.
Computational Efficiency Analysis. Beyond performance metrics, scaling to larger datasets requires efficient handling of increased graph sizes. We analyze computational efficiency by measuring training time across varying node counts (10-140 nodes per graph). Figure compares GotenNet with competitive taasetion-bbsed basclines including Geoformer (Wang et al., 2023a), Equiformer (Liao & Smidt, 2023), and EquiformerV2 (Liao et al., 2024). Full experimental setup details are provided in Appendix G. The æ-axis shows the node count, while the y-axis shows the training time per batch in milliseconds. The results show GotenNet maintains efficient scaling at higher node counts, while baseline methods like Geoformer, despite strong performance on smaller graphs, become computationally intensive due to their dense O(n²) representations. Both GotenNets and GotenNet variants maintain consistent efficiency across all node counts, demonstrating their suitability for large-scale applications where computational overhead is critical.
4.3
MD22 DATASET
Dataset. The MD22 dataset (Chmiela et al., 2023) contains molecular dynamics trajectories for seven systems, with atom counts from 42 to 370, across four biomolecule and supramolecule classes. It presents challenges in system size, flexibility, and nonlocality, making it key benchmark for scalability and accuracy in molecular force field models. Following the data splits from (Chmiela al., 2023), we evaluate GotenNet against several baselines, including sDGML (Chmiela al., 2018), ET (Thölke & De Fabritüs, 2022), Allegro (Musaelian et al.. 2023), MACE (Batatia et al., 2022b), Equiformer (Liao & Smidt, 2023). ViSNet (Wang et al., 2024), QuinNet ( (Wang al., 2023b), SO3KRATES (Frank et al., 2024), and LSRM (Li et al., 2024) along with its variants E-LSRM (Equiformer-LSRM) and V-LSRM (ViSNet-LSRM)).
Results. The MD22 dataset poses significant challenges due to its wide range of molecule sizes, requiring accurate predictions of both energy and forces. We show results of MD22 dataset in Table 3. Our proposed model, GotenNet, consistently outperforms tfae-oofthhe-ar methods across all evaluated molecules, demonstrating superior performance in both energy and force predictions.
For molecules such as Tetrapeptide and AT-AT, GotenNet achieves notable reductions in energy errors, with improvements of 18.6% and 29.5% over the previous best models, respectively. Simultancously, force prediction errors are reduced by up to 30.4%, underscoring GotenNet's balanced performance across both metrics. In more complex cases, such as the Buckyball catcher and Double- walledd nanotube, GotenNet sets new benchmarks, reducing energy errors by over 35% and force errors by up to 31.3%. These results highlight the robustness and versatility of GotenNet in handling diverse molecular structures, establishing it leading model in both energy and force prediction.

---

Published as a conference paper at ICLR 2025
4.4 RMD17 DATASET
Dataset. The rMD17 dataset (Christensen & Von Lilienfeld, 2020) is revised version of the MD17 benchmark, featuring 10 small organic molecules with 100,000 conformations per molecule. It serves as a key benchmark for evaluating machine learning models' ability to predict molecular energies and forces across diverse conformations. We follow the standard split (Christensen & Von Lilienfeld, 2020) of 950 training, 50 validation, and the remaining conformations for testing. The results are averaged over five predefined splits to ensure robust evaluation.
Table 4: The table presents MAE for energy (kcal/mol) and forces (kcal/mol/AA on the rMD17 dataset.
Molecule
NequiP
ACE
UNITE
Allegro
BCTTNet
MACE
TensoeNet
GotenNer
Aspitia
F
E
0.1891
0.0530
0.1407
0:4128
0.1753
0.0553
0.1683
0.0530
0.1960
0.0530
0.1522
0.0507
0.2052
0.0553
0.1306
0.0358
Azohenzene
E
0.0161
0.0669
0.2514
0.0830
0.0060
6,0254
0.0277
0.0000
0.0161
0.0277
0.0692
0.0161
0.0715
0.0121
Benzene
E
0.0009
1000000
8.0016
0.009
0.0077
0.0092
0.0005
0.0005
F
0.0069
0.0115
0.0168
0.0046
0.0069
0.0069
0.0069
0.01-17
Ethanol
E
0,0092
0.0277
0.0143
0.0092
0.0092
0.0092
0.0115
0.0071
0.06446
0.1683
0.0853
0.0144
0.0484
0,0807
Maloraldehyde
E
0.0184
0.0392
0.0254
0.0388
0.014
0.0184
0.0184
0.0129
0.1176
0,2560
0.1522
0.0830
0.1338
0.0945
0.1245
0,0830
Naphthalene
E
0,0208
0.0208
0,0106
0.0046
0.0046
0,0115
0.0046
0,0039
0.0300
0.0600
0.0208
0.0115
0.0369
0.0369
0,0210
Paracetamol
E
0.0323
0.0922
0.0138
0.0346
0,0300
0,0300
0,0300
0.0212
F
0.1361
0.2929
0.1637
0.1130
0.1338
0.1107
0.1361
0.0928
Salicylic acid
E
0.0161
0.0115
0,0168
0.0208
0.0184
0,0208
0.0184
0,0141
0.0922
0.0715
0.1061
Tohene
E
0.0069
0.0255
0.0104
0.0022
0.0092
0.0115
0.0069
0,0044
F
0.0369
0.1499
0.0577
0.0415
0.0138
0.0346
0.0392
0.0261
Uracil
E
0.0092
0.0254
0.0134
0.0138
0.0092
0.0115
0.0092
0.0064
0.0715
0.0876
0.0738
0.0484
0.0715
Results. As shown Table 4, GotenNet outperforms other models in 80% of tasks and ranks second in the remaining, excelling in both energy and force predictions. GotenNet sets new benchmarks for molecules such as Aspirin, Azobenzene, Ethanol, Paracetamol, and Toluene, demonstrating balanced improvements across energy and force predictions. These results highlight GotenNet's 's robustness and its ability to accurately model molecular properties, outperforming prior methods on rMD17 dataset.
4.5 ABLATION STUDY
Table Ablation study on QM9 dataset.
Table 5 presents the results of the ablation study, high- lighting the impact of various components on the per- formance of GotenNet. The inclusion of structural embedding (SE), self-attention (SEA), geometric en- coding (GE), and HTR generally leads to improved results, as shown in rows 1, 7, and 8, where the model achieves the lowest std MAE and log MAE. The re- moval of any one of these components results in a significant degradation performance, particularly in the cases without geometric encoding (row 4) or reducing T.max (row 2). The full model with 12
I
SE
SEA
GE
HTR
std
log
v
0.67
-6.21
0.68
-6.17
2
x
d
V
V
0.67
-6.17
x
V
0.65
-6.23
2
V
X
0.83
-5.6
2
v
V
X
0.64
-6.20
0.61
-6.26
12
2
0.56
-6.34
layers (row 8) achieves the best performance, with the lowest std MAE of 0.56 and log MAE of -6.34. This demonstrates the combined effectiivnness of all components for model scalability.
CONCLUSION
We presented GotenNet, a framework for modeling 3D molecular structures that strikes a balance between expressiveness and efficiency by integrating geometric tensor representations with innovative components, including unified structure embedding, geometry-aware tensor attention, and hierarchical tensor refinement. GotenNet consistently outperforms state-of-the-art methods across four benchmark datasets. It also demonstrates scalability and computational efficiency, making it highly suitable for large-scale molecular systems. These results establish GotenNet a versatile and powerful framework for 3D equivariant graph neural networks. Future work could further enhance its scalability to larger molecular systems and explore applications in molecular dynamics and materials science.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Junyi An, Chao Qu, Zhipeng Zhou, Fenglei Cao, Xu Yinghui, Yuan Qi, and Furao Shen. Hybrid direc- tional graph neural network for molecules. In The Twelfth International Conference on Learning Representations, 2024. URL https: opwpreeiee.ee.eeeeeoeeeee wew. wttp:/oommmmmdmBB/-m--
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32, 2019.
Sarp Aykent and Tian Xia. GBPNet: Universal Geometric Representation Learning on Protein Structures. In Pruceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery Data Mining, KDD 22, Washington, DC, USA, 2022. Association for Computing Machinery. ISBN 978-1-45033-3355-0022200
Sarp Aykent and Tian Xia. SaVeNet: A Scalable Vector Network for Enhanced Molecular Represen- tation Learning. In seirth-eeventh Conference on Neural Information Processing Systems, 2023, URL httphhtt/////
htttps://peerreeeee opwpreveeeeeeeeeeeeeeeeeeeeeeeee net ntt//oruneteee forum? id= L0-0OIIBCFsds
Ilyes Batatia, Simon Batzner, Dávid Péter Kovács, Albert Musaclian, Gregor NC Simm, Ralf Drautz, Christoph Ortner, Boris Kozinsky, and Gābor Csányi The design space of e (3)-cquivariant atom-centered interatomic potentials. arXiv preprint arXiv:2205.0666 2022a.
Ilyes Batatia, David P Kovacs, Gregor Simm, Christoph Ortner, and Gabor Csanyi. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 11423-11436
Curran Associates, Inc., 2022b.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Korblluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)equiaarian graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):2453, 2022.
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling, Ge- ometric and physical quantities improve e(3) equívariant message passing. International Conference on Learning Representations, 2022. URL https: /opprreviee.....c//m net/ forum?
id= [t=__w28@0BBeV.
Jiacheng Cen, Anyi Li, Ning Lin, Yuxiang Ren, Zihe Wang, and Wenbing Huang. Are high- degree representations really unnecessary in equivariant graph neural networks? In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: // //openreview. net worumorriiio Darrmmidd-M0nnnWuGG..
Lifan Chen, Xiaoqin Tan, Dingyan Wang, Feisheng Zhong, Xiaohong Liu, Tianbiao Yang, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, and Mingyue Zheng. Transformerepi: improving compound- protein interaction prediction by sequence-based deep learning with self-attention mechanism and label reversal experiments. Bioinformatics,
44116.44442220 2020.
Stefan Chmiela, Huziel E Sauceda, Klaus-Robert Müller, and Alexandre Tkatchenko. Towards exact molecular dynamics simulations with machine-learned force fields. Nature communications, 9(1): 2887,2016
Stefan Chmicla, Valentin Vassilev-Galinde Oliver T. Unke, Adil Kabylda, Huziel E. Sauceda, Alexandre Tkatchenko, and Klbes-Robert Müller. Accurate global machine learning force fields for molecules with hundreds of atoms. Science Advances, 9()xaadf0873, 2023. doi: 10. 126/sciadv, adf0873. URL https: / http://swienc.e.eeeeee.teeccecet org/ doi abs 1126/ sciadv adf0873.
Anders S Christensen and O Anatole Von Lilienfeld. On the role of gradients for machine learning of molecular energies and forces. Machine Learning: Science and Technology. 1(4):045018, 2020.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for so (3)-equiva networks. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 2200-11209, 2021.
11

---

Published as a conference paper at ICLR 2025
Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie- Yan Liu. Se (3) equivariant graph neural networks with complete local frames. In International Conference on Machine Learning, pp. 5583-5608 PMLR, 2022.
Weitao Du, Yuanqi Du, Limei Wang. Dieqiao Feng, Guifeng Wang, Shuiwang Ji, Carla P Gomes, and Zhi-Ming Ma. A new perspective on building efficient and expressive 3d equivariant graph neural networks. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URI. /ttttp::///penevvete net forum? (d=hWPNYNKKPP.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
J Thorben Frank, Oliver T Unke, Robert Müller, and Stefan Chmiela. A euclidean transformer for fast and stable machine learned force fields. Nature Communications, 15(1):6539, 2024.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3attransformmr:: 3d roto- translation equivariant attention networks. Advances in neural information processing systems, 33: 1970-1981, 2020.
Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and uncertainty-awar directional message passing for non-equilibrium molecules. In Machine Learn- ing for Molecules Workshop, NeurIPS, 2020a.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations, 2020b. URL https:// openreview , net/ forum? BdeBBeebxttBP...
Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks. arXiv preprint arXiv:2207.09453. 2022.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp.
1272- PMLR, 2017.
Jiaqi Han, Jiacheng Cen, Liming Wu, Zongzhao Li, Xiangzhe Kong, Rui Jiao, Ziyang Yu, Tingyang Xu, Fandi Wu, Zihe Wang. Hongteng Xu, Zhewei Wei, Yang Liu, Yu Rong, and Wenbing Huang. survey of geometric graph neural networks: Data structures, models and applications, 2024.
URL https::p//// /ttt///axxxi //taxaaxvvooog.//eoa org/ abs /2003. 00485.
Kexin Huang, Tianfan Fu, Lucas M Glass, Marinka Zitiik, Cao Xiao, and Jimeng Sun. Deeppurpose: deep learning library for drug-target interaction prediction. Bioinformatics, 36(22-23):5545- 5547, 2020.
Rui Jiao, Jiaqi Han, Wenbing Huang, Yu Rong, and Yang Liu. Energy- equivariant pre- training for 3d molecular graphs. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 8096-8104 8104, 2023.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021. URL https: http:///eeeeeeeeeeeeeeeee wtt.ri/eweee/ttnnn forum??
(d==YYLDDV5xxJ4
Johannes Klicpera, Florian Becker, and Stephan Günnemann. GemNet: Universal directional graph neural networks for molecules. In A. Beygelzimer, Y. Dauphin. P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:// openreview net forum? dd=ss_sOaSaSSSS 90_S_SKK.
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch Clebsch-gordan nets: fully fourier space spherical convolutional neural network. In Advances in Neural Information Processing Systems 32, pp. 10117-10126, 2018.
Tuan Le, Frank Noé, and Djork-Amé Clevert. Equivariant graph attention networks for molecular property prediction. arXiv preprini 200.0920.,,099/ 2022.
12

---

Published as a conference paper at ICLR 2025
Yunyang Li, Yusong Wang, Lin Huang, Han Yang. Xinran Wei, Jia Zhang, Tong Wang, Zun Wang, Bin Shao, and Tie-Yan Liu. Long-short-ranee message passsagepassing: A physics-informet framework to capture non-local interaction for scalable molecular dynamies simulation. In The Twelfth International Conference on Learning Representations, 2024. URL https:///peeettt /pppnrevie...... net/forum weerummidd-roe devDgeedMno...
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In International Conference on Learning Representations, 2023. URL https:// openreview n forum? d=KwmP FARgOTD.
Yi-Lun Liao, Brandon M Wood, Abhishek Das, and Tess Smidt. Equiformery2: Improved equivariant transformer for scaling higher-degree representations. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https:://////. /tte:///pppeeiieeeee net/ fesetfforuee forummrmmmmmmm id= mCOBKZmrZD.
Shengchao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with SE(3)innvriint de- noising distance matching. In The Eleventh International Conference on Learning Representations, 2023. URL https: htt///ppnrrviieeee net/ forum? d=C THVoldvR
Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang. Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3D molecular graphs. In International Conference Learning Representa- lions, 2022. URL https::////... / opwnreview net/ fest//form?? i di= www.gvsssssss... 9r.
Yurou Liu, Jiahao Chen, Rui Jiao, Jiangmeng Li, Wenbing Huang, and Bing Su. Denoise VAE: Learning molecule-adaptive noise distributions for denoising- baased based 3d molecular are-training In The Thirteenth International Conference on Learning Representations, 2025. http::
//operreiiee net forum? d=ymm pr333QQ..
Shengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie-Yan Liu, Liwei Wang, and Di He. One transformer can understand both 2d & 3d molecular data. arXiv preprint arXiv:2210.0/765, 2022.
Albert Musaelian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. Nature Communications, 14(1):579, 2023.
Thin Nguyen, Hang Le, Thomas P Quinn, Tri Nguyen, Thuc Duy Le, and Svetha Venkatesh. Graphdta: predicting drug-target binding affinity with graph neural networks. Bioinformatics, 37 (8):1140-1147, 2021.
Yuyan Ni, Shikun Feng, Xin Hong. Yuancheng Sun. Wei- Yai-Ying Ma. Zhi-Ming Ma, Qiwei Ye, and Yanyan Lan. Pre-training with fractional denoising to enhance molecular property prediction. Nature Machine Intelligence, pp. 1-10, 2024a.
Yuyan Ni, Shikun Feng, Ying Ma, Zhi-Ming Ma, and Yanyan Lan. Sliced denoising: A physics- informed molecular pre-training method. In The Twelfth International Conference on Learning Representations, 2024b. URL https: wpwp//ppeeeew..eeeoo/pppe http://wwwee//oomemmeeeeeeeee
Sergio
Santiago Morandi, Rodrigo A Vargas Hernan-Hr Kjell Jorner, Žarko Ivković, Núria López, and Alán Aspuru-Guzik. Fast evaluation of the adsorption energy of organic molecules metals via graph neural networks. Nature Computational Science, 3(5):433-442, 2023.
Saro Passaro and C Lawrence Zitnick. Reducing so (3) convolutions to so (2) for efficient equivariant gnns. In International Conference on Machine Learning, pp. 27420-27433 27438. PMLR, 2023.
Can Polat, Hasan Kurban, Erchin Serpedin, and Mustafa Kurban. TDCM25: A multi-modal multi- task benchmark for temperaturreddeeeddent crystalline materials. In A/ for Accelerated Materials Design- ICLR 2025, 2025. URL httpt:ttp/.c http:////eeeeeeeeeeeeeeee net./ forum? wwrrrnindinna... d=NNBBSTTTQ..
Zhuoran Qiao, Anders S Christensen, Matthew Welborn, Frederick R Manby, Anima Anandkumar, and Thomas F Miller III. Informing geometrie deep learning with electronic interactions to acceler- ate quantum chemistry. Proceedings of the National Academy of Sciences, 119(31):e220222119 2022.
13

---

Published as a conference paper at ICLR 2025
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):-77, 2014.
Patrick Reiser, Marlen Neubert, André Eberhard, Luca Toresi,, Chen Zhou, Chen Shao, Houssam Metni, Clint van Hoesel, Henrik Schopmans, Timo Sommer, et al. Graph neural networks for materials science and chemistry. Communications Materials, 3(1):93, 2022.
Lars Ruddigkeit, Ruud van Deursen, Lorenz C. Blum, and Jean- Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. 17. Journal of Chemical Information and Modeling, 52(1 11:28)44222888 2875, 2012. doi: 10.1021/c1300415d URL
https:: / doii. wrggr1///0 10. /t01//c/000000 1d. PMID: 23088335.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) Equivariant Graph Neural Networks. In Proceedings of the 38th International Conference on Machine Learning, pp. 9323- 9332. PMLR, 2021. URL https: httpss///ooceedieeee ml www.m.......//3333..m http://rrrooeennnnn..rrrrrr/e/re html.
Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning,
pp. 9377-9388. PMLR, 2021.
Kristof Schütt,, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. 30, 2017.
Philipp Thölke and Gianni De Fabritis.. Equivariant transformers for neural network based molecular potentials. In International Conference on Learning Representations, 2022.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation -qquivarann neural networks for 3d point clouds. arXiv preprint arXiv: 1802.02219 2018.
Oliver T Unke and Markus Meuwly. PhysNet: A neural network for predicting energies, forces, dipole moments, and partial charges. 15(6))3678-366 2019.
Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. ComENet: Towards complete and efficient message passing for 3D molecular graphs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022. URL htt.:httss///ppeeeeiieeeee
net forum? 1d=mC2MgeWSFF.
Yusong Wang, Shaoning Li, Tong Wang, Bin Shao, Nanning Zheng, and Tie- Yin Liu. Geometric trans- former with interatomic positional encoding. In Shirty-seventhh Conference on Neural Information Processing Systems, 2023a. URL https::/p//. httpt:://pppeeeeeeeeee wttp///wwpeeeeeeee////e/ee net/ wttp://wwwwee/ttotiid/deedo/dd wowpoooddooooooeoeeooooeee
Yusong Wang, Tong Wang. Shaoning Li, Xinheng He, Mingyu Li, Zun Wang, Nanning Zheng, Bin Shao, and Tie-Yan Liu. Enhaneing geometric representations for molecules with equivariant vector-scalar interactive message passing. Nature Communications,
[.]::31,,0024,
Zun Wang, Guoqing Liu, Yichi Zhou, Tong Wang, and Bin Shao. Efficiently incorporating quintuple interactions into geometric deep learning force fields. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL https:/ http:///ppeeeeiiieeeeeeee net forum?id bPJmu1PbZD.
Tian Xia and Wei-Shinn Ku, Geometric graph representation learning on protein structure prediction. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining,
KDD '21, pp. 1873-1883, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 4773833333325. doi: 10. 1411443444548.44673333 URL //dp::::///d..../ wrw./..ggl.000 11451 http:///ooooooooo.oo/oo/ooo
34475 848..44..0 64633333.
Tian Xie and Jeffrey C. Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical Review Letters, 2018.
14

---

Published as a conference paper at ICLR 2025
Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3d: A benchmark for predicting 3d geometries from molecular graphs. CoRR, abs/2110.01717, 2021. URL https: httppp////axxv..or//...oooo org/
2110.01711
Ziduo Yang, Weihe Zhong, Lu Zhao, and Calvin Yu-Chian Chen. Mgraphdta: deep multiscale graph neural network for explainable drug-targe targgtt binding affinity prediction. Chemical science, 13(3):
2022.
Sheheryar Zaidi, Michael Schaarschmidt, James Martens, Hyunjik Kim, Yee Whye Teh, Alvaro Sanchez-Gonzalez, Peter Battaglia, Razvan Pascanu, and Jonathan Godwin. Pre-training via denoising for molecular property prediction. In The Eleventh International Conference Learning Representations, 2023, URL https: / wpppreveeeeeeeeeeeeeeeeeeeeeeeee net forum ? id=tYIMM
Xiao-Meng Zhang, Li Liang, Lin Liu, and Ming-Jing Tang. Graph neural networks and their current applications in bioinformatics. Frontiers in genetics, 12:690049, 2021.
Zehong Zhang, Lifan Chen, Feisheng Zhong, Dingyan Wang. Jiaxin Jiang, Sulin Zhang, Hualiang Jiang, Mingyue Zheng, and Xutong Li. Graph neural network approaches for drug-targe interac- tions. Current Opinion in Structural Biology, 73: 13:122227, 2022.
Zheng Zhang and Liang Zhao. Representation learning on spatial networks. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, volume 34, pp. 2303-2318. Curran Associates, Inc., 2021. URL https: http:::://proeediigg.nerrr.pppp http://s....e...s.pppp.pp....... flle:/22255288122222222222333332
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A PROOF: EQUIVARIANCE OF GEOMETRIC TENSOR
Equivariance of Geometric Tensor: Let T: M T be a geometric tensor field, and g E B(3) be an element of the Euclidean group. Then 7" is equivariant under g if:
gg (p)) = ( p)
where p a representation of E(3) on the space of tensors
Proof. To prove the equivariance of the geometric tensor field, we start by recalling the definition of equivariance. A tensor field T M + T is equivariant under the group action of g E E(3) if, for all points p E M,
T(g p) = ppg)T(p),
where p(g) a representation of the group element 9 on the space T.
Now, consider the action of the group element g E E(3) on T. By the definition of the group action of on the tensor field T, we have:
(g) T(g-1.p)
Next, apply the representation p(g) to the transformed tensor T(g p). By the equivariance condition, we require that:
(g (p) P(g)T(g-'
This completes the proof, as we have shown that the transformed tensor field g is related to the original tensor field by the representation p(g). satisfying the equivariance condition.
B PROOF: GOTENNET PRESERVE THE EQUIVARIANCE PROPERTY
Proof. We will prove that layer operations in GotenNet preserve the equivariance property of geo- metric tensor fields under the action of the Euclidean group (33). To this end we will prove that each component used GotenNet preserves the invariance/equiivarince properties.
First we will prove that the initial node and edge representations are invariant under the action of the Euclidean group E3). We'll 'lll consider each component separately. The node embedding process consists of two main steps: message passing and representation update. We'll Wl'll show that both steps are invariant under (3). The message passing equation in node embedding is:
m,
Σ JEN())
Aj
(1((7()) Wmlp ((7(0) :))).
Under the action of F?(3) zj invariant it's atomic number. Paj,ini is invariant it's the distance between nodes i and j. .N(i) is invariant as the set of neighbors doesn't change under rigid transformations. Therefore, m, is invariant under E(3). The node representation update is defined as:
h (C((NN((E)A na ) Wma)
W
mru
Here, z; is invariant, m, invariant (as shown above), and other operations (concatenation, linear transformations, layer normalization, and activation) are invariant. Thus, h; is invariant under E(3). The node embedding is followed by edge embedding and the initial edge representation is computed as:
tn (h. -...inis) (p)()(()) W
We' 'e already shown that Ami is invariant. The distance Fij.init is also invariant under E(3). The operations ş2, and linear transformations Thus, we have shown that both the initial node representations hiiin and edge representations inji invariant under the action of the Euclidean group F(3).
all invariant. Therefore, iij is invariant under E(3).
16

---

Published as a conference paper at ICLR 2025
After the initialization step the next module is GATA. Therefore, will prove that the GATA module and the subsequent operations the interaction layer preserve equivariance under the action of the Euclidean group E3). The query, key, and value Computation defined in Equation (5), we have:
q: h. Wg: k h, by = (hg))
h; and h scalar (O-degree steerable features) representations, which are invariant under F(3). The linear transformations W, and WA. and the function "e. preserve this invariance. Thus, qt. kij- and vy are invariant under E3). The attention coeflicients defined in Equation (6), the attention coefficient Oij is computed as:
a3 = q: (k3 o (t,y Wre))².
We've ve shown that and j are invariant. tij is an edge embedding, which is invariant under F(3). The operations σk o, and matrix multiplication preserve invariance. Therefore, Qij invariant under E(3). The self-attention operation in Equation (6) defined as:
exp(C(t) Σ CREN( exp(a:k) 4/-
sea.j
Since oiy and are invariant, and the softmax operation preserves invariance, sea;y is invariant under E3).. Geometric Encoding defined in Equation (7), we have:
d.11) Timas (.(1) o 1o 1111 .oo 1==1 split(sea,,) (th Wrs) 7 (h,)) ((F dnc)..
seaij. Lig. hy, and (0) are all invariant under E(3). The operations Wr., 7a. Φ. o, and split preserve (o)11)) [=] ya and {ory (,(0) Lmnxx are invariant under E(3). Finally, node L
this invariance. Therefore, of (o tensor representation updated with Equation (8), we have:
N[=1"
Ah;
+ (oit)) jEN())
A
+
Oo d0
+o
100 g
JEN())
d.(0) Loax and (o (,(1) bmax are invariant under E(3). Specifically, {o M=11 Here, hes o.j. {0 11=1
d.(1) Lmoz
and for; (.1) banx are sets of invariant coefficients, where for each degree of steerable features, we have distinct invariant scalars o, d.(1) and 0 (.(1) +(0) and X, are high-degree steerable features that transform equivariantly under E(3). The operation between invariant sealars and equivariant tensors preserves equivariance. Specifically, since multiplies the same invariant values over the spatial dimension the high-degree tensors, this operation preserves equivariance. To see this, let g E F33) be transformation, be invariant scalar, and T be an equivariant tensor. Then:
g(s T) g(TT) (T)
(T)).
This shows that the operation commutes with the group action, preserving equivariance. The permutation-invarian aggregation function Q (such as summation or averaging) preserves equivari- ance because it operates independently on each degree of the tensor, maintaining their transformation properties under E(3). Therefore, the scalar Ah; remains invariant, while the high-degree steerable features (l) transform equivariantly under F(3).. Thus, we have shown that the interaction layer preserves the equivariance of the input representations under the action of the Euclidean group F3(). Thus, we have shown that the interaction layer preserves the equivariance of the input representations under the action the Euclidean group (3).
Next, we will prove that the HTR component preserves equivariance under the action of the Euclidean group E3). We'll consider each operation the HTR component. Tensor projections as defined in Equation (10), we have:
EQ,'
W ogg
EK
( nk:
for E
0 and ( high-degree tensors that transform equivariantly under F(3). The projection
matrices W... og and ((() apply uniform weights across the spatial dimensions for each representation
17

---

Published as a conference paper at ICLR 2025
dimension. This uniform application preserves equivariance because it commutes with the action of E(3). Let g E E(3) be a transformation:
8(x: W vq) - (o( wxg) = (o( eeg)).
The same holds for EK, Therefore, EQ; and EK; transform equivariantly under E(3). Aggregation of angular and magnitude information from Equation (11), we have:
Aggi=1 Ximax (BQQ (1) EK," ())
Wij
(1) (8)
The operation (EQ,) EK;
is an inner product between equivariant tensors. Crucially, both
(1) EQ: and EK; are subject to the same global rotation and translation under the action of E(3).
Let g E E(3) be a transformation. Then:
")) (o (e(EQ, )) g( (EK, (1)
(1) = (D) EQ; T(D(0))))))))))))))))))))))))))) EK,
(1) =EQQ (1) (D ( (r) EK, (1)
= =(EQ.) (1) EK; -r'.
Here, (r) represents the Wigner D-matrix of degree which the appropriate representation for the transformation of spherical tensors under rotations. The matrix (r) unitary, meaning (r) ( = I. The factor of +1 accounts for the parity transformation, where for improper rotations and y 1 for proper rotations, with the exponent determining the overall sign based on the spherical harmonic degree. This demonstrates that the inner product transforms covariantly under the full 0(3) symmetry group, with the parity factor properly accounting for improper rotations. The aggregation ① over these covariant scalars preserves the transformation properties, ensuring that Way transforms appropriately under both rotations and inversions. The edge representation update is given by:
bu t- Atyj, At. = Tw "e (e(g)))
Wij is invariant, as shown in the previous step, and t.j is an edge embedding, which is also invariant under (3). Since both W and bij are invariant, the application of the differentiable functions 7 and preserves this invariance. Moreover, the Hadamard product of invariant quantities is itself invariant. Hence, we have
g( = (w))) o (e(())) = The (w),) (k(g)) = Atij
Thus, Alyj invariant under E(3). Thus, we have shown that the Hierarchical Tensor Refinement component preserves the equivariance of the input representations under the action of the Euclidean group E3)). The high-degree tensors transform equivariantly, while the scalar quantities and edge embeddings remain invariant.
Finally, we will prove that the EQFF (Equivariant Feed-Forwarde blocks preserve equivariance under the action of the Euclidean group E(3). Consider each operation in the EQFF component described in Equation (13). The input scalars h is invariant under E(3) as it represents scalar features, while X transforms cquivariantly under E33 as it represents high-degree steerable features. Next, we examine the computation of m, and m2:
m,, m2 (plit,(7m (I) X (" Woul2,, h)).
The operation X ( Wew preserves equivariance as it applies the same linear transformation across all spatial dimensions. The Le norm this equivariant tensor field, (1) (W......) null2. invariant under E(3). To see this, let E E3) be a transformation. Then l[g( xx W..))l = | (X....lllll Wo.)lla (0 where R is the rotation matrix corresponding to g. The translation component doesn't t affect the norm. The concatenation [J] X W Weull2,, h)) is of two invariant quantities, resulting in an invariant vector. As Ym is applied an invariant input, its output is also invariant. Finally, splitting this invariant vector results in invariant components my and m2. Now, we analyze the EQFF operation:
EQFF(h, x0) = ((h+ (h+mm)). (x(1)
(x + (m2 X (0 W..)).
18

---

Published as a conference paper at ICLR 2025
The term h + m is sum of two invariant quantities, resulting in invariant scalar. The operation m2 X W... preserves equivariance because m2 invariant scalar and ( is equivariant. The element-wise product of an invariant scalar with an equivariant tensor is equivariant. Conse- quently, x0)) + (m2 o X W wu) is a sum of two equivariant tensors, resulting an equivariant tensor. Hence, EQFF returns tuple of updated representations for scalar and high-degree steerable features.
Thus, have shown that the EQFF operation preserves the equivariance of the input representations under the action the Euclidean group (3). The scalar part remains invariant, while the high-degree steerable features transform equivariantly.
Hence, we have shown that each major component the GotenNet architecture preserves the equivariance property under the action of the Euclidean group E(3). The initial node and edge embeddings are invariant under E33). The GATA module, including its self-attention mechanism and geometric encoding, maintains the invariance of scalar quantities and the equivariance of high-degree steerable tensors. The Hicrarchical Tensor Refinement (HTR) component preserves equivariance in its tensor projections and ensures that edge updates remain invariant. Finally, the EQFF blocks maintain the overall equivariance structure by preserving the invariance of scalar parts and the equivariance of high-degree steerable features. In each of these components, we have demonstrated how the various operations interact with the group action of E(3) to preserve the required invariance and equivariance properties.
C PROOF: EQUIVARIANCE OF GOTENNET
Equivariance of GotenNet: If all layers of GotenNet are equivariant, then the entire network is equivariant.
Proof. We will prove this by induction on the number of layers in the network.
Base Case (Layer 1): Let T M - T represent the output of the first layer of the network. Since the first layer is equivariant by assumption, we have that for all g E E(3).
(g T (T) = p(g)T p).
Thus, the first layer preserves equivariance.
Inductive Step: Assume that the output of layer 1, denoted by T is equivariant. That is, for all g E E(3).
(g T = ((
We need to show that layer ( + 1, denoted by 7 also equivariant. Let +/+1 represent the operation of the (1 + 1)-th layer. Then 7 = + 11 o T Since the (1 1)-th layer is also assumed to be equivariant, we have:
(g (g-1++))pp) = p(g) p).
The equivariance of T follows from the equivariance of T and +1+1, as the composition of equivariant functions also equivariant. Explicitly:
(g- T (p)) (g ($'+1 o T))))))
gl+2((g T))
= (10)
((
=1
Hence, by the principle of mathematical induction, have shown that if all individual layers in the GotenNet are equivariant, then for any number of layers, the final output of the network, being the composition of these equivariant layers, is also equivariant. This result relies the property
19

---

Published as a conference paper at ICLR 2025
that the composition of equivariant functions is itself equivariant. Specifically, have shown that T = 4'+ oT', and the equivariance of both Φ'+1 and ensures the equivariance of T Hence, the entire network is equivariant. This theorem, combined with the previous proof of component-wisg equivariance (see Appendix B), establishes the overall equivariance of the GotenNet architecture, ensuring its consistency under Euclidean transformations of the input space.
D REPRODUCIBILITY STATEMENT
The details on components of the architecture, in Section E. The code used reproduce the experiments is available. All datasets used in this study are publicly available; the access instructions will be included in the source code. We have included information on the computational resources used for our experiments, including hardware specifications and software versions, to facilitate reproducibility of our results.
and model variations are outlined
Table 6: Hyper-parameters for the datasets GotenNet compared against the baselines. The parameters for GotenNet if multiple variations exists.
Hyper-parameters
QM9
Molecule3D
MD22
rMD17
Optimizer
AdamW
AdamW
Adam
Adam W
Learning rate scheduling
Linear warmup with reduce on plateau
Warmup steps
10,000
5,000
1,000
1,000
Maximum learning rate
[6e-5, 5, 1e-4] [e-4]
le-4
[4e-5, 5. 1e-44
2e-4
Learning rate decay
0.8
0.8
0.8
0.8
Learning rate patience
15
5
30
30
Loss function Gradient clipping
MSE
T
MSE
MSE
10
5
10
Batch size
32
256
4
Number of epochs
1,000
300
3,000
3,000
Weight decay
0.01
0.01
0.01
0.01
Dropout rate
0.1
0.1
0.1
0.1
Node dimension (dne)
256
384
[256, 384]
192
Edge dimension (dea)
256
384
[256, 384]
192
Edge refinement dimension (dapa)
256
384
768
768
Lmax
2
2
2
Number of Layers
6
12
[6, 8]
12
Number of RBFs
64
32
32
32
Number of Attention Heads Cutoff radius
8
8
8
8
5.0
5.0
[4.0, 5.0]
5.0
E TRAINING DETAILS AND
Table presents the comprehensive set of various datasets. These parameters were carefully selected to optimize model performance and ensure fair comparisons with baseline methods. For the GotenNet architecture, we primarily report the parameters for the base (GotenNet.)) variation where multiple model sizes exist, The optimization process utilized the AdamW optimizer across all datasets, coupled with linear warmup strategy and learning rate reduction on plateau. This adaptive learning rate approach allows for more stable training and improved convergence. Furthermore, spherical harmonics were computed using the e3nn (Geiger Smidt, 2022) library.
employed in our experiments across
The learning rates were fine-tuned for each dataset, with QM9 and MD22 employing range of maximum learning rates to account for the diverse nature of their target properties. Molecule3D and rMD17 datasets, on the other hand, used fixed maximum learning rates of le-4 and 2e-4, respectively. To mitigate overfitting and promote generalization, we implemented weight decay (0.01) and dropout 10.1) consistently across all datasets. The choice of loss function varied, with mean squared error (MSE) being the predominant choice, except for Molecule3D, which utilized the L1 loss.
20

---

Published as a conference paper at ICLR 2025
Table 7: Performance comparisons of GotenNet variations on QM9 dataset.
Task
Ag meV
CHOMO mcV
J mD
C2 midl
G
II
πt moñ
U
Un mcV
ZPVE
std.
log
Units
mag
mcV
meV
mcV
mcV
meV
%
GotenNets
25.4
18.4
15.7
7.5
4.17
3.97
3.89
0.67
GotenNets
21.2
16,9
13.9
7.5
3.70
29
3.67
3.71
1.09
0.60
-6.29
Improvement
16.5
8.2
11.5
0.0
4.8
11.3
12.1
7.6
4.6
10.5
1.3
G
33
23.0
16.4
14.4
7.8
20
5.42
3.74
32
3.76
3.76
1.10
0.61
-6.26
GotenNeta
32
20.5
15.2
13.0
7.2
19
5.19
3.44
27
3.49
3.43
1.09
0.56
-6.35
% Improvement
3.0
10.9
7.3
9.7
7.7
5.0
4.2
8.0
15.6
7.2
8.8
0.9
8.2
1.4
Go GotenNetz.
20.7
14.3
13.3
7.7
19
5.27
3.47
25
3.58
3.67
1.08
0.56
-6.34
28
19.8
13.4
12.2
6.7
19
4.98
3.30
24
3.41
3.37
1.08
0.52
-6.41
Improvement
6.7
4.4
6.3
13.0
0.0
5.5
4.9
4.0
8.2
7.1
The model architecture parameters, such as dne, ded, and dayd. were adjusted based on the complexity of the dataset and the specific prediction tasks. For instance, MD22 and rMD17 datasets, which involve more complex molecular dynamics simulations, employed larger edge refinement dimensions (768) compared to QM9 and Molecule3D (256 and 384, respectively). The number of layers in the model also varied, with Molecule3D and rMD17 using deeper architectures (12 layers) compared to QM9 and MD22 (6 layers for the base model).
is worth noting that for QM9, we experimented with different model sizes by varying the number of layers (4, 6, and 12 for S, B, and L variations, respectively). Similarly, for MID22, we explored more compact model variation by reducing the number of layers to 4 and halving the representation dimensions. These variations allow us investigate the trade-off's between model complexity and performance across different molecular property prediction tasks.
The consistent use of 8 attention heads and a Lmax of 2 across all datasets suggests that these parameters provide a good balance between computational efficiency and model expressiveness for a wide range of molecular modeling tasks. The cutoff radius, predominantly set at 5.0 (with some variations in MD22), was chosen capture relevant atomic interactions while maintaining computational feasibility.
F ABLATION STUDY: SHARED AND INDIVIDUAL COEFFICIENTS
In this section, we investigate the impact of using shared coefficients for spherical harmonics as compared to learning individual coefficients for each degree. Table 7 presents comprehensive performance comparison on the QM9 dataset for the two configurations. Variants denoted with a hat (e.g., Go, Got GotenNet,) )) utilize shared coefficients across all degrees, whereas the corresponding non-hat variants (e.g., GotenNets, GotenNetg. GotenNet,, learn separate coefficients for each spherical harmonic degree.
The results indicate that the models with individual coefficients consistently yield lower error metries across several tasks. For instance, the GotenNetg variant outperforms its shared-coefficient counterpart in terms of a, As, and molecular orbital energies, with similar trends observed for the base and large model configurations. This suggests that allowing independent coefficient learning provides greater expressiveness in capturing geometric nuances.
However, it is important to note that the increased flexibility of individual coefficients comes at the cost of higher number of parameters. In resource-constraing settings, the shared coefficient approach may offer favorable trade-off between model complexity and performance. Overall, these findings highlight the need to balance expressiveness and computational efficiency when selecting the appropriate configuration for a given application.
G EFFICIENCY EXPERIMENT DETAILS
The efficiency experiments are conducted by sampling graphs from the Molecule3D dataset. Using a real-world dataset provides realistic neighborhoods for the nodes computed with a radius graph, which is more beneficial than using synthetic dataset. The graph sampling process involves selecting
21

---

Published as a conference paper at ICLR 2025
Table 8: Computational complexity comparison of different methods. Training and inference latency are measured with batch size of 128 samples. Training time is reported in GPU days (min/avg/max//imit). Time per epoch in seconds. Latency measurements are in milliseconds. Best results are in bold.
Model
Batch Size
Time per Epoch (s)
Training Time (GPU days)
Training Latency
Inference Larency
Truinable Parameters
Std.
Log
min
max
limit
MAE
MAE
Equiformer
128
425
1.48
1.48
1.48
1.48
421
150
3.5M
0.70
-5.82
EquifomnerV2
64
821
2.85
2.85
2.85
2.65
918
341
11.2M
0.67
-5.87
EquifonnerV2
48
847
2,94
2,94
2,94
2.65
918
341
11.2M
0.67
-5.87
Geoformer
759
50.6M
-6.12
Go
32
117
0.41
0.75
1.34
1.35
80
6.1M
0.67
-6.21
GotenNet.
0.75
1.15
1.92
9.2M
GotenNet,
291
1.37
1.87
2.33
3.37
244
112
18.3M
0.56
-6.34
graphs with a predetermined node count. If no graph is available for given node count, we sample the larger graphs with the closest node count and subsample the nodes satisfy the node limitation. the number of graphs is less than desired, we oversample the graphs for experimentation.
After creating dataset for each node count, we conduct experiments by first warming up the models without timing for 5,000 steps. Then, we start timing the forward and backward passes for the subsequent 5,000 steps. The final values obtained by averaging the timings over the total number of steps. It is important to note that the timings are measured per batch. Each batch consists of a fixed number of graphs, and the reported timings represent the average time taken process single batch during the training process.
This experimental setup ensures fair comparison the scalability properties of GotenNet and the baseline models, providing insights into their efficiency in handling graphs of varying sizes. By measuring the training time per batch, we can accurately assess how the computational overhead scales with increasing graph sizes, independent of the number of epochs or the total dataset size.
H COMPUTATIONAL COMPLEXITY ANALYSIS
We present comprehensive analysis of computational efficiency across state-of-thh-at models, examining training requirements, inference speed, and model complexity. Here, we analyze the performance of GotenNet under shared-coefficiee paradigm for the spherical harmonic outputs fo /11 Lynox and (o 1=11 Lmax (denoted GotenNet,y in Section F), highlighting its computational efficiency and effectiveness. Table 8 compares these metrics across different architectures under standardized conditions.
Our analysis reveals several key insights about computational efficiency across models. First, training protocols show significant variation across architectures from Equiformer's relatively lightweight approach to Geoformer's more intensive requirements. Equiformer (Liao & Smidt, 2023) and EquiformerV2 (Liao et al., 2024) employ 300 epochs with batch sizes of 128 and 48/64 respectively, requiring 1.48 and 2.85/2.94 GPU days for completion. Geoformer (Wang et al., 2023a) utilizes batch size of 32 for up to 600 epochs with early stopping, theoretically requiring 5.05 GPU days for full training. This variation reflect different trade-offs between computational demands and model expressiveness.
GotenNet demonstrates superior efficiency across multiple metrics. The smallest variant, GotenNets: achieves competitive performance with just 1M parameters while requiring minimal computational resources (0.75 GPU days average training time). This efficiency extends to both training and inference latencies, with GotenNets achieving the lowest latencies in both categories (80ms and 37ms respectively).
Notably, even as model capacity increases, GotenNet maintains its efficiency advantages. GotenNet, (18.3M parameters) demonstrates remarkable scalability, requiring only 1.87 GPU days average training time while achieving 42% faster inference and 25% faster training compared to the closest competitor, Equiformer.
22

---

Published as a conference paper at ICLR 2025
These results demonstrate that GotenNet's architectural innovations particularly its efficient handling of geometric tensor representations - translate to practical advantages across all operational metrics. The consistent performance improvements across model scales suggest that GotenNet's approach to balancing expressiveness and efficiency is fundamentally sound, making it particularly suitable for real-world applications where computational resources constrained.
I COMPREHENSIVE PERFORMANCE COMPARISON ON QM9 DATASET
Table 9 provides comprehensive comparison of the performance of various baseline models against our proposed GotenNet on the QM9 dataset. This extended table includes full list of baseline methods, offering detailed assessment across all molecular property prediction tasks. The results showcase how GotenNet consistently surpasses existing models in both energy and force predictions, further highlighting its robustness and scalability. By including a wider range of baseline comparisons in this appendix, we aim to give clearer view of GotenNet's advantages in different metrics and provide a more exhaustive evaluation for the QM9 dataset.
Table Performance comparisons on QM9 dataset. i denotes using different data partitions.
Task
As meV
EhOMO meV
C2
G meV
II
U mcV
Uo mcV
ZPVE meV
std. %
log
Units
mag
meV
mD
maal
mcV
Invariant models
Cormorant
85
61
34
38
38
26
20
21
961
21
22
2.03
2.14
-4.75
NMP
92
69
43
38
30
40
19
17
180
20
20
1.50
1.78
-5.08
DimeNete+
44
32.6
24.6
19.5
29.7
23
7.56
6.53
331
6.28
6.32
1.21
0.98
-5.67
ComENer
45
32.4
23.1
19.8
24.5
22
7.98
6.86
259
6.82
6.69
1.20
0.93
-5.69
SphereNet!
46
31.1
22.8
18.9
24.5
22
7.78
6.33
268
6.36
6.26
1.12
0.91
-5.73
Scalarization-buseel models
ClofNet
40
27
9
1.37
-5.37
EGNN
48
29
29
31
12
12
106
12
11
1.55
1.23
-5.43
PaiNNt
45.7
27.6
20.4
12.0
24
7.35
5.98
66
5.83
5.85
1.28
1.01
-5855
LEFTNet
48
40
24
18
12
23
6
109
7
6
1.33
0.91
-5882
EQGAT
53
32
20
16
11
24
23
24
382
25
25
2.00
0.86
-5.28
ET
59
36.1
20.3
17.5
11
26
7.62
6.16
33
6.38
6.15
1.84
0.84
-5.90
Geoformer
40
33.8
18.4
15.4
10
22
6.13
4.39
28
4.41
4.43
1.28
0.75
-6.12
SaVeNet-Bi
39
24.8
18.4
16.3
9.3
23
6.64
5.43
58
5.48
5.43
1.18
0.69
-6.04
High-degree steerable models
SEGNN
24
23
16
1.08
-5.27
HDGNN
8.12
8.34
0.80
-5.64
Equiformer
15.4
14.7
12
7.63
6.63
251
6.74
6.59
1.26
0.70
-5.82
Equiformeryz
29.0
14.4
13.3
9.9
23
7.57
6.22
186
6.49
6.17
1.47
0.67
-5.87
Pre-trained models
SB(3)-DDM
46
40.2
23.5
19.5
15
24
7.65
7.09
122
6.99
6.92
1.31
0.93
-5.76
3D-EMGP
37.1
21.3
18.2
20
26
9.30
8.70
8.60
8.60
1.38
0.92
-5.68
Transformer-M
27.4
17.5
16.2
37
22
9.63
9.39
9.41
9.37
1.18
0.86
-5.74
Coord
31.8
17.7
14.3
20
6.45
6.11
6.57
1.71
0.76
-5.75
Frad(VRN)
27.7
17.9
13.8
6.01
354
5.35
5.41
1.63
0.71
-5.85
Frad(RN)
27.8
15.3
13.7
20
5.55
5.62
5.33
1.42
0.66
-5.91
DenoiseVAE
26.0
7.9
15
4.19
4.31
-6.18
SliDe
26.2
12.3
8.7
19
5.37
4.26
4.29
4.28
-6.02
Spherriallsecalarizttiooo models
GotenNets
33
21.2
16.9
13.9
7.5
20
5.50
3,70
29
3,67
3.71
1.09
0.60
-6.29
GotenNet
32
20.5
15.2
13.0
7.2
19
5.19
3.44
27
3.49
3.43
1.09
0.56
-6.35
GotenNet,
28
19.8
13.4
12.2
6.7
19
4.98
3.30
24
3.41
3.37
1.08
0.52
-6.41
J MD22 VISUALIZATIONS
Figure visualizes the mean absolute errors (MAE) for energy and forces across molecules in the MDD2 dataset. The æ-axis represents the energy (kcal/mol), and the y-axis denotes the force
23

---

Published as a conference paper at ICLR 2025
enror (kcal//mol/A). Each point corresponds model, with performance improving the point approaches the origin (0,0), where lower values indicate better performance for both metrics. Our proposed GotenNet consistently outperforms the baseline models, achieving the best performance across all molecules, as evidenced by its closer proximity to the origin compared to competing methods.
0.10
Energy Error Iecalmmoll
(kcakmooll
0.15
(kcalmmol)
0.225
0.09
0.200
0.14
0.08
0.12
0.07
Energy
0.10
Energy
0.125
0.06
0.08
0.100
0.05
0.08
3.06
0.07
3.08
0.09
0.10
0.045
8.055
0.060 6.5 0.070
0.04
0.06
0.07
Force Emoo (kcallmal(A)
Force Error (kcallmoljjA)
Force Error (kcal/mol/A)
(a) Tetrapeptide
(b) DHA
(c) Stachyose
2.2
0.16
0.58
2.0
(kcalmmlt
Energy (kcalmal)
Eneroy (icalmol))
0.14
0.50
1.8
0.12
0.45
1.6
1.4
Energy
0.10
0.40
1.2
0.08
0.35
0.08
0.30
0.07
Force Enror (kcallmol(A)
Force Error (kcalmollAA)
Force Error (Jeca (mca//oook))
(d) AT-AT
(e) Buckyball catcher
(f) Double- nanotube
Allegro
Equiformer ViSNet
QUINNet
VisNet-LSRM
MACE
Equiformer-LSRM
GotenNet
Figure 4: Mean absolute error of the molecules for energy and forces.
K RMD17 VISUALIZATIONS
Figure 5 presents the MAE for energy and force predictions across nine molecules in the rMD17 dataset, including Aspirin, Azobenzene, Benzene, Ethanol, Malonaldehyde, Naphthalene, Paraceta- mol, Uracil, and Toluene. The r-axis represents energy error (mkcal/mol), and the y-axis denotes force error (mkcal/mol/A)) Each point corresponds model's performance specific molecule, where better performance is indicated by proximity to the origin (0,0) reflecting lower errors in both metrics. Our proposed GotenNet demonstrates consistent superiority over baseline models, achieving the lowest errors across all nine molecules, as evidenced by its closer alignment with the origin compared other methods.
L FUTURE WORK AND EXTENSIONS
While GotenNet has demonstrated strong performance molecular property prediction, several promising directions exist for future research and extensions. The architecture naturally extend to other spatial data where geometric relationships significantly influence node interactions, such as point cloud processing, protein structure analysis, and dynamic molecular simulations. These applications
24

---

Published as a conference paper at ICLR 2025
NequiP
UNITE
Allegro
BOTNet
MACE
TensorNet
GotenNet
55.0
Energy (mkcaalmon)
52.5
26
Energy Error (miccalmal)
50.0
24
47.5
22
45.9
20
42.5
Energy
18
40.0
2.
37.5
14
12
35.0
140
200
60
12
14
16
Force Emor (mkcal/mmt(A)
Force Eror (miccal/mal(A)
Force Emor (miccal/ma(AA)
(a) Aspirin
(b) Azobenzene
(c) Benzene
14
20.0
Energy Error (mkcatmmo)
23
Energy Error (micalmmll)
Energy (mkcalmmol)
15.0
51
12.5
30
18-
1.6
7.5
Force Error (mkcal/moliA)
Force Error (miccal/motik)
Force Error (mkcalmmlll))
(d) Ethanol
(e) Malonaldehyde
(f) Naphthalene
Energy Exror (miccalmoll
Energy Enror (mkcalmmll)
Energy (micaalmol)
10-
12-
11
10
30
100
14c (mkcalimol/A)
160
40
45
5.
50
60
(mkccalmmo(AA
Erer (miccal/moliA)
(g) Paracetamol
(h) Toluene
(i) Uracil
Figure 5: Mean absolute error of the molecules rMD17 dataset for energy and forces.
share the fundamental requirement of processing geometric relationships while preserving symmetries, making them natural candidates for framework. From architectural perspective, the model could be enhanced through the incorporation of scale equivariance, exploration of higher-order features beyond second degree, and development of sparse implementations for larger systems. Memory efficiency improvements could also enable applications to even larger-scale systems. On the theoretical front, future work could focus on developing formal analyses of the expressiveness- efficiency trade-off, understanding generalization properties of geometric tensor representations. These potential extensions maintain GotenNet's core principle of balancing expressiveness and efficiency while broadening its applicability across different domains of geometric deep learning.
25


Reviews:
[{"review_id": "x9E0AMYlRn", "reviewer": "Reviewer_4mUv", "summary": "The paper introduces GotenNet, a Geometric Tensor Network, that advances molecular representation learning by addressing computational challenges in 3D graph neural networks. GotenNet's novel tensor embedding strategy eliminates the need for complex irreducible representations and Clebsch-Gordan transformations, enhancing computational efficiency while preserving model expressiveness. Its Geometry-Aware Tensor Attention mechanism enables refined edge representations, capturing complex geometric relationships for better molecular property predictions. Additionally, the Hierarchical Tensor Refinement approach allows the model to adapt across scales, accommodating both broad patterns and detailed molecular features. Evaluated on benchmark datasets such as QM9 and Molecule3D, GotenNet consistently outperforms existing methods, demonstrating robustness and scalability.", "questions": "Please see the weakness part.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a novel framework, GotenNet, which innovatively combines geometric tensor representations with advanced attention mechanisms. This approach addresses the expressiveness-efficiency trade-off in 3D graph modeling, a challenge that has been inadequately tackled in prior works.\n2. The paper is well-structured and clearly articulates the problem being addressed, the proposed solutions, and the significance of the findings.", "weaknesses": "1. **Comparison of computational complexity with previous methods**: Could you provide details on the model size, as well as training and inference times, in comparison with existing methods? This information would help highlight GotenNet’s computational efficiency relative to other models in the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XhQCy8XJcy", "reviewer": "Reviewer_bNe3", "summary": "This paper proposed a new network archtiecture that is SE(3) equivariant and builds on Transformers and previous works on equivariant Transformers. The results on various datasets show good performance and speedup.", "questions": "> Writing\n\n1. I think the title is too general. We can always rethink something and make them more efficient. Be very specific to your proposed method.\n2. Line 16 -- 18: Mention what is the difference from previous works clearly.\n3. Line 22 -- 23: Mention the datasets you tested so that readers can know the scale of experiments in this work.\n4. Figure 1: I think the x and y axes are similar. Also mention which direction is better.\n5. Line 62: What is \"this\" in \"Some recent works have sought to address this by...\"? Make sure the sentence is clear.\n6. Line 113: Equivariant networks model translational \"invariance\".\n7. Line 159: \"Effective Representations\" -> too general. Be specific to what makes it effective.\n8. Figure 2: Please double checkt the errors in the caption.\n9. Line 195 -- 215: Better to link to existing literature since they are similar to (or basically the same as) vector spaces of irreps.\n10. Line 220 -- 221: The notation of h, r, t has no meaning and should be replaced by other variables that directly reflect what they are presenting.\n11. Line 263: $z$ denotes the \"one-hot encoding\" of the atomic number.\n12. Section 3.3: This is unclear. It would be better to first mention the high-level concept and then go to the details. Also state the differences from previous works.\n13. Line 323 or Equation 8: This is essentially an SO(3) linear layer. It would be great to link to previous works.\n14. Line 373: Geoformer Wang et al. -> make sure the format of citation is correct.\n15. Figure 3: Mention the dataset you tested.\n16. Line 423: \"The\" proposed method\n17. Line 448: I don't get the definition of Scalability in the paper. Should be better to use \"Evaluation on Efficiency\"?\n\n> Question \n\n1. Line 66 -- 67: How do you define scalability? I think that means you invest more compute, you always get better results. Efficiency should be the correct term to use if you are saying some models are slow or take much memory.\n2. Table 1: Somewhat hestitated to trust the results here. From the method section, it is unclear to me what are the differences that can improve the results. Besides, it would be great to discuss some potential oversmoothing, which prevents deeper networks from performing better.\n3. It would be good to report training time for completeness.\n4. Have you conducted any experiments on materials/catalyst datasets? I think OC20 IS2RE dataset would be good to test given the compute requirement is not that high and the train/val/test splits are well-defined.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The experimental results are great and intensive.", "weaknesses": "1. The writing of the paper should be greatly improved. Please see \"Questions\" below.\n2. The paper does not discuss why the proposed architecture differs from other works and why this is better in terms of accuracy and efficiency even though the results are seemingly good. Also a reference implementation would be great to make sure the better results are reproducible.\n3. The proposed architecture should be simplified (partially because of the way it is presented).\n4. The comparison is somewhat unfair. For example, on QM9, this work trains the proposed model for 1,000 epochs while some previous works only train for 300 epochs. Besides, the batch size is 32 on QM9 (some previous works are 128), and smaller batch sizes on QM9 can sometimes lead to better results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MQiHTC9LRM", "reviewer": "Reviewer_BR7w", "summary": "The importance of equivariance in neural networks has become ubiquitous\nin machine learning research. Neural networks operating on input features\nwith certain symmetries, such as rotations, translation and permutation\npredict with higher certainty and better quality if the architecture respect\nthese symmetries. In Graph Neural Networks equivariance with respect to\npermutation of the nodes has been the foundation upon which a plethora\nof ML approaches have been developed. The authors present a deepening\nof this paradigm, by combining the success of transformers, permutation\ninvariance and E(3) equivariance and distilling these three components into\nan architecture for molecular learning. The main contribution of the paper\nis the development of Geometry Aware Tensor Attention layer that defines\nan E(3) equivariant transformer layer for both node and edge embeddings. A\nstrong highlighted advantage is computational speed, compared to architectures\nusing Chlebsch-Gordan decompositions or architectures based on irreducible\nrepresentation.\n\nThrough a thorough set of experiments, the authors show that their\narchitecture, GotenNet, is able to outperform a wide variety of benchmarks on\nmolecule property predictions. Their overall performance and scaling in terms\nof computations is also shown.", "questions": "Some questions: \n\n-  How  does  the  current architecture  generalize  to  different tasks? As\npresented, it  would work only on  molecule type graphs, is it also possible to\ngeneralize the architecture to for instance social  networks or  other types\nof graphs? In  other words,  how general  is the  method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a rather strong experimental section showing that the\nauthors have taken the time to set up a good set of experiments. In particular\nthe fact that all comparison partners have been retrained and compared to\nthree versions of the model shows dedication to the and has been a pleasure\nto read. The experiments addressing the inference times is an important\ncomplementary addition, since often there is a rather steep tradeoff between\npracticality and usability, where better performance on metrics is achieved\nthrough exponential increases in parameter count or long inference times. To\nsee both good results and good inference time / scalability is reassuring to\nthe reviewer. The introduction of geometric tensors as a replacement of CG\ncoefficient and the computation irreducible representation is a novel and\noriginal viewpoint which combined with the favorable tradeoff between good\nperformance and accuracy is an impactful contribution. Writing is clear,\nconcise and easy to follow.", "weaknesses": "Since the reviewer is not too familiar with the literature, the current\nwriting makes it somewhat hard to compare to the current available literature.\nFor instance, what would be the comparable difference between GotenNet and an\nequivariant transformer (such as equiformer) or the Graph Attention Networks.\nIs the current architecture a mix of the two? In that sense some more context\nmight be good in the introduction or overview of related work.\n\nThe paper as written in its current form is of great quality, and could be\npotentially be further strengthened through the incorporation of the points\nabove and by addressing the questions below in a section discussing the current\nlimitations of the method and potential future avenues of research. For\ninstance it seems that the current method is _specifically_ tailored to the\nprediction of molecule properties and how would that generalize? Another\npoint to potentially discuss is how this method would extend beyond $E(3)$\nand $SE(3)$ equivariance (maybe scaling for instance) and/or if the method\nwould generalize to point clouds (where this line of work is also highly\nrelevant). Since the paper is already of great quality addressing these\ntopics in the rebuttal would be sufficient, however addressing them in the\nrevision would be beneficial for the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AVHFMNNtHo", "reviewer": "Reviewer_hFv3", "summary": "This paper propose a novel Geometric Tensor Network (GotenNet) to address  trade-offs between expressiveness and computational efficiency. The core of this model is to use inner-product in Eq. (9) to avoid  tensor products or CG coefficients.\n***\n\nIn order to better explain my comments and facilitate the understanding of the author, other reviewers, and AC, I will sort out the timetable of related models that are not mentioned in the article, mainly involving the following references:\n1. SE(3)-Transformer [a] (in NeurIPS 2020): In Eq. (11) of [a], the calculation of attention $\\alpha$used $\\bf{q}^\\top\\bf{k}$, which sum all inner-products of high-degree steerable features together.\n2. SE(3)-Transformer implemented by e3nn [b]: See code below\n```python\ndot = o3.FullyConnectedTensorProduct(irreps_query, irreps_key, \"0e\")\n...\nexp = edge_weight_cutoff[:, None] * dot(q[edge_dst], k).exp()\n```\nAnd the main difference is the module $\\texttt{o3.FullyConnectedTensorProduct}$, which introduces different learnable weights to each inner-product. However, now the model still uses tensor products to generate the key and value pair and is limited by efficiency.\n\n3. ViSNet [c] (in Nature Communications: 05 January 2024, preprint at arXiv:2210.16518): In Eq. (6-7) of [c], authors introduced the connection between inner-products and Legendre polynomials (though a mathematical common sense). But in methodology, they only use the 1st-degree steerable features (Cartesian vectors) to build their models.\n\n4. SO3KRATES [d] (in Nature Communications: 06 August 2024, preprint at arXiv:2309.15126): The SO3KRATES technique in Eq. (14), CG coefficient used in such form, but is equivalent to the inner-products of high-degree steerable features. This paper proposed an Euclidean transformer and achieved nice performance in property prediction tasks (e.g. rMD17 and MD22).\n\n5. HEGNN [e] (in NeurIPS 2024, preprint at arXiv:2410.11443): During studying symmetric geometric graphs, the authors proposed a equivariant GNN model that uses the scalarization trick (i.e., inner product) to introduce high-degree steerable features in Eq. (6) of [e], and pointed out the connection between this approach and SO3KRATES. They used a method similar to Deepsets and Legendre polynomials to prove that this method can recover the information of all angles between each pair of edges, achieved good performance in dynamics tasks (e.g. N-body and MD17), and demonstrated that the introduction of high-degree steerable features can make the model more robust.\n\n[a] SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks\n\n[b] https://docs.e3nn.org/en/stable/guide/transformer.html \n\n[c] Enhancing geometric representations for molecules with equivariant vector-scalar interactive message passing\n\n[d] A Euclidean transformer for fast and stable machine learned force fields\n\n[e] Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks?\n\nIn subsequent comments, I will use the names of the above models without citation.\n\n---\n> **[First Comment, Rating 3, Confidence 5, with 4/1/4 of Soundness/Presentation/Contribution]**\n\nThe model and experiment in this paper are excellent and deserve a clear acceptance or even an award (8-10), but the terrible presentation of this version is totally unworthy of such an achievement. If the author can improve the presentation, I will be very happy to increase the score in the rebuttal. I am very confident that this is a paper that will have a profound impact on the entire field, so I hope it will be accepted in a very perfect manner.\n\n>**[Second Comment, Rating 10, Confidence 5, with 4/2/4 of Soundness/Presentation/Contribution]**\n\nThe authors have provided a sincere and thoughtful response, addressing most of my concerns. The remaining questions, primarily related to experimental hyperparameters (e.g. higher-degree steerable features), would likely require significant time to resolve and may not be feasible during the rebuttal period, which I fully understand. Regarding the shortcomings in the presentation originally raised, the revised version has been significantly improved, and the motivation and logic are now clearer and more coherent. Although the presentation still needs further improvement, it is no longer worth over-focusing on during the discussion. Overall, these minor details do not detract from the paper’s value. I therefore give a **strong accept** recommendation.", "questions": "Before anything else, is the author willing to provide an anonymous link to the project as soon as possible (this will not affect my subsequent rating increase) so that the results can be reproduced? \n\n> **Q1. Why use the same weight for steerable features of different degrees (in Eqs. (7) and (11))? Can they be different?**\n\nNote that there are several formula items: $o\\_{ij}^d\\circ r\\_{ij}^{[1,L\\_{max}]}$, $o\\_{ij}^t\\circ h\\_{j}^{l,[1,L\\_{max}]}$, $m\\_2\\circ h^{l,[1,\\hat{L}\\_{max}]}\\bf{W}\\_{vu}$. They seem to be multiplied by the same coefficient. Steerable features of different degrees represent different orthogonal bases, so their weights should not be the same (as HEGNN does). Can the authors change it to have different weights for each degree and test the performance of the corresponding model?\n\n> **Q2. Why does Eq. (9) use a permutation-invariant operator?**\n\nThe results of different degrees should be ordered, so there seems no need for permutation invariance. What are the benefits of using permutation-invariant operations?\n\n> **Q3. Is it possible to explore the expressive power of GotenNet using high-degree steerable features?**\n\nOne of the great benefits of this model is that the complexity grows slowly with the feature degree. Can authors show the experimental results of using higher-degree features (e.g. $L\\_{\\max}\\in\\\\{6,8\\\\}$)? In fact, similar HEGNN and EquiformerV2 have demonstrated the benefits of using such higher-degree features.\n\n> **Q4. Is it possible to explore the expressive power of GotenNet on large-scale geometric graphs and dynamics tasks?**\n\nIs it possible to add speed attributes like EGNN so that GotenNet can be applied to dynamics tasks? In addition, it is easy to associate the fast multipole method with the possibility that high-degree representations may have good effects on large-scale geometric graphs. Can the authors try the effect of GotenNet on large-scale geometric graphs (e.g. 100-body in HEGNN)?\n\n> **Q5. Is it possible to test the effect of $L\\_{\\max}=1$?**\n\nAlthough the benefits of high-degree steerable features are obvious, in many cases a low-degree model is good enough. Can authors give a case where only Cartesian vectors ($L\\_{\\max}=1$) are used?\n\n> **Q6. Is it possible to provide an e3nn version of GotenNet implementation later?**\n\nFrom the full text, I guess the author may not know the e3nn library, which can easily implement the interaction of multi-channel high-degree steerable features. I believe it will be easier for everyone to follow the work of this article. However, this project may be very large, and I hope the author can provide this version after the article is accepted.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "For the increasingly important discipline of AI for Science, dealing with equivariance is undoubtedly very important. According to the classification of [f], there are two mainstream models: scalarization-based models and high-degree steerable models. The timeline I introduced in the summary (and this article) is actually doing something very valuable, that is, bridge the gap between these two types of models.\n\nIn fact, the only papers that truly achieve this are SO3KRATES, HEGNN, and GotenNet (this paper). Although the core formulas of these three works are equivalent, their publication dates are close together—especially since HEGNN's preprint appeared after the ICLR submission deadline. I believe this is merely a case of normal concurrent discovery, and therefore I recognize the originality of this paper. Moreover, equivariance is a very strong constraint, and I would even guess that this form is the only feasible form, which makes the contribution seem small in terms of formula form, but in fact it is a very important contribution.\n\nMoreover, these three works exhibit significant differences. Both SO3KRATES and GotenNet employ equivariant attention, but the inner product formulation used in GotenNet is more concise and yields far better results than SO3KRATES. While both HEGNN and GotenNet point out the inner product formulation, HEGNN’s contributions lie more in theoretical research and model framework, with experiments focusing on dynamical problems, creating a complementary relationship with GotenNet. From my perspective, although GotenNet is on par with these two other works and shares an equivalent formulation, its superior performance makes it an excellent contribution.\n\n[f] A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applications", "weaknesses": "I think the whole article is amazingly good except for the presentation, and there is no major weakness. Here I mainly give the shortcomings of the presentation that I think need to be improved. For other parts (such as models and experiments), there are only some suggestions for the author to choose whether to adopt (see questions).\n\n> **W1. Lack of a good review of relevant literature.**\n\nThis article should introduce several works in the timeline I sorted out in the summary section, especially SO3KRATES and HEGNN, which should be introduced in detail. The results of SO3KRATES should be added to the experimental results for easy comparison (although GotenNet outperform SO3KRATES).\n\nFor models such as eSCN and EquiformerV2 that use spherical activation functions, a simple comparison should be made. For example, the use of Fibonacci grid requires a large number of sampling points, which brings a large constant in complexity, resulting in a theoretical complexity of $\\mathcal{O}(L^3)$, which is actually quite time-consuming (and this can only achieve qusi-equivariance).\n\n> **W2. Weird math definitions and confusing math symbols.**\n\nI don't know what the meaning of Def. 1, Theo. 1-2 in the main text is. I think it is just to make it look like there is some theory. Theo. 3 and Prop. 1, which have some meaning, are too trivial. They can be explained in one sentence: \"The product of an invariant scalar and other steerable features does not change its equivariant form.\" However, the key initialization by spherical harmonics is not only placed in the appendix, but also a clear formula is not given (while Eq. (11) in SO3KRATES and Eq. (5) in HEGNN are clearly written). I suggest replacing this with a discussion of the model architecture. In fact, it is a very valuable point to explain why GotenNet performs so much better than SO3KRATES with a similar architecture. Theoretical analysis may be very difficult, so it is not required, but it would be great if the authors could come up with some engineering tricks.\n\nMoreover, this article will surely have many followers in the future, and a good symbolic system will help future generations understand. The number of superscripts is sometimes one, sometimes two (though it has been explained), and the letters that appear are similar (the $l$ for the number of layers and the $L$ for high-degree steerable features are used at the same time), which is confusing. In addition, the current symbolic representation cannot well show which are invariant scalars and which are high-degree steerable features. It is recommended that the author adopt the HEGNN symbol system (e.g. using $\\Delta\\boldsymbol{h}$ to represent inter-layer updates, and using fonts such as $\\tilde{\\boldsymbol{v}}^{(l)}$ to represent $l$-degree steerable features).\n\n> **W3. Others.**\n\n- It is recommended to unify the terminology of the article, such as high-degree steerable features, and not to mix order and degree, otherwise it always gives readers the feeling that the authors are studying multi-body interactions. \n\n- There are too many crossing lines in Fig. 2 (b-d), which makes it very difficult to understand. Can it be simplified?\n\n- The rotation matrix $R$ in Line 879 (in fact, it should be emphasized that it is an orthogonal matrix) should be the Wigner-D matrix $\\boldsymbol{D}^{(l)}(r)$. In addition, the inversion factor should be multiplied according to the parity of the vector itself. For details, refer to [g].\n\n[g] https://docs.e3nn.org/en/stable/api/o3/o3_irreps.html", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)