SE3Set: Harnessing Equivariant Hypergraph Neural Networks for Molecular Representation Learning
Authors: Hongfei Wu
Lijun Wu
Guoqing Liu
Zhirong Liu
Bin Shao
Zun Wang
Venue: Accepted by TMLR ()

Abstract: In this paper, we develop SE3Set, an SE(3) equivariant hypergraph neural network architecture tailored for advanced molecular representation learning. Hypergraphs are not merely an extension of traditional graphs; they are pivotal for modeling high-order relationships, a capability that conventional equivariant graph-based methods lack due to their inherent limitations in representing intricate many-body interactions. To achieve this, we first construct hypergraphs by proposing a new fragmentation method that considers both chemical and three-dimensional spatial information of the molecular system. We then design SE3Set, which incorporates equivariance into the hypergraph neural network. This ensures that the learned molecular representations are invariant to spatial transformations, thereby providing robustness essential for the accurate prediction of molecular properties. SE3Set has shown performance on par with state-of-the-art (SOTA) models for small molecule datasets like QM9 and MD17. It demonstrates outstanding performance on the MD22 dataset, achieving a remarkable ~20\% improvement in accuracy across all molecules. Furthermore, on the OE62 dataset, SE3Set outperforms all short-range models. We also conducted a detailed analysis of OE62, highlighting the prevalence of complex many-body interactions in large molecules. This exceptional performance of SE3Set across diverse molecular structures underscores its transformative potential in computational chemistry, offering a route to more accurate and physically nuanced modeling. The code of this work is available at https://github.com/Navantock/SE3Set.

Published in Transactions on Machine Learning Research (04/2025)
SE3Set: Harnessing Equivariant Hypergraph Neural Networks for Molecular Representation Learning
Hongfei Wu
College of Chemistry and Molecular Engineering, Peking University, Beijing, 100871, China
Lijun Wu
Microsoft Research A14Science, Beijing, 100084, China
Guoqing Liu Microsoft Research AI4Science, Beijing, 100084, China
Zhirong Liu
LiuZhhiHoon@@puu.cee edw.cckcu.durr.r
College of Chemistry and Molecular Engineering, Peking University, Beijing, 100871, China
Bin Shao
binshaao@miicrroofl.ccmm
Microsoft Research A14Science, Beijing, 100084, China
Zun Wang
zww.ang@miccroooft..cmm
Microsoft Research A14Science, Beijing, 100084, China
Reviewed on OpenReview: https: // openreview. net/ forum? id= muwEt1
Abstract
In this paper, we develop SE3Set, an SE(3) equivariant hypergraph neural network architec- ture tailored for advanced molecular representation learning. Hypergraphs are not merely an extension of traditional graphs; they are pivotal for modeling high-order relationships, capability that conventional cquivariant graph-based methods lack due to their inherent limitations in representing intricale many-body interactions. To achieve this, we first con- struct hypergraphs by proposing a new fragmentation method that considers both chemical and three-dimensionall spatial information of the molecular system. We then design SE3Seet which incorporates equivariance into the hypergraph neural network. This ensures that the learned molecular representations are invariant to spatial transformations, thereby providing robustness essential for the accurate prediction of molecular properties. SE3Set has shown performance par with state-offthe-arl (SOTA) models for small molecule datasets like QM9 and MD17. It demonstrates outstanding performance on the MD22 dataset, achieving a remarkable 20% improvement in accuracy across all molecules. Furthermore, on the OE62 dataset, SE3Set outperforms short-range models. We also conducted a detailed
*These authors did this work during internship Microsoft Research AT4Science.

---

Published in Transactions on Machine Learning Research (04/2025)
analysis of OE62, highlighting the prevalence of complex many-body interactions in large molecules. This exceptional performance of SF3Set. across diverse molecular structures underscores its transformative potential in computational chemistry, offering route to more accurate and physically muanced modeling. The code this work is available at http:://ggiatbb.co//laaattttttt
1 Introduction
Molecular representation (Mathews & Challee 2012 David et al. 2020; Wigh et al. 2022 is pivotal for cheminformatics (Fourches et al. 2010), impacting the prediction of molecular properties in drug discovery and material science. Traditional descriptors like fingerprints capture basic struetural and energetic aspects of molecules by considering mainly one- and two-body interactions. However, they often miss complex electronic correlations and collective behaviors important for understanding phenomena such as chemical reactivity and protein folding. To address this, advanced methods that include many-body interactions are crucial for a more comprehensive molecular characterization. These methods enhance the predictive capabilities of computational models by more accurately reflecting the intricate dynamies and properties of molecules, which are essential for a deeper understanding of their functionality and reactivity in cheminformaties.
Graph neural networks (GNNs) (Zhou el al. 2020 Wu et al. 2020) are a foundational tool for representing structured data in molecular sciences with atoms as nodes and chemical bonds as edges, respectively. GNN
models excel in tasks ranging from property prediction to reaction simulation (Do ct al. 2019 Xiong
et al. 2021 Reiser et al. 2022)- GNNs can capture higher-order molecular interactions through message
passing Gilmer et al.) 2017) but face overfitting and inefficiency challenges (Godwin et al. 2021 Rusch et al.
2023 Architectural improvements in GNNs facilitate the modeling of complex interactions, overcoming
some limitations of deep networks (Gasteiger et al. 2019 Schütt et al. 2021 Batzner et al. 2022).
Advances demonstrate the potential of architectural enhancements in GNNs to represent complex interactions Gaasteiger
ct al. 2020 2021 Thölke & De Fabritiis 2021 Batatia et al. 2022: Musaelian et al., 2023 Wang et al. 2024).
but efficiently integrating many-body interactions into these networks an ongoing challenge (Wang et al,
2023
An effective approach to describe many-body interactions in chemical molecules involves leveraging predefined substructures, such as functional groups in organic chemistry. This strategy has demonstrated significant success in various chemical molecule-related tasks, including pre- gre-training (Li & Jiang 2021 Li et al. 2023
and representation learning (Hajiabolhassan et al. 2023 Wu et al., 2023 Furthermore, these predefined
chemical substructures can be naturally modeled as hyperedges within a hypergraph structure. Hypergraphs offer a compelling alternative to complex GNN architectures. Hypergraphs, with hyperedges connecting multiple vertices, can naturally represent many-body phenomena like electronic delocalization and collective vibrations. This allows for a more accurate modeling of molccular intricacics beyond the limitations of traditional graphs. Integrating hypergraphs with machine learning, particularly through Hypergraph Neural Networks (HGNNs), is an emerging research area. HGNNs manage the flow of information across hyperedges, capturing complex multi-atom interactions and enriching molecular representations. This technique promises to balance model expressiveness with computational efficiency. By innately encoding many-body interactions, HGNNs stand to significantly advance cheminformatics, offering a new approach to molecular property prediction and simulation that resonates with the actual behavior of chemical systems.
By leveraging the relationship between hypergraphs and many-body interactions, aim to incorporate the advantages of equivariant graph neural networks into hypergraph structure learning, thereby developing a more comprehensive molccular representation learning model grounded in domain knowledge. In this work, we introduce SE3Set, an innovative approach that enhances traditional GNNs by exploiting hypergraphs for modeling many-body interactions, while ensuring SE(3) equivariant representations that remain consistent regardless of molccular orientation. Our key contributions arc:
new fragmentation method for hypergraph construction that seamlessly integrates 2D chemical and 3D spatial information, enriching the molecular structure representation.
2

---

Published in Transactions on Machine Learning Research (04/2025)
The deployment of hypergraph neural networks to capture many-body interactions, providing deeper insight into molecular behavior that surpasses conventional pairwise modeling.
The incorporation of SE(3) equivariance within our hypergraph framework, guaranteeing orientation- independent molccular representations.
SE3Set underwent comprehensive benchmarking process, exhibiting comparable outcomes to state-oftthe-art (SOTA) models on small molecule datasets QM9 and MD17. Tt demonstrated exceptional performance on the larger molecule dataset MD22, where higher-order interactions are more evident, surpassing SOTA models with a significant reduction in mean absolute errors (MAEs) by an average of roughly 20%. This confirms SE3Set's efficacy in capturing the complexity of molecular representations.
These advances establish SE3Set as formidable tool for molecular representation learning, with implications for computational chemistry and beyond.
2
Related works
2.1 Graph neural networks
Mcssage passing neural networks (MPNNs)) a class of graph neural networks, are essential for learning node features by transmitting information along graph edges, a process crucial for interpreting structured data like molecules (Gilmer al.
2017). Equivariant GNNs are especially important for molecular modeling.
They adopt either group representation methods, aligning architectures to symmetry groups for improved
interaction modeling (Thomas et al. 2018 Anderson et. al. 2019 I Fuchs et al. 2020 Batzner el al. 2022 Liao
& Smidf, 2022 Liao et al. 2023
Musaelian et al. 2023 : or pirection-based methods that incorporate spatial
information for accurate molecular representations (Schütt et al. 2017
Kindermans & Müller 2018 Coors
al.
2018 Gasteiger
2019
2020
Schütt al. 2021 Thölke & De Fabritiis 2021 Gasteiger al.
2021
Wang et al.] 2022 Du al.
2024
Aykent & Xia 2024
Wang al. 2024 h and have been engineered to
handle intricate up to five-body interactions (Wang et al.
2023
2.2
Hypergraph neural networks
Hypergraph Neural Networks (HGNNs) enhance GNNs by incorporating multi-node hyperedges to better capture complexity in data from various domains. They advance GNNs' implicit many-body interactions with methods like clique expansion for compatibility with existing algorithms (Agarwal al.! 2005 Zhou al., 2006) and employ tensor techniques for improved hypergraph-base
feature learning (Li et al. 2013 Pearson & Zhang, 2014 Benson et. al. 2017 Chien et al. 2021a Tudisco et al. 2021 While equivariant HGNNs adeptly handle node permutations, preserving data symmetries (Kim et al. 2021 2022a)). they often miss 3D spatial transformations, crucial for physical system modeling. In computational chemistry, hypergrapl algorithms simulate complex behaviors and optimize molecules through hypergraph grammar (Cui et al. 2023 Tavakoli et al. 2022 Kajino 2019). providing multidimensional insights into molecular structures (Nachmani & Wolf 2020 Chen et al. 2021 Chen & Schwaller 2023). Despite their promise, these methods still face
hurdles integrating spatial information effectively.
2.3 Fragmentation methods
Fragmentation methods break down complex molecules for simpler ab inétio QM computations of properties,
later combining these for a holistic view (Gordon et al. 2012 Collins & Bettens 2015). Leveraging the localized nature of chemical reactions, these techniques aim for scalable algorithms suitable for large molecule analysis. While instrumental in computational pretraining (Du et al. 2021 Kim et al, 2022b Luong & Singh 2023), they typically neglect the fusion of 2D structures and 3D spatial data. Hence, we advocate for refined fragmentation approach that merges chemical properties with spatial context, potentially advancing hypergraph-based chemical modeling.
3

---

Published in Transactions on Machine Learning Research (04/2025)
3 Preliminaries
3.1
Equivariance
Consider a function L that maps imputs from space X' to outputs in space V C is called G-equivariant if it. preserves the symmetry a group G across mappings, meaning for each 9 E G, we have:
Lo (g)
(1)
where Dt represents the group G's action on X. This ensures that the function L reflects changes made to inputs by G in its outputs.
3.2 Hypergraph
Hypergraphs elegantly capture the essence of higher-order interactions among multiple entities, making them an invaluable tool for representing complex relational data. Let G = (V, E) be hypergraph with N vertices and M hyperedges, where V represents a set of nodes and E is set of hyperedges. Distinguishing itself from a traditional graph, hyperedge can encompass multiple nodes, not limited to two, i.e. each hyperedge e E E is a non-empty subset of V.
3.3 AllSet
The AllSet framework (Chicn et al. 2021b). an advancod HGNN model, addresses houristic propagation rule limitations in HGNNs by integrating Deep Sets (Zaheer et al. 2017 and Set. Transformers (Lee el al.) 2019 principles. It uses task-optimized dual multiset functions that maintain permutation invariance, crucial for hypergraph learning. The update rules in AllSet are:
(((1),. fV-E "00"", (x!"),
(2)
x(++1) = 8,
x!!).
(3)
Here, fv-4 and fe-v are the key multiset functions mapping node and hyperedge features. For example, fV-+E(S) MLP (TP EEe MLP((s) is used in AllDeepSets. The notation Ve,x and E,,z represent multisets of node and hyperedge features, respectively. The AllSet approach updates nodes and hyperedges in the hypergraph by leveraging their features in conjunction with those of adjacent hyperedges or nodes, enabling rich representation of the hypergraph structure. The method could differentiate node v from its multiset, allowing for sophisticated fcature aggregation.
4 Methods
We introduce the SE3Set model to leverage hypergraph neural networks for capturing complex molecular interactions, integrating both 2D chemical and 3D spatial structures (Sec. 4.1p. J. It builds upon the AllSet framework (Chien et al. (2221b) and the Equiformer (Liao Smidt (022).. Upcoming sections will delve into the specifics of molecular fragmentation and the SE3Sct architecture.
4.1 Fragmentation algorithm
To harness the power of hypergraph neural networks for molecular representations, we need map molecules onto hypergraph structures through refined fragmentation algorithm. Our strategy intertwines molecular topology and spatial geometry to create hyperedges that capture groups of atoms, reflecting their functional and spatial characteristics. More specifically, the topology incorporates the chemical definitions of covalent bonds, functional groups, elc., allowing us to initially delineate fragments by chemical prior knowledge of chemistry to identify the important substructures. And the 3D spatial structure aids in modifying the fragments to capture the interactions short range which are not specified by covalent bonds in the conformation. Thus, in crafting this fragmentation approach for hypergraph-bar based molecular representation, propose set of principles for fragmentation approach:
4

---

Published in Transactions on Machine Learning Research (04/2025)
(a)
(b)
HzN
H2N
N
N
HN
HN
OH
OH
OH
OH
(c)
HUN
HN
H2N
H2N
HN
OH
Figure Folic acid fragmentation illustrated with CID 135398658 from PubChem. (a) Preprocessing identify cleavable bonds for fragmentation. (b) Initial fragments formed using BFS, color-coded by functional groups (blue), rings (orange), and single atoms (green). (c) Fragments merged to satisfy atom count criteria, detailed in Appendix C (d) Expansion of fragments shown with directional arrows.
The design should merge topological chemistry with 3D structural data into a unified hypergraph representation, ensuring hyperodges accurately embody the molccule's chemical and spatial properties.
2. Controlling fragment size is vital for the SE3Set model to balance capturing meaningful interactions and computational efficiency. Optimal fragment sizes are key for model performance and learning capabilities.
To preserve the key chemical information for the molccule's properties and behavior, the fragmentation better only selectively break single bonds and maintain important functional groups and ring integrity.
4. Fragment overlap is essential to maintain functional group effects on local charge distribution and to ensure hyperedge interaction within the SE3Sct model for improved molecular learning.
Based on these principles, we propose a fragmentation method for molecules combined with chemical-defined substructures whose definitions and usages are described in detail in Appendix Generally, fragmentation algorithm improves molecular representations by combining bond order, functional groups, and substructures, including SMARTS-identifeed smaller rings and merged adjacent groups. Overcoming the drawbacks of non-overlapping fragmentation, it uses 3D spatial data and allows overlaps, preserving local effects for precise charge distribution and enhancing hypergraph neural network learning of molecular interactions.
Definition 4.1. A molecular fragment, denoted as 3, is defined specific subset of atoms within molecule, characterized by being cohesive assembly of predefined substructures linked in sequential concatenation.
Our fragmentation method meticulously dissocts a given molecule into meaningful subsets of atoms, and this process unfolds through four steps (corresponding to the pipeline in Fig. 1
5

---

Published in Transactions on Machine Learning Research (04/2025)
1. Pre-processing by analyzing the molecule's bond order matrix mask high-order bonds and those within functional groups or rings, and merging adjacent functional groups for a streamlined structural representation.
2. Core substructures are delineated from the remaining bonds using a Breadth- First Search algorithm, establishing the basic units of the molecular framework.
3. These substructures are then aggregated into larger molecular fragments according to predefined rules that maintain a minimum atom count within each fragment. This step could be optional.
To enhance fragment counectivity, we expand each by incorporating adjacent groups, using interaction strength metries based on interatomic distances to guide this process. Here we set the cutoff value denoted as Cw of an interaction strength metric to intercept the extended fragment.
Furthermore, step leads to substantial computational overhead for hypergraph neural net when processing larger molecular systems. To enhance the efficiency of our model for such expansive molecular systems, we introduce revised strategy for step
4* For each atom i, identify the neighboring atoms M, that fall within a specified radial cutoff re- A fragment is considered to be adjacent to atom i there an overlap of at least one atom between I and N₁. For ease of reference, the set of fragments adjacent to atom is represented as NT,, which implies that NT N ø).
We designate the application of step 4 as an explicit overlap and the application of step 1 as an implicit overlap. These approaches introduce nuanced variations in the mathematical expressions of our model, as reflected in Eq. 8 Eq. 9 and Eq. 16 For the detailed step-by-step methodology, please refer Appendix
4.2
SE3Set
Building upon our aforementioned fragmentation algorithm, we now turn to outline the architecture of SE3Set. The SE3Set model, influenced by AllSet (Chien et al., 2021b) and built on the Equiformer (Liao Smidt 2022), incorporates 3D spatial equivariance (proof refers to Appendix F in our hypergraph neural network, improving the capture of many-body interactions for precise molecular structure representation. SE3Seet consists of an embedding layer, attention blocks, and an output head, as shown in Fig, 2 (a). Morcover, we give a brief discussion in Appendix K on the advantages of the hypergraph structure we implement when considering the efficiency of message passing, comparing with the vanilla graph neural networks using a simple illustrative example.
4.2.1 Embedding
As depicted in Fig. 2 (b), the embedding block generates detailed node and hyperedge features reflecting molecular structures. Node features blend intrinsic properties with degree embeddings from connected hyperedges, while hyperodge features aggregate node embeddings and the corresponding relative position vectors depending on the node which the hyperedge feature located. SH (Spherical Harmonie) functions are used to project normalized relative position vectors Fij between node i and j into the irreducible representations (irreps) feature space with different orders I, i.e. SH(F)) y F.s
The features also F
mapped onto the same L-order irreps space for SE3 equivariance and updated separately. Hyperedges capture nodes' positional relationships, assigning a distinct feature h. to each node i in hyperedge 3a. reinforcing structural fidolity. Nodes v; intograte hyperodge information, harmonizing uniqueness with interconnections. Attention mechanisms then refine node and hyperedge interactions for accurate molecular and structural reprosentation.
4.2.2 Equivariant hypergraph attention blocks
As presented in Fig. 2 (f), the attention mechanism comprises two essential components: the Vertex-to- Edge (V2E) and Edge-to- Vertex (E2V) attention blocks, based on the AllSet framework (Chien et al. 2021b).

---

Published in Transactions on Machine Learning Research (04/2025)
(8)
Fg
X
[1]
Fly
Finbalding
Layer Norm
Layer Norm
VIE Module
Lincer
Linesr
Linear
Linze
E2V Medule
Rescale
Linzar
Linear
Layer Norm
Amenion Module
Attesion Module
Food Forward
Rescale
+
Ostput Hoal
Linear
Linca
(c)
(
+
Reshape
Norm
(Not Used)
Norm
Food Formand
Fu-a
fc=2
Feed Furwazd
LeakyyReLU
LeakyReLU
Linear
Lincar
Linear
A 22001
(8)
Sofimax
Lincer
Sofimax
Lincer
Linear
Reshupe
Rolupe
Figure 2: Overall architecture of SE3Set. (a) SE3Set begins with node and hyperedge embeddings, cycles through V2E and E2V attention modules for itcrative updates, and concludes with normalization and a leed-forward block for output. (b) Embedding. Atomic numbers and position vectors are transformed inlo initial embeddings for nodes and hyperedges. (c) V2F Module. Utilizes node features and their relative positions update hyperedge features. (d) E2V Module. Employs hyperedge features
refresh node features. (e) Attention Block. Merges feature sets with positional or hyperedge data for feature processing. (f) A variant of the Attention Module in E2VModule. This Attention Module does not introduce hyperedge information itself. This architecture corresponds to the result of E2V Summation in ablation study 5.4 (g) Fordaro Block. Enhances feature sets through a streamlined network. Symbols &, @ and @ in figures denote depth-wise tensor product, summation, and Hadamard multiplication, respectively. 4 represents hyperedge features, 2; is for node features, superscript n indicates the number of updates, and Fy is the relative position vector between nodes and j.
The V2F block refines hyperedge features, while the E2V block updates node features, both operating with an equivariant hypergraph attention mechanism. To improve training and enable deeper network structures, we incorporate normalization layers and residual connections to prevent gradient issues. The attention module's output passes through feed-forward block (Fig. 2 (d)), enhancing representation complexity. Node and hyperedge features maintain equivariance to molecular geometry, preserving data symmetrics and the integrity of representations, thus boosting the model's expressiveness in capturing complex structural interactions. (The concepts of irreducible representations and tensor products can be referenced in the Appendix
V2E attention The SE3Sct model uses geometrically invariant attention weights dij, derived from / - 0 irreps acting as scalars under geometric transformat ions. These weights are computed from scalar features fyl1= using an MLP with LeakyReLU activation and softmax normalization, reflecting node relationships within the hypergraph. Node and hyperodge features undergo non-linear transformations represented by tensor products of irreps with the quantum number The features combine through direct tensor products (DTP), yielding non-linear values Vi (Fig 2 (c)). Hyperedge features are updated by aggregating features
7

---

Published in Transactions on Machine Learning Research (04/2025)
from connected nodes, utilizing SH and radial basis functions hyperedge features. The model calculates initial features /3 and V2E attention weights ag via ML.ss with non-linear values oi emerging from similar transformations.
[ (Zinear(z;) + Linear(zz) (zy)
(4)
J Linear(") DTP ((((F (H)))
(5)
6j Softmaxy(a Leaky
(6)
0 Linear(Gate(ft) @ @ F.)I)) SH(T)))
(7)
Ultimaeey, the SE3Set model updales hyperedge features h by accumulating the weighted features of nodes within the same hyperedge and applying a linear transformation to the aggregated information. For the explicit overlap fragmentation method,
A4ς Linear
Σ
aiy Dij
where ny denotes the node with index and 3a denotes the fragment with index a as each fragment could be considered as a hyperedge the hypergraph. Due to the frequent occurrence of a high number of explicitly overlapping aloms, this scenario commonly results in increased computational complexity. Consequently, when adopting the implicit overlap approach, may opt for equation of the form:
Ah Linear
Σ
(9)
WJEBLAB.ENN
where NY is delineated in step 4 of the implicit overlap method. This characteristic renders a more computationally efficient scheme for to-tex-toEEdge (V2E) attention mechanisms. The detailed architecture of the V2E attention block is shown in Fig. 2 (c) and (e).
E2V attention Following the V2E attention module, the E2V attention module (Fig. 2 (()) updates node features by transforming them with a tensor product the updated hyperedge feature, followed by a linear layer. Attention weights are then calculated using softmax-applied, LeakyReLU-aactieee features, ensuring node features are refined after hyperedge updates.
f Linear((Tineanl & DTP hg)
(10)
a Softmax, (a LeakyReLU(fft-o
(11)
These atlention weights direct the synthesis of information, culminating in the calculated value:
of
Linear(Gate( @ DTP (T)).
(12)
Furthermore, we propose an alternative method for constructing the E2V attention block as shown in Fig.
(g).
S Linear(Linear(r:) Linear(h?))
(13)
a Softmax,, T Ue
(14)
0 = Linear Linear(Gate((""))
(15)
However, practical experiments reveal that the previous method yields superior results, with detailed findings presented in Sec. 5.4
8

---

Published in Transactions on Machine Learning Research (04/2025)
Then the node aggregates all the hyperedge features corresponding to itself to update the node feature,
Explicit overlap: Ar; Linear
Σ
aj
os
(16)
aa:i3a
Implicit overlap: Ar. Linear
Σ
(17)
The visualized architecture of the E2V attention block is also shown Fig. (d), (e), and (f).
4.2.3 Output head
The SE3Set model employs node features to generale predictions, using leed-forward network to transform these features into the target label's irreps dimension. A summation strategy aggregates node features into a single hypergraph-level representation, which is then processed by linear layer to output the model's final predictions.
5 Results
We Lested our equivariant hypergraph neural network on QM9 (Ruddigkeit et al. 2012 Ramakrishnan el al. 2014), MD17 (Chmiela al. 2017) (see Appendix G). MD22 (Chmiela et al. 2023 and OE62 dStuke et al. 2020) to assess its molecular representation learning. QM9 and MD17 gauge small molecule property prediction, while MD22 and OE62 evaluate larger systems with complex many-body interactions (Wang et al, 2023). An ablation study was also conducted to pinpoint the contributions of fragmentation and architecture to our method's performance, offering insights into the network's efficacy and arcas for enhancement. In the Appendix H we have also provided a detailed analysis of the model's complexity. Additionally, we have evaluated the robustness of our model on different tasks by some summary statistic metrics in Appendix M
5.1 QM9
The QM9 dataset [Ruddigkeit et al. 2012 Ramakrishnan et al. 2014 consists of 134k small organic molecules calculated at the /3LYP/6-31G(2d,, p) level. SE3Set, after training on 110k QM9 molecules and validation on 10k, achieves low mean absolute errors (MAEs) 12 tasks, performing on par with leading models, detailed in Table [ For completeness, some pre-trained molecular representation learning models are also selected basclines (sec Appendix J. In small molecular systems, higher-order many-body interactions are less pronounced and the molecular properties can be well approximated by the GNN-based models designed with some of the higher-order interactions. As a result, SE3Set does not significantly outperform other state-of-the-art (SOTA) models. Nevertheless, we will demonstrate that, while SE3Set maintains good performance with small molccules, the rate of performance degradation significantly decrcases as the size of the molecules increases (see Section 5.3
5.2 MD22
Recognizing the prominence of higher-order many-body interactions larger molecules (Wang et al. 2023
SE3Set was tested on the comprehensive MD22 dataset (Chmiela al. 2023) This dataset spans four classes
of biomolecules and supramolecules, from 42-atom peptide to 370- 370-atom nanotube, with high-resolution
sampling at 400-500 K using the PBE-MBD (Perdew et al., 1996. Tkatchenko et al. 2012) framework for energy and force computations. Our fragmentation method, which maintains functional groups and rings, selectively excludes structures like the Buckyball catcher and Double-walled nanotube from MD22, thus concentrating on the other five molecular lypes. We partition the training/ /est set following QuinNet (Wang et al. 2023). As Table 2 shows, SE3Set outperforms other SOTA models in these cases, reducing MAEs by an average of 20%, underscoring its exceptional ability capture molocular intricacies. Moreover, our results indicate that incorporating higher-order many- hany-body interactions is crucial for representing the non-local features of larger molecules within the MD22 dataset.

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: A comparative analysis was performed to assess the Mean Absolute Errors (MAEs) on the QM9 dataset when training SE3Set on a configuration comprising 110,000 training samples and 1,000 validation samples. Bolding shows the best model and the underlining shows the second best model and the underlining tilde shows third best model. All the baseline results are extracted from the origin papers (Schütt al, Gasteiger al.
2017
2020;
Schütt et al.
2021
Coors et al.) 2018 Wang ct al. 2022 Thölke & De Fabritiis
2021
Musaelian et al.
2023
Wang et al.
2024
Liao & Smidt
2022
Wang et al. 2023).
UNIT
SCHNET
DIMENET++
PAINN
SPRERENET
CoMENNT
ET
ALLecao
VISNET
QUISNET
EQUIFORSER
SE3SET
D
0.033
0.012
0.026
0.0245
0,771
0.011
<b
0.235
0.044
U.645
0.046
0.0452
0.058
0.041
0.047
0.046
0,042
HOMO
meV
11
20
23
23
17.3
20.4
15
15
LUMO
meV
34
28
18
17.6
GAP
meV
63
33
46
32
32
36
31.7
28.2
30
29
%5
0.073
0.331
0.006
0.259
0.1333
0.030
0.194
0.251
0.197
ZPVE
m
1,70
1.28
1.12
1.20
1.84
1.56
1,26
1.26
1.40
Uo
mcV
14
5.85
6
6.59
6.15
4.7
4.23
7.6
6.50
5.74
5.87
6.82
8.4
5.00
H
meV
14
5.98
6
6.86
6.16
4.1
4.52
7.6
6.63
0,70
meV
14
7.35
7.98
7.82
5.7
5.86
8.5
7.63
6.03
koal
0.033
0.024
0.021
0.024
0.025
Table 2: A comparison of Mean Absolute Errors (MAEs) across various benchmarked models. SE3Sct is trained on the five molecules of the MD22 datasel with specific number of training/val validation. Bolding shows the best model and underlining shows the second best model. The improvements column shows the improvement of our model over the previous SOTA model in percentage terms. The MAÉs reflect the precision of energy predictions in units of kcal/mol /mol and forces in units of kcal/ (mol-A). AA. The results of the
baseline models refer to the origin papers (Chmiela al., 2023 Thälke & De Fabritiis
2021 Musaelian et al.
2023
Batatia et al. 2022 Liao & Smidt
2022
Wang et al. 2024 2023
Li et al. 2024].
Moroeraa
TRAM
SCD3LL
ToncHMD-NET
Astaono
MACE
Bottronsen
VISNET
QuisNer
EQuPORSEE-LSRX
VISNET-LERM
SESSCT
IIFROVEAGNTS
NHME
000
Poacee
Essaor
0.2012
6.7908
0.1121
0.2015
0-105
0.0524
01061
0.959
0.0081
0.054
I03780
0.0909
C.M154
0J0645
0.0499
20.03
19.53
STACEVOSE
7600/
Rosawr
Fonce
0,6744
0,1921
0.136
0.2685
0.007
0.1234
3.0576
0.104
0.0633
0.1283
0.0569
0.0048
0.20
1.1.52
0.0632
5.005
0,0007
0.0762
0.0424
21.05
27.8%
AT-AT
2000 500
EstRor
Donch
0.7255
0.8811
0.1130
0.8036
0.1425
0.0832
0.1003
3.0983
0.130
0.0997
0.1688
0.1070
0.14
B.1007
DOS11
5,0072
0.0688
24.23
19.13
A1-XT-0G-CG
6,7028
5.3
OE62
The OE62 dataset (tuke et al. 2020) contains about 62k large organic molecules with annotated DFT- computed energies in the unit of eV. We randomly selected 50,000 data points as the training set and 6,000 data points as the validation set following the work of Kosmala et al. (2023). Here we report the MAE of energy as the performance metric on the test dataset. Since OE52 is considered to have significant long-range interactions (Kosmala et al. 2023), we selected model results without explicit long-range modeling as a baseline. The results (Table 3) demonstrate that SE3Set outperforms all baseline models (which focus on short-range interactions), highlighting the potential of our model for capturing many-body interactions.
To further evaluate the performance of our model, conducted additional tests on the OF62 dataset to analyze the correlation between model performance (SE3Set and the baseline models) and molecular size, as reflected by the number of atoms in molecule. First, we reproduced the results the basclinc model using open-source code"] as shown in parentheses in the Table 3 The reproduced results are largely consistent with those extracted from the original work. The analysis of each model's performance in representing molecules of different sizes illustrated in Fig. We grouped the test set based on the number of atoms in the molecules and reported the MAE for each group. To facilitate clearer comparison among different models, we defined the relative difference (RD) metric across all m models.
MAE M Σm MAE,. 1
RD,,
(18)
M Σn MAE,,
http:://githuub.cc//aattrrrrrree
10

---

Published in Transactions on Machine Learning Research (04/2025)
Table 3: comparison of MAEs across some benchmarked models on OE62 dataset in the unit of meV. The result of the best model is bolded. The bascline results are extracted from Ref. Kosmala et al. (2023). while the results in parentheses represent the reproduced experimental outcomes.
MODEL
ScnNET
PAINN
DIMENET++
GEMNET-T
SE3SET(3 LAYERS)
MAE (meV)
131.3 (130.0)
63 (62.6)
53.8 (52.3)
53.1 (54.9)
51.7
(4)
(b)
(c)
Sertet
FARk
Dimeelate
1200
3525
SESSet
Crang PMC1000
Charge MMC 1960
2320
800
302-
200
320
09 341
(S 651
(VA se1
196, 1291
1125,
1501
14796
MGRM
(d)
(6)
(1)
ScrNct
Fanih
Dimelistee
Lnoog Nam Iveen
Eraryy MAE 10.
25
70
Mumber of
301
120,
050 500
(50, 1200
(122, 158)
Seviil 3030cl
Figure 3: The correlation between the mean absolute error (MAE) of energy predictions by different SE3Set models and varying molecular sizes or fragment counts. We use the number of atoms to represent molecular size, dividing them into bins at intervals of 30 atoms. For each interval, we calculate the MAE and the standard deviation of the MAE for the model's energy predictions. The relative difference (RD), as defined in Eq. 18 is intended to indicate model performance compared to the average level. The results for the baseline model are from our replicated experiments, and SE3Set uses a 3-yayer configuration, which corresponds to (a) The MAEs of the total energy of the whole molecule. (b) The RD of different (c) The MAFs of the total energy of SE3Set with different numbers of fragments. (d) The MAEs of the energy relative to each atom. (e) The RD of different models on the energy relative to each atom. (f) The MAEs of the energy of SE3Set relative to each atom with different number of fragments.
the results in Table
3
models on total energy.
We compare the MAE of the total energy per molecule and the MAF of the energy relative to each atom separately. For all models, both the MAEs of the total energy and the MAEs averaged to each atom become progressively larger as the size of the molecule increases as shown in Fig. 3 (a) and (d), which shows the interactions, especially the high order interactions are more complicated and significant larger molecules. However, the MAEs of SE3Set grow significantly slower than the other bascline models. Additionally, the RD curves in Fig. (b) and (e) indicate that SE3Set achieves better performance on large molecules and has the lowest MAEs when the number of atoms interval lower bound is greater than 60. It is also worth noting that the advantage of SE3Set becomes progressively more apparent as the size of the molecule increases, reflected by the decrease in its RD with an increasing number of atoms. These results further support our speculation that SE3Set is more effective compared to other models when learning on larger molecules.
To further analyze the impact of the number of hyperedges on the model performance in addition to the number of nodes (which corresponds the number of atoms), we also show the results of the variation of MAFs of SE3Set with the number of fragments in Fig. (c) and (f). Since the fragmentation method
11

---

Published in Transactions on Machine Learning Research (04/2025)
to construct the hypergraph structure, the number of fragments is associated with the number of hyperedges. The MAFs of the total energy grow larger with the increasing fragment numbers, more fragments are suggestive of errors accruing from larger molecular systems. When it comes to the MAEs of energy per atom, the increase in MAE with the number of fragments very slight. This result indicates that our method can handle systems with different fragment numbers more lairly. Considering that molecules with similar atomic numbers may have differences in fragment numbers, this demonstrates that our model more robust for different molecular structures.
60
5.4 Ablation studies
52.3
50
To better understand SE3Sct, we conduct ablation studies focusing on fragmenta- tion and model architecture. For the explicit overlap fragmentation method, we explore how different fragmenta- tion techniques affect SE3Set's training and compare with the non-overlapping BRICS (Degen el al.) 2008 Landrum et al., 2020) strategy on QM9's homo energy task. As Fig. indicates, tests QM9's homo energy task showed SE3Set's robustness to variations in the fragmentation method. The results show that our method surpasses BRICS, demonstrating the importance of hyper- edge interaction. Furthermore, we per- formed ablation studies on the model ar- chitecture. Among two design variants in the E2V attention section, the one using tensor product interactions between nodes and hyperedges proved superior, emphasizing the value of our tensor product-based mechanism and architecture design in enhancing molecular property predictions. Additionally, a 6-layer SE3Set model outperformed its 3-layer counterpart.
HOMD MAE (maV)
40
21.6
20
16.9 16.8 16.7
15.1 15.0 15.1
17.8
0-
3 Layers
Layers
Go 0.1
Gy = 0.05
Co 0.01
BRICS
E2V Summation
Figure 4: Ablation studies on the QM9 dataset's HOMO task (units: meV). The variable cw represents the threshold for expan- sion the fourth step of fragmentation, guided by the fragment bond order defined in Eq. 19 The term BRICS denotes another fragmentation method implemented in RDKits. Additionally, the E2V summation refers to the architectural framework specified
from Eq. 13 to Eq 15
Table 4: The MAE for energy (unit: kcal/mol) and forces (unit: kcal/mol A) the AT-AT-CG-CC dataset using 3 layers SE3Set with different cutoff radii (re) in the implicit overlap method.
IMPLICIT r((A)
4.0
5.0
ENERGY
0.2123
0.1153
0.11033
FORCE
0,1559
0.1019
0.0937
Unlike the explicit overlap method, the fragment size the implicit overlap method depends on the choice of Te- We test the effect of ra on the model training results on the AT-AT-CG-CG molecule of the MD22 dataset. (Table 4.. The SE3Set performs better when using higher re it will include more fragments generate implicit overlaps. The effect of re fragment size is pronounced for the implicit method than for the explicit method, but it gives good performance for different parameters compared with the baseline models.
6 Conclusion
In conclusion, this study demonstrates the efficacy of SE3Set, cutting-edgr hypergraph neural network architecture, in the realm of molccular representation learning. By meticulously crafting a fragmentation method that coalesces two-dimensional chemical knowledge with three-dimensional spatial information, we establish a robust foundation for constructing hypergraphs that faithfully capture the complex nature of molccular structures. The SE3Sct architecture, drawing inspiration from the AllSet framework and the Equiformer, adeptly processes these hypergraphs and preserves the essential invariances and symmetries. SF3Set demonstrates performance comparable to SOTA models in small molecular systems and significantly
12

---

Published in Transactions on Machine Learning Research (04/2025)
outperforms SOTA models in large molecular systems where higher-order many-body interactions are pronounced. The results of our research affirm the potential of SF3Set to model high-order many-body interactions, providing a powerful tool for molecular representation.
7 Impact Statements
This paper advances machine learning techniques for molecular representation, with potential impacts in drug discovery, materials science, and chemical engineering. By improving the modeling of large molecules, our work could enhance drug development, lead to new materials, and improve the understanding of chemical interactions. Additionally, it may aid personalized medicine through better predictions of molecular interactions. We recognize the importance considering ethical implications, such as data privacy and responsible AT use, and encourage ongoing discussion on these broader impacts.
References
Sameer Agarwal, Jongwoo Lim, Lihi Zelnik-Manor, Pietro Perona, David Kriegman, and Serge Belongie. Beyond pairwise clustering. In 2005 IEEE Computer Sociely Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 2, pp. 838 845. IEEE, 2005.
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32, 2019.
Sarp Aykent and Tian Xia. Savenct: scalable vector network for enhanced molecular representation learning. Advances in Neural Information Processing Systems, 36, 2024.
Tyes Batatia, David Kovacs, Gregor Simm, Christoph Orlner, and Gábor Csányi. MACE: Higher order equivariant message passing neural networks for fast and accurate force fields. Advances in Neural Information Processing Systems, 35:1 55:11423-11433 2022.
Simon Batzner, Albert Musaclian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nat. Commun., 13(1):2453, 2022.
Austin Benson, David F Gleich, and Lek eek-Heng Lim. The spacey random walk: A stochastic process for higher-order data. SIAM Rev., 59(2):321-344 2017.
Adam J. Bridgeman and Christopher J Empson. Bond orders between molecular fragments. Chemistry, 12 8: 2252 62, 2006.
He Cao, Zijing Liu, Xingyu Lu, Yuan Yao, and Yu Instructmol: Multi-modal integration for building a versatile and reliable molecular assistant drug discovery. arXiv preprint arXiv:2311.16208, 2023.
Fangying Chen, Junyoung Park, and Jinkyoo Park. A hypergraph convolutional neural network for molecular properties prediction using functional group. Preprint at
2021.
Junwu Chen and Philippe Schwaller. http://arxiv http://org/bbbbbaaeeeee 2212. 13136 2023.
Molecular hypergraph neural networks.
Preprint at
Eli Chien, Pan Li, and Olgica Milenkovic. Landing probabilities of random walks for seed-set expansion in hypergraphs. In 2021 TEEE Information Theory Workshop (TTW), pp. 1-6. IEFE, 2021a.
Eli Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovic. You are AllSe:: A multiset function framework for hypergraph neural networks. In International Conference on Learning Representations, 2021b.
Stefan Chmicla, Alexandre Tkatchenko, Huziel E. Saucoda, Igor Poltavsky, Kristof T Schütt, and Klber-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Sci. Adv., 3(5):11633015,
2017.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Stefan Chmicla, Valentin Cassilev-Galindo Oliver T Unke, Adil Kabylda, Huziel E Sauceda, Alexandre Tkatchenko, and Klaus-Robert Müller. Accurate global machine learning force fields for molecules with hundreds of atoms. Science Advances, (()) 2023.
Michael A Collins and Ryan PA Bettens. Energy-based molccular fragmentation methods. Chem. Rev., 115 (12):5607 5642, 2015.
Benjamin Coors, Alexandru Paul Condurache, and Andreas Geiger. SphereNet: Learning spherical rep- resentations for detection and classification in omnidirectional images. In Proceedings of the European conference computer vision (ECCV), pp. 518 533, 2018.
Shicheng Cui, Qianmu Li, Deqiang Li, Zhichao Lian, Jun Hou, et al. Hyper-mol: Molecular representation learning via fingerprinlbbaeed hypergraph. Comput. Intell. Neurosci. 2023, 2023.
Laurianne David, Amol Thakkar, Rocío Mercado, and Ola Engkvist. Molecular representations in AL-driven drug discovery: review and practical guide. J. Cheminform., 12(1):1-22 22, 2020.
Jörg Degen, Christof Negscheid-Gerlach Andrea Zaliani, and Matthias Rarey, On the of compiling and ch'hmmia using' drug-like- chemical fragment spaces. ChemMedChemm Chemistry Enabling Drug Discovery, 3(10):
2008.
Kien Do, Truyen Tran, and Svetha Venkatesh. Graph transformation policy network for chemical reaction prediction. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 750-760, 2019.
Boxin Du, Changhe Yuan, Robert Barton, Tal Neiman, and Hanghang Tong. Hypergraph pre-training with graph neural networks. Preprint at sttp:://axxvv.org/bbb///22 1086/ 10862 2021.
Yuanqi Du, Limei Wang, Dieqiao Feng, Guifeng Wang, Shuiwang Ji, Carla P Gomes, Zhi-Ming Ma, et al. A new perspective building efficient and expressive 3d equivariant graph neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. Preprint at http://arxiv http::/aartv..orr//a///100. 02428 2019.
Denis Fourches, Eugene Muratov, and Alexander Tropsha. Trust, but verify: on the importance of chemical structure curation in cheminformatics and qsar modeling rescarch. J. Chem. Inf. Model., 50(7):1189, 2010.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, SE (3)-Lransformers: 3D Roto-translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-185 1981, 2020.
Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations, 2019.
Johannes Gasteiger, Shankari Giri, Johannes T Margraf, and Stophan Günnemann. Fast and uncertainly-aware directional message passing for non-equilibrium molecules. Preprint attp://arxxv.orr//bb///00 14115 2020.
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790 6802, 2021.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263 -172.. PMLR, 2017.
Jonathan Godwin, Michael Schaarschmidt, Alexander L Gaunt, Alvaro Gonzazz, Yulia Rubanova, Petar Veličković, James Kirkpatrick, and Peter Battaglia. Simple GNN regularisation for 3D molecular property prediction and beyond. In International Conference Learning Representations, 2021.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Mark S Gordon, Dmitri G Fedorov, Spencer R Pruitt, and Lyudmila V Slipchenko. Fragmentation methods: A route to accurate calculations on large systems. Chem. Rev., (12(1):32-6672 2012.
Hossein Hajiabolhassan, Zahra, Taheri, Ali Hojatnia, and Yavar Taheri Yeganeh. Funqg: Molecular repre- sentation learning via quotient graphs. Journal of chemical information and modeling, 63((1):3275-32M5) 2023.
Xiaohong Ji, Zhen Wang, Zhifeng Gao, Hang Zheng, Linfeng Zhang, Guolin Ke, et al. Uni-mol2: Exploring molecular pretraining model at scale. arXiv preprint arXiv:2406.1 149.9999,,,,, 2024.
Hiroshi Kajino. Molccular hypergraph grammar with its application to molecular optimization. In International Conference on Machine Learning, pp. 3183 3191. PMLR, 2019.
Jinwoo Kim, Sacyoon Oh, and Seunghoon Hong. Transformers generalize deepsets and can be extended to graphs & hypergraphs. Advances in Neural Information Processing Systems,, 34:28016 11220066-2008,, 2021.
Jinwoo Kim, Saeyoon Oh, Sungjun Cho, and Seunghoon Hong. Equivariant hypergraph neural networks. In European Conference on Computer Vision, pp. 86 103. Springer, 2022a.
Seojin Kim, Jaehyun Nam, Junsu Kim, Hankook Lee, Sungsoo Ahn, and Jinwoo Shin. Contrastive learning of molecular representation with fragmented views. 2022b.
P-J Kindermans and K-R Müller. Schnet deep learning architecture for molecules and materials. J. Chem. Phys., 148(24), 2018.
Arthur Kosmala, Johannes Gasteiger, Nicholas Gao, and Stephan Günnemann. Ewald-basod long-range message passing for molecular graphs. In International Conference on Machine Learuing (ICML), 2023.
Greg Landrum, Paolo Tosco, Brian Kelley, sriniker, gedeck, NadineSchncider, Riccardo Vianello, Ric, Andrew Dalke, Brian Cole, AlexanderSaveller, Matt Swain, Samo Turk, Dan N, Alain Vaucher, Eisuke Kawashima, Maciej Wójeikowski, Daniel Probst, guillaume godin, David Cosgrove, Axel Pahl, JP, Francois Berenger, strets123, JLVarjo, Noel O'Boyle, Patrick Fuller, Jan Holst Jensen, Gianluca Sforna, and DovidthGavid. rdkit/rdkit 2020 03 1 (q1 2020) release, March 2020. URL httts:///oo.oo.... http://i..oog///////o/oo 10. http://eooood.333333
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorck, Soungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutalionninnarian neural networks. In International conference on machine learning, pp. 3744 3753. PMLR, 2019.
György Lendvay. On the correlation of bond order and bond length. J. Mol. Struct., 501:389 09000-9333100 2000.
Biaoshun Li, Mujie Lin, Tiegen Chen, and Ling Wang. Fg-bert: generalized and self-supervised fune- tional group- based molecular representation learning framework for properties prediction. Briefings in Bioinformatics, 24(6):bbad398, 2023.
Guoyin Li, Liqun Qi, and Gaohang Yu. The Z-cigenvalues of symmetric tensor and its application spectral hypergraph theory. Numer. Linear Algebra Appl., 20(6):1001 1029, 2013.
Juncai Li and Xiaofei Jiang. Mol-bert: An effectivo molccular representation with bert for molccular property prediction. Wireless Communications and Mobile Computing, (171(1)7,,8115,, 2021.
Yunyang Li, Yusong Wang, Lin Huang, Han Yang, Xinran Wei, Jia Zhang, Tong Wang, Zun Wang, Bin Shao, and Tie- Yan Liu. Long-short-range message-passing: A non-local interaction for scalable molecular dynamics simulation. In The Twelfth International Conference on Learning Representations, 2024. URL
framework to capture
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3D atomistic graphs. In The Eleventh International Conference on Learning Representations, 2022.
Yi-Lum Liao, Brandon Wood, Abhishek Das, and Tess Smidt. EquiformerV2: Improved equivariant transformer for scaling to higher-degree representations. Preprint at http://arxivvoorg/ale 2306 12059 2023.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Kha-Dinh Luong and Ambuj Singh. Fragment-base baase pretraining and finetuning molecular graphs. Preprint at http::/arxiv. org/abss 23..//3231.03024 2023.
Jonathan P Mathews and Alan L Chaffee, The molecular representations of coal-a a review. Fuel, 96:1-14,
2012.
David L Mobley, Caitlin C Bannan, Andrea Rizzi, Christopher Bayly, John D Chodera, Victoria T Lim, Nathan M Lim, Kyle A Beauchamp, David R Slochower, Michael R Shirts, et al. Escaping atom types in force fields using direct chemical perception. J. Chem. Theory Comput., 14(11):6076 6092, 2018.
Albert Musaclian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. Nat. Commun, 14(1):579, 2023.
Eliya Nachmani and Lior Wolf. Molecule property prodiction and classification with graph hypernetworks. Preprint at httt:://arxa httpaarxivv..ogaaai org/abs/2002...o.. 00240 2020.
Kelly J Pearson and Tan Zhang. On spectral hypergraph theory of the adjacency tensor. Graphs Combin, 111123--1488 2014.
John P Perdew, Kieron Burke, and Matthias Ernzerhof. Generalized gradient approximation made simple. Phys. Rev. Lett., 77(18):3865, 1996.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Sci. Data, 1(1):1-7, 2014.
Patrick Reiser, Marlen Neubert, André Eberhard, Luca Torresi, Chen Zhou, Chen Shao, Houssam Metni, Clint van Hoesel, Henrik Schopmans, Timo Sommer, et al. Graph neural networks for materials science and chemistry. Commun. Mater., 3(1):93, 2022.
Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean- Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database GDB-17. J. Chem. Inf. Model., 52(11))2864 2875, 2012.
T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. Preprint at htttp://arxv..rrrr///e/ www.:/wwwwwwraab/////aaaa 10993. 2023.
Kristof Schütt, Pieter-Jan Kindermans, Huxiel Enoe Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Claus-Robert Müller. Schnet: A continuous-flle convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pp. 9377-9388. PMLR, 2021.
Annika Stuke, Christian Kunkel, Dorothea Golze, Milica Todorović, Johannes T Margraf, Karsten Reuter, Patrick Rinke, and Harald Oberhofer. Atomic structures and orbital energies of 61, 4,9 crystal-forming organic molecules. Scientific data, 7(1):58, 2020.
Mohammadamin Tavakoli, Alexander Shmakov, Francesco Coccarelli, and Pierre Baldi. Rxn hy- pergraph: a hypergraph attention model for chemical reaction representation. Preprint http///awvvvv.//////// https///22...11101 011966 2022.
Philipp Thölke and Gianni De Fabritiis. Equivariant transformers for neural network based molecular potentials. In International Conference on Learning Representations, 2021.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and censllationeqqiariinnn neural networks for 3d point clouds. Preprint www.a//ww..orrabb//////0/ 08219 2018.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Alexandre Tkatchenko, Robert A DiStasio Jr, Roberto Car, and Matthias Scheffter. Accurate and efficient method for many-body van der Waals interactions. Phys. Rev. Lett., 108(23):236402, 2012.
Francesco Tudisco, Austin R Benson, and Konstantin Prokopchik. Nonlinear higher-order label spreading. In Proceedings of the Web Conference 2021, pp. 2402-2413, 2021.
Jeff Wagner, Matt Thompson, David L. Mobley, John Chodera, Caitlin Banman, Andrea Rizzi, trevorgokey, David L. Dotson, Josh A. Mitchell, jaimergp, Camila, Pavan Behara, Christopher Bayly, JoshHorton, Iván Pulido, Lily Wang, Victoria Lim, Sukanya Sasmal, SimonBoothroyd, Andrew Dalke, Daniel Smith, Brent Westbrook; Josh Horton, Lee-Ping Wang, Richard Gowers, Ziyuan Zhao, Connor Davel, and Yutong Zhao. openforceficld pttptoorreeildddpenffffff lowenff-toolkff 0.15.1. Testing updates, January 2024. URL 10lodo.10595583
Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. ComENct: Towards complete and efficient message passing for 3D molecular graphs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Yusong Wang, Tong Wang, Shaoning Li, Xinheng He, Mingyu Li, Zun Wang, Nanning Zheng, Bin Shao, and Tin-Yan Liu. Enhancing geometric representations for molecules with equivariant vector-scalar interactive message passing. Nat. Commun., 15(1):313, 2024.
Zun Wang, Guoqing Liu, Yichi Zhou, Tong Wang, and Bin Shao. Efficiently incorporating quintuple interactions into geometric deep learning force fields. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Daniel S Wigh, Jonathan M Goodman, and Alexei A Lapkin. A review of molecular representation in the age of machine learning. Wiley Interdiscip. Rev. Comput. Mol. Sci., 12(5))e1603, 2022.
Zhenxing Wu, Jike Wang, Hongyan Du, Dejun Jiang, Yu Kang, Dan Li, Peichen Pan, Yafeng Deng, Dongsheng Cao, Chang- Yu Hsieh, et al. Chemistry-intuitee explanation of graph neural networks for molecular property prediction with substructure masking. Nature Communccations, 14(1):2585, 2023.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprchensive survey on graph neural networks. IEEE Trans. Neural. Netw. Learn. Syst., 32(1):4 24, 2020.
Jiacheng Xiong, Zhaoping Xiong, Kaixian Chen, Hualiang Jiang, and Mingyue Zheng. Graph neural networks for automated de novo drug design. Drug Discov. Today, 26(6):1382 1393, 2021.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhuldinos and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Dengyong Zhou, Jiayuan Huang, and Bernhard Schölkopf. Learning with hypergraphs: Clustering, classifica- tion, and embedding. Advances in neural information processing systems, 19, 2006.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molccular representation lcarning framework. In International Conference on Learning Representations, 2023.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al open, 1:57-81, 2020.
17

---

Published in Transactions on Machine Learning Research (04/2025)
A Details of fragmentation steps
Before delving deeper into the specifies of our fragmentation method, it's important to establish a foundational understanding through key definitions and concepts. We first give the definition of bond order here, Definition A.1. The bond order represents the multiplicity or the number of shared electron pairs that constitute a covalent bond between two atoms.
The bond order matrix B is an N x N representation of bond strength between atoms in a molecule, with higher bond order values indicating stronger bonds. This symmetric matrix (Bij = B,,) is crucial for studying the molecular structure and reactivity, capturing bond nuances including delocalized and resonance bonds in computational chemistry.
Based on the design principles in Sec. 4.1 the detailed step-by-step methodology of explicit overlap fragmen- tation method is shown as follows,
The pre-processing step begins by analyzing the given molecule through bond order matrix, denoted as B. Identify and mask bonds that are part of functional groups or rings, as well as those with bond order of Biy > 2. Functional groups are then identified using predefined SMARTS patterns for accurate matching. To achieve more generalized representation of functional groups, topologically adjacent functional groups are merged into single entity. This aggregation allows to focus on specific subfunctional groups that are of particular interest, simplifying the complexity of the molecular structure for subsequent analysis.
2. Following the masking of selected bonds, the Breadth-First Search (BFS) algorithm is employed to reconstruct the substructures, denoted as (G}, from the remaining unmasked bonds. These groups represent the core structural units of the molecule as discussed at the outset this section.
3. Consolidate the previously identified groups {G} into larger molecular fragments, applying a set of predefined rules to guide the merging process. These rules are meticulously designed to ensure that each resulting fragment, now denoted as {8}, contains at least a minimum specified nunber of atoms. For a comprehensive understanding of the merging criteria, one can refer to the detailed rules outlined in
4. Extend each fragment (3) by incorporating adjacent groups from {6} to enrich the connectivity between molecular fragments, thus intentionally creating regions of overlap among the fragments. This expansion controlled by a cutoff threshold, denoted as Own which is typically a function based on interatomic distances. The fragment bond order (Lendvay] 2000 Bridgeman & Empsonn] 2006), symbolized by Wfa, is used to quantitatively assess the interaction strength between a fragment 3, and an adjacent substructure S,. This method reflects the interaction strength based the proximity of atoms in different fragments, expressed by the following equation:
Σ exp (- (dy di,)) ) (0.25 A)
W₁+ =
(19)
JEG,
where di represents the interatomic distance between atoms i and j, and di, stands for the equilibrium distance typically expected for such bond. This equation is utilized to determine which substructures should be included in the expansion of fragment, based on the strength of their interactions governed by the distance function. Additionally, in alignment with Pauling's concept of 'chemist's 's bond order" [Lendvay] 2000), an alternative method is introduced to calculate the bond order using a single exponential function,
WS--
Σ exp (-(des - d(,)). HEE,
(20)
where Wfs encapsulates the bond order between atoms belonging to a fragment By and a substructure G.. In this context, di signifies the actual measured distance ween atom and atom j. The term
18

---

Published in Transactions on Machine Learning Research (04/2025)
di refers to the theoretical equilibrium covalent bond length, which is estimated by summing the empirical covalent radūí of the two atoms involved, given by:
d
12 Tays
(21)
where rs, the empirical covalent radius of an atom with atomic number 2,. This function provides simplified yet effective representation of bond order, allowing to gauge the bonding interactions within the molecular structure concerning the proximity of the atoms.
The implicit overlap fragmentation method only changes the step 4 the details of the changed fourth step have been spelled out in
B Functional groups SMARTS
In the initial phase of fragmentation approach, identify functional groups using the SMARTS pattern-matching language. In Table 5 present the complete list of SMARTS patterns utilized, which have been expanded upon from the default set found within the Open Force Field toolkit (Mobley et al. 2018: Wagner et al. 2024 (accessible at:
oreef hwtoooonffffaraeee fragmenter/ json].
blob/ mwwm/mpenf/offfffemmommm httoo//mmi/ppppff/////eeerrr////
Table 5: SMARTS patterns for functional groups employed in the preprocessing stage of fragmentation.
FUNCTIONAL GROUPS NAME
SMARTS
HYDRAZINE
|XX3:1
NX3:2
HYDRAZONE
NNX:1
NX2:2
NITRIC OXIDE
N:1]-[0:2)
AMIDE
##:21 (-:33)) [
AMIDE NEGATIVE ION
[#7:1]]44:2[[[[--33]
ALDEHYDE
[CX3HL:][=[(::]]4666]
SULFOXIDE
SULFONYL.
#10X:]||==0X1::))0000131
SULFINIC ACID
SULFONIC ACID
# #4B24|||-00112220100000000000000
SULFINAMIDE
PHOSPHINE OXIDE
PHOSPHONATE
[P:
PHOSPHATE
CARBOXYLIC ACID
([X0]]==002])XXXHH0XXXH1
NX3 [33=:1][-00))00033]] [|01-0002210033/133333=-00-0---- [NX3:][=[0:2]=[0033
NITRO
[CX3:[[[[22]]0x2HH00]
ESTER
TRI-HALIDE
LL.,,, (F..LBB.)))(..CC.LD.)))(((CCCC.) (F.C..LBB
HYDROXYT.
[#8:1 (8:1)-#1:22
C
Merge process of fragmentation
During the third step of our fragmentation method, introduce strategy to enlarge substructures, ensuring that each initial fragment contains least Rmmin atoms, with 7min being a predefined integer. To maintain permutation invariance for a molccule, we incorporate weights, Wfs, to guide the sequence of merging, The process is outlined in the pseudocode (Algorithm D. The calculation of W based on either Eq, 19 or Eq. 20 By considering the sum of bond orders to other groups, we assess each group's centrality. The groups are then ordered first by the number of atoms they contain, followed by the summation of their bond orders, ensuring that the fragmentation merge process is permutation invariant when following this specified sequence. The algorithm then assists smaller groups in merging with others to achieve size of at least Manin atoms.
19

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm 1 Pseudo code of Iragmentation merge step.
Input: groups {8), minimum atoms number Reiia maximum atoms number Mmax Topological bond order matrix B, isolated threshold Cis
m = -[
Isolate groups -
Calculate fragmentation bond order matrix We.s,
Sort {5) in descending order based on the following attributes: number of atoms, Es, 06 Wo6. repeat
Pop last fragment as By from {6}
for = 1 to 1 do
= MAX INT, merge idr = -1 if any Big 2 1, E G;j E 6₁ and [6,] < and a + [B,] < mmax then a= |=,6, merge idr =
end if end for if merge
idr - then for i = 1 to 1 do
if any We.8, 2 Cis and [8,] < a then a a= 1, merge ids = -1
end if end for end if
if merge idr # 1 then
Merge Gg to 6 merge idæ
Resort {5} by the same priority and update W.
else
Add {Gx} to {
end
until [ > mmin
(3) U
Initially, consider topologically adjacent groups with the fewest atoms. If a target group lacks topological neighbors, we procoed to merge based on the bond order from W. We introduce a threshold Cı. that allows a group to remain isolated is significantly distant from others. It should be noted that isolated groups may not meet the minimum atom number requirement; however, they could be further expanded in the subsequent fragmentation step, depending on the chosen thresholds for Ga and cw (refer to Sec. 4.1h. Overall, this algorithm ensures a ietpuuttttioiiivviiinn merging process.
Distribution of fragmentation dataset
Different parameters used in the fragmentation process can lead to variety hyperedges, which turn result in distinct hypergraphs utilized for training our model. To illustrate the variances attributed different
fragmentation parameters or methods (such BRICS implemented in RDKit (Degen et al. 2008 Landrum)
et al.) 2020 use the QM9 dataset (Ruddigkeit et al. 2012 Ramakrishnan et al. 2014 to demonstrate how the data distributions attached to hypergraphs may change.
The impact of adjusting fragmentation parameters on the composition of hyperedges can be observed in Fig. 3 Altering the expansion threshold ew within a certain range has a minimal effect on fragment expansion. However, when utilizing the Lendvay bond order (Eq. 19). fragments tend comprise fewer atoms comparod when using the Exponential bond order (Eq. 20). This difference is likely due to the more gradual decline in the exponential function, which results in greater cumulative contribution to the weights Wss-
20

---

Published in Transactions on Machine Learning Research (04/2025)
(a)
Atoms Number Molecule
0.25
25
0.20
Frequency
0.15
15
0.10
10-
0.05
5
0.00
8
2 3
Number Fragments
Number of Fragments
(d)
0.125
0.125
0.15
Frequency
Frequency
Frequency
0.10
0.075
0.075
0.050
0.05
0.025
0.025
0.000
0.000
20
10
10
20
Average Atoms Number in Fragment
Average Atoms Number in Fragment
Average Atoms Number Fragment
0.20
0.20
Frequency
0.15
Frequency
0.15
0.10
0.10
0.05
0.05
0.00
0.00
Average Atoms Number in Fragment
Average Atoms Number in Fragment
(h)
(i)
30
0.3
Atoms Number Moleculs
0.10
25
Frequency
0.2
20
Frequency
0.08
0.06
15
0.1
10
0.02
5
0.00
6
10
Fragments
Number of Fragments
Average Atoms Number in Fragment
Figure Distribution of fragments in QM9 dataset. (a) Fragment Count Distribution. The distribution remains consistent regardless of the value of Cw the bond order calculation method employed. (b) Molecule Size vs. Fragment Count Distribution. Generally, the more atoms molecule has, the more fragments will generate. It is also invariant for e, or bond order calculation scheme. Average Atom Count per Fragment Distribution (c) cu 0.1, (d) Cu = 0.05, (c) cw = 0.01 for Lendvay bond order and (f) cw = 0.4 and (g) Cp = 0.2 for exponential bond order, respectively. (h) BRICS Fragment Count Distribution. (i) BRICS
Molecule Size vs. Fragment Count Distribution (j) BRICS Average Atom Count per Fragment.
Our ablation study (Sec. 5.4 also includes a comparison with the BRICS fragmentation method. Fragments generated by the BRICS method are observed contain significantly fewer atoms since this approach does not create overlapping regions between different. fragments.
21

---

Published in Transactions on Machine Learning Research (04/2025)
E Proof of SE(3) equivariance
The SE3Set consists of the basic modules including linear, depth-wise tensor product, gate activation, and layer normalization. Here we will prove the SE3 equivariance for these modules. As the inputs of SE3Set are all invariant to translation in 3D Euclidean space, we only need to prove the SO(3) equivariance.
Let E (3(3) and the D denote the group representation of SO(3). Besides, we denote the irreps feature as f, and f' denotes the hth feature vector irreducible representations space of SO(3) with order.
Lincar Linear module deploys separated lincar operations for each I in the irreps feature. For cach 1, we consider the output channel f. Then have
= Linear(f') = -] Ewighh
(22)
where why denotes the linear combination weight. When acting D(g) at the input fl, we can find that
(C)) = Linear(D(g)Ft
(23)
Σ why P(g)
(24)
D(g) Σ wlyth
(25)
= D)
(26)
Therefore, the linear module is SO(3) equivariant.
Depth-wise Tensor-product Tensor product is equivariant operalion for D(g)
(Lg) @ = = (D(g) (D(g)
(27)
The depth-wise lensor product differs from the tensor product only in that one order / vector in output irreps feature depends only one order feature, where 1' is equal to different from I. Hence the SO(3) equivariance still holds for depth-wise tensor product.
Gate As the 0 vector is invariant for D(g), we have
Cate(D(g){{)
(etivation(D(g)
(28)
Activation(
(29)
D(g) Activation( G(te(())(
(30)
(31)
We use the non-linear output from the Activation(P()){E) (i.e. Sigmoid in SE3Set) as the weight which multiplics 1 >> 0 vector to implement the gate function. Thus for I 0, we also have
Gate(D(g)ff,) = APtiv
(32)
Activation(f,, (D)
(33)
G(geGate(TE,
(34)
So the gate function is SO(3) equivariant for all / vectors.
Layer Normalization For = 0, as it is SO(3) invariant, the general layer normalization adapted
LN(
+ B
(35)
(
22

---

Published in Transactions on Machine Learning Research (04/2025)
where C' denotes the channel number corresponding and || denotes the 2-norm for each channel vector. μ is the mean value of fo. Learnable weight and learnable bias | are also deployed. The module is SO(3) invariant as fo is SO(3) invariant. The layer normalization for >> O vector has the following form
RMSc((I(()) )
LN(C))
(36)
We can prove the module is SO(3) equivariant as
((
LN(D())
(37)
RMSC(IP(())PTI
D(g)
(38)
(MSC((IF
D(O)
(39)
F Concepts of irreps features and tensor product
Irreps features The SE3Set model utilizes the special orthogonal group SO(3) to capture three-dimensional rotational symmetries in molecular structures. This approach similar to Equiformer (Liao Smidt 2022 Liao et al. 2023). It employs irreducible representations (irreps) of SO(3), parameterized by an integer I, which correspond spherical harmonics (SH) functions Y These functions imbue feature vectors with rotational information, ensuring the model's equivariance rotations and enabling consistent geometric property analysis. This approach is key to the model's ability to accurately represent and predict molecular and other rotationally invariant systems.
Tensor product To boost the model's expressive power, we consider interactions between irrep features of different angular momenta through the tensor product, which merges two irreps I and l2 into a new irrep with angular momentum 1z. This achieved using Clebsch-Gordar coefficients in an expansion weighted by
Wmy
M/a (5th @ f' ms
Wmi.mz m2 Igmms, ma my flz
da mas
(40)
mi mi.22
To reduce complexity, depth-wisc tensor product @ DTP is adopted from the Equiformer (Liao & Smidt 2022
Liao el. al. 2023), utilizing internal weights to streamline computations. Input-dependent lensor product
weights are denoted as DTP ensuring computational efficiency while preserving equivariance for feature interactions.
G Results of MD17
The MD17 dataset (Chmicla et al. 2017] features wide variety of molecular configurations simulated 500 K, with high-resolution trajectories and labeled with energies and forces from the PBE+vdW-TS
method (Perdew et al. 1996 Tkatchenko et al. 2012 SE3Set's performance on this dataset is shown in
Table 0 SE3Set outperforms Equiformer in accuracy, highlighting its refined force calculation capabilities. In small molecular systems, higher-order many-body interactions are less pronounced, and as result, SE3Set does not significantly outperform other state-of-the-art (SOTA) models.
Complexity Analysis
The computational complexity mainly depends on the V2E module and the E2V module.
The V2E module aggregates the information from each atom in one fragment generate atom-wiss wise hyperedge features. Considering the system is split into fragments and each fragment has n, atoms, this module
23

---

Published in Transactions on Machine Learning Research (04/2025)
Table 6: A comparison of Mean Absolute Errors (MAEs) across various benchmarked models. SE3Set is trained on the MD17 dataset with a configuration of 950 training samples and 50 validation samples. Bolding shows the best model and underlining shows the second best model and the underlining tilde shows third best model. The MAEs reflect the precision of energy predictions in units of kcal/mol and forces in units of
kcal/ (mol-A)). The results of each baseline come from the corresponding articles (Schütt et al. 2017
Gasteiger
et al.
2019
Schütt et al.; 2021 Thölke & De Fabritii
2021 Gasteiger et al. 2021 Batzner et al. 2022
Wang et al.
2024
Liao & Smidt
2022
Wang et al. 2023
ScaNET
DISENET
PAINN
ET
GRMNET
NEQUIP (1-3)
VISNET
QUINNET
EqUIrOMERR
SE3Ser
ASPIRIN
ENERGY Foncn:
0.37
0.204
0.167
0.123
0.131
0.116
0.119
9.122
0.130
1.35
0.499
0.338
0.253
0.217
0.184
0.155
0.145
0.152
0.153
ETHANOL
ENERGY 0.08
0.064
0.064
0.062
0.051
0.051
0.050
0.051
0.054
FORCE
0.39
0.230
0.221
0.109
0.085
0.071
0.080
0.060
0.067
0.002
MALOKALDERYDE
ENERGY FORCE
0.13
0.104
0.091
0.077
0.076
0.075
0.078
0.074
0.074
0.66
0.383
0.319
0.169
0.155
0.129
0.100
0.097
0.125
0,103
ENERGY 0.16
0.122
0.116
0.085
0.113
0.085
0,101
0.085
0.113
NAPITHALRNI
FORCE
0.58
0:215
0.077
0.061
0.051
0.039
0.039
0.039
0.046
0.039
SAMOCIIC ACID
ENERGY
0.20
0.134
0.116
0.083
0.106
0.092
0.101
0.099
0.108
FORCE
0.85
0.374
0.195
0.129
0.125
0,090
0.084
0.080
0.090
0,090
TOLUENE
ENERGY
0.12
0.302
0.095
0.074
0.092
0.074
0.080
0.085
0.093
FORCE
0.57
0.216
0.094
0.067
0.060
0.046.
0.039
0.039
0.048
0,046
URACI
ENERGY
0.14
0.115
0.106
0.095
0.104
0.095
0
0.099
0.103
FORCE
0.56
0.301
0.139
0.095
0.097
0.076
0.062
0.062
0.076
0.087
includes Σ." nn(n, - 1) pair-wise messages for the attention architecture. Actually, this number depends on the fragment hyperparameter mmin and Amazz in particular for the explicit overlap method, also on Cur- For the explicit overlap method, this module would have higher complexity than the implicit overlap method because the explicit overlap has higher average number of atoms in each fragment.
For each atom, the E2V module aggregates all the hyperedge features that an atom possesses. Thus the calculated pair-wise message number depends on the number of fragments per alom shared, corresponding to the introduced overlap degrees. Then the computational complexity of the explicit overlap method in this module depends on the Cw for the explicit overlap method and ra for the implicit overlap method.
To give general approximate complexity, we consider system with N atoms. When using the explicit overlap method, if the system has fragments with average пex atoms in one fragment, the V2E modulc's complexity will be 0(mm2xp) according to the previous analysis. For the E2V module, the number of fragments to which each atom belongs on average can be represented as mnxxp/ /N. Thus, the complexity will be O(mnexp) when considering N atoms.
Similarly, for the implicit overlap method, the complexity the V2E module will be O(1 where nf represents the average number of neighbor fragments within cutoff per atom and napp denotes the average atoms in one fragment. And the complexity of the E2V module is On,
Compared to the explicil overlap method, the implicit overlap method has lower computational complexily. As we can approximate that 7=P & Rexp (overlap degree is similar) and determine m7 = N (implicit overlap method has same fragment number m but without extra count of atoms in 7mmp)- On the other hand, we can determine mmxepp > as the explicit overlap will count atoms more times. Hence the explicit overlap method has higher complexity than the implicit overlap on the V2E Module according to the deduction below
mixepp > NTeppp & nytlimp N mnxxp mmyfiimp R n
2
(41)
(42)
To better support the analysis, we have tested the inference speed of SE3Set on part of QM9 and MD22 datascts as shown in Table 7 Larger fragments, with smaller cw, reduce speed, more so in larger molecules. Speed drops with increased fragment size (smaller eu or larger re), and the explicit overlap method slower than the implicit, leading to our preference for the latter in MD22. This highlights the need to optimize
24

---

Published in Transactions on Machine Learning Research (04/2025)
Inference Speed (its)
25
20
5
Ac-Ala3-NHMe
DHA
Stachyose Molecule
AT-AT
AT-AT-CG-OG
SE3Set
NequIPicutoff=6.C.. À)
NequIP(cutoff=4.00
Equiformer
MACE
Figure 6: Inference speeds of different models the MD22 dataset in units iteration/s. The average inference speeds were calculated for each molecule dataset. All tests were performed on single Tesla A100 80G. The SE3Set includes 3 layers and different cutoff is set (4 Aand 6 Á) for NeuqIP. The other models. The other models use parameter configurations that correspond to the results reported in their respective studies.
Iragment size for a trade-off between delailed interaction capture and speed, particularly big molecular systems. Additionally, the explicit overlap method slower than the implicit overlap method, which confirms our previous analysis.
Table 7:
The inference speed (unit: iterations/s) of SE3Set with different Go re- The tests on the MD22 dataset only use the AT-AT-CG-CG subset.
EXPLICIT Cu
IMPLICIT re
DATASET
0.1
0.05
0.01
4.0
5.0
6.0
QM9
11.60
11.54
11.37
INPERENCE SPEED MD22 (AT-AT-CG-CG)
2.62
2.29
1.68
2.92
2.90
2.87
Moreover, we have evaluated the inference time with different baseline models. The results in Fig. e show SE3Set runs slightly slower than the other models. This is reasonable as we have included more pairwise node information in hypergraph. The improvement, in model's performance indicates that model requires further optimization to achieve better balance between efficiency and performance.
The number of parameters of SE3Set and other baseline models are showed in Fig. SE3Set achieves better performance using fewer parameters.
Table 8: Number of parameters with different models.
MODEL
VISNET
QUINNET
SE3SET(6 LAYERS)
SE3SET(3 LAYERS)
No. OF PARAMETERS (MILLIO))
10
9
6.3
3.5
Training details
This section outlines the training specifics, encompassing the fragmentation parameters, SE3Sct hyperparam- eters, and certain implementation nuances utilized in our experimental setup.
25

---

Published in Transactions on Machine Learning Research (04/2025)
Our dataset construction is founded PyTorch Geometric (eey & Lenssen 2019) augmented with our fragmentation process (Sec. 4.1p. Due to inconsistencies in molecular topology identified through RDKit's sanitization routine (Landrum et al. 2020), 1,403 data points were excised from the original dataset. We designated 110,000 data points for the training set and 10,000 for the validation set, selected at random.
We used an explicit overlap scheme on the QM9 and MD17 datasets because of their relatively small molecular systems. We implemented two distinct schemes for calculating fragment bond orders, following either Eq. [9 or Eq. 20 The parameters for fragmentation are detailed below. Given that the MD17 molecules relatively not actually utilized. However, present the
small, the merge step in the fragmentation process fragmentation parameters here for the sake of completeness.
Besides, we adopt an implicit overlap scheme on the MD22 dataset and OE62 dataset to reduce computational resource consumption. The details of cutoff ge introduced in 4 can be found in Table I1
On the QM9 and MD17 datascts, our model was trained using a single Tesla V100 GPU with 32GB of memory, except for the 6-layer model employing the exponential bond order the QM9 dataset, which was trained on two Tesla V100 GPUs with 32GB each. For MD22 dataset and OE62 dataset, model was trained on a single Tesla A100 GPU with 80GB of memory.
We selected 1 = 2 for our irreducible representations (irreps) feature, which includes both node and hypergraph features. For the radial basis function (RBF), we utilized Gaussian basis functions or Bessel basis functions for the QM9 dataset (Ruddigkeit et al. 2012 Ramakrishnan et al. 2014) and exponential basis functions for the MD17 dataset, MD22 dataset and OE62 dataset (Chmiela al. 2017p. Details can be found in Table 9 and Table 10
Table 9: Syper--aramettrs for training SE3Set model. In the context of hyperparameter settings for dimensions, the symbols e and are used to denote even and odd parity, respectively.
QM9
MD17
MD22
OE62 VALUR OR DISCRIPTIONS
PARAETTAR
VALUR OR DESCRIPTIONS
VALUE on DESCRIPTIONS
VALUR OR DESCRIPTIONS
OPTIMIZER Panin
ADAMW COSINE
ADAMW COSINE
ADAMW COSINE
ADAMW COSINE
LEARNING RATE SCHEDULER
WARM EPOCHS Omax
MINIMUM LEARNING RATE
1.0 x 10-6
1.0 x 10-6 8 1500
1.0 x 10 8
1.0 10-6
BATCH SIZE
32, 128
32
NUMBER OF EPOCHS
400
WRIGHT
5.0 x 10
1.0 10
1.0 10
1.0 10
DROPOUT RATE
0.1
0.0
D.D
0.0
RBF CUTOFF (A)
42.0
MAX DISTANCE OF USED ATOM PAIRS
BASIS
GAUSSIAN BESSEL
EXPONENTIAL 32
EXPONENTIAL
EXPONENTIAL
NL MIESI OF BASES
128 OR 8
32
32
NUMBER OP BLOCKS
3 on 6
3 on
3
NODE EMREDDING DIMENSION
[[288 0c), [64. 1o), [32, 2c)
HYPEREDDE EMBEDDING DIMENSION
[128, 0c), (64. 1o), (32, 2c)
ATTENTION HEAD DIMENSION
[322 0e), [16, 10). [8, 2c)]
FEED FORWARD DIMENSION
(384, De), (192, lo], (96, 2c]]
OUTPUT FEATURE DIMENSION
[[122 0c)
Table 10: on the MD22 dataset because they adapt implicit overlap scheme.
for fragmentation. The expanded threshold does not work for models training
BOND ORDER METHODS
BOND ORDER BY LENDVAY 19
FRAGMENTATION BY EXPONENTIAL 20
MINIMUM ATOMS NUMBER Manin
2
2
MAXIMUM ATOMS NUMBER 7maa
6
6
ISOLATED THRESHOLD (c..)
0.1
0.4
EXPAND THRESHOLD (Cw)
0.1
0.2, 0.4
26

---

Published in Transactions on Machine Learning Research (04/2025)
Table 11: Hxype--araaneter for step
4
of implicit overlap scheme in MD22 and OE62 experiments.
DATASET
MD22
OE62
ACME
DHA
STACHYOSE
AT-AT
AT-AT-CG-CG
DISTANCE CUTOFF Te (A)
5.0
4.0
4.0
6.0
6.0
5.0
Table 12:
Performance comparing with some pre-trained models on 3 tasks (HOMO, LUMO, GAP) of QM9
dataset. We use the average mean absolute error performance here following the article of baselines. All the
basclinc results extracted from the origin articles (Zhou et al. 2023 Ji ct al. 2024
Cao ct al. 2023
MODEL
UNIMOL
UNIMOL-2
INstruCTMol
SE3SET
AvG. MAE (MEV)
127
95
136
19
J Comparison with pre-trained baselines
Pre-trained model has become more popular in molecular representation learning recently, For completeness, have added the performance comparison with some state-of-tho-art pre-trained models in this section. We choose the reported average MAE on 3 targets (HOMO, LUMO, gap) of QM9 as metric, our model performs better than those pre-trained models on this metric as shown in Table [2
K
Comparison with graph neural networks
The hypergraph could incorporate more information than a graph with binary edges thcoretically, as the hyperedge denotes a set of nodes in graphs, which means the graph can be treated as a special case of hypergraph as shown in Fig. 7 (a). To further elaborate on the features and advantages of the hypergraph neural network architecture we use, we show the difference between our model and the graph neural network in this section, using 5 linear arranged nodes as an example Fig. (b)-(d). The black lines indicate the edges in the origin graph. This structure can correspond to the heavy atoms of the molecule malononitrile (NC-CHp-CN). We focus on the message passing from the orange node to the green node in the graph. For a graph neural network, such message passing requires 4 layers. For the graph neural network with the addition of 3-hop node virtual edges, 2 layers are required. For SE3Set, on the other hand, only 2 layers are needed using the explicit overlap method, and only 3 layers are needed for the implicit overlap method, even in the case where the 3-hop nodes are not included inside the hyperedges either. This demonstrates the efficiency of SE3Set in message passing.
L Fragmentation method compared to BRICS
In order demonstrate the difference of our proposed fragmentation method compared to previous methods like BRICS more clearly and intuitively, we use ethylene glycol diacetate as an example molecule for illustration in this section (shown in Figure D To directly show the difference we choose the explicit overlap method. Our fragmentation method preserves the predefined functional cluster substructure and generates overlaps between fragments to achieve message passing between different hyperedges in the hypergraph structure, whereas BRICS does not generate overlaps, which results in the hypergraph structure directly corresponding to BRICS not being able to In addition, BRICS generates molecular fragments based on a chemical reaction library that determines whether to break chemical bonds or not, which may lead to incomplete functional groups fragments, such the carboxyl group in this case.
M Summary Statistics
To further evaluate the performance in our different runs on different targets, we defined the summary statistics std. MAE following Gasteiger et al. (2019) which reflects the average error compared to the standard
27

---

Published in Transactions on Machine Learning Research (04/2025)
(a)
(b)
(c)
V1
V3
16
22
V5
44
(d)
(e)
V1
V3
V6
2
V5
V4
Figure 7: Comparison message passing efficiency of with GNN (graph neural networks), and the GNN adding virtual bodes between 3-hop neighborhoods. The black line between the two circles denotes the binary edge in the graph. We consider the message passing process from the orange node to the green node and the nodes in the shaded area belong the same hyperedge. The red dashed arrows indicate the nodes where the source information reaches after each passing layer. (a) The structural differences between graph (above) and hypergraph (below). The hypergraph contains hyperedges defined by set of nodes. (b) Simple GNN architecture. (c) The GNN architecture adding virtual bodes between 3-hop neighborhoods. hypergraph in SE3Set built with explicit overlap method. overlap method.
(d) The (e) The hypergraph in SE3Set built with implicit
Ours
BRICS
Figure 8: The difference between the proposed fragmentation method (left) and BRICS (right) shown by case molecule ethylene glycol diacetate. For convenience, we use an explicit fragmentation method and set the minimum number of atoms Demin to 0.
deviation of cach target by Eq.
M
N
1 (m)
(m))
std. MAE
Σ
(43)
M
N
Fm
m=
i=1
28

---

Published in Transactions on Machine Learning Research (04/2025)
with target index m in total M targets, dataset size N, the predict value of the model trained by m-th target, and the corresponding ground truth y; (m)
Furthermore, to evaluate the robustness of the performance, especially the MAE, we further propose std. MAE std. by Eq. 44 which reflects the degree of data volatility of MAE on different tasks.
1 M
y) (m)
y (m)
std. MAE std.
std. MAE
(44)
N
Fm
We compute these metrics on QM9 (M = 12), MD17 (M = 7) and MD22 (M 5) respectively. The OE62 is ignored as there is only one task its experiment. For some energy-related labels (Up, U, H, G) in QM9, the atomic level reference energy is subtracted from the origin value generate the ground truth. For MD17 and MD22 datasets, as the forces are calculated by the derivation of energy, which means the energy MAE and force MAE belong the same trained model, we calculate the metrics of energy and force separately with different molecules different targets. All the results are shown in Table 13 To present the results more clearly, we also show the average value and standard deviation of the origin ground truth as well, which are presented in Table
29

---

Published in Transactions on Machine Learning Research (04/2025)
Table 13: The std. MAE and std. MAE std. different datasets calculated across different targets.
MD17
MD22
DATASET
QM9
ENERGY
FORCE
ENERGY
FORCE
STD. MAE (%) STD. MAE STD. (%)
0.67
1.88
0.28 0.13
0.64 0.12
0.20
0.82
0.28
0.05
Table 14: The average ground truth and standard deviation for different targets in the QM9 datasct.
TARGET
HOMO
LUMO
GAP
R² ZPVE
Un
C,
UNIT
eV
eV
eV
a5
eV
eV
eV
eV
eV
mork
kral
Ava.
2.7061
75.1917
-6,5300
0.3027
6.8328
1189.5256
1.0115
-75. -75.9234 9233
-76.3839
-76.8200
-70.6561
31 6009
STD.
1.5304
8.1876
0.6022
1.2772
1.2931
279.7507
0.9054
10.3775
10.4
10.3440
9.5490
4.0624
Table 15: The average ground truth and standard deviation for different molecules in the MD17 dataset. Energy (E) is in unit of kcal/mol and force (F) is in unit of keal/(mol A).
MOLECULE
ASPIIIN
ETHANOL
MALONALDEHYDE
NAPITHALENE
SALICYLIC ACID
TOLCENE
URACIL
E. Ava.
406733.2765
977995.914
-16750L8334
-241898.788,
-311033. 7587
-170223.8472
-200107.3
E STD.
5,9468
4.1920
4.1399
5.5867
5.4163
5.0996
4,9230
F AvG.
0.0000
0.0002
0.0000
-0,0001
0,0000
0,0000
0.0000
F STD.
27 .284
26.2370
28.6374
28.6544
28.5897
27.3297
30.0238
Table 16: The average ground truth and standard deviation for different molecules in the MD22 datasct. Energy (E) is in unit of kcal/mol and force (F) is in unit of keal/(mol- A).
MOLECULE
AC-ALA3-NIMME
DIIA
STACHYOSE
AT-AT
AT-AT-CG-CG
E AvG.
1620662.7117
-631480.1115
-1578838.9200
-115-4896.66 6660
-2329950.
E STD.
8.2038
9.5559
13.7595
10.9584
15. 10.7047
F AvG.
0.0000
0.0000
0.0000
0.0000
0.0000
F STD.
26.0387
25.9614
25.6038
27.9251
27.7151
30


Reviews:
[{"review_id": "07965DRuWX", "reviewer": "Reviewer_YbKk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors developed a new machine learning model that outperforms against others to predict molecular properties.\n2. The authors proposed a new fragmentation method.\n3. The authors provide comprehensive evaluations across various datasets, which shows that the model can handle complex many-body interactions.\n\nWeaker element:\n1. Not a real weak element. I believe it would be interesting to see the transferability of this model. Especially with the usage of fragmentation, I am curious to see how the model trained with small molecules performs on larger molecules (even polymers or proteins?).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript introduces SE3Set, a novel SE(3)-equivariant hypergraph neural network for molecular representation learning that can naturally capture many-body interactions. By including both chemical and 3D structural information in a new fragmentation method, SE3Set outperforms on both small molecule datasets and complex molecule datasets against other state-of-the-art models. The manuscript provides new insights into leveraging hypergraph neural networks with equivariance for prediction of molecular properties.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None. I think the manuscript is in good shape to publish.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zx1TKLuvTX", "reviewer": "Reviewer_QFS8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths: \n1.  results on multiple datasets covering small to medium sized molecules\n2.  have good result on MD22 and OE62 dataset\n3. the downstream experiments and ablation studies are thorough and comprehensive.\n\nweakness\n1. the relatively not SOTA result on QM9 raises some questions. it is against common believe that small molecule should be easier to predict compared to large molecules. please provide further discussions about that.\n2. another question is regrading the  fragmentation, since different size function groups pooled into a single node, molecule with similar atom count could lead to very different node count. Would like to know how the model performance changes with node count instead of atom count.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work focuses on the task of learning molecular representations from 3D molecular structures and proposes a new model called SE3Set. Basically, SE3Set uses hypergraphs constructed based on a fragmentation strategy to represent many-body interactions and then develops an SE(3)-equivariant network to learn from the hypergraph. It’s an extension of hypergraph neural network (which was proposed for graphs without 3D information) and tensor product-based equivariant models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. the relatively not SOTA result on QM9 raises some questions. it is against common believe that small molecule should be easier to predict compared to large molecules. please provide further discussions about that.\n2. another question is regrading the  fragmentation, since different size function groups pooled into a single node, molecule with similar atom count could lead to very different node count. Would like to know how the model performance changes with node count instead of atom count.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c4S6DfAsVM", "reviewer": "Reviewer_UEqG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method stems from clear motivation and the authors have also made it easy to understand the key novelty of their method\n- The proposed method generally achieves satisfying performance, especially on data sets with many large molecules\n\nWeaknesses:\n- The authors can add more explanations on the connection between the proposed method and priors from chemical knowledge", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The contribution of this work can be divided into two parts: the first part is about a fragmentation method on molecules that considers both chemical and 3D spatial information, the other part is about a novel architecture SE3Set, which is based on hypergraph neural networks on the fragmented molecules and incorporates equivariance to better model higher-order many-body interactions. Empirical results on different data sets demonstrate the effectiveness of proposed method, especially for property prediction on rather large molecules.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Generally I think this submission does not need much further revision, though I have one question from the highlighted point 3 in Section 4.1: “To preserving key chemical information for the molecule’s properties and behavior, the fragmentation better only selectively break single bonds and maintain important functional groups and ring integrity”. While I am not an expert in this field, it might be better if the proposed fragmentation method can extend beyond manually defined functional groups and identify some novel and interesting sub-structures. The authors can consider comparing the proposed method with other fragmentation methods that do not enforce such constraints from chemical knowledge and see which one works better. \n\nI also note that some grammatical errors, e.g., “To preserving”->”To preserve” for the above sentence. The authors may need to go through the whole submission and fix all possible typos and grammatical errors.", "broader_impact_concerns": "The authors have added an impact statement sections, though it seems to be directly copied from some templates from ICML. While it may not be a significant issue, the authors are still encouraged to think further on possible broader impacts of better modeling rather large molecules.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are satisified by the revision.
Decision: Accept (poster)