Clifford Group Equivariant Simplicial Message Passing Networks
Authors: Cong Liu
David Ruhe
Floor Eijkelboom
Patrick Forré
Venue: ICLR 2024 poster (2024)
Keywords: Clifford Algebra
Geometric Algebra
Graph Neural Networks
Simplicial Message Passing
Topological Deep Learning
Geometric Deep Learning
Equivariance
Primary area: learning on graphs and other geometries & topologies

Abstract: We introduce Clifford Group Equivariant Simplicial Message Passing Networks, a method for steerable $\mathrm{E}(n)$-equivariant message passing on simplicial complexes. Our method integrates the expressivity of Clifford group-equivariant layers with simplicial message passing, which is topologically more intricate than regular graph message passing. Clifford algebras include higher-order objects such as bivectors and trivectors, which express geometric features (e.g., areas, volumes) derived from vectors. Using this knowledge, we represent simplex features through geometric products of their vertices. To achieve efficient simplicial message passing, we share the parameters of the message network across different dimensions. Additionally, we restrict the final message to an aggregation of the incoming messages from different dimensions, leading to what we term *shared* simplicial message passing. Experimental results show that our method is able to outperform both equivariant and simplicial graph neural networks on a variety of geometric tasks.

Published as a conference paper at ICLR 2024
CLIFFORD GROUP EQUIVARIANT SIMPLICIAL MES- SAGE PASSING NETWORKS
Cong Liu 2, David Ruhe Floor Eijkelboom 14 Patrick Forré AMLab, University of Amsterdam {c. 1iu4, d. ruhe, f.ei jkk boom, p.d. forre)@uva...
ABSTRACT
We introduce Clifford Group Equivariant Simplicial Message Passing Networks, a method for steerable E(m)-equivvriant message passing on simplicial com- plexes. Our method integrates the expressivity of Clifford group-equivariant lay- ers with simplicial message passing, which is topologically more intricate than regular graph message passing. Clifford algebras include higher-order objects such bivectors and trivectors, which express geometric features (e.g., areas, volumes) derived from vectors. Using this knowledge, we represent simplex fea- tures through geometric products of their vertices. To achieve efficient simplicial message passing, we share the parameters of the message network across different dimensions. Additionally, we restrict the final message to aggregation of the incoming messages from different dimensions, leading to what we term shared simplicial message passing. Experimental results show that method is able outperform both equivariant and simplicial graph neural networks on a variety geometric tasks. Our implementation is available on GitHub
INTRODUCTION
Graph Neural Networks (GNNs) have established themselves as effective tools for learning repre- sentations of relational data from variety of domains, including social networks (Pan al. 2019).
bioinformatics (Zitni&& & Leskovec 2017). and physics (Battaglia et al. 2016). Recently, there has been surge of interest in the theoretical foundations of GNNs, with particular focus on their expressive power (Geerts & Reutter 2022) Standard message passing leverages the sparsity of the underlying graph by exchanging 'messages' between nodes only when they are adjacent. As such, nodes with local structures that are alike will learn similar representations, restricting the expressivity of this framework. This limitation has been formalized by Xu et al.] (2019); Morris et al. (2019), showing that Message Passing Neural Networks (MPNNs) are at most expressive as (1-WL) test at distinguishing unattributed graphs. As a consequence, such networks cannot identify graph structures such triangles or their higher-dimensional equivalents (Chen et al. (020).. In response, Bodnar et al. (202I) developed simplicial message- message-passing net- works operating on simplicial complexes, resulting in both theoretical and empirical improvements regarding expressive power. Further, their contributions illuminate mathematical intersections with algebraic and differential topology, as well geometry (Ghrist 2014).
the Weisfeiler-Lehman
In the context of geometric graphs, the focus lies on graphs that are embedded in a geometry, such as metric space or a manifold. These data points are often accompanied by geometric features, like positions or velocities. Such quantities transform predictably. though nontrivially, under rigid opera- tions like rotations, reflections, or translations. E(n) Equivariant Graph Neural Networks (EGNNs) (Satorras et al. 2021) are designed to respect these symmetries by either exhibiting equivariance or invariance characteristic tailored to the specific task. This field has witnessed continual advance- ments (Huang et al. 2022 Thölke & Fabritiis 2022] Finzi et al., 2020 Brandstetter et al. 2022] Batzner et al. 2022 with one of the latest being the introduction of Clifford Group Equivariant Neural Networks (CGENNs): neural network architecture operating on the Clifford (or geometric)
algebra (Ruhe et al. 2023a| Brehmer et al.) 2023). While these methods reap the benefits of geo- metric equivariance, they still limited to the expressive power of traditional message- message-passing.
"Contributed equally. AMML... "Al4Science Lah. Anton Panneknek Institute. "UvA-Boseh Delta Lah.

---

Published as a conference paper at ICLR 2024
Combining the merits of both simplicial message passing and geometric equivariance, Eijkelboom
et al. (2023) developed a method based on EGNNs, called E(n) Equivariant Message Passing Sim- plicial Networks (EMPSNs). However, we can identify two limitations of this method. First, the higher-dimensional simplices are initialized using manually calculated geometric information. Sec- ond, EGNN is an architecture that falls under the umbrella of scalarization methods (Han et al.) 2022), which operate mainly with invariant features and update vector features only through scaling. In this work, we go a step further and develop steerable Clifford algebra-based simplicial message- passing networks: Clifford Group Equivariant Simplicial Message Passing Networks (CSMPNs).
The Clifford algebra contains higher-order elements like bivectors and trivectors. These objects are computed through vector composition and can represent geometric quantities like areas and volumes. Analogously, simplices are also fully defined by their constituent vertices. The geometric product and the Clifford algebra are well-defined for any inner product space of any dimension. This versatility makes the geometric product suitable option for computing simplex features and enables embedding simplices with geometric features across various spaces and dimensions.
As such, we initialize the higher-order simplices using geometric products (as well as linear com- binations) of their vertices. The network then refines these simplices by passing messages between simplices of different order. To do so efficiently, unlike EMPSN, which uses a separate model for each type of communication, various simplex orders, conditioning it on the dimensionalities of the source and target simplices. Additionally, we restrict the final message enabling the use of modern parallelized graph reduction operations in the simplicial setting. We call the resulting algorithm shared simplicial message passing. Equivariance achieved by restrict- ing all message and update neural networks to be Clifford group equivariant networks (Ruhe et al.) 2023a). Experimental results show that our method can outperform both equivariant and simplicial graph neural networks in geometric tasks spanning multiple domains and dimensionalities, includ- ing convex hulls volume prediction task, human walking motion prediction, molecular motion prediction, and NBA player trajectory prediction.
share the parameters of the message passing function across
aggregation of the incoming simplicial messages,
2 BACKGROUND
We provide a brief introduction to Clifford group equivariant neural networks and simplicial com- plexes. For review on equivariant message passing networks, consider Appendix B
2.1 CLIFFORD GROUP EQUIVARIANT NEURAL NETWORKS
We start by introducing the Clifford algebra, also known as the geometric algebra, which is a pow- erful mathematical object with applications in various areas of science and engineering. For full development, we refer the reader to Ruhe et al. (202a)). While the theory has been generally de- veloped, we restrict our attention to the case where the underlying vector space is Rd The Clifford algebra CI(Rd, q) is the "biggest" unitary, associative, such that for all gd, x² = q(v). where q : _d + non-commutative algebra generated by Rd quadratic form. In other words, vectors square to scalars. has an associated bilinear form : Rd x Rd + R. The algebra's 's elements, gen- erally called multivectors, are linear combinations of commuutmtee products of vectors while respecting the quadratic form: x E CI(Rd, .q), = Eier G U/, ... U2 Here, the index set is finite, c, E IR are scalars, and U.j E Rd are vectors. Clifford multiplication is referred to as the geometric product. The Clifford algebra forms a graded vector space, with
d
CI(Rd, .)
① (R) (TR .g). k=
dim (k) (TR. (g)gg)-
c kk)
where and dim CI(RR q) 2d These subspaces are called grades, where elements of k 0 are called scalars, elements of & = 1 are called vectors, elements
= 2 are called bivectors, and so on. A multivector x E CI(Rd q) can thus written as a sum of its grades: x = Σi= (()) where (R()) (1 (RC. .). The operation (-)(+) CI(Rd, .) + (1()) (R() g) is called grade projection, which selects the grade-k component of x. Beyond scalars and vectors, which only express magni-
tude and direction, respectively, bivectors express oriented area, trivectors express oriented volume,
and forth. Note that we really have the identifications: (TEd) ( q) R and (1(1) .g) Rdd.
2

---

Published as a conference paper at ICLR 2024
p(w)
Rotate
p(w)
Rotale
Figure 1: Illustration of our proposed architecture. Top left: set of vertices (and edges) is lifted to a simplicial complex. We highlight three simplex types: vertices (0-simplices. ) edges (1- simplices, - ), and triangles (2-simplices, A ). In this case, the vertex feature is vector-valued and embedded as the grade part of Clifford algebra element: multivector. three dimensions, multivector has scalar (O), vector ( >)), bivector f and trivector ( ( components. Higher-order simplices are initialized using the geometric product of their constituent vertices. As such, edges in the top left visualization are bivector-valuedt and triangles are trivector-valued. The simplicial message- pressage-assing framework, denoted by é, refines the cealtiveccto-valuuce simplices, as portrayed in the bottom-left, by passing messages between simplices of different order. Crucially, Φ maintains equivariance to the Clifford group's orthogonal action p(w), representing rotation here. In doing so, our method ensured to respect the geometric symmetries of the input data.
Ruhe (203a)) then define the Clifford group I q). Its elements are invert- ible homogeneous" multivectors of CI(R', that preserve vectors under the group action p(w): CI(Rd + CI(Rd ,q), called the twisted conjugation. That is, if x (1 9) Rd then (x) e (Rd q) Rd.
further shown that all p(w) preserve the quadratic form Therefore, each p(w) defines an orthogonal automorphism. Let F E R(TT..... Tel be polyno- mial (non-comen commutaattee, where multiplication performed by the geometrie product) in variables, T(Rd g). and Ti,..., Ie E CI(Rd. g). We then have the following equivariance properties (Ruhe et al. (203aa).
Theorem 2.1 (All polynomials are Clifford group equivariant). Let F E TT...... Te) be polynomial l variables with coefficients in IR, [(R .9) Further, consider é elements IT. E CI(Rd, .). Then we have the following equivariance property:
p(w) (F(x1...., = ((p(w))())).....pee
(2)
Theorem 2.2 (All grade projections are Clifford group equivariant). For w E T(Rd, .). e CI(RA, .4 and ls D....,d have the following equivariance property:
(z=) (z))
(3)
In particular, for x e (1))) .q) we also have (x))(x) E (R
(Rd
That is, polynomials in multivectors, as well as their grade projections, are Clifford group equivari- ant, and therefore equivariant the orthogonal group. It is worth mentioning that p(w) not only preserves vectors, but is generally
Using these two properties, Ruhe et (2023a) then introduce several linear, bilinear (through the geometric product), and nonlinear equivariant neural network layers. Composing these layers, we obtain a Clifford group equivariant neural network.
We mean homogeneous with respect to parity. That is, multivectors that only have nonzero grades (k: mod 2) nonzero odd grades.
3

---

Published as a conference paper at ICLR 2024
Several other works that incorporate geometric algebra in deep learning (Melnyk et al. 2021
Spellings 2021; Brandstetter et al. 2023; Ruhe et al., 2023b). Finally, Brehmer et al. (2023 intro-
duce the geometric algebra transformer, method that uses the projective geometric algebra (Gunn 2016) in combination with the transformer (Vaswani et al. (017) architecture.
2.2 SIMPLICIAL COMPLEXES
Definition 2.3 (Simplicial Complex). Let V be a finite set. An abstract simplicial complex K is subset of the power set 2 that satisfies:
1. Vo e Eu
2. Vo E : CAT
In words, it is space formed by collection of subsets a V, called simplices, which always including all the singletons, such that if a E K and C a, then T E K as well. In other words, K closed under taking subsets. For example, if a triangle (dimension 2 simplex) is in K, then all its edges and vertices (dimensions 1 and O, respectively) are also in K. We also have dim ø := |o] and dim K := maxfdin a a K}. Note that the 1-skeleton consisting of only vertices and edges, showing that simplicial complexes are a natural generalization of graphs. The act of creating a simplicial complex from lower dimensional ones, like vertex set, is called lifting transformation. By constructing rich adjacency structure on the simplicial complex, we can model interactions not only between vertices, but also between higher-dimensional simplices. E.g. triangle in simplicial complex can be thought of as a meta-veriex that represents the interaction between its three vertices. [Bodnar et al. (2021) introduce message passing networks on simplicial complexes lifted from regular graphs. Exploring the connectivity within a simplicial complex, they identify the following adjacency types.
graph simplicial complex
Definition 2.4 (Simplicial Adjacencies). Let us define the boundary relation T := T C dim = dim o 1. We define the following adjacency structures of a simplicial complex K:
1. Boundary udjacencies B(a) = {r E K <o);
2. Coboundary adjacencies C(a) = (T K T o};
3. Lower adjacencies (1(a) = (T E K
38 E K : 5 < < < a):
4. Upper adjacencies N+(a) == {r E
E : K
8
For example, if a is triangle, then B(o) is the set of edges that make up the triangle. Lower adjacencies are simplices of the same dimensionality but with common lower boundary, while upper adjacencies share common upper boundary. Note that regular message passing uses only the upper adjacencies of nodes. Simplicial message passing networks satisfy the following theorems.
Theorem 2.5 (Bodnar et al. (2021).. Simplicial Weisfeiler-Lehman (SWL) with a clique complex lifting is strictly more powerful than Weisfeiler-Lehman (WL.), and is not less powerful than 3-WI..
Theorem 2.6 (Bodnar et al. (202T)). Message Passing Simplicial Networks (MPSNs) with suffi- cient layers and injective neighborhood aggregators are as powerful as SWL. Moreover, MPSNs are strictly more powerful than WI.
The Weisfeiler-Lehmal algorithm (Leman & Weisfeiler, 1968) is a well-known graph isomorphism test. Here, clique complex lift turns every clique a graph into simplex in simplicial complex. We depict two isomrrphic geometric graphs in Appendix C
3 CLIFFORD GROUP EQUIVARIANT SIMPLICIAL MESSAGE PASSING NETWORKS
3.1 CREATING GEOMETRIC SIMPLICIAL COMPLEXES
Starting with a finite set V, potentially part of a (geometric) graph G = EV, several ways exist to construct simplicial complex K. For instance, form simplicial complex by creating
4

---

Published as a conference paper at ICLR 2024
simplex for every possible subset of V. That is, we include all edges, triangles, tetrahedra, and so on. However, the resulting simplicial complex would have (.41) number of n-simplices. One can easily see that this number can quickly become prohibitively big.
As such, various methods exist to lift a set of points to a simplicial complex in a more tractable way. Examples are Vietoris-Rips, Čech, manual and algorithmic lifts. Furthermore, we have access to graph structure, we can use a clique lift. For a more detailed discussion of these, consider Ap- pendix[D To create a Vietoris-Ripp lift, we consider a node position x" E X attached to V, together with a distance function d: X x X + R. Then, the Vietoris-Rips complex Vietoris- (V, d, e) is the simplicial complex containing all simplices whose vertices are c-close for some e > (see Figure 55. The manual lift defines a simplicial complex by hand. For example, shape of a polyhedral object by its vertices, edges, and faces, making a simplicial complex also known mesh. Another alternative is let expert knowledge guide us. Take, for example, the molecule of water HzO. The bond angle between the two hydrogen atoms, governed by triangular interaction involving all three atoms, crucial for the molecule's 's properties. Finally, recent work has been done on algorithmically constructing simplicial complexes from data, for example, through the mapper procedure (Hajij et al., 2018). In this work, we use the Vietoris-Rip and manual lifts in our experiments. We typically cap the maximal simplex dimension to 2 for efficiency reasons.
define the
3.2 EMBEDDING SIMPLICIAL DATA IN THE CLIFFORD ALGEBRA
In the following, we elaborate on how we embed the usual scalar and vector features of each node V E V in the Clifford algebra, as well as how we create simplex features.
First, we have node features Ve V he E R* @ (R4) Here, let hj hi...., hg IR denote scalar features, or invariants. Consider, for example, particle's mass or charge in a physics setting, They transform trivially under the Clifford group action. On the other hand, let i........ h2 R" be vector features, such as position, velocity, and acceleration. They transform nontrivially un- der the Clifford group action. One of these is usually the position of the vertex, which (e.g.. through distances) is used to construct a simplicial complex. Recall that the Clifford subspace (1 (R' .)) IR is the subspace of all scalars, and therefore we can embed our scalar features in this subspace. C1 (Rd 9) = Rd is the vector subspace, and therefore we can embed our vector features here. Further, if we have access to higher-order features such as bivectors, we can embed them in the other subspaces of the Clifford algebra. As such, after embedding, we now denote Ve E V : 1t" CI(RS,q)" where m = k l
We now consider simplicial complex K lifted from V. Our goal is, analogously to the node fea- tures, to obtain a Clifford feature h" for each simplex E K. For the singletons {u,} E K, we can directly put h(*² == ht For the edges {v., 0z} E K, we can put hiveey) := h,,, de- noting the geometric product of the two Clifford features. This process extends to triangles and nigher-dimensional simplices, multiplying Clifford features of all vertices with geometric product. In Figure | we depict this embedding, illustrating how edge and triangle simplices relate to bivector- and trivector-valued features. Since there are multiple ways embed simplices, we learn the em- bedding through Clifford group-equivariant layers, i.e., we take the Clifford simplicial features as input learnable Clifford group-equivariant layers and use the outputs the learnable Clifford simplicial features. These can be decomposed into parameterized linear combinations as well as parameterized geometrie products, resulting in analogous embeddings to the ones described above, but including learnable parameters. To ensure permutations of geometric products. A more intricate approach involves passing messages between the vertices of a k-simplex. The aggregated readout is then used as the initialized feature for the k-simplex.
embeddings, one can aggregate
Eijkelboom et al. 2023) take a similar approach as here, but they manually have to define the sim- plicial embeddings for all simplices. Specifically, distances, volumes, and angles between simplices are calculated. All such quantities can be expressed using linear combinations and geometric prod- ucts (Doran& & Lasenby| 2003), which are computed by Clifford group equivariant layers. Moreover, Eijkelboom et al. (2023) can only embed geometrically covariant information through relative posi- tions, which are computed additively. In contrast, the use of geometric products and the fact that any polynomial in multivectors is Clifford group equivariant, we can also embed covariant information
5

---

Published as a conference paper at ICLR 2024
Figure 2: Left: show how simple graph (three fully-com collec-ed nodes) is lifted to simplicial complex. Using simplicial message passing, we allow communication between objects of different dimensions. That is, between vertices (0 0)
nodes and edges (0 and 1 0)
edges (1 <> 1) ------ and between edges and triangles (1 - 2 and + 1) as left, but top-down view. illustrates the hypergraph associated with the complex with several meta-vertices representing the simplices of various dimensionality. Instead of running message passing separately for all different communication types, we share the parameters of single neural network operating on the extended graph. By conditioning on the message type, it is still able to leverage the simplicial complex.
Right: same
extracted from three or more node positions multiplicativell. Since these networks generalize to inner-product spaces of any dimension, these notions also hold for Clifford algebras over higher- dimensional spaces.
3.3 EQUIVARIANT SHARED SIMPLICIAL MESSAGE PASSING
have Clifford feature he CI(R* g)"h. We propose two techniques that enable efficient (equivariant) message passing on simplicial complexes. In doing so, we require access parameterized message function ddhh and update function Φ² First, the message m will he an equivariant aggregation of all information (processed by a neural network) from several adjacencies of different dimensions as defined in Definition 2.4 In contrast to, e.g., Eijkelboom et al. )2023); Bodnar et al. (2021), who iteratively run message passing for different adjacency types, can leverage existing parallel implementations of classical message passing. In other words, consider the adjacency matrix of the simplicial complex's 's corresponding hypergraph, where we have several mertivertices that represent the different types simplices. This corresponds considering the O-simplices of the barycentric subdivision of the simplicial complex, which is a common way for refining simplicial complexes (Ghrist. 2014). This idea visualized Figure
For all E K,
Secondly, instead of considering a different pa- rameterization for each type of communication, define single message funcctio 6" that can handle all types of communication. How- ever, by conditioning nt the type of mes- sage, it can still leverage the simplicial com- plex. In doing so, efficiently share parame- ters between different types of communication, which is in contrast with previous methods that defined a different neural network for each type of communication.
Algorithm Shared Simplicial Message Passing
Require: K, Vo E hh" 60 Φh
Repeat:
m + Agg TEB()) (1², hh,, dim o, dim TEC(e)
(e
[
haT t- 6h (he m² dimo))
'Intriguingl, Bodnar et al. 2021 prove that only the boundary and upper adjacencies are required for full expressivity. However, only use the boundary, coboundary, and upper adjacencies keep consistent with Fijkelboom al. 2023

---

Published as a conference paper at ICLR 2024
MSE (4)
MPNN (Gilmer et al.
2017)
0.212
GVP-GNN Jing VN (Deng et al. 2021
2021
0.097
0.046
EGNN (Satorras et al.
2021)
0.011
CGENN (Ruhe et 2023a)
0.013
EMPSN Eijkelboom et al.
2023)
0.007
Figure 3: In the convex hulls experiment, the
CSMPN
0.002
task is to estimate the volume of the convex hull of eight five-dimensional random points. Here, we display three-dimensional exam- ple, which is easier visualize.
Table I: MSE (L) of the tested models on the convex hulls experiment.
To make the overall method equivariant, utilize Clifford group equivariant neural networks from Ruhe et al. (2023a). Then, as long as the simplicial embedding, the aggregation operation (e.g., a summation). and 6th and & are equivariant, the overall method is Clifford group equivariant (see Appendix B) This then makes it equivariant to rotations, reflections, and other orthogonal transfor- mations in any dimension. The algorithm is summarized in Algorithm I Note that it generalizes typical message passing, which only considers messages from the upper adjacencies between O- simplices, i.c. nodes. For a quick overview of how this compares to typical message passing and simplicial message passing, consider Appendix F
EXPERIMENTS
We selected of geometric experiments that involve different types of data from several domains and include both invariant and equivariant predictions. Currently, the QM9 (Ramakrishnan et al. 2014) and MD17 (Chmiela et al., (017) datasets are highly popular benchmarks for equivariant graph neural networks. We found, however, that the state-of-the-an art models incorporate a lot of domain knowledge, which beyond the scope of the current work. Note that we ensure a fair comparison by maintaining similar scale of parameters between CSMPN and the baseline models across all experiments. More experimental details than presented here can be found in Appendix
4.1 5D CONVEX HULLS
We run this experiment based on the convex hull volumetric experiment of Ruhe et al. (2023a). We consider a five-dimensional space, where we sample eight points from standard normal distribution. The task is to estimate the volume of the convex hull of these points. We give an example of the dinee-dimeniional case in Figure 3 Note that this is an E(5)-invariant task. For the simplicial networks (EMPSN (Bijkelboom et al. 2023) and ours), we use the representation of the hull as a of 4-simplices, and extract all their 1- and 2-simplices. We then pass messages between these as well as the O-simplices (the points). As shown in Table CSMPN outperforms the other models, showing that in this setting the simplicial structure is significantly improved way of presenting the data to the network.
4.2 CMU MOTION CAPTURE
In this experiment, we evaluate our models on the CMU Human Motion Capture dataset (Gross Shi 200T). We demonstrate that CSMPN exhibits greater performance motion prediction compared to equivariant architectures reliant on regular graphs. In agreement with, e.g.. Huang cet al. (0022), we use the 35th human subject of the dataset. Each graph in the dataset contains 31 equally connected nodes, with each representing specific position on the human body during walking. The objective is to use the node positions of a random frame to predict the node positions after 30 timesteps. We manually lift the graph to simplicial complex. Specifically, connect elbow joint nodes with shoulder and palm nodes to create edges and triangles. Similarly, hip, knee, and heel nodes are linked, forming another set of edges and triangles.
7

---

Published as a conference paper at ICLR 2024
Targets
Predictions
Method
MSE (4)
Radial Field (Köhler et al.
2020)
197.0
TEN TThoas et al. 2018
66.9
SH(3)- (Fuchs Fuuhs
2020)
60.9
GNN Gilmer et al.
67.3
EGNN (200K) Satorras et al.
202
31.7
GMN (200K) (Huang al. 2022
17.7
EMPSN (200K)
15.1
CGENN (200K)
9.41
CSMPN (200K)
7.55
10
Table Left: MSE (10-2) of the tested models on the CMU motion capture dataset. Right: Depic- tion (not cherry-picked/ of an instance (the ground-truth target positions) vs. CSMPN prediction.
Aspirin
Benzene
Ethanol
Malonaldehyde
Radial Field (Köhler et al.
2020)
17.98 / 26.20
7.73 12.47
8.10 / 10.61
16.53 / 25.10
TFN (Thoma al. 2018
15.02 / 21.35
7.55 12.30
8.05 / 10.57
15.21 / 24.32
SE (ruchs et al. 2020)
15.70 / 22.39
7.62 12.50
8.05 10.86
15.44 / 24.47
EGNN Satorras et al.
14.61 / 20.65
7.50 12.16
8.01/
15.21 / 24.00
S-LSTM Alahi et al. 2016
13.12 18. 14
3.56
7..8
11.93 / 18.43
NRI (Kipf et al. 2018
12.60 / 18.50
1.89
6.69/
12.79 / 19.86
NMMP (Hu et a... 2020)
10.41 14.67
2.21/3.33
6.17 77.86
9.50 14.89
GroupNet (Xu et al. 2022
10.62 / 14.00
2.09
6.00 7.88
7.99 12.49
GMN-L (Huang et al. 2022
9.76 /-
48.12/-
4.83 /-
13.11/-
EqMotion (300K) (Xu et al.
2023
5.95 / 8.38
1.18 1.73
5.05 / 7.02
5.85 / 9.02
EMPSN (300K)
9.53 12.63
1.03 1.12
8.80 9.76
7.83 10.85
CGENN (300K)
3.70 / 5.63
1.03 1.59
4.53 / 6.35
4.20 6.55
CSMPN (300K)
3.82 5.75
1.03 1.60
4.44/ 6.30
3.88 / 5.94
Table 3: ADE FDE (10 ?) (J) of the tested models the MD17 atomic motion dataset.
We incorporated all baselines from Huang et al. (2022) and added EMPSN and CGENN ourselves. From Table 2] we note that EMPSN, the simplicial version of EGNN, already enjoys significant boost from simplicial message passing. On top of this, Clifford layers further improve the perfor- mance, which CSMPN then again surpasses. We ensured that model sizes, in terms the number of parameters, are comparable.
4.3 MD17 ATOMIC MOTION PREDICTION
We turn to the molecular domain with the MD17 dataset Chmiela et al. (2017). However, we do use the standard task of predicting the energy of molecule, but instead we predict the motion of the atoms. In accordance with Han et al. 2022), we select four molecules: aspirin, benzene, ethanol, and malonaldehyde. The aim to assess how well CSMPN can model molecular dynamics by directly predicting atom positions in future time steps. We take the starting positions of the heavy atoms from ten separate time frames for each molecule and predict their positions in the next ten frames. For aspirin, the atom positions. The other molecules are fully connected. Since we have access to this graph structure, use clique complex lift form the simplicial complex.
k-nearest neighbors with k 3 to construct the regular graphs from
The Average Displacement Error Final Displacement Error (ADE / FDE) of the methods are dis- played in Table Here, ADE is the RMSE of the location predictions averaged over the number of
8

---

Published as a conference paper at ICLR 2024
Attack
Defense
STGAT (Huang et al. 2019
9.94 / 15.80
7.26 / 11.28
Social-Ways (Amirian al. 2019)
9.91 / 15.19
7.31 / 10.21
Weak-Supervvsson (Zhan et al.
201 9
9.47 / 16.98
7.05 / 10.56
DAG-Net (200K) (Monti et al.
2020
8.98 / 14.08
6.87 / 9.76
CGENN (200K)
9.17 / 14.51
6.64 / 9.42
CSMPN (200K)
8.88 14.06
6.44 9.22
Table 4: ADE/ FDE (4) of the tested models on the VUSport NBA player trajectory dataset.
time-steps, and FDE the RMSE of the final time-step. We take all baselines from Xu et al. (2023) and included EMPSN and CGENN ourselves. Inspecting the results, we see that simplicial message passing yields a significant boost for EMPSN, the simplicial version of EGNN. Clifford layers then further improve the performance, the only exception being the case of benzene. We hypothesize that the restricted expressivity of EMPSN forms a good inductive bias here, since the molecule is rigid and planar. CSMPN achieves outstanding scores across the board, slightly surpassing clifford the cases of ethanol and malonaldchyde. We ensured that the simplicial structure used in the comparison of EMPSN and CSMPN is equal, and that the number of parameters is comparable. Similarly, the architecture of the CGENN and CSMPN models is roughly identical.
4.4 NBA PLAYERS 2D TRAJECTORY PREDICTION
Finally, we subject our CSMPN to testing on the STATS SportVU NBA Dataset (STATS Perform 2023). This two-dimensional dataset contains the tracking positions of the NBA team players in regular seasons. Our preprocessing approach aligns with Monti et al. (2020), representing each player's position as a two-dimensional coordinate. Considering the distinct behaviors of players in offensive or defensive roles, we assess our model on these different contexts, utilizing ten observed time frames to predict the subsequent forty. Motion uncertainty and interactions between players make the task challenging. Each player's position is represented as a node in our graph, each node connected all the other nodes in the graph to make the regular graph fully-connected. We also include one fixed reference point in our graphs indicating the orientation of the basketball court. We create the simplicial complex using Vietoris-Rip with infinite with a maximum simplex dimension of 2. We compare our CSMPN with baselines provided by Monti et al. (2020) as well as CGENN and present the results in Table
5 CONCLUSION
We presented Clifford Group Equivariant Simplicial Message Passing Networks (CSMPNs), a class of Clifford algebra-based neural networks that are E(n)-equivariant and operate simplicial com- plexes. Our method links the combination of vertices to form higher-dimensional simplices to the combination of vectors to form multivectors, combining the expressiveness of Clifford group equiv- ariant neural networks with the topological intricacy of simplicial message passing. To do so effi- ciently, we reduce simplicial message passing to message passing on a hypergraph. but condition on the dimensionality of the source and target simplices. As such, we can share the parameters of the message passing functions across various simplex orders, thereby maintaining the topological struc- ture of the complex. Experimental results showed that CSMPNs can outperform both equivariant and simplicial graph neural networks on a variety of tasks. In some cases, however, the perfor- mance of CSMPNs is comparable to that of the other methods. Future research might explore which scenarios specifically benefit from utilizing simplicial geometric message passing. While a limita- tion to method is the increased computational cost of simplicial message passing, have already made significant strides by sharing parameters across simplex orders and are optimistic about future improvements in this direction.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The author C.L. was financially supported by Health-Holland, Top Sector Life Sciences Health (LSH-TKI), project number LSHM22023, which realizes public-private partnership (PPP) be- tween the University of Amsterdam and Janssen Vaccines Prevention B.V. We thank SURF (www.surf.nl) for the support in using the National Supercomputer Snellius.
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social LSTM: Human Trajectory Prediction in Crowded Spaces. In Computer Vision and Puttern Recognition (CVPR), pp. 961-971, 2016.
Javad Amirian, Bean-Bernard Hayet, and Julien Pettre. Social ways: Learning multi-modal distribu- tions of pedestrian trajectories with gans, arXiv, 2019.
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction Networks for Learning about Objects, Relations and Physics. In Con- ference Neural Information Processing Systems (NeurIPS), pp. 4502-4510, 2016.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P. Mailoa, Mordechai Ko- rnbluth, Nicola Molinari, Tess E. Smidt, and Boris Kozinsky. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications, 13(1), 2022.
Cristian Bodnar, Fabrizio Frasca, Yuguang Wang, Nina Otter, Guido F. Montúfar, Pietro Lió, and Michael M. Bronstein. Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networks. In International Conference on Machine Learning (ICML), volume abs/2103.03212.
pp. 1026-1037, 2021.
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Ge- ometric and Physical Quantities improve E(3) Equivariant Message Passing. In International Conference on Learning Representations (ICLR), 2022.
Johannes Brandstetter, Rianne van den Berg, Max Welling, and Jayesh Gupta. Clifford Neural Layers for PDE Modeling. In International Conference on Learning Representations (ICLR), 2023.
Johann Brehmer, Pim De Haan, Sönke Behrends, and Taco Cohen. Geometric algebra transformers. arXiv preprint arXiv:2305. 18415, 2023.
M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velivckovi'e. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. arXiv, 2021.
Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can Graph Neural Networks Count Substructures? In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Stefan Chmiela, Alexandre Tkatchenko, Huziel Sauceda, Igor Poltavsky, Kristof Schütt, and Klaus-Robert Müller. Machine learning of accurate energy seeggycconserving molecular force fields. Science Advances, 3(5), 2017.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J. Guibas. Vector Neurons: A General Framework for Equiv-aqqi Networks, In IEEE Inter- national Conference on Computer Vision (ICCV), pp. 12180-12189, IEEE, 2021.
Chris Doran and Anthony Lasenby. Geometric Algebra for Physicists. Cambridge University Press, 2003.
Floor Eijkelboom, Rob Hesselink, and Erik Bekkers. E(n) equivariant message passing sim- plicial networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engel- hardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Con- ference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 1071-9081. PMLR, 23-29 Jul 2023. URL https: http::///prceeedingg. mlr ptw.sp//esss2222. e jkke beellboom2a... .ttml
10

---

Published as a conference paper at ICLR 2024
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph Neural Networks for Social Recommendation. In The World Wide Web Conference. ACM, 2019.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv: 1903.02228 2019.
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing Convolu- tional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data. In Inter- national Conference on Machine Learning (ICML), pp. 3165-3176, 2020.
Fabian Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. Transfrranformmrrs: 3d Roto- Translation Equivariant Attention Networks. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
F. Geerts and Juan L. Reutter. Expressiveness and Approximation Properties of Graph Neu- ral Networks. In International Conference on Learning Representations (ICLR), volume abs/2204.046on 2022.
Robert W. Ghrist. Elementary Applied Topology, volume 1. Createspace Seattle, 2014.
Justin Gilmer, Samuel Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp- 1263-1272. PMLR, 2017.
Ralph Gross and Jianbo Shi. The emu motion of body (mobo) database. Technical Report CMU- RI-TR-01-18, Camegie Mellon University, Pittsburgh, PA, June 2001.
Charles G. Gunn. Doing Euclidean Plane Geometry Using Projective Geometric Algebra. Advances in Applied Clifford Algebras, 27(3): 2016.
Mustafa Hajij. Paul Rosen, and Bei Wang. Mapper on graphs for network visualization. arXiv,, 2018.
Jiaqi Han, Yu Rong, Tingyang Xu, and Wenbing Huang. Geometrically equivariant graph neural networks: A survey, arXiv,, 2022.
Yue Hu, Siheng Chen, Ya Zhang, and Xiao Gu. Collaborative Motion Prediction via Neural Motion Message Passing. In Computer Vision and Puttern Recognition (CVPR), pp. 6318-6327, 2020.
Wenbing Huang, Jiaqi Han, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Equivariant Graph Mechanics Networks with Constraints. International Conference on Learning Repre- sentations (ICLR), 2022.
Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. Stgat: Modeling spatial- temporal interactions for human trajectory prediction. In 2019 IEEE/CVF International Confer- on Computer Vision (ICCV), pp. 6271-6280, 2019. doi: 10.1109/CCV2221.00066...
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron O. Dror. Learning from Protein Structure with Geometric Vector Perceptrons. In International Conference on Learning Representations (ICLR), 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv, 2017.
Thomas N. Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard S. Zemel. Neural Relational Inference for Interacting Systems. In International Conference on Machine Learning (ICML), pp. 2693-2702, 2018
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant Flows: Exact Likelihood Generative Learn- ing for Symmetric Densities. In International Conference on Machine Learning (ICML), pp- 5361-5370, 2020.
Andrey Aleksandrovich Leman and Boris Weisfeiler. A reduction of graph to canonical form and an algebra arising during this reduction. Cauchnoo-Techiccessten Informatsiya, 12-11 1968.
11

---

Published as a conference paper at ICLR 2024
Pavlo Melnyk, Michael Felsberg, and Mårten Wadenbäck. Embed me if you A geometric perceptron. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 1276-1284, 2021.
Alessio Monti, Alessia Bertugli, Simone Calderara, and Rita Cucchiara, Dag-net: Double attentive graph neural network for trajectory forecasting. arXiv, 2020.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and Leman Go Neural: Higher-Order Graph Neural Net- works. In AAAI Conference on Artificial Intelligence (AAAI), volume 33, pp. 4602-4609, 2019.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quan- tum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1(1), 2014.
David Ruhe, Johannes Brandstetter, and Patrick Forré. Clifford group equivariant neural networks. arXiv preprint arXiv:2305.1114// 2023a.
David Ruhe, Jayesh K. Gupta, Steven De Keninck, Max Welling, and Johannes Brandstetter. Ge- ometric Clifford Algebra Networks. In International Conference on Machine Learning (ICML), pp. 29306-29337, 2023b.
Victor Garcia Satorras, E. Hoogeboom, and M. Welling. E(n) Equivariant Graph Neural Networks. In International Conference on Machine Learning. 2021.
Matthew Spellings. Geometric algebra attention networks for small point clouds. arXiv preprint
arXiv:2110.02393, 2021.
STATS Perform Sportvu, 2023.
URL https:
www.sstt.tttessro.t..mmmmm com/
Philipp Thölke and G. D. Fabritiis. Torch MD-NET: Equivariant Transformers for Neural Network
based Molecular Potentials. ArXiv, 0tbs/2202.0254111 2022.
Nathaniel Thomas, T. Smidt, S. Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick F. Ri- ley. Tensor Field Networks: Rotation- and Translation- -qquiaaiinn--ivvaiaian Neural Networks for 3d Point Clouds. arXiv, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Conference on Neural In- formation Processing Systems (NeurIPS), pp. 5998-6008, 2017.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nel- son, Eric Jones, Robert Kern, Eric Larson, C J Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E, A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-276 2020. doi: (0930//41111--09-99900
Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale Hy- pergraph Neural Networks for Trajectory Prediction With Relational Reasoning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6498-6507, 2022.
Chenxin Xu, Robby Tan, YuHong Tan, Siheng Chen, Yu Guang Wang, Xinchao Wang, and Yanfeng Wang. Eqmotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reason- ing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are Graph Neural Networks? In International Conference on Learning Representations (ICLR), 2019.
Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, and Patrick Lucey. Generating multi-agent trajectories using programmatic weak supervision. arXiv, 2019.
Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33(14):110-i1998 2017.
12

---

Published as a conference paper at ICLR 2024
A REPRODUCBILITY & BROADER IMPACT
We will make the complete codebase utilized our experiments publicly available, encompassing aspects like model architectures, data preprocessing, training configurations, hyperparameters, and evaluation methodologies. This open-source approach is aimed at ensuring straightforward repro- ducibility.
Enhancing current graph-based models carries the promise of advancing numerous scientific do- mains, including but not limited to computational biology, materials science, and computer vision. These advancements have the potential to catalyze discoveries and innovations that contribute to the betterment of society by deepening comprehension of intricate, organized datasets.
B EQUIVARIANT MESSAGE PASSING NETWORKS
Symmetry is crucial in various branches of mathematics and science, such as geometry, physics, and chemistry, for understanding the underlying structures and properties of objects and systems. Groups are mathematical formalizations of symmetries, where each element of the group corresponds to symmetry of the object, and the group operation combines these symmetries.
In geometric deep learning (Bronstein et al., 2021), equivariance refers to property where the output of model symmetry group is applied to the input. Let w be element of an abstract group G. We have representation px (w) (e.g., rotation matrix on a vector space X, such that px xx(( X + X. we have
X Y should transtorm predictably when a transformation from some
= py
(4)
for all x X and G, then ø is called G-equivariant.
Equivariance desirable property of neural networks since it ensures that they respect certain symmetries present in the data. Instead of representations breaking down, equivariant neural net- works ensure that their output transforms faithfully. This stability is crucial for neural networks to generalize well to unseen data. Moreover, equivariant models are more data-efficient, not all possible symmetries need to be learned from data.
Message Passing Neural Networks (Gilmer et al. 2017) neural algorithms for learning graphs. We denote a graph G = (V, F)) to be a tuple of nodes and edges. A node E V or edge = (v, w) E E can have attached features h" or & (v,w), respectively. We then have the following update rules:
m. AggwEN(w)
(5)
Mi1 = 6b 00 (1i mf),
(6)
where N(v) denotes the neighbor set of e and hi == h" Here, 8th and &² are neural networks, called the message and update functions, respectively. Further, Agg denotes permutation-invarian aggregation function, e.g., a summation or average.
Locality is physics principle that states that object is influenced only by its immediate surround- ings. Message passing allows a model to capture local information by symmetrically characterizing the relationships between central node and its adjacent nodes. This mechanism enables the model to respect the inherent spatial hierarchies and dependencics present within the data. Further, sharing parameters of message and update networks across different nodes in graphs reduces parameters of networks but also provides consistency in learning structural relationships.
Note that px (w) does not necessarily have to be linear function; can also exist as non-linear function within different function space.
13

---

Published as a conference paper at ICLR 2024
Equivariant message passing be achieved by restricting the message and update functions, to- gether with the aggregation, to be G-equivariant in all their arguments:
p(w)(mi) p(w) (AgEWEN(w) (T hi ( blww)))
(7)
AgSwEN(v) (p(w) (h) k(mw)))
(8)
((w)
N(v)
Similarly, for the update equation.
p(w) = p(w) (o" (hf, mî))
(10)
= = = (p() ((()))))))))
(11)
Here, suppressed the representation space which p(w) acts for brevity. By composing these layers, we obtain a G-equivariant neural network that ensures that all outputs are transformed pre- dictably under G-action applied to the input.
C GEOMETRIC GRAPH ISOMORPHISM: EXAMPLE
In Figure 4 we display two non-isomorphic ge- ometric graphs. That is, the node attributes may contain geometric information, e.g., their position in space. A regular graph neural net- work would not be able to distinguish such two graphs. That is, their representations would be equal. However, by lifting the graphs to simpli- cial complexes and passing messages between different types of simplices, we can distinguish between the two graphs. Furthermore, if we apply geometric transformation the geometric fea- tures, such as rotation, the representations of the non-equivariant models would break down. For equivariant models, the representations would be transformed accordingly, respecting the fact that the chosen frame or reference is arbitrary.
Figure 4: Two non-isomorphic geometric graphs.
D CONSTRUCTING SIMPLICIAL COMPLEXES
Since data is typically not presented as graphs or point clouds rather than simplicial complexes, a choice needs be made when constructing simplicial complex based the underlying graph. We outline four different approaches. We assume that our input consists of either geometric graphs G (V, F)) such that for each V we have some coordinate X, point clouds {a; VN of N coordinates. Note that for the second case, there is no underlying graph structure present that can be leveraged create the complex.
Clique Lifts If we have a geometric graph, a standard graph lift or clique lift can be used. Since n-simplex defined by n+ points that are fully connected (e.g., a triangle or 2-simplex consists of three fully connected points), we can naturally associate to each clique of n + 1 points in the graph a simplex.
Manual Lifts In the situation where we do not have graph want to augment an existing graph, we sometimes can use domain knowledge to construct the simplicial complexes. That is, we have some intuition about where adding simplices in the graph could be beneficial, simplicial complex by manually adding simplices to the data. Take, for example, the molecule of water H,O. The bond angle between the two hydrogen atoms is crucial for the properties of water, and this interaction involves all three atoms. Modeling this requires a structure that goes beyond pairwise interactions.
create
Moreover, manual lift can be initiated with the Vietoris-Rips lift to form simplicial complexes. Subsequently, one can impose manually set thresholds for different dimensional simplices (like
14

---

Published as a conference paper at ICLR 2024
Figure 5: Left: a two-dimensional point cloud with e-balls around each point. Right: the corre- sponding Vietoris-Rips simplicial complex. In this case, is equal to the Čech complex.
edge lengths for L-simplices, areas of triangles for 2-simplices, and volumes of tetrahedra for 3- simplices). A group of (geometrically) close nodes form higher-order chemical structures, such as functional groups (think: aldehydes, ketones, nitrates, etc). E.g. we want to predict some prop- erties of (amino) acids, it could be useful to have "simplex" correspond with hydrophilic group s.a. carboxyl. Additionally, since molecular graphs often contain carbon rings, it's logical to break these rings into 2-dimensional or higher-order simplicial complexes better represent the rings in a topological sense. Moreover, higher-order simplices can be manually constructed based on expert knowledge capture the specific chemical and physical properties of the molecules. For instance, in biochemical structures like proteins or DNA, certain arrangements of atoms or residues have spe- eific functional implications. By manually constructing higher-order simplices, one can incorporate these domain-specific insights into the model. This approach allows for the explicit representation of essential molecular structures such as hydrogen bonding patterns, aromatic systems, which are critical for understanding molecular behavior and interactions.
similar approach is commonly leveraged in computer graphics, where objects are typically crafted and then approximated with meshes, creating 2-dimensional simplicial complex embedded in RS
Viertoris-Ripe and Čech Complexes In the case of no graph or domain information available, we can use Viertoris-Rip complex or Čech complex. For this, suppose our point cloud embedded in some metric space (X, d) where d X x X + R is some arbitrary distance function; e.g., the norm between vectors in R", Both approaches construct e-balls around points to form a simplicial complex based on the geometry of the space. Formally, we have for e
Vietoris- RicrissRiip((ee V, d. e) == (a V Vu, w a xx") <c}
(12)
That is, the Vietoris-Rips complex Vietoris- Tietoris-Rips(XX X, d, e) is the simplicial complex containing all simplices whose vertices are e-close. Similarly, we can construct Cech complex as
Čech(V, d,e) = fa c v 1 B(s, + #0,
(13)
vEO
where = X d(x",z) S c} is the closed ball of radius e around x". See Figures for illustrative example of these two lifts.
Intuitively, we consider the topology formed by the union of c-bals,, the case can be made that Čech complexes more intuitively resemble the topology on the data since Čech(V, d, e) homotopy- equivalent to the topology formed by combining these e-balls. However, the runtime for constructing a Vietoris- Rips complex is significantly less in practice, as illustrated in Eijkelboom et al. (2023). Moreover, since it holds that for any
we have
Čech(V d, e) c Vietoris-Rips(44 d,e) c (Ccch(V,4, 2c),
(14)
we know that if we can find some e such that the data is well described by the respective Čech complexes, then will be by Vietoris-Rips complex.
15

---

Published as a conference paper at ICLR 2024
Čocli Complex
Vietoris Rips Complex
Figure 6: We show three nodes u, 0, w, their respective e-balls, as well as the resulting Čech complex and Vietoris-Rips complex. Since the intersection of the balls is empty, the corresponding Čech complex does not contain triangle simplex, whereas the Vietoris-Rips complex does.
EXPERIMENTAL DETAILS
E. 5D CONVEX HULLS
The hulls are generated by randomly choosing eight nodes around the origin. We then use the convex hull functionality of Scipy (Virtanen et al. (2200) to realize it. The task is to predict the volume of the convex hulls given just the positions of the nodes. Graph neural network baselines operate fully-connected graph created from the positions.
An n-dimensional convex hull be represented by set of (n 1)-simplices. In our case, 5. We then consider all sub-simplices of these (n -)-sippliess up to dimension 2 and pass messages between them.
The finalized dataset contains 16384 entries for training, validation, and test sets. To maintain objectivity in the comparison, is ensured that all the baseline models and CSMPN have roughly equivalent number of parameters (200K). We use three simplicial message passing layers where the message and update functions are Clifford group-equivariant MI.s with 28 hidden features. Training of CSMPN is achieved through an Adam optimizer (Kingma & Ba, 2017) with learning rate x 10 10-3, 3 We train baselines with 10 steps with a batch size of 512. CSMPN uses a batch size of 16 in the training process.
E.2 CMU MOTION CAPTURE
We use the same preprocessing as Huang et al. (2022). The processed dataset has 200 entries in the training set and 600 entries both in the validation set and test set. We manually lift the triangles formed by elbow, shoulder, and forearm nodes. Consequently, the edges and nodes that form the triangles are also lifted as simplices. Triangles formed by hip nodes, knee joint nodes, and heel nodes also lifted to simplices. We also lift the rest of the nodes to zero-dimensiona simplices and keep their connectivity intact. The architecture of CSMPN (200K) is kept similar to the Convex Hulls experiment. In Figure depict the initial, ground truth, and predicted positions by CSMPN.
16

---

Published as a conference paper at ICLR 2024
Initial Positions
Targets
30
Predictions
10
44 2
12
Figure 7: Example initial, ground truth target, and predicted positions of CMU motion data sample.
The Adam optimizer with learning rate x 10 4 and weight decay x 10 was used for training. We train all models with 10 steps with batch size 100.
E.3 MD17 ATOMIC MOTION PREDICTION
The same preprocessing method as Xu et al. (2023) applied to the MD17 dataset. We sample and extract the positions of the heavy atoms from each of the molecules to form the final dataset. We use k-nearest neighbors to construct the Aspirin graph and fully connect the other molecules. We then clique-lift the graphs to form the simplicial complexes.
The training set and test set each molecule have 5000 and 2000 instances, respectively. For Aspirin, we have 1303 validation instances. For the other molecules, have 2000 instances to validate and tune the model performance.
We similar structure of CSMPN (300K) but with 5 message passing layers, each using Clifford group-equivarin networks with 32 hidden neurons as message and update functions. An Adam optimizer is used to train CSMPNs with learning rate 10-3 and weight decay 1 x 10-6, We train all models with 105 steps, with each batch containing 100 instances.
E.4 NBA PLAYERS TRAJECTORY PREDICTION
The same preprocessing steps from MMnnti al. 2020) applied obtain the final trajectory dataset. 8420 entries are used to train the CSMPN. We have 2806 entries the validation test sets, respectively. All five players clique complex lift is used to form the simplcial complexes from regular graphs. We use 4 simplicial message-passing layers with Clifford group-equivariant networks with 32 hidden neurons mes- sage and update functions, yielding roughly 200K parameters. An Adam optimizer with learning rate 5 x 10 is used train CSMPN. We train all models with 10a steps, with each batch containing 100 instances.
connected to each other form fully connected graphs. A
ALGORITHMS
In Algorithm Algorithm 3. and Algorithm 4 we denote message passing, simplicial message passing, and shared simplicial message passing, respectively. Here, Embed and Readout neural networks that embed input data and project the final node features to a single output, respectively. We also included graph and simplicial pooling aggregators. Considering that the I-skeleton of
17

---

Published as a conference paper at ICLR 2024
simplicial complex is graph, and regular message passing only uses the upper adjacencies, it is clear that standard message passing is a special case of simplicial message passing. Further, we see that effectively share parameters in the shared setting while leveraging the simplicial complex by conditioning on the source and target simplices. Finally, by reducing the incoming message to be aggregation of messages coming from varying simplex dimensions, can existing CUDA- optimized graph reduction operations already present in graph neural network libraries, c.g., Fey & Lenssen (2019).
Algorithm 2 Standard Message Passing
Require: = EV, Vu V : hi 6Th o
he t Emboddh
for f = 0, , L 1 do
# Message Passing
my AggwEN(w) (Kg, m²)
(
h + end for
G + b
how <- Readout (hC)
return hout
Algorithm 3 Simplicial Message Passing
Require: K, Vo E hig 6T Φ
hg < Embod(
for l = 0, ... L 1 do
# Message Passing
m ma((a)
A on (hg, h2)
m
(a) + Aggrecio) 00 (hg h2)
Nr (a) + AggrEN,(o) 2N (1g,11)
m
No (a) + (o) NN
m
(h2 h1))
h2t1 -
m B (o), my (0), m (a). N- (o)) Nt
end for
R + Aggo 11
howt + Readout (h return hout
Algorithm Shared Simplicial Message Passing
Require: Ko E K hig 6m oh
ng + Embod(h for = O,..., L do # Message Passing
mg +- Agg reB(o) dt (hg hT, dim a, dim
reC(a)
(o))(0)
TEN1(a)
h,+11 & (he m "asmmmdinn ,iim)) end for h.K + Aggoekh1 18 Nout <- Readout () return Braat
18

---

Published as a conference paper at ICLR 2024
Method
Dataset
Seconds/Step
EMPSNs
Aspirin
0,04
CEGNNs
Aspirin
0.08
CSMPNs (separated)
Aspirin
0.37
CSMPNs (shared)
Aspirin
0.22
EMPSNs
Benzene
0.03
CEGNNs
Benzene
0.07
CSMPNs (separated)
Benzene
0,40
CSMPNs (shared)
Benzene
0.24
EMPSNs
Ethano!
0.03
CEGNNs
Ethanol
0.08
CSMPNs (separated)
Ethanol
0.29
CSMPNs (shared)
Ethanol
0.09
EMPSNs
Malonaldehyde
0.03
CEGNNs
Malonaldehyde
0.07
CSMPNs (separated)
Malonaldehyde
0.32
CSMPNs (shared)
Malonaldehyde
0.16
Table inference time of Clifford, Shared and Separate message passing networks on MD17 atomic motion dataset.
G SHARED VS SEPARATE MESSAGE PASSING LAYERS
To provide practical perspective on the shared message passing scheme, we present a table on the inference times of CSMPNs when applied to the MD17 atomic motion dataset. Averaged in- ference time is measured for a batch of 100 samples running on a GeForce RTX 3090 GPU. For comprehensive understanding, Equivariant Graph Neural Networks) and CSMPNs with separate simplicial message passing net- works, serving comparative benchmarks.
also include performance metrics of CEGNNs (Clifford Group
19


Reviews:
[{"review_id": "rPuMIm4xup", "reviewer": "Reviewer_atRt", "summary": "The paper introduces Clifford Group Equivariant Simplicial Message Passing Networks (CSMPNs), a method designed for steerable $E(n)$-equivariant message passing on simplicial complexes. The approach combines the expressivity of Clifford group-equivariant layers with simplicial message passing, which is more expressive than regular graph message passing. To implement it, the paper discusses methods for lifting a set of points to a simplicial complex and embedding simplicial data in the Clifford algebra, where various types of lifts like Vietoris-Rips and manual lifts are considered. The results and evaluations of the CSMPNs are across various domains showing the effectiveness of this method.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a combination of Clifford group-equivariant layers and simplicial message passing, providing a strong theoretical foundation for the method. The use of Clifford algebras to represent geometric features with simplices is natural and mathematically sound.\n2. The model is designed to be applicable across various domains, from geometry to molecular dynamics. This broad applicability is a strong point, especially for a mathematical audience interested in universal structures.", "weaknesses": "1. I would like to see the detailed inference time of CSMPN compared to others, e.g., in the MD17 atomic motion dataset. I believe the time complexity of CSMPN is highly relevant to the number of simplices, and I would like to know if this is an issue in the implementation.\n2. As the authors mention experiments on MD17 and QM9 are beyond the scope of their research; I hope the authors give a discussion or possible directions on how the manual lift could be implemented for small and medium-sized molecules to strengthen the paper's applicability, not just using H2O as an example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YHXXCQQAtZ", "reviewer": "Reviewer_q1Xe", "summary": "In this paper, the authors proposed a Graph Neural Networks method that combines both simplicity message passing and Clifford group equivariants neural networks. It is understood that this is an extension of the recent study of E(n) equivariant message passing simplicity networks, which has the limitation of manual initialization and restriction of updating methods. Numerical experiments show that the proposed method outperform existing methods in most cases.", "questions": "Maybe the authors can explain by \"learn\" the embedding through Clifford group-equivariant layers in Sec. 3.2.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "As I have summarized above, the min contribution of this paper is to propose an alternative message passing network to the existing EMPSN method, and eliminate two main limitations of EMPSN. \n\nAlthough neither utilizing Clifford group in neural networks  nor simplicial message  passing is new, the authors combine them to achieve an advancement in GNN.", "weaknesses": "In the key technical session 3.2, embedding simiplicial data to Clifford algebra, the embedding of the simplices is not fully and clearly described, thus renders a full understanding of the proposed method difficult. The authors did point to Figure 1 for depiction of the method, but that is open for different interpretations. \n\nWhile numerical results indeed show that new method, CSMPN, achieves the best performance in most experiments, the improvement over existing methods can not be characterized as significant, thus not leading to a convincing argument for its usage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aMQ3uJ1i1r", "reviewer": "Reviewer_u7om", "summary": "- The authors propose a new architecture for geometric graphs that combines simplical message-passing, Clifford algebras, and equivariance.\n- On a high level, their approach is based on linking the expansion in the number of vertices that occurs in simplices with the expansion in grades in a Clifford algebra.\n- A given geometric graph is first lifted to simplices, then the outputs are computed with a message-passing algorithm between the simplices. Different from Bodnar et al (2021), the proposed algorithm uses Clifford algebras as representations, the geometric product to compute simplex features, and Clifford Group Equivariant NNs (Ruhe et al, 2023) to construct messages. These changes add a geometric inductive bias to the algorithm and make it equivariant.\n- The algorithm is demonstrated on experiments ranging from simple toy data to motion capture data, molecular dynamics, and even NBA data.", "questions": "- Could you comment on what I wrote in the first point in the Weaknesses section? It's well possible I missed something here.\n- On page 2, what do you mean with \"geometric products are meaningful in all such cases\"?\n- On page 7, you mention that you use an equal parameter count for CSMPN and the baselines. Is that really fair when compairing equivariant and non-equivariant baselines? Usually, equivariant methods have far less parameters for the same expressivity.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea is novel and interesting. I like how the authors link two previously disjointed ways of describing expansions to multiple objects.\n- The experimental evaluation on a range of different problems highlights the versatility of the approach. The results demonstrate that the method works.\n- The paper is well-written, the structure clear, and the figures of a high quality.", "weaknesses": "- The authors overstate the naturality of their method. While the approach feels intuitive at a quick glance, it mixes two different concepts: Simplices are a topological concept, while the geometric product is related to the geometric space that the graph is embedded in. This leads to a number of issues. Even in 3D space, one can construct 100-simplices, but the Clifford algebra only goes up to grade 4. While intuitively higher-order simplices encode more information, higher-order grades eventually become lower-dimensional. More generally, the geometric product is lossy: different simplices will be described by the same multivector. These all seem like undesirable properties for a simplex representation. I wish the authors would acknowledge and discuss these issues, which to me seem central to this paper.\n- While I like the experiments, it would be useful to see the performance on more standard benchmarks of geometric deep learning. The authors acknowledge that they do not achieve SOTA results on some of these tasks. That's fine, but it would still be useful to see how the method performs compared to other generic equivariant methods.\n- The paper's contributions are somewhat thin: the authors essentially plug together two existing ideas (simplical message passing and Clifford group equivariant networks). Of course, the same can be said about many other papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)