E(n) Equivariant Topological Neural Networks

## Abstract

Abstract Graph neural networks excel at modeling pairwise interactions, but they cannot flexibly accommodate higher-order interactions and features. Topological deep learning (TDL) has emerged recently as a promising tool for addressing this issue. TDL enables the principled modeling of arbitrary multi-way, hierarchical higher-order interactions by operating on combinatorial topological spaces, such as simplicial or cell complexes, instead of graphs. However, little is known about how to leverage geometric features such as positions and velocities for TDL. This paper introduces E(n)-Equivariant Topological Neural Networks (ETNNs), which are E(n)-equivariant message-passing networks operating on combinatorial complexes, formal objects unifying graphs, hypergraphs, simplicial, path, and cell complexes. ETNNs incorporate geometric node features while respecting rotation, reflection, and translation equivariance. Moreover, ETNNs are natively ready for settings with heterogeneous interactions. We provide a theoretical analysis to show the improved expressiveness of ETNNs over architectures for geometric graphs. We also show how E(n)-equivariant variants of TDL models can be directly derived from our framework. The broad applicability of ETNNs is demonstrated through two tasks of vastly different scales: i) molecular property prediction on the QM9 benchmark and ii) land-use regression for hyper-local estimation of air pollution with multi-resolution irregular geospatial data. The results indicate that ETNNs are an effective tool for learning from diverse types of richly structured data, as they match or surpass SotA equivariant TDL models with a significantly smaller computational burden, thus highlighting the benefits of a principled geometric inductive bias. Our implementation of ETNNs can be found here .

## 1 Introduction

Graph Neural Networks (GNNs) are employed across various fields, including computational chemistry (Gilmer et al., ), physics simulations (Shlomi et al., ), and social networks (Xia et al., ), to highlight a few. Their effectiveness stems from merging neural network adaptability with insights about data relationships through the graph topology. Research on GNNs covers a broad range of categories, mainly divided into spectral (Bruna et al., ) and non-spectral (Gori et al., ). In both cases, GNNs aim to learn representations for node (and/or) edge attributes through local aggregation driven by the graph topology, essentially defining message passing networks (MPN) (Gilmer et al., ). Utilizing this capability, GNNs have delivered significant achievements in tasks like node and graph classification (Kipf & Welling, ), link prediction (Zhang & Chen, ), and specific challenges such as protein folding (Jumper et al., ).

Figure: Figure 1: Overview of the $E(n)$-Equivariant Topological Neural Networks framework. An input geometric graph/point cloud with (possibly) non-geometric features is provided. A combinatorial complex, whose elements are called cells, is constructed from the input geometric graph/point cloud to encode higher-order hierarchical interactions, based on topological or domain-specific considerations. If available, higher-order cell features can be injected. Geometric invariants for the cells (e.g. pairwise distances, Hausdorff distances, volumes, …) are computed. Finally, $E(n)$ Equivariant Topological Neural Networks update both geometric and non-geometric features to improve a downstream task while respecting rotation, reflection, and translation equivariance.
Refer to caption: extracted/5899612/Figures/gabstract3.png

However, GNNs usually struggle to model higher-order and multi-way interactions that the intrinsically pairwise structure of graphs cannot capture.
Consequently, the field of Topological Deep Learning (TDL) (Barbarossa & Sardellitti, ; Hajij et al., ; Papamarkou et al., ) started to gain significant interest as it seeks to address these limitations. Although the general scope of TDL is designing models for data defined over a variety of topological spaces, the main focus is usually on combinatorial topological spaces (CTS), i.e., topological spaces that can be described combinatorially, such as cell complexes (Grady & Polimeni, ; Bodnar, ; Battiloro, ). Leveraging this combinatorial structure, it is possible to design more sophisticated adjacency schemes among cells, i.e., single nodes or groups of nodes, than the usual adjacency in graphs. Message passing networks over CTS have been proven to be more expressive (Bodnar et al., ; Battiloro et al., ; Bodnar et al., ; Truong & Chin, ; Jogl et al., ) (using the Weisfeiler-Lehman test criterion (Xu et al., )), more suitable to handle long-range interactions (Giusti et al., ; ), and more effective to handle heterophilic data settings (Bodnar et al., ; Hansen & Gebhart, ) than classical GNNs. Cell complexes are powerful objects to model certain classes of higher-order interactions along with a hierarchical structure among them, while general combinatorial objects like hypergraphs are useful to model arbitrary higher-order interactions, but without any hierarchy. Hajij et al. () introduced the notion of Combinatorial Complex (CC) to provide a formal, flexible, yet simple structure that pseudo-generalizes simplicial/path/cell complexes and hypergraphs, thus enabling the design of arbitrary higher-order interactions along with a hierarchical structure. They also introduced message passing networks over combinatorial complexes.

Little is known about how to integrate and leverage geometric information of the data in such TDL models. In this context, nodes are embedded in a manifold or, in general, in a metric space, and have geometric features, such as positions or velocities. In the case of geometric graphs, a common practice is designing message passing networks able to respect some suitable symmetry, i.e., being equivariant w.r.t. the action of some symmetry group. Here we focus on $E(n)$-equivariant neural networks, i.e., networks being equivariant w.r.t. rotations, reflections, and translations. The $E(n)$-Equivariant Graph Neural Network (EGNN) from Satorras et al. () is one of the simplest yet powerful equivariant models for graph-structured data. Eijkelboom et al. () generalized the approach from Satorras et al. () to simplicial complexes (a particular instance of cell complexes), introducing $E(n)$-Equivariant Message Passing Simplicial Networks (EMPSNs), able to handle graph or simplicial-structured data. However, as we will show in the paper, both graphs and simplicial complexes are combinatorial objects often not flexible enough to model arbitrary higher-order interactions, and a general framework for designing
equivariant neural networks over CTS is missing. Appendix B presents a review of related works.

Contribution. We generalize the approaches from Satorras et al. () and Eijkelboom et al. () to design $E(n)$-Equivariant Topological Neural Networks (ETNNs), i.e., $E(n)$-Equivariant Message Passing Networks over Combinatorial Complexes. CCs represent a pseudo-generalization of graphs, simplicial complexes, path complexes, cell complexes, and hypergraphs, therefore ETNNs represent a legitimate framework for designing arbitrary $E(n)$-Equivariant Message Passing Networks over combinatorial topological spaces or, more in general, over combinatorial objects used to model interactions among entities, such as hierarchical graphs. An ETNN layer is made of a message passing round over the CC comprising scalar invariants of the geometric features and a learnable update of the geometric features added up in the current directions. As such, ETNNs are a scalarization method (Han et al., ; ). We study the improved expressiveness of ETNNs by evaluating their ability to distinguish $k$-distinct geometric graphs (Joshi et al., ), showing that they are at least as expressive as other well-known scalarization-based $E(n)$ equivariant graph neural networks. Furthermore, we empirically show that ETNNs are also able to distinguish most of the counterexample structures from (Pozdnyakov et al., ; Joshi et al., ) that are indistinguishable using $k$-body scalarization. Generalizing (Hajij et al., ) to the geometric setting, we
show how several $E(n)$-equivariant variants of TDL models can be directly derived
from our framework. As an additional contribution, we test ETNNs in applications where domain knowledge induces combinatorial complex structures that are more natural and parsimonious than the graphs or higher-order structures that are usually employed to model the same phenomena. Specifically, we show the efficacy of ETNNs on i) molecular properties prediction on the QM9 benchmark, and on ii) land-use regression for hyper-local estimation of air pollution with multi-resolution irregular geospatial data, representing also a completely new challenging benchmark for TDL models. Interestingly, ETNNs match or surpass the results of EMPSNs with less than half of the memory usage and almost half of the runtime per epoch. The obtained results jointly validate ETNNs, our CC-based approach, and, in general, the injection of geometric information in TDL models.

## 2 Combinatorial Complexes and Equivariance

We review the fundamentals of combinatorial complexes (CCs), fairly general objects providing an effective and flexible way to represent higher-order multi-way interactions. Such structures generalize graphs, simplicial complexes, path complexes, cell complexes, and hypergraphs. We first define our domain of interest -CCs-, the relations among its elements -neighborhood functions-, the data defined on it -topological signals-, and a class of deep networks operating on it -CC message passing networks (CCMPNs)-(Hajij et al., ). We finally introduce the formal notion of equivariance, showing some properties of CCMPNs.

Combinatorial Complex. A *combinatorial complex* (CC) is a triple $(\mathcal{S},\mathcal{X},\mathrm{rk})$ consisting of a set $\mathcal{S}$, a subset $\mathcal{X}$ of $\mathcal{P}(S)\backslash\{\emptyset\}$, and a function $\mathrm{rk}:\mathcal{X}\rightarrow\mathbb{Z}_{\geq 0}$ with the following properties:

- 1.
for all $s\in\mathcal{S},\{s\}\in\mathcal{X}$;
- 2.
the function $\mathrm{rk}$ is order-preserving, i.e., if $x,y\in\mathcal{X}$ satisfy $x\subseteq y$, then $\mathrm{rk}(x)\leq$ $\mathrm{rk}(y)$.

The elements of $\mathcal{S}$ and $\mathcal{X}$ are nodes and cells, respectively, and $\mathrm{rk}(\cdot)$ is the rank function. $\mathcal{X}$ simplifies notation for $(\mathcal{S},\mathcal{X},\mathrm{rk})$. The rank function flexibly organizes the cells hierarchically. Therefore, CCs can model both hierarchical (as in simplicial and cell complexes) and set-type (as in hypergraphs) relations. For each singleton cell $\{s\}$ in a CC, we set $\mathrm{rk}(\{s\})=0$, aligning CCs with simplicial and cell complexes. The rank of a cell $x\in\mathcal{X}$ is $k:=\mathrm{rk}(x)$, and we call it a $k$-cell. The dimension $\mathrm{dim}(\mathcal{X})$ of a CC is the maximal rank among its cells.

Neighbhorhood functions. Combinatorial Complexes can be equipped with a notion of neighborhood among cells. In particular, we can define a “neighborhood" function $\mathcal{N}:\mathcal{X}\rightarrow\mathcal{P}(\mathcal{X})$ on $\mathcal{X}$ as a function that assigns to each cell $x$ in $\mathcal{X}$ a collection of “neighbor cells" $\mathcal{N}(x)\subset\mathcal{X}$. Usually, two main types of neighborhood functions are considered (Hajij et al., ): *adjacencies*, in which all of the elements of $\mathcal{N}(x)$ have the same rank as $x$, and *incidences*, in which its elements are from a higher or lower rank. A generally applicable choice for these functions is as follows. First, up/down incidences $\mathcal{N}_{I,\uparrow}$ and $\mathcal{N}_{I,\downarrow}$ are defined by the *containment* criterion

$$ $\mathcal{N}_{I,\uparrow}(x)=\{y\in\mathcal{X}|\mathrm{rk}(y)=\mathrm{rk}(x)+1, x\subset y\},\quad\mathcal{N}_{I,\downarrow}(x)=\{y\in\mathcal{X}|\mathrm{rk}( y)=\mathrm{rk}(x)-1,y\subset x\}.$ (1) $$

Therefore, a $k+1$-cell $y$ is a neighbor of a $k$-cell $x$ w.r.t. to $\mathcal{N}_{I,\uparrow}$ if $x$ is contained in $y$; analogously, a $k-1$-cell $y$ is a neighbor of a $k$-cell $x$ w.r.t. to $\mathcal{N}_{I,\downarrow}$ if $y$ is contained in $x$.
The incidences induce up/down adjacencies $\mathcal{N}_{A,\uparrow}$ and $\mathcal{N}_{A,\downarrow}$ by the *common neighbor* criterion

$$ $\displaystyle\mathcal{N}_{A,\uparrow}(x)=\{y\in\mathcal{X}|\mathrm{rk}(y)= \mathrm{rk}(x),\exists z\in\mathcal{X}:\mathrm{rk}(z)=\mathrm{rk}(x)-1,z \subset y,\text{ and }z\subset x\},$ $\displaystyle\mathcal{N}_{A,\downarrow}(x)=\{y\in\mathcal{X}|\mathrm{rk}(y)= \mathrm{rk}(x),\exists z\in\mathcal{X}:\mathrm{rk}(z)=\mathrm{rk}(x)+1,y \subset z,\text{ and }x\subset z\}.$ (2) $$

Therefore, a $k$-cell $y$ is a neighbor of a $k$-cell $x$ w.r.t. to $\mathcal{N}_{A,\uparrow}$ if they are both contained in a $k+1$-cell $z$; analogously, a $k$-cell $y$ is a neighbor of a $k$-cell $x$ w.r.t. to $\mathcal{N}_{A,\downarrow}$ if they both contain a $k-1$-cell $z$.

While this choice of neighborhood functions applies to any CC, other neighborhood functions are more natural in specific applications. This scenario will be illustrated in 4 Real-World Combinatorial Complexes.

Topological signals. A topological signal (or feature) over $\mathcal{X}$ is a mapping $f:\mathcal{X}\rightarrow\mathbb{R}$ from the set of cells $\mathcal{X}$ to real numbers. Therefore, the feature vectors $\mathbf{h}_{x}\in\mathbb{R}^{F}$ and $\mathbf{h}_{y}\in\mathbb{R}^{F}$ of cells $x$ and $y$ are a collection of $F$ CC signals, i.e., $\mathbf{h}_{x}=[f_{1}(x),\dots,f_{F}(x)]$ and $\mathbf{h}_{y}=[f_{1}(y),\dots,f_{F}(y)]$.

Message passing networks over CCs. Let $\mathcal{X}$ and $\mathcal{CN}$ be a CC and collection of neighborhood functions on it. The $l$-th layer of a CC Message Passing Network (CCMPN) updates the embedding $\mathbf{h}^{l}_{x}$ of cell $x$ as

$$ $\mathbf{h}_{x}^{l+1}=\beta\left(\mathbf{h}^{l}_{x},\bigotimes_{\mathcal{N}\in \mathcal{CN}}\bigoplus_{y\in\mathcal{N}(x)}\psi_{\mathcal{N},\mathrm{rk}(x)} \left(\mathbf{h}^{l}_{x},\mathbf{h}^{l}_{y}\right)\right),$ (3) $$

where $\mathbf{h}_{x}^{0}:=\mathbf{h}_{x}$ are the initial features, $\bigoplus$ is an intra-neighborhood permutation invariant aggregator, $\bigotimes$ is an inter-neighborhood (possibly) permutation invariant aggregator, and the rank- and neighborhood-dependent message functions $\psi_{\mathcal{N},\mathrm{rk}(x)}$ and the update function $\beta$ are learnable functions. In other words, the embedding of a cell is updated in a learnable fashion through aggregated messages with its neighboring cells over a set of neighborhoods, generalizing what has been done with Graph MPN (Gilmer et al., ). Interestingly. the message functions can be customized to incorporate additional information, such as the orientation of the cells (Barbarossa & Sardellitti, ; Sardellitti et al., ) (if available), or the embeddings of "common" cells (if present) like $x\cup y$ or $x\cap y$. Finally, please notice that CCMPN as in (3) are inherently able to model heterogeneous settings, as two cells can be neighbors in multiple neighborhoods, but messages among them will be dependent on the neighborhood itself. Thus, this property allows us to model different relation types through different (overlapping) neighborhoods. In Appendix C, we show how to model graphs, simplicial complexes, cell complexes, and hypergraphs through the CC framework, deriving the corresponding message passing architectures from the CCMPN in (3).

Equivariances of CCMPNs. Symmetries are framed as groups (Bronstein et al., ). Let $G$ be a group with an action $a:G\times\mathcal{Y}\rightarrow\mathcal{Y}$ on a set $\mathcal{Y}$. Given a function $e:\mathcal{Y}\rightarrow\mathcal{Y}$, it is said to be equivariant w.r.t. the action of $G$ if, for all $x\in\mathcal{X}$ and $g\in G$, it holds

$$ $e(a(g,x))=a(g,e(x)).$ (4) $$

If $e(a(g,x))=e(x)$, $e$ is said to be invariant w.r.t. the action of $G$. As Graph MPNs (Gilmer et al., ), CCMPNs are permutation equivariant too (Hajij et al., ), i.e., they are equivariant w.r.t. the action of the symmetric group $\mathrm{Sym}(\mathcal{X})$ on $\mathcal{X}$. In other words, CCMPNs are equivariant to the relabeling of cells in the complex.

## 3 E(n) Equivariant Topological Neural Networks

Let us now introduce the setting in which nodes (0-cells) are embedded in some Euclidean space, i.e., they come with both non-geometric and geometric features, such as positions or velocities. In the following, we assume to have only positions for the sake of exposition, but velocities can be directly injected into our model as explained in Appendix D. We denote the non-geometric features with $\mathbf{h}_{x}\in\mathbb{R}^{F}$ for cell $x\in\mathcal{X}$ and the position with $\mathbf{x}_{z}\in\mathbb{R}^{n}$ for cell $z\in\mathcal{S}$. Since many problems exhibit $n$-dimensional rotation, reflection, and translation symmetries, it is desirable to design models that are equivariant w.r.t. the action of $E(n)$ (Satorras et al., ; Eijkelboom et al., ; Han et al., ). An element of the $E(n)$ group is a tuple $(\mathbf{O},\mathbf{b})$ consisting of an orthogonal matrix $\mathbf{O}\in\mathbb{R}^{n\times n}$ (the rotation/reflection) and a vector $\mathbf{b}\in\mathbb{R}^{n}$ (the translation). The action $a$ of $E(n)$ on a vector $\mathbf{x}\in\mathbb{R}^{n}$ is given by

$$ $a((\mathbf{O},\mathbf{b}),\mathbf{x})=\mathbf{O}\mathbf{x}+\mathbf{b}.$ (5) $$

As has been done for graphs (Satorras et al., ) and simplicial complexes (Eijkelboom et al., ), we represent equivariance by scalarization (Han et al., ). Therefore, geometric features undergo an initial transformation into invariant scalars. Subsequently, they are processed in a learnable way (e.g., MLPs) before being combined along the original directions to achieve equivariance.

E(n) Equivariant Topological Neural Networks. Let $\mathcal{X}$ and $\mathcal{CN}$ be a CC and collection of neighborhood functions on it. The $l$-th layer of an $E(n)$ Equivariant Topological Neural Network (ETNN) updates the embeddings $\mathbf{h}^{l}_{x}$ of every cell $x\in\mathcal{X}$ and the position $\mathbf{x}_{z}^{l}$ of every node $z\in\mathcal{S}$ as

$$ $\displaystyle\mathbf{h}_{x}^{l+1}=\beta\left(\mathbf{h}^{l}_{x},\bigotimes_{ \mathcal{N}\in\mathcal{CN}}\bigoplus_{y\in\mathcal{N}(x)}\underbrace{\psi_{ \mathcal{N},\mathrm{rk}(x)}\left(\mathbf{h}^{l}_{x},\mathbf{h}^{l}_{y},\mathrm {Inv}\left(\{\mathbf{x}_{z}^{l}\}_{z\in x},\{\mathbf{x}_{z}^{l}\}_{z\in y} \right)\right)}_{\mathbf{m}_{x,y}^{\mathcal{N}}}\right),\text{ for all }x\in \mathcal{X}$ (6) $\displaystyle\mathbf{x}_{z}^{l+1}=\mathbf{x}_{z}^{l}+C\sum_{\mathcal{N}\in \mathcal{CN}}\sum_{t\in\mathcal{S}:\{t\}\in\mathcal{N}(z)}\left(\mathbf{x}_{z} ^{l}-\mathbf{x}_{t}^{l}\right)\xi\left(\mathbf{m}_{z,t}^{\mathcal{N}}\right), \text{ for all }z\in\mathcal{S},$ (7) $$

where $\mathbf{h}_{x}^{0}=\mathbf{h}_{x}$, $\mathbf{x}_{z}^{0}=\mathbf{x}_{z}$, and $\xi$ is a scalar learnable function. $\mathrm{Inv}$ takes as input the positions of the nodes forming the involved cells, and it is an invariant function w.r.t. the action of $E(n)$, i.e., $\mathrm{Inv}(\{\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}\}_{z\in x},\{\mathbf{O}
\mathbf{x}_{z}^{l}+\mathbf{b}\}_{z\in y})=\mathrm{Inv}(\{\mathbf{x}_{z}^{l}\}_
{z\in x},\{\mathbf{x}_{z}^{l}\}_{z\in y})$. We refer to a specific $\mathrm{Inv}$ as a geometric invariant. The update in (7) again naturally allows for neighborhood heterogeneities: two nodes can be neighbors via more than a single neighborhood function, carrying different messages. This is common in many applications, as illustrated in 4 Real-World Combinatorial Complexes.

The following theorem demonstrates the equivariance property of ETNNs.

###### Theorem 1 .

An ETNN layer as in (6)-(7), synthetically denoted as $\{\mathbf{h}_{x}^{l+1}\}_{x\in\mathcal{X}},\{\mathbf{x}_{z}^{l+1}\}_{z\in
\mathcal{S}}=\mathrm{ETNN}\left(\{\mathbf{h}_{x}^{l}\}_{x\in\mathcal{X}},\{
\mathbf{x}_{z}^{l}\}_{z\in x}\right)$, is $E(n)$ equivariant, that is

$$ $\{\mathbf{h}_{x}^{l+1}\}_{x\in\mathcal{X}},\{\mathbf{O}\mathbf{x}_{z}^{l+1}+ \mathbf{b}\}_{z\in x}=\mathrm{ETNN}\left(\{\mathbf{h}_{x}^{l}\}_{x\in\mathcal{ X}},\{\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}\}_{z\in x}\right),$ (8) $$

for all $(\mathbf{O},\mathbf{b})\in E(n)$.

###### Proof.

See Appendix E.
∎

When only (6) is applied but the update step in (7) is not performed, then ETNNs are $E(n)$-invariant. At this point, it is important to discuss geometric invariants. As has been shown in (Satorras et al., ; Eijkelboom et al., ), geometric invariants should make use of the underlying topological structure. For instance, the pairwise distance of connected nodes has been used in EGNN (Satorras et al., ), while the simplex volume has been used in EMPSN (Eijkelboom et al., ). However, both EGNN and EMPSN operate on rigid domains, such as simplicial complexes, while CCs can have a much more arbitrary set of relations. In the following, we present a series of geometric invariants that can be used in ETNNs, without assuming any specific structure for the CC. We consider two cells $x$ and $y$, along their corresponding node positions $\{\mathbf{x}_{z}\}_{z\in x}$ and $\{\mathbf{x}_{z}\}_{z\in y}$.

Permutation invariant functions of pairwise distances. Any function of the form

$$ $\bigoplus_{z\in x,t\in y}\tau\left(\|\mathbf{x}_{z}-\mathbf{x}_{t}\|\right)$ (9) $$

is a geometric invariant, with $\bigoplus$ being a permutation invariant aggregator, and $\tau$ a (possibly) learnable function. A basic instance is $\bigoplus=\sum$ and $\tau=\mathrm{Id}$, i.e., the sum of pairwise distances.

Distances of permutation invariant functions. Any function of the form

$$ $\tau\left(\Bigg{\|}\bigoplus_{z\in x}\mathbf{x}_{z}-\bigoplus_{t\in y}\mathbf{ x}_{t}\Bigg{\|}\right)$ (10) $$

is a geometric invariant, with $\bigoplus$ being a linear permutation invariant aggregator, and $\tau$ a (possibly) learnable function. A basic instance is $\bigoplus=\frac{1}{|\cdot|}\sum$ and $\tau=\mathrm{Id}$, i.e., the distance between centroids.

Hausdorff distance. The Hausdorff distance, defined as

$$ $\max\left\{\max_{z\in x}\min_{t\in y}\|\mathbf{x}_{z}-\mathbf{x}_{t}\|,\max_{t \in y}\min_{z\in x}\|\mathbf{x}_{t}-\mathbf{x}_{z}\|\right\}$ (11) $$

is a geometric invariant. The Hausdorff distance measures the mutual proximity of two sets whose elements live in a metric space. It does so by computing the maximal distance between any point of one set to the other set. Moreover, the two $\max\min$ terms inside the outer $\max$, usually referred to as directed Hausdorff distances, can be used as geometric invariants themselves. This geometric invariant has been used in various machine learning applications, especially in the area of computer vision (Aydin et al., ; van Kreveld et al., ).

Volume of the convex hull. Any function of the volumes of the convex hulls of $\{\mathbf{x}_{z}\}_{z\in x}$ and $\{\mathbf{x}_{z}\}_{z\in y}$ is a geometric invariant. Computing the volume of a convex hull is quite an expensive procedure, thus it is not always suitable to be performed at each training step. However, it can still be used if the cells have small cardinality, or if ETNNs are used in an invariant fashion, i.e., without the update in (7), because the volume needs to be computed only once before training. Finally, the volume of convex hulls reduces to the volume of simplices in case the considered CC is a (geometric) simplicial complex, generalizing what has been done in (Eijkelboom et al., ).

###### Remark 1 .

It is noteworthy that if higher rank cells, not only nodes, come with attached geometric features, they can be seamlessly integrated into ETNNs. In that case, there would be geometric invariants of node geometric features, geometric invariants of higher-rank geometric features, and, possibly, geometric invariants of both nodes and higher rank cells. The equivariant update in (7) would be straightforwardly modified to include all the cells with geometric features in the sum.

Expressiveness of ETNN. The expressive power of
neural networks over CTS is proportional to their ability to distinguish non-isomorphic CTSs. As the renowned Weisfeiler-Leman (WL) test for GNNs (Xu et al., ), several generalized WL tests have been developed for CTSs (e.g. simplicial (Bodnar et al., ), cell (Bodnar et al., ), or path (Truong & Chin, ) complexes) and have been shown to be upper bounds on the expressiveness of the corresponding TDL models. These tools also demonstrated that TDL models are usually more expressive than vanilla GNNs when it comes to distinguishing isomorphic graphs. For geometric graphs, the Geometric-WL (GWL) test (Joshi et al., ) has been recently introduced as an upper bound on the expressiveness of invariant/equivariant geometric GNNs. In Appendix F, we introduce the notion of Geometric Augmented Hasse Graph and we combine it with the GWL to show that ETNNs are, in general, more expressive than scalarization-based graph methods. We also perform experiments to validate our claim, including testing ETNN on the counterexample structures from (Pozdnyakov et al., ; Joshi et al., ). Next, we present an informal statement of our result. Formal statement and proof are in Appendix F.

Proposition 2 . (Informal) An ETNN is at least as powerful as an EGNN (Satorras et al., ) in distinguishing $k$-hop distinct graphs. In most of the cases, an ETNN is strictly more powerful than an EGNN. The quantitative gap between ETNNs and EGNNs depends on how the graphs are lifted into a CC, i.e., how the set of cells $\mathcal{X}$ and the collection of neighborhood functions $\mathcal{CN}$ are defined.

Formal Statement and Proof . See Appendix F.

Computational complexity of ETNN. In Appendix G, we present an analysis of the computational complexity and runtime of ETNNs. In particular, we show that, if the connectivity of the CC is sufficiently sparse, which is usually the case, then the overall complexity is linear in the number of cells. In the worst case, all the cells all connected among them, resulting in a quadratic cost in the number of cells. In the same Appendix G, we also empirically validate these results by comparing the runtimes of ETNN with EGNN (Satorras et al., ) and EMPSN (Eijkelboom et al., ).

Figure: Figure 2: Real-World Combinatorial Complexes. *(Left)*: A molecular CC where 0-cells are atoms, 1-cells are bonds, and 2-cell represent rings and functional groups. *(Right)*: A geospatial CC where 0-cells are grid points, 1-cells are road polylines, and 2-cells are census tract polygons. For visibility, the geospatial CC only shows one road (rank 1) and its incident lower and higher rank cells.
Refer to caption: extracted/5899612/Figures/MolCC+SpatCC.png

###### Theorem 1 .

###### Proof.

###### Remark 1 .

## 4 Real-World Combinatorial Complexes

Designing equivariant models for CCs tailored for specific applications is an exciting direction since CCs can accommodate hierarchical higher-order features that the literature’s most used CTSs cannot readily incorporate, including graphs, simplicial complexes, path complexes, cell complexes, and hypergraphs(^1^11Appendix C provides an overview of the aforementioned CTSs.). This section presents two vastly different applications.
The first application considers molecular data. While this data has been investigated under the lens of CTS, our approach allows us to naturally integrate their higher-order structures such as rings and functional groups. Next, we present a novel TDL framework for geospatial data. We show that the interactions between geographical objects in geospatial data such as polylines and polygons can be modeled via CCs. In the next section, we will use instances of these CCs in experiments evaluating the ETNN architecture.

Molecular combinatorial complexes. We introduce the notion of Molecular Combinatorial Complexes, showing that they can jointly retain all the properties of graphs, cell complexes, and hypergraphs, summarized in Appendix H, thus overcoming their mutual limitations. In a molecular CC, the set of nodes $\mathcal{S}$ is the set of atoms, while the set of cells $\mathcal{X}$ and the rank function $\mathrm{rk}$ are such that: atoms are 0 0-cells, bonds and functional groups made of two atoms are $1$-cells, and rings and functional groups made of more than two atoms are $2$-cells (see Figure 2).

Geospatial combinatorial complexes.
Points, polylines, and polygons constitute some of the most basic geometric objects in geospatial data and are widely used throughout geographic information systems. These objects can be modeled as cells of a CC and their rank correspond to their intrinsic dimensionality. Fig. 2. illustrates this scenario, where 0-cells are points where air quality monitoring units are located (e.g., in a city), 1-cells are traffic roads represented as polylines, and 2-cells are census tracts represented as polygons, geographic areas with homogeneous demographic characteristics. This CC is an example of a *Geospatial CC*, characterized by a spatial representation and a spatially-determined neighborhood system. In general, and formally, let $\mathcal{T}\subset\mathbb{R}^{n}$ represent a geospatial domain represented as a compact subset of Euclidean space. Then, there exists a function $s:\mathcal{X}\to\mathcal{T}$ mapping each cell to its geographic space $s(x)\subset\mathcal{T}$. Cells of a geospatial CC have a dual view. From the CC perspective, a cell $x$ such as a road (polyline) or census tract (polygon) is represented by the points it contains. On the other hand, neighborhood functions can be constructed from the geographic space $s(x)$ corresponding to $x$, such as:

$$ $\displaystyle\mathcal{N}_{A}(x)=\{y\in\mathcal{X}\mid\mathrm{rk}(y)=\mathrm{rk }(x),s(x)\cap s(y)\neq\emptyset\}$ $\displaystyle\mathcal{N}_{I,\uparrow}(x)=\{y\in\mathcal{X}\mid\mathrm{rk}(y)= \mathrm{rk}(x)+1,s(x)\cap s(y)\neq\emptyset\}$ $\displaystyle\mathcal{N}_{I,\downarrow}(x)=\{y\in\mathcal{X}\mid\mathrm{rk}(y) =\mathrm{rk}(x)-1,s(x)\cap s(y)\neq\emptyset\}.$ (12) $$

For instance, $\mathcal{N}_{A}(x)$ can represent adjacencies between geographic spaces that share a portion or a boundary, while the up and down incidences $\mathcal{N}_{I,\uparrow}(x)$ and $\mathcal{N}_{I,\downarrow}(x)$ can capture the notion of road crossing a county, or two geographic areas of different rank overlapping. Evidently, more than one neighborhood function can be used or combined with the standard choices in (1) -(2 Combinatorial Complexes and Equivariance) according to the application. It is not always necessary to use points and polylines. One can also consider geographic divisions (polygons) with multiple resolution levels, each mapping to a different rank. For example, in the United States,
census tracts are smaller than zip codes, which are smaller than counties, and so on. A census tract can intersect multiple zip codes, and zip codes can belong to multiple counties, partitioning space irregularly. As an example, Fig. 2 portrays a road polyline traversing multiple census tract polygons. Unlike classical multi-resolution spatial modeling
(Kolaczyk & Huang, ; Hengl et al., ), we don’t require regular structures such as grids or nested partitions. More importantly, we allow each resolution to have unique features. This point illustrates a crucial difference and advantage over graph modeling, which requires all data to be pre-aggregated at the same rank (typically the 0-cells), see Appendix H.
This work is the first to explore combinatorial topological modeling of multi-resolution irregular geospatial data, which are the backbone of geographic information systems.
The dataset and hyper-local air pollution downscaling task that we will present in the next section will not only propose a novel technique for learning from irregular multi-resolution geospatial data, but can be used as a benchmark for TDL.

## 5 Experiments

Molecular property prediction. We evaluate our model on molecular property prediction using the QM9 dataset (Ramakrishnan et al., ), a comprehensive collection of quantum chemical calculations for small organic molecules. It includes geometric, energetic, electronic, and thermodynamic properties for 134k stable molecules with up to nine heavy atoms (C, O, N, F). See Appendix I.1.1 for a summary.

We consider several different variants of ETNNs (see Appendix J), but the most general molecular combinatorial complexes (CCs) that we design is the one as in Fig. 2. It employs three ranks to capture different levels of molecular interactions: i) atoms as 0-cells, ii) bonds, i.e., pairs of atoms, as 1-cells, and iii) rings and functional groups of size greater than two, i.e., at least triplets of atoms, as 2-cells.
Similar to methodologies for global connections (e.g. EGNN(Satorras et al., ) and VN-EGNN (Sestak et al., )), we generalize the notion of virtual node by incorporating a single cell, that we refer to as virtual cell, containing the whole set of 0-cells (the atoms), and having maximum rank, e.g. in the most general case it would have rank 3. As neighborhood functions, we employed the up/down adjacencies and incidences from (1)-(2 Combinatorial Complexes and Equivariance), and a "max" adjacency defined as

$$ $\mathcal{N}_{A,\textrm{max}}(x)=\{y\in\mathcal{X}|\mathrm{rk}(y)=\mathrm{rk}(x ),\exists z\in\mathcal{X}:\mathrm{rk}(z)=\underset{z\in\mathcal{X}}{\textrm{ max }\mathrm{rk}(z)},y\subset z,\text{ and }y\subset z\}$ (13) $$

The virtual cell together with the max adjacency in (13) acts as a hub across all cells of the same rank, connecting each pair, following a common practice (Satorras et al., ; Eijkelboom et al., ). However, the max adjacency is an object of independent interest, even when the virtual cell is not in the complex. In that case, it would allow cells of lower ranks to communicate based on the highest order cells they are part of, e.g. atoms belonging to the same ring or functional group. In this setting, it is clear how ETNN naturally handles heterogeneity, e.g., the same pair of bonds could be connected because part of the same functional group (up adjacency) and the virtual cell (max adjacency), but the two messages will be different across the neighborhoods.
Similarly, ETNN allows us to equip each rank with unique features without any ad-hoc aggregations at the atom or bond levels. We consider four distinct types of non-geometric feature vectors: atom features, bond features, ring features, and functional group features. We provide a full list of the non-geometric feature vectors we used in Appendix I.1.2. As geometric features, we use the atoms’ positions. The complete list of the model parameters is in Appendix I.1.3. A description of all the configurations and a detailed reproducibility discussion is found in Appendix J. In all the experiments, following (Satorras et al., ; Eijkelboom et al., ), we use the invariant version of ETNN.

**Table 1: Mean Absolute Error for the molecular property prediction benchmark in QM9 dataset. ETNN-* corresponds to the configurations, among the ones we tested, that produced the best results for each property. ETNN-single-* corresponds to the configuration, among the ones we tested, that produced the average best results across properties. ETNN-graph-W and EMPSN* are EGNN (Satorras et al., ) and EMPSN (Eijkelboom et al., ) derived as instances of ETNNs through our codebase. All the models except ETNN-w/o VC use the virtual cell. We report the improvement of ETNN-* over the original EGNN. TopNet^† is the best performing variant of TopNets (Verma et al., ). The other baselines are from (Satorras et al., ). Details and full results in Appendix J.**
| Task | $\alpha$ | $\Delta\varepsilon$ | $\varepsilon_{\mathrm{HOMO}}$ | $\varepsilon_{\mathrm{LUMO}}$ | $\mu$ | $C_{\nu}$ |
| --- | --- | --- | --- | --- | --- | --- |
| Units ($\downarrow$) | bohr^3 | meV | meV | meV | D | cal/mol K |
| NMP | .092 | 69 | 43 | 38 | .030 | .040 |
| Schnet | .235 | 63 | 41 | 34 | .033 | .033 |
| Cormorant | .085 | 61 | 34 | 38 | .038 | .026 |
| L1Net | .088 | 68 | 46 | 35 | .043 | .031 |
| LieConv | .084 | 49 | 30 | 25 | .032 | .038 |
| DimeNet++* | .044 | 33 | 25 | 20 | .030 | .023 |
| TFN | .223 | 58 | 40 | 38 | .064 | .101 |
| SE(3)-Tr. | .142 | 53 | 35 | 33 | .051 | .054 |
| Equiformer (SotA) | .046 | 30 | 15 | 14 | .011 | .023 |
| TopNet^† | .083 | 47 | 37 | 24 | .035 | .032 |
| EGNN | .071 | 48 | 29 | 25 | .029 | .031 |
| ETNN-* | .062 | 45 | 26 | 22 | .022 | .030 |
| ETNN-graph-W | .067 | 46 | 27 | 25 | .030 | .036 |
| ETNN-w/o VC | .161 | 73 | 49 | 48 | .306 | .051 |
| EMPSN* | .061 | 42 | 29 | 25 | .030 | .028 |
| ETNN-single-* | .069 | 46 | 26 | 23 | .025 | .033 |
| \hdashlineImprovement over EGNN | -13% | -6% | -10% | -12% | -26% | -3% |

In Table 1, we present the results for the first 6 molecular properties, from which it is clear that ETNN significantly outperforms EGNN (Satorras et al., ). Details and full results for all the properties are described
in Appendix J. ETNN generally improves the prediction performance of EGNN, with notable improvements on $\alpha$ and $\mu$, achieving state-of-the-art performance for $\mu$ despite being a general framework not tailored for chemical tasks. To ensure further fairness in the comparison, the table shows the results of a 1.5M EGNN obtained through our framework and codebase (EGNN-graph-W), as explained in Appendix J. Comparing this with the original EGNN, we observe performance improvements. Finally, the low prediction performance of ETNN-w/o VC (ETNN without the virtual cell) highlights the significant impact of the virtual cell. The reproduced EMPSN* (Eijkelboom et al., ) is obtained by running ETNN on a combinatorial complex being the same exact Vietoris-Rips simplicial complex of the original paper, similarly using the up/down adjacencies and incidences plus the max adjacency, and with architectural hyperparameters as similar as possible to the original architecture. See Appendix J for a detailed discussion. As the reader can see, ETNN generally matches or outperform the reproduced EMPSNs. However, ETNN employs almost two orders of magnitude less cells. In particular, the original VR complex of dimension 2 used in EMPSN has 555.25 cells per molecule on average (18 nodes, 157.9 edges, 379.36 triangles) and it is also expensive to compute in the preprocessing step, while our molecular CC (in the worst case) has 39.8 cells per molecule on average (18 nodes, 19 edges, 1.8 rings, and 1.3 functional groups) and it requires slightly more than a lookup table to be computed. As a consequence, ETNNs have half of the memory usage and almost half of the runtime per epoch w.r.t. EMPSNs, as we describe in Appendix G. This fact ultimately showcases the effectiveness of the principled inductive bias given by the molecular CC.

Hyperlocal air pollution downscaling.
Next, we introduce a novel benchmark for TDL based on the geospatial framework introduced in 4 Real-World Combinatorial Complexes. We first provide an overview of the data and tasks for completeness.

**Table 2: Root Mean Square Error for the *air pollution downscaling* benchmark.**
| Baseline | RMSE |
| --- | --- |
| Linear | 1.040 |
| MLP | 0.954 |
| GNN | 0.972 |
| EGNN | 0.979 |
| ETNN | 0.939 |

The task consists of predicting $\textrm{PM}_{2.5}$ air pollution at a high resolution of $0.0002^{\circ}\approx 22m$. The prediction targets of $\textrm{PM}_{2.5}$ measurements are obtained from a publicly available dataset (Wang et al., ) consisting of measurements by mobile air sensors installed on cars, corresponding to the last quarter of 2021 in the Bronx, NY, USA. The targets are aggregated at the desired spatial resolution. The resulting Geospatial CC consists of 3,946 point measurement units (0-cells), 550 roads (1-cells), and 151 census tracts (2-cell). Fig. 3 provides an overview of the features for each rank, including demographics, land-use information, daily traffic rates and traffic composition per road, and distance to schools and other points of interest. The most important predictor for the downscaling task is the coarse-level annual PM 2.5 from the previous year (Di et al., ), available at a coarser 1 km resolution which is projected to the roughly equivalent census-tract area level. In total, we include 5 point-level features, 2 road-level features, and 17 tract-level features. The full list of features and their data sources is in Section I.2.2. All the pre-processing code is included in our code repository.

We create a geospatial CC using the neighborhood functions described by 4 Real-World Combinatorial Complexes. Since only ETNN can support multi-level features, we create point-level features for MLP, GNN, and EGNN by concatenating the features of each point with the features of the 1-cell and 2-cell that the point belongs to. For the base ETNN, we again employ a virtual cell as defined in the previous paragraph. Since the task is at the node level, the base implementation uses the equivariant update from 1. We will perform ablation experiments over these defaults. Section I.2.3 presents in detail the hyper-parameter and architecture configurations used for the experiments.x

The experiment results are shown in Table 2(a). The results are averaged over 3 seeds. The table indicates that ETNN outperforms EGNN, GNN, and MLP, emphasizing the benefits of multi-level modeling combined with $E(n)$ equivariance. We additionally conduct ablations studies on ETNN, presented in Table 2 (b). The results show that, as expected, not including geometric features decreases performance. Perhaps surprisingly, the ablation of ETNN using only the geospatial CC structure without geometric features (invariants) performed slightly better than using the features but not using the position update in (7), i.e., using the invariant version of ETNN. Nonetheless, the largest decrease came from not using the virtual cell.

Figure: Figure 3: Overview of features used in the *air pollution downscaling* benchmark. The task is to predict the local $\mathrm{PM}_{2.5}$ measured at the 0-cells, which have an approximate resolution of $22\times 22$ meters. As such, this is a node regression task.
Refer to caption: extracted/5899612/Figures/downscaling_features_hyperlocal.png

## 6 Conclusion

We introduced E(n) Equivariant Topological Neural Networks (ETNNs), a framework for designing scalarization-based equivariant message passing networks on combinatorial complexes. Combinatorial complexes flexibly represent arbitrary hierarchical higher-order interactions, thus ETNNs are a legitimate proxy for designing equivariant models on a wide class of combinatorial topological spaces, such as simplicial or cell complexes, or, more in general, over combinatorial objects used to model interactions among entities, such as hierarchical graphs. We showed that ETNNs are at least as expressive as existing scalarization-based equivariant architectures for geometric graphs. Experiments on the QM9 molecular benchmark demonstrated ETNNs’ effectiveness for richly structured molecular data.
Moreover, we introduced a novel geographic regression task, highlighting ETNNs’ ability and showing for the first time how combinatorial complexes can be used to model irregular multi-resolution geospatial data. We discuss the limitations and future directions in Appendix A. ETNNs represent, overall, a unifying framework showcasing the benefits of a principled synergy between combinatorial and geometric equivariances over beyond-graph domains. We believe that our methodological contribution, together with the easy-to-use and solid codebase we provide, can mark a step in boosting and standardizing research for scalarization-based equivariant TDL.

## Reproducibility Statement

We include all the details about our experimental setting, e.g. the choice of hyperparameters and the specifications of our machine, in Appendix I.
The code, data splits, and virtual environment needed to replicate the experiments and easily use the ETNN framework are provided [here](https://github.com/NSAPH-Projects/topological-equivariant-networks)(^2^22[https://github.com/NSAPH-Projects/topological-equivariant-networks](https://github.com/NSAPH-Projects/topological-equivariant-networks)). An in-depth discussion about the reproducibility of the molecular property prediction task is in Appendix J.

## Appendix A Limitations & Future Directions

To our knowledge, ETNN is the first unifying framework for scalarization-based $E(n)$ equivariant networks over combinatorial topological spaces or, more in general, over combinatorial objects used to model interactions among entities. In this work, we introduced ETNNs as a significantly more general, expressive, and flexible framework compared to prior geometric/topological deep learning models operating on geometric graphs (Satorras et al., ) or simplicial complexes (Eijkelboom et al., ). The promising results open several avenues.

Methodological. First, while we introduced several useful invariants like pairwise distances, centroids, Hausdorff distances, and convex hull volumes, there are likely many other meaningful geometric quantities that could be leveraged, especially for specific application domains. Therefore, developing a more comprehensive set of geometric invariants, as well as learning methods to discover them from data automatically, is an important future direction. In this sense, generalizing the approach from (Liu et al., ) of integrating Clifford group-equivariant layers with message passing layers. i.e., going beyond scalarization, is a possible solution, although requiring to solve possible scalability issues. Second, extending our framework to make it scale invariant could be useful in geospatial tasks. Third, at this stage ETNNs cannot directly handle time-varying scenarios, thus we aim to go beyond the static settings and develop dynamic or temporal variants of ETNNs. This could enable new applications in areas like spatio-temporal forecasting, and physical simulations. Fourth, and finally, ETNNs assume that (potentially) each rank should satisfy the same $E(n)$ equivariance, but another interesting direction is studying the setting in which each rank is required to satisfy different symmetries, pointing towards a principled use of products of groups.

Computational. While our experimental validation of ETNNs covered both supervised inductive and semisupervised transductive tasks, future works could tailor ETNNs to fit challenging specific applications at scale. In this sense, unsupervised or self-supervised training methods for ETNNs could enable the learning of meaningful representations in a data-driven manner without requiring labeled data for every task. This could allow ETNNs to better leverage large, unlabeled geometric datasets. Moreover, though computationally tractable, ETNNs may benefit from further improvements to effectively scale on very large datasets. The main bottlenecks are MP operations and geometric invariants computation. A possible solution could be neighbor samplers (Zhang et al., ), or developing $E(n)$ equivariant message passing-free ETNNs, building on works like (Maggs et al., ).

## Appendix B Related Works

Topological Deep Learning. TDL is informed by foundational work in Topological Signal Processing (TSP) (Barbarossa & Sardellitti, ; Schaub et al., ; Roddenberry et al., ; Sardellitti et al., ), highlighting the value of analyzing multi-way relationships in data. The works in (Bodnar et al., ; ) extended the Weisfeiler-Lehman graph isomorphism test to simplicial and regular cell complexes (Bodnar et al., ), respectively, showing that message passing over these spaces are more expressive than message passing over graphs. Convolutional (Ebli et al., ; Yang et al., ; Hajij et al., ; Yang & Isufi, ; Roddenberry et al., ; Hajij et al., ) and attentional architectures (Battiloro et al., ; Giusti et al., ; Goh et al., ; Giusti et al., ) over simplicial and cell complexes have been introduced as well. Additionally, the application of message passing or diffusion on cellular sheaves (Hansen & Ghrist, ) over graphs (Hansen & Gebhart, ; Bodnar et al., ; Battiloro et al., ; ; Barbero et al., ) has proven effective in heterophilic scenarios. Models without message passing have been introduced for simplicial complexes (; Gurugubelli & Chepuri, ; Maggs et al., ) and hypergraphs (Tang et al., ). An architecture for inferring a latent regular cell complex to improve a downstream task has been introduced in (Battiloro et al., ). Gaussian Processes over cell complexes have been introduced in (Alain et al., ). An extensive review of TDL is available in (Papillon et al., ). A comprehensive framework for TDL was outlined in (Hajij et al., ), proposing combinatorial complexes and message-passing networks over them. Finally, software for neural networks on combinatorial topological spaces has been presented in (Hajij et al., ; Telyatnikov et al., ).

Equivariant Neural Networks
The effectiveness of Convolutional Neural Networks (He et al., ), and their first group extension to the SO(2) group (Cohen & Welling, ) initially demonstrated the advantages of equivariant models, and paved the way for a series of works exploring how to design models respecting some symmetries (Weiler et al., ). Loosely speaking, some of these works employ natively equivariant function spaces (Thomas et al., ; Fuchs et al., ), or port the
spatial space into high-dimensional spaces (Cohen et al., ; Finzi et al., ; Hutchinson et al., ; Batatia et al., ). An alternative strategy is designing neural networks (Köhler et al., ; Schütt et al., ; Brandstetter et al., ; Batzner et al., ) that perform equivariant operations directly within the original space, including the (VN-)EGNN from (Satorras et al., ; Sestak et al., ). A characterization theorem for equivariant networks has been presented in (Pacini et al., ). Recent works showed how to achieve equivariance by leveraging neural network architectures operating on the Clifford
algebra (Ruhe et al., ; Brehmer et al., ). Each of these approaches has limitations and advantages, and the effectiveness of a specific design choice is related to the considered application and computational constraints (Han et al., ).

Our paper builds on both classes of works. To the best of our knowledge, the most relevant works merging TDL models and $E(n)$ equivariant neural networks are (Eijkelboom et al., ; Liu et al., ; Verma et al., ). In particular, (Eijkelboom et al., ) generalizes the approach from (Satorras et al., ) and introduces $E(n)$ Equivariant Simplicial Message passing Networks (EMPSNs). The framework in (Verma et al., ) injects information about persistent homology (PH) on top of (equivariant) simplicial message passing networks, thus interestingly combining TDL and Topological Data Analysis. Finally, (Liu et al., ) adopts the approach from (Ruhe et al., ) by combining Clifford group equivariant layers with simplicial message passing networks. However, no other combinatorial topological spaces are considered, and, overall, a general framework for designing equivariant neural networks over combinatorial topological spaces is missing. Here we address this important gap by introducing our ETNNs.

## Appendix C Combinatorial Topological Spaces as Combinatorial Complexes

In this appendix, we revisit a subclass of mathematical objects (not always formal topological spaces) that can be described combinatorially, message passing networks operating on them, and how they can be cast in the ETNN frameworks. As we did for CCs, we will often overload the notation $\mathcal{X}$ in the following subsections, incurring some notation abuses for the sake of exposition and consistency.

### C.1 Graphs

The first combinatorial objects we describe are obviously graphs. They are remarkably useful in several fields, across several tasks, thanks to their ability to encode prior data knowledge, i.e., their networked structure.

Graphs. A graph $\mathcal{X}$ is a pair $(\mathcal{S},\mathcal{E})$, where $\mathcal{S}$ is the set of nodes, and $\mathcal{E}$ is the set of edges (i.e., pairs of nodes).

Neighbhorhood Functions in a Graph. Given a graph $\mathcal{X}$, for all $x\in\mathcal{S}$, the up adjacency $\mathcal{N}_{A,\uparrow}$ (i.e., the usual node adjacency) is defined as

$$ $\mathcal{N}_{A,\uparrow}(x)=\{y\in\mathcal{S}|\exists z\in\mathcal{E}:x\subset z \textrm{ and }y\subset z\}.$ (14) $$

Graphs as Combinatorial Complexes. Graphs can be easily cast as combinatorial complexes. Let us denote the set of singletons $\widetilde{\mathcal{S}}=\{\{s\}\}_{s\in\mathcal{S}}$, a graph $\mathcal{X}$ is a combinatorial complex $(\mathcal{S},\mathcal{X},\mathrm{rk})$, where $\mathcal{S}$ is the set of nodes, the set of cells $\mathcal{X}=\widetilde{\mathcal{S}}$ is the set of singletons, and the rank function $\mathrm{rk}$ assigns rank 0 to each singleton. Please notice that there is no need to include edges as cells in the complex, differently from, e.g., simplicial complexes, as detailed in the next subsection, because a CCMPN as in (3) with $\mathcal{CN}=\{\mathcal{N}_{A,\uparrow}\}$ recovers the standard Graph Message Passing Network from (Gilmer et al., ). An ETNN as in (6) with the same collection of neighborhoods $\mathcal{CN}$ recovers the $E(n)$ Equivariant Graph Neural Network (EGNN) from (Satorras et al., ).

### C.2 Simplicial Complexes

Graphs are capable of explicitly managing only pairwise interactions among nodes. However, in numerous applications (Giusti et al., ; Kanari et al., ; Patania et al., ; Sardellitti et al., ), including the ones presented in this work, higher-order multiway interactions need to be taken into account. Simplicial complexes are one of the most intuitive objects to start incorporating them.

Simplicial Complexes. Given a finite set of nodes $\mathcal{S}$, a $k$-simplex $x$ is a subset of $\mathcal{S}$ with cardinality $k+1$. A face of $x$ is a $k-1$-simplex being a subset of it, thus a $k$-simplex has $k+1$ faces. A coface of $x$ is a $(k+1)$-simplex being a superset of it (Barbarossa & Sardellitti, ; Lim, ). A simplicial complex (SC) $\mathcal{X}$ of order $K$ is a collection of $k$-simplices $x$, $k=0,\ldots,K$ such that, if a simplex $x$ belongs to $\mathcal{X}$, then all its subsets $y\subset x$ also belong to the complex (inclusivity property). In most of the cases, the focus is on complexes of order up to two, thus having a set of nodes, a set of edges, and a set of triangles. To give a simple example, in Fig. 4 we sketch a simplicial complex of order $2$.

Figure: Figure 4: A simplicial complex of order 2, comprising of nodes (cells of rank 0), edges (cells of rank 1), and triangles (cells of rank 2).
Refer to caption: extracted/5899612/Figures/sc.png

Neighbhorhood Functions in a Simplicial Complex. Given a SC $\mathcal{X}$, the coboundary and boundary neighborhood functions $\mathcal{N}_{C}$ and $\mathcal{N}_{B}$, respectively, are defined, as

$$ $\mathcal{N}_{C}(x)=\{y\in\mathcal{X}|y\textrm{ is a coface of }x\},\quad \mathcal{N}_{B}(x)=\{y\in\mathcal{X}|y\textrm{ is a face of }x\}.$ (15) $$

The up and down adjacencies $\mathcal{N}_{A,\uparrow}$ and $\mathcal{N}_{A,\downarrow}$, respectively, are defined, as

$$ $\displaystyle\mathcal{N}_{A,\uparrow}(x)=\{y\in\mathcal{X}|y\textrm{ shares a coface with }x\},\;\mathcal{N}_{A,\downarrow}(x)=\{y\in\mathcal{X}|y\textrm{ shares a face with }x\}.$ (16) $$

Simplicial Complexes as Combinatorial Complexes. It is easy to frame SCs in the CC framework. In particular, a simplicial complex $\mathcal{X}$ is a combinatorial complex $(\mathcal{S},\mathcal{X},\mathrm{rk})$, where $\mathcal{S}$ is the set of nodes, the set of cells $\mathcal{X}$ is the set of simplices, and the rank function $\mathrm{rk}$ assigns to each cell its cardinality minus one. A CCMPN as in (3) with $\mathcal{CN}=\{\mathcal{N}_{C},\mathcal{N}_{B},\mathcal{N}_{A,\uparrow},
\mathcal{N}_{A,\downarrow}\}$ recovers the Message Passing Simplicial Network from (Bodnar et al., ). An ETNN as in (6) with the same collection of neighborhoods $\mathcal{CN}$ recovers the $E(n)$ Equivariant Message Passing Simplicial Network (EMPSN) from (Eijkelboom et al., ).

### C.3 Cell Complexes

Simplicial complexes are capable of managing relationships of varying degrees, yet the inclusion property often binds them. This property mandates that if a set is part of the space, then all its respective subsets must also be included in that space. In numerous applications, this restriction is not only unnecessary but also lacks a justified basis. Consequently, cell complexes are frequently employed instead (Mulder & Bianconi, ; Hirani et al., ).
These structures are versatile and support a hierarchical organization.

Regular Cell Complexes. A regular cell complex (or regular CW complex) is a topological space $\mathcal{X}$ together with a partition $\{\mathcal{X}_{x}\}_{x\in\mathcal{H}_{\mathcal{X}}}$ of subspaces $\mathcal{X}_{x}$ of $\mathcal{X}$ called cells, where $\mathcal{H}_{\mathcal{X}}$ is the indexing set of $\mathcal{X}$, such that

- 1.
For each $x$ $\in$ $\mathcal{X}$, every sufficient small neighborhood of $x$ intersects finitely many $\mathcal{X}_{x}$;
- 2.
For all $x$,$y$ we have that $\mathcal{X}_{x}$ $\cap$ $\overline{\mathcal{X}}_{y}$ $\neq$ $\varnothing$ iff $\mathcal{X}_{x}$ $\subseteq$ $\overline{\mathcal{X}}_{y}$, where $\overline{\mathcal{X}}_{y}$ is the closure of the cell;
- 3.
Every $\mathcal{X}_{x}$ is homeomorphic to $\mathbb{R}^{k}$ for some $k$;
- 4.
For every $x$ $\in$ $\mathcal{H}_{\mathcal{X}}$ there is a homeomorphism $g$ of a closed ball in $\mathbb{R}^{k}$ to $\overline{\mathcal{X}}_{x}$ such that the restriction of $g$ to the interior of the ball is a homeomorphism onto $\mathcal{X}_{x}$.

From 2, $\mathcal{H}_{\mathcal{X}}$ has a poset structure, given by $y$ $\leq$ $x$ iff $\mathcal{X}_{y}$ $\subseteq$ $\overline{\mathcal{X}_{x}}$, and we say that $y$ bounds $x$. This is known as the face poset of $\mathcal{X}$. From 4, all of the topological information about $\mathcal{X}$ is encoded in the poset structure of $\mathcal{H}_{\mathcal{X}}$.
Then, a regular cell complex can be identified with its face poset. From now on we will indicate the cell $\mathcal{X}_{x}$ with its corresponding face poset element $x$. The dimension or order $\textrm{dim}(\sigma)$ of a cell $\sigma$ is $k$, and we call it a $k-$cell. The dimension or order of a regular cell complex is the largest dimension of any of its cells. In the following, we will refer to regular cell complexes simply as cell complexes or CW complexes. Cell complexes can be described via an incidence relation (boundary relation) with a reflexive and transitive closure that is consistent with the partial order introduced in the above definition. In particular, we have the boundary relation $y$ $\prec$ $x$ iff $\dim({y})$ $\leq$ $\dim({x})$ and there is no cell $z$ such that $y$ $\leq$ $z$ $\leq$ $x$. In other words, the boundary of a cell $x$ of dimension $k$ is the set of all cells of dimension less than $k$ bounding $x$. The k-skeleton of a regular cell complex $\mathcal{X}$ is the subcomplex of $\mathcal{X}$ consisting of cells of dimension at most $k$. Usually, CW complexes $\mathcal{X}$ are constructed through a hierarchical
gluing procedure starting from a set of nodes $\mathcal{S}$ (Hansen & Ghrist, ; Bodnar et al., ). Therefore, we can interpret the 0-skeleton of a cell complex $\mathcal{X}$ as the set $\mathcal{S}$ of 0-cells (nodes) and the 1-skeleton as the set of nodes together with the set of 1-cells (edges), thus a graph. For this reason, given a graph $\mathcal{G}$, i.e., a realization of an order-1 cell complex, it is possible to lift it to an order-2 cell complex $\mathcal{X}$ whose 1-skeleton is isomorphic to $\mathcal{G}$. A way to do it is by attaching order 2 cells to the/a set/subset of simple/induced cycles of $\mathcal{G}$. Cells of order greater than 2 can be attached as well using the same gluing procedure mentioned above (Sardellitti et al., ; Grady & Polimeni, ), but there is usually little interest in them. In Fig. 5, we sketch an order 2 CW complex.

Figure: Figure 5: A CW complex of order 2, comprising of nodes (cells of rank 0), edges (cells of rank 1), and cycles (cells of rank 2).
Refer to caption: extracted/5899612/Figures/cw.png

###### Remark 2 .

It is easy to notice that a CW complex considering cycles of length up to 3 is equivalent to a simplicial complex of order 2 having the same set $\mathcal{S}$ of 0-simplices (nodes), the same set of 1-simplices (edges), and the same set of 2-simplices (triangles). In general, simplicial complexes are special cases of CW complexes for which any $k$-cell is composed of exactly $k+1$ nodes.

###### Remark 3 .

The reader may have noticed that we gave no explicit characterization of a simplicial complex as a topological space, although it is pretty easy to particularize cell to simplicial complexes. This is because our definition of SC in Sec. C.2 is, to be completely rigorous, the definition of an abstract simplicial complex, that itself is not a topological space. However, when we refer to a simplicial complex as a topological space, we are implicitly talking about the geometric realization of an abstract simplicial complex. Therefore, we are implicitly stating that the SC has a topology coherent with its simplices, which are "topologized" as homeomorphisms of the standard simplices living in Euclidean space. Please refer to (Barbarossa & Sardellitti, ) for a detailed but accessible description. We will not incur any ambiguity by referring to abstract simplicial complexes as just simplicial complexes.

Neighbhorhood Functions in a Cell Complex. Given a CW complex $\mathcal{X}$, the coboundary and boundary neighborhood functions $\mathcal{N}_{C}$ and $\mathcal{N}_{B}$, respectively, are defined, as

$$ $\mathcal{N}_{C}(x)=\{y\in\mathcal{X}|y\prec x\},\quad\mathcal{N}_{B}(x)=\{y\in \mathcal{X}|x\prec y\}.$ (17) $$

The up and down adjacencies $\mathcal{N}_{A,\uparrow}$ and $\mathcal{N}_{A,\downarrow}$, respectively, are defined, as

$$ $\displaystyle\mathcal{N}_{A,\uparrow}(x)=\{y\in\mathcal{X}|\exists z\textrm{ such that }x\prec z,y\prec z\},\;\mathcal{N}_{A,\downarrow}(x)=\{y\in\mathcal{ X}|\exists z\textrm{ such that }z\prec x,z\prec y\}.$ (18) $$

Cell Complexes as Combinatorial Complexes. It is again straightforward to frame CW complexes in the CC framework. In particular, a cell complex is a combinatorial complex $(\mathcal{S},\mathcal{X},\mathrm{rk})$, where $\mathcal{S}$ is the set of nodes, $\mathcal{X}$ is the set of cells, and the rank function $\mathrm{rk}$ assigns to each cell its dimension $\mathrm{dim}$. A CCMPN as in (3) with $\mathcal{CN}=\{\mathcal{N}_{B},\mathcal{N}_{A,\uparrow}\}$ on a CW complex recovers the Molecular Message Passing Network from (Bodnar et al., ). ETNNs as in (6) with the same collection of neighborhoods $\mathcal{CN}$ give rise to $E(n)$ Equivariant Molecular Message Passing Networks, not present in literature(^3^33This approach was recently explored in (Kovač et al., ), appeared online only after we released the first preprint of ETNN.).

###### Remark 4 .

Simplicial Complexes and Cell Complexes are rich topological objects. Neighborhoods in a simplicial or a cell complex are strongly rooted in arguments from algebraic topology. In particular, Hodge theory (Griffiths & Schmid, ) is used to derive formal notions of boundary, coboundary, and Hodge Laplacians (Barbarossa & Sardellitti, ). More detailed descriptions can be found in (Grady & Polimeni, ; Barbarossa & Sardellitti, ; Battiloro et al., ; Ribando-Gros et al., ). In this work, we are interested in the definition of cells (in the CC sense) and neighborhood functions that these objects (SCs and CWCs) induce. For this reason, a reader familiar with algebraic topology could find our definitions and treatment not totally comprehensive. However, we aim to provide a unifying framework in the context of (equivariant) TDL, thus our presentation is sufficiently consistent and strikes a good tradeoff between generality, rigor, and applicability.

###### Remark 2 .

###### Remark 3 .

###### Remark 4 .

### C.4 Hypergraphs

Cell complexes have their formal hierarchical and topology-grounded characterization as their main strength. However, their structure is still constrained by the homeomorphism requirements, preventing them from modeling arbitrary higher-order hierarchical interactions. On the other hand, hypergraphs are combinatorial objects that have total flexibility, but without any hierarchical and (in the general case) topological structure.

Hypergraphs. A hypergraph (HG) on a nonempty set of nodes $\mathcal{S}$ is a pair $(\mathcal{S},\mathcal{E})$, where $\mathcal{X}$ is a subset of the power set of $\mathcal{P}(\mathcal{S})\backslash\{\emptyset\}$ of $\mathcal{S}$. Elements of $\mathcal{S}$ are called hyperedges. In Fig. 6, we sketch a hypergraph with 3 nodes and 3 hyperedges.

Neighbhorhood Functions in a Hypergraph. Let us again denote the set of singletons $\widetilde{\mathcal{S}}=\{\{s\}\}_{s\in\mathcal{S}}$. Given a HG $\mathcal{X}$, for all $x\in\widetilde{\mathcal{S}}\cup\mathcal{E}$, the up and down incidences $\mathcal{N}_{I,\uparrow}$ and $\mathcal{N}_{I,\downarrow}$, respectively, are defined as

$$ $\mathcal{N}_{I,\uparrow}(x)=\{y\in\widetilde{\mathcal{S}}\cup\mathcal{E}|x \subset y\},\quad\;\mathcal{N}_{I,\downarrow}(x)=\{y\in\widetilde{\mathcal{S}} \cup\mathcal{E}|y\subset x\}.$ (19) $$

Therefore, if $x$ is a node, the down and up incidences evaluated at $x$ are the empty set and the hyperedges containing $x$, respectively. If $x$ is a hyperedge, then the down and up neighborhood functions evaluated at $x$ are the nodes contained in $x$ and the empty set, respectively.

Hypergraphs as Combinatorial Complexes. Even in this case, framing HGs in the CC framework is straightforward. In particular, a hypergraph $\mathcal{X}$ is a combinatorial complex $(\mathcal{S},\mathcal{X},\mathrm{rk})$, where $\mathcal{S}$ is the set of nodes, the set of cells $\mathcal{X}=\widetilde{\mathcal{S}}\cup\mathcal{E}$, and the rank function $\mathrm{rk}$ assigns rank 0 to the nodes and rank 1 to the hyperedges. A CCMPN as in (3) with $\mathcal{CN}=\{\mathcal{N}_{I,\uparrow},\mathcal{N}_{I,\downarrow}\}$ recovers the Hypergraph Message Passing Networks from (Feng et al., ) (in the convolutional realization) and (Heydari & Livi, ). ETNNs as in (6) with the same collection of neighborhoods $\mathcal{CN}$ give rise to $E(n)$ Equivariant Hypergraph Message Passing Networks, not present in literature.

Figure: Figure 6: A hypergraph, comprising of nodes (cells of rank 0), hyperedges (cells of rank 1).
Refer to caption: extracted/5899612/Figures/hg.png

## Appendix D ETNNs with Velocity Type Inputs

As EGNNs, ETNNs can handle settings in which nodes come with velocity type geometric features. In this case, we have non-geometric feature $\mathbf{h}_{x}\in\mathbb{R}^{F}$, and positions $\mathbf{x}_{z}\in\mathbb{R}^{n}$ and velocities $\mathbf{v}_{z}$ for cell $z\in\mathcal{S}$. We simply need to modify the position update in (7) to include the velocity update. Let us denote the velocity of the $z$-th node at layer $l$ with $\mathbf{v}_{z}^{l}$. At the $l$-th layer, an ETNN updates the embeddings $\mathbf{h}_{x}^{l}$ of every cell $x\in\mathcal{X}$ as in (6), and the position $\mathbf{x}_{z}^{l}$ and velocity $\mathbf{v}_{z}^{l}$ of every node $z\in\mathcal{S}$ as

$$ $\displaystyle\mathbf{v}_{z}^{l+1}=\zeta\left(\mathbf{h}_{z}^{l}\right)\mathbf{ v}_{z}+C\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in\mathcal{S}:\{t\}\in \mathcal{N}(z)}\left(\mathbf{x}_{z}^{l}-\mathbf{x}_{t}^{l}\right)\xi\left( \mathbf{m}_{z,t}^{\mathcal{N}}\right),\quad\mathbf{x}_{z}^{l+1}=\mathbf{x}_{z} ^{l}+\mathbf{v}_{z}^{l+1}.$ (20) $$

## Appendix E Proof of Theorem 1

The proof of Theorem 1 is a straightforward generalization of the proof provided for Equivariant Graph Neural Networks (EGNN) in (Satorras et al., ). In particular, we can directly state that the update of the features in (6) is $E(n)$ invariant because the positions $\{\mathbf{x}_{z}\}_{z\in x}$ are used only as input to the geometric invariant(s) $\mathrm{Inv}$ and the features $\mathbf{h}_{x}$ contain no geometric information, i.e.

$$ $\mathbf{h}_{x}^{l+1}=\beta\left(\mathbf{h}^{l}_{x},\bigotimes_{\mathcal{N}\in \mathcal{CN}}\bigoplus_{y\in\mathcal{N}(x)}\underbrace{\psi_{\mathcal{N}, \mathrm{rk}(x)}\left(\mathbf{h}^{l}_{x},\mathbf{h}^{l}_{y},\mathrm{Inv}\left( \{\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}\}_{z\in x},\{\mathbf{O}\mathbf{x}_{z }^{l}+\mathbf{b}\}_{z\in y}\right)\right)}_{\mathbf{m}_{x,y}^{\mathcal{N}}} \right),$ (21) $$

for all $x\in\mathcal{X}$ and $(\mathbf{O},\mathbf{b})\in E(n)$. Therefore, we just need to show that the update of the positions in (7) is $E(n)$ equivariant, i.e., we want to prove that

$$ $\mathbf{O}\mathbf{x}_{z}^{l+1}+\mathbf{b}=\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf {b}+C\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in S:\{t\}\in\mathcal{N}(z)} \left((\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b})-(\mathbf{O}\mathbf{x}_{t}^{l}+ \mathbf{b})\right)\xi\left(\mathbf{m}_{z,t}^{\mathcal{N}}\right),$ (22) $$

for all $z\in\mathcal{S}$ and $(\mathbf{O},\mathbf{b})\in E(n)$. By direct computation, as showed in (Satorras et al., ), we can write

$$ $\displaystyle\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}+C\sum_{\mathcal{N}\in \mathcal{CN}}\sum_{t\in S:\{t\}\in\mathcal{N}(z)}\left((\mathbf{O}\mathbf{x}_{ z}^{l}+\mathbf{b})-(\mathbf{O}\mathbf{x}_{t}^{l}+\mathbf{b})\right)\xi\left( \mathbf{m}_{z,t}^{\mathcal{N}}\right)$ $\displaystyle=\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}+\mathbf{O}C\sum_{ \mathcal{N}\in\mathcal{CN}}\sum_{t\in S:\{t\}\in\mathcal{N}(z)}\left(\mathbf{x }_{z}^{l}-\mathbf{x}_{t}^{l}\right)\xi\left(\mathbf{m}_{z,t}^{\mathcal{N}}\right)$ $\displaystyle=\mathbf{O}\left(\mathbf{x}_{z}^{l}+C\sum_{\mathcal{N}\in\mathcal {CN}}\sum_{t\in S:\{t\}\in\mathcal{N}(z)}\left(\mathbf{x}_{z}^{l}-\mathbf{x}_{ t}^{l}\right)\xi\left(\mathbf{m}_{z,t}^{\mathcal{N}}\right)\right)+\mathbf{b}= \mathbf{O}\mathbf{x}_{z}^{l+1}+\mathbf{b}.$ (23) $$

Therefore, the proof is concluded as the updates in (6)-(7) are $E(n)$ invariant and equivariant, respectively.

The proof is slightly different if velocities $\mathbf{v}_{z}$ are available. In this case, we want to prove that the updates in (20) are $E(n)$ equivariant. Therefore, for the velocity update, we want to show that

$$ $\mathbf{O}\mathbf{v}_{z}^{l+1}=\zeta\left(\mathbf{h}_{z}^{l}\right)\mathbf{O} \mathbf{v}_{z}^{0}+C\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in S:\{t\}\in \mathcal{N}(z)}\left(\left(\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}\right)- \left(\mathbf{O}\mathbf{x}_{t}^{l}+\mathbf{b}\right)\right)\xi\left(\mathbf{m} _{z,t}^{\mathcal{N}}\right),$ (24) $$

for all $z\in\mathcal{S}$ and $(\mathbf{O},\mathbf{b})\in E(n)$. Again by direct computation, we can write

$$ $\displaystyle\zeta\left(\mathbf{h}_{i}^{l}\right)\mathbf{O}\mathbf{v}_{z}^{0}+ C\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in S:\{t\}\in\mathcal{N}(z)}\left( \left(\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}\right)-\left(\mathbf{O}\mathbf{x }_{t}^{l}+\mathbf{b}\right)\right)\xi\left(\mathbf{m}_{z,t}^{\mathcal{N}}\right)$ $\displaystyle=\mathbf{O}\zeta\left(\mathbf{h}_{i}^{l}\right)\mathbf{v}_{z}^{0} +\mathbf{O}C\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in S:\{t\}\in\mathcal{N}( z)}\left(\mathbf{x}_{z}^{l}-\mathbf{x}_{t}^{l}\right)\xi\left(\mathbf{m}_{z,t} ^{\mathcal{N}}\right)$ $\displaystyle=\mathbf{O}\left(\zeta\left(\mathbf{h}_{i}^{l}\right)\mathbf{v}_{ z}^{0}+C\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in S:\{t\}\in\mathcal{N}(z)} \left(\mathbf{x}_{z}^{l}-\mathbf{x}_{t}^{l}\right)\xi\left(\mathbf{m}_{z,t}^{ \mathcal{N}}\right)\right)=\mathbf{O}\mathbf{v}_{z}^{l+1}.$ (25) $$

It is trivial to see that the position update in (20) is $E(n)$ equivariant. Therefore, the proof is concluded as the updates in (6)-(20) are $E(n)$ invariant and equivariant, respectively.

## Appendix F Expressiveness of ETNNs

The ability of Graph Neural Networks (GNNs) to differentiate between non-isomorphic graphs is typically employed as an expressivity metric. In particular, it is well known that vanilla GNNs are at maximum as powerful as the Weisfeiler-Leman (WL) test (Weisfeiler & Leman, ) in discriminating isomorphic graphs. As such, the WL test sets a limit on the expressiveness of GNNs. To broaden the use of this method to include geometric graphs, the Geometric Weisfeiler-Leman test (GWL) has been introduced (Joshi et al., ). This test evaluates whether two (undirected) graphs are geometrically isomorphic, as defined next.

Geometrically Isomorphic Graphs.(Joshi et al., ; Sestak et al., ) Two geometric graphs $\mathcal{G}_{1}=(\mathcal{S}_{1},\mathcal{E}_{1})$ and $\mathcal{G}_{2}=(\mathcal{S}_{2},\mathcal{E}_{2})$ with $|\mathcal{S}_{1}|=|\mathcal{S}_{2}|$, non-geometric and geometric $i$-th node features $\mathbf{h}_{x_{i}}^{\mathcal{G}_{j}}$ and $\mathbf{x}_{x_{i}}^{\mathcal{G}_{j}}$ ($j\in\{1,2\}$), are called geometrically isomorphic if there exists an edge-preserving bijection $b:\mathcal{I}_{1}\rightarrow\mathcal{I}_{2}$ between their corresponding node indices set $\mathcal{I}_{1}$ and $\mathcal{I}_{2}$, such that their positions are equivalent up to the $E(n)$ group action, i.e.:

$$ $\mathbf{h}_{x_{b(i)}}^{\mathcal{G}_{2}},\mathbf{x}_{x_{b(i)}}^{\mathcal{G}_{2} }=\mathbf{h}_{x_{i}}^{\mathcal{G}_{1}},\mathbf{O}\mathbf{x}_{x_{i}}^{\mathcal{ G}_{1}}+\mathbf{b}$ (26) $$

for all $i\in\mathcal{I}_{1},(\mathbf{O},\mathbf{b})\in E(n)$.

$\mathbf{k}$-hop Distinct Graphs.(Joshi et al., ; Sestak et al., ) Two geometric graphs $\mathcal{G}_{1}$ and $\mathcal{G}_{2}$ are said to be $k$-hop distinct if for all graph isomorphisms $b$, there is some node $i\in\mathcal{I}_{1},b(i)\in\mathcal{I}_{2}$ such that the corresponding $k$-hop neighborhood subgraphs are distinct (not geometrically isomorphic). Otherwise, if they are identical up to $E(n)$ actions for all $i\in\mathcal{I}_{1}$, we say $\mathcal{G}_{1}$ and $\mathcal{G}_{2}$ are said to be $k$-hop identical.

To study Combinatorial Complexes (CCs) and the expressiveness of Equivariant CCMPNs (ETNNs), we introduce the notion of Geometric Augmented Hasse Graph representation of CCs, generalizing (Hajij et al., ) and the notion of Hasse diagram for finite partially ordered sets (Demey & Smessaert, ) to the geometric setting.

Geometric Augmented Hasse Graph of a Combinatorial Complex. Let $(\mathcal{S},\mathcal{X},\mathrm{rk})$ be a CC where each node $z\in\mathcal{S}$ comes with positions $\mathbf{x}_{z}$, and $\mathcal{CN}$ a collection of neighborhood functions. The Geometric Augmented Hasse graph $\mathcal{G}_{\mathcal{X}}$ of $(\mathcal{S},\mathcal{X},\mathrm{rk})$ is a (possibly) directed geometric graph $\mathcal{G}_{\mathcal{X}}=$ $\left(\mathcal{X},\mathcal{E}\right)$ with cells as nodes, edges given by

$$ $\mathcal{E}=\{(y,x)|x\in\mathcal{X},y\in\mathcal{X},\exists\mathcal{N}\in \mathcal{CN}:y\in\mathcal{N}(x)\},$ (27) $$

and positions of cell $x$ being a linear permutation invariant function $\bigoplus_{z\in x}\mathbf{x}_{z}$ of its corresponding node positions$\{\mathbf{x}_{z}\}_{z\in x}$, $z\in\mathcal{S}$.
Therefore, the geometric augmented Hasse graph of a CC is a graph representation of it obtained by considering the cells as nodes, by inserting edges among them if the cells are neighbors in the CC, and by assigning positions to higher-order cells as functions of the nodes (of the CC) they contain. Fig. 7 shows an example of a CC and the corresponding geometric augmented Hasse graph. Given a CC $\mathcal{C}$, a standard graph message-passing network that runs over the augmented Hasse graph $\mathcal{G}_{\mathcal{X}}$ induced by a collection of neighborhoods $\mathcal{CN}$ is equivalent to a specific CCMPN derived as a particular case of (3) and running over $\mathcal{X}$ using $\mathcal{CN}$ (Hajij et al., ). The following also holds.

Figure: (a)
Refer to caption: extracted/5899612/Figures/1_1_cc_ex_expr.png

###### Proposition 1 .

Let $(\mathcal{S},\mathcal{X},\mathrm{rk})$ be a CC where each node $z\in\mathcal{S}$ comes with positions $\mathbf{x}_{z}$, $\mathcal{CN}$ a collection of neighborhood functions, and $\mathcal{G}_{\mathcal{X}}$ the corresponding geometric augmented Hasse graph. Assuming that:

- 1.
a single message function for all neighborhoods and ranks, i.e., $\psi_{\mathcal{N},\mathrm{rk}(x)}=\psi$ in (6);
- 2.
the inter- and intra-neighborhood aggregation functions are the same, i.e., $\bigotimes=\bigoplus$ in
(6);
- 3.
the only employed geometric invariants are sum of distances of linear permutation invariant functions as in (10), e.g. sum or mean.

Then ETNNs over $(\mathcal{S},\mathcal{X},\mathrm{rk})$ are equivalent to EGNNs (Satorras et al., ) over $\mathcal{G}_{\mathcal{X}}$ with the following modified update for positions of node $x$ (of the Hasse graph):

$$ $\mathbf{x}_{x}^{l+1}=\begin{cases}\mathbf{x}_{x}^{l}+C\sum_{\mathcal{N}\in \mathcal{CN}}\sum_{t\in S:\{t\}\in\mathcal{N}(z)}\left(\mathbf{x}_{x}^{l}- \mathbf{x}_{t}^{l}\right)\xi\left(\mathbf{m}_{x,t}\right),\text{ if }x\in \mathcal{S},\\ \mathbf{x}_{x}^{l+1}=\bigoplus_{z\in x}\mathbf{x}_{z}^{l+1},x\in\mathcal{X} \setminus\mathcal{S}\end{cases}$ (28) $$

###### Proof.

The proof is trivial and is obtained by simple direct substitution in (6)-(7).
∎

Intuitively, the modified update is required because higher-order cells do not come with positions, thus they cannot be updated as the positions of the nodes in $\mathcal{S}$.

Lifting of a Graph into a CC. Given a graph, many ways exist to lift it into a combinatorial complex. As shown in Appendix C.1, denoting the set of singletons $\widetilde{\mathcal{S}}=\{\{s\}\}_{s\in\mathcal{S}}$, a graph $\mathcal{G}(\mathcal{S},\mathcal{E})$ is a combinatorial complex $(\mathcal{S},\mathcal{X},\mathrm{rk})$, where $\mathcal{S}$ is the set of nodes, the set of cells $\mathcal{X}=\widetilde{\mathcal{S}}$ is the set of singletons, the rank function $\mathrm{rk}$ assigns rank 0 to each singleton, and the up adjacency from (14) is employed. However, we can enrich the representation by including the edges $\mathcal{E}$ as cells, i.e., $\mathcal{X}=\widetilde{\mathcal{S}}\bigcup\mathcal{E}$, and assign rank 1 to them. In this setting, it is possible to make nodes communicate with edges and vice-versa, rather than just making nodes communicate through edges. In general, a natural question is how to define higher-order cells, i.e., cells of rank greater than 0. We have already shown in Section 4 Real-World Combinatorial Complexes that domain knowledge can be injected by properly modeling higher-order cells. However, we are now interested in understanding which conditions should higher-order cells respect to result in an increased expressive power of ETNN. To perform a fair analysis, we consider only skeleton-preserving lifts (Bodnar et al., ), i.e., lifts in which the original connectivity among the nodes in $\mathcal{S}$ is preserved. In other words, we don’t directly rewire the underlying graph $\mathcal{G}$. As shown in Fig. 7, a straightforward example of skeleton-preserving lifts is choosing (1)-(14) as neighborhood functions without necessarily adding the edges as cells. Another example is choosing (1)-(2 Combinatorial Complexes and Equivariance) as neighborhood functions including the edges as cells.

Expressivity of ETNN. Due to Proposition 1, given two geometric graphs $\mathcal{G}_{1}$ and $\mathcal{G}_{2}$ and two combinatorial complexes $\mathcal{X}_{\mathcal{G}_{1}}$ and $\mathcal{X}_{\mathcal{G}_{2}}$ obtained by respectively lifting them, we can study how expressive ETNNs are by analyzing how the GWL performs on the geometric augmented Hasse graphs $\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{1}}}$ and $\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{1}}}$ of $\mathcal{X}_{\mathcal{G}_{1}}$ and $\mathcal{X}_{\mathcal{G}_{2}}$, respectively, w.r.t. how it would perform directly on the underlying graphs $\mathcal{G}_{1}$ and $\mathcal{G}_{2}$. Since the GWL operates on undirected graphs, we need to assume that the produced Hasse graph is undirected. The two examples of skeleton-preserving lifts clearly result in undirected Hasse graphs. We would obtain a directed Hasse graph if, e.g., only one of the up/down incidences in (1) is used as a neighborhood function. Similar to the standard graph WL test, the GWL method updates node colors iteratively based on the features of nodes in the local neighborhood. Additionally, GWL maintains $E(n)$-equivariant hash values that capture each node’s local geometry. Therefore, any $k$-hop distinct, $(k-1)$-hop identical geometric graphs can be distinguished by $k$ iterations of GWL, i.e., when the updated hash values that capture each node’s local geometry differ across the two graphs for the first time, or by $k$ layers of an EGNN, i.e., when the geometric invariants differ across the two graphs for the first time (Joshi et al., ).

###### Proposition 2 .

Assume to have two $k$-hop distinct geometric graphs $\mathcal{G}_{1}=(\mathcal{S}_{1},\mathcal{E}_{1})$ and $\mathcal{G}_{2}=(\mathcal{S}_{2},\mathcal{E}_{2})$ where the underlying graphs are isomorphic in the standard sense, two combinatorial complexes $\mathcal{X}_{\mathcal{G}_{1}}$ and $\mathcal{X}_{\mathcal{G}_{2}}$ and a collection $\mathcal{CN}$ of neighborhood functions obtained via a skeleton-preserving lift and leading to undirected geometric Hasse graphs $\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{1}}}$ and $\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{2}}}$, respectively. An ETNN operating over $\mathcal{X}_{\mathcal{G}_{1}}$ and $\mathcal{X}_{\mathcal{G}_{2}}$ respecting Assumptions 1-3 can distinguish $\mathcal{G}_{1}$ and $\mathcal{G}_{2}$ in $M$ layers, where $M$ is the number of layers required to have at least one cell in $\mathcal{X}_{\mathcal{G}_{1}}$/$\mathcal{X}_{\mathcal{G}_{2}}$ whose receptive field is the whole set of nodes in $\mathcal{G}_{1}$/$\mathcal{G}_{2}$.

###### Proof.

To prove the result, we study the behavior of the GWL on the geometric augmented Hasse graphs $\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{1}}}$ and $\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{2}}}$ of $\mathcal{X}_{\mathcal{G}_{1}}$ and $\mathcal{X}_{\mathcal{G}_{2}}$, respectively. For 1-hop distinct graphs, one iteration of GWL suffices to distinguish them even without any lifting into a CC and, thus, the proposition holds.

Now, let us assume that $\mathcal{G}_{1}$ and $\mathcal{G}_{2}$ are $k$-hop distinct and $(k-1)$-hop identical for any $k>1,k\in\mathbb{N}$. As a direct consequence of Proposition 1, $M$ GWL iterations are required for (at least one) node of the Hasse graphs $\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{1}}}$/$\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{2}}}$ to have the entire set of nodes $\mathcal{S}_{1}$/$\mathcal{S}_{2}$ in its receptive field.
Denote that node with $x_{i}\in\mathcal{G}_{1}$ and $x_{b(i)}\in\mathcal{G}_{2}$, where $b$ is any isomorphism between $\mathcal{G}_{1}$ and $\mathcal{G}_{2}$. Due to the k-hop distinctness of the graphs, the hash values corresponding to node $x_{i}\in\mathcal{G}_{1}$ and $x_{b(i)}\in\mathcal{G}_{2}$ will differ in the $M$-th iteration, thus $M$ iterations of GWL can distinguish the graphs.
Therefore. Combining the fact that EGNNs are as powerful as the GWL in distinguishing $k$-hop distinct graphs (Sestak et al., ; Joshi et al., ) with the result from Proposition 1, the proof is completed.
∎

###### Remark 5 .

The reader may have noticed that, in proving Proposition 2, we used the result from Proposition 1 ignoring the modified position update rule in (28), i.e., implicitly assuming that also the positions of higher-order cells, when seen as nodes of the corresponding augmented Hasse graph, are updated as in (7), following the standard EGNN. This is not relevant, as the modified update rule in (28) is actually more powerful than the standard rule in (7), because the positions of the higher-order cell $x$ at layer $l$ is updated using the already updated node positions $\bigoplus_{z\in x}\mathbf{x}_{z}^{l+1}$ and not the $\mathbf{x}_{z}^{l}$s. For this reason, Proposition 2 is somehow a pessimistic scenario for the practical use of ETNNs, and there could be corner cases in which ETNNs can distinguish $k$-hop distinct geometric graphs with a lower number of layers than prescribed. Finally, in practice, we observed it is not important to stick to the homogeneous setting (Assumption 1), to a unique aggregation function (Assumption 2), or to a certain geometric invariant (Assumption 3). These assumptions are useful to simplify the tractation and have a perfect match with the geometric augmented Hasse graph formalism.

Discussion. Proposition 2 provides formal hints about how different lifts of graphs into combinatorial complexes can impact the expressive power of ETNNs. Our result can be seen as a proper generalization of Proposition 2 in (Sestak et al., ) about the expressivity of VN-EGNNs. Indeed, adding a virtual node connected to all the other nodes can be interpreted as a combinatorial complex in which the cells are the set of singletons (the nodes) plus a cell containing all the nodes (the virtual node). A (simplified) variant of a VG-EGNN is then obtained by using the adjacency induced by the edges of the original graph and the up/down incidences as neighborhood functions. In this case, as prescribed from Proposition 2, the receptive field of the virtual node is the whole graph by definition, making VN-EGNNs able to distinguish $k$-hop distinct graphs in one layer. Our results also go in the direction of (Jogl et al., ), in which the expressivity of standard graph message passing networks on transformed/lifted graphs is systematically studied. This work provides a first, preliminary intuition on how the framework from (Jogl et al., ) could be extended to the geometric graph setting. Moreover, it is worth noting that our expressivity analysis could be enriched through the same GWL framework from (Joshi et al., ), eventually leading to a topological/higher-order generalization of it. An interesting direction is studying the expressivity of ETNNs when the lifts result in directed geometric augmented Hasse graphs. Finally, our result could shed light on which lifts should be preferred. In the current literature (Bodnar et al., ; Hajij et al., ; Battiloro et al., ; Eijkelboom et al., ) except for (Battiloro et al., ), the lifts are always decided based on simple criteria (e.g., if the domain is a simplicial complex, all the triangles of the underlying graphs are considered as 2-simplices). However, lifts could strike desirable tradeoffs between expressivity and learning performance (e.g. if the domain is again a simplicial complex, only some triangles could be chosen without expressivity drop).

Figure: Figure 8: $k$-chain geometric graphs, i.e., $k$ aligned nodes and two end nodes with opposite orientations.
Refer to caption: extracted/5899612/Figures/kchains.png

Experimental Validation. To validate our theoretical claims, we follow the approach from (Joshi et al., ; Sestak et al., ). In particular, we employ pairs of $k$-chain geometric graphs, where each graph pair consists of $k$ nodes linearly aligned and two endpoints with different orientations, as shown in Fig. 8. These graphs are clearly $\left(\left\lfloor\frac{k}{2}\right\rfloor+1\right)$-hop distinct, and thus should be distinguishable by $\left(\left\lfloor\frac{k}{2}\right\rfloor+1\right)$ EGNN layers or the same number of GWL iterations (Joshi et al., ; Sestak et al., ). To show the correctness of our result, we consider $k=4$, and we lift each pair of graphs into a CC in different ways, listed below. In all the cases, we use the mean to assign positions to cells in the Hasse graph (Assumption 3 of Proposition 1). Without loss of generality, we use $\mathcal{N}_{A,\uparrow}$ as in (14) (the usual node adjacency) and the up/down incidences $\mathcal{N}_{I,\uparrow}$/$\mathcal{N}_{I,\downarrow}$ in (1) as neighborhood functions, i.e., we keep the connectivity among nodes as in the original pair of graphs without including edges as cells and we allow higher-order cells to communicate only with nodes but not among them (thus resulting in a hypergraph-like structure). The readout classifier takes as input the geometric invariants computed on the final updated positions.

- •
Lift 1a: we consider a set of cells composed of the nodes of the original graphs plus one 1-cell containing all the nodes, as shown in Fig. 9(a) (a). In Figure 9(b), we show the corresponding geometric Hasse graphs. All the nodes will have the whole graph as their receptive field before the first iteration, thus we expect the GWL on the Hasse graph, hence ETNN, to distinguish the graphs using one layer. This case would be solved even by just feeding the initial geometric invariants to the readout classifier (using "0" layers).
- •
Lift 2a: we consider a set of cells composed of the nodes of the original graphs plus two 1-cells of the same cardinality being overlapping paths of length 4, as shown in Fig. 10(a). In Figure 10(b), we show the corresponding geometric Hasse graphs. It is clear that nodes $\{p_{1}\}$,$\{p_{2}\}$ will have the whole graph as their receptive field in one iteration, thus we expect the GWL on the Hasse graph, hence ETNN, to distinguish the graph using one layer.
- •
Lift 3a: we consider a set of cells composed of the nodes of the original graphs plus two 1-cells of different cardinality and not overlapping, as shown in Fig. 11(a). In Figure 11(b), we show the corresponding geometric Hasse graphs. It is clear that nodes $\{p_{1}\}$,$\{p_{2}\}$,$\{o_{1}\}$,$\{p_{2}\}$ will have the whole graph as their receptive field in two iterations, thus we expect the GWL on the Hasse graph, hence ETNN, to distinguish the graph using two layers.

Per each lift, we trained ETNNs with an increasing number of layers to classify the pairs of graphs. Due to possible oversquashing effects (Joshi et al., ), we also trained ETNNs with 5 different hidden dimensions and averaged over 10 different seeds. In Table 3, we report the results. We also report the results for EGNN (taken from (Sestak et al., )) and the GWL on the original pair of graphs without any lifting into a CC. As the reader can see, empirical results confirm the validity of our analysis, and the clear advantage, in terms of expressivity, of using CCs and ETNNs over standard EGNNs. Further empirical advantage is reported, pointing at higher-order interactions as a practical solution to oversquashing (Giusti et al., ; Di Giovanni et al., ).

**Table 3: Classification Accuracy on $4$-chain geometric graphs.**
|  |  | Number of Layers |  |  |  |
| --- | --- | --- | --- | --- | --- |
| $k=4$ | Hidden Dim. | 1 | 2 | 3 | 4 |
| GWL |  | $50\%$ | $50\%$ | $100\%$ | $100\%$ |
| EGNN from (Sestak et al., ) | 32 | $50\%$ | $50\%$ | $56.5\%$ | $50\%$ |
| 64 | $50\%$ | $50\%$ | $100\%$ | $99\%$ |  |
| 128 | $50\%$ | $50\%$ | $96.5\%$ | $98.5\%$ |  |
| ETNN Lift 1a | 32 | $100\%$ | $100\%$ | $100\%$ | $100\%$ |
| 64 | $100\%$ | $100\%$ | $100\%$ | $100\%$ |  |
| 128 | $100\%$ | $100\%$ | $100\%$ | $100\%$ |  |
| ETNN Lift 2a | 32 | $100\%$ | $100\%$ | $100\%$ | $100\%$ |
| 64 | $90\%$ | $100\%$ | $100\%$ | $100\%$ |  |
| 128 | $90\%$ | $100\%$ | $100\%$ | $100\%$ |  |
| ETNN Lift 3a | 32 | $50\%$ | $85\%$ | $100\%$ | $100\%$ |
| 64 | $50\%$ | $80\%$ | $95\%$ | $85\%$ |  |
| 128 | $50\%$ | $80\%$ | $95\%$ | $95\%$ |  |

**Table 4: Classification Accuracy on the counterxamples graphs.**
|  |  | Body Order |  |  |
| --- | --- | --- | --- | --- |
|  |  | 2-body | 3-body | 4-body |
| EGNN from (Joshi et al., ) |  | $50\%$ | $50\%$ | $50\%$ |
| ETNN Lift 1b | Inv. | $100\%$ | $100\%$ | $50\%$ |
| Equiv. | $100\%$ | $100\%$ | $50\%$ |  |
| ETNN Lift 2b | Inv. | $100\%$ | $50\%$ | $50\%$ |
| Equiv. | $100\%$ | $100\%$ | $50\%$ |  |
| ETNN Lift 3b | Inv. | $100\%$ | $100\%$ | $50\%$ |
| Equiv. | $100\%$ | $100\%$ | $50\%$ |  |
| ETNN Lift 4b | Inv. | $100\%$ | $100\%$ | $50\%$ |
| Equiv. | $100\%$ | $100\%$ | $50\%$ |  |

###### Remark 6 .

The lifts we used in the experiments above are handcrafted lifts, employed to validate Proposition 2 in a easy and consistent way. However, our result obviously holds in all the cases, even with more ranks and more sophisticated collections of neighborhood functions (e.g. induced by skeleton-preserving lift to simplicial, cell, or path complexes(Bodnar et al., ; Giusti et al., ; Battiloro et al., ; Truong & Chin, ; Bodnar et al., )). Moreover, it is worth it to notice that, with the experiments above, we investigated the role of the choice of the cells given the collection of neighborhood functions. However, it is interesting to study even the opposite case, in which the cells are fixed and the neighborhood functions are chosen. For example, in Lift 2a, adding the lower adjacency from (2 Combinatorial Complexes and Equivariance) would have led to solving the task with just one layer because the cells $c_{1}$/$c_{2}$ and $d_{1}$/$d_{2}$ would have had the whole graph as their receptive field before the first iteration. On the other hand, in Lift 3a, if we used the up adjacency from (2 Combinatorial Complexes and Equivariance) without including the edges as cells, we would have not been able to solve the task, because we would have obtained a disconnected graph, since $p_{1}$/$p_{2}$ and $o_{1}$/$o_{2}$ would have not been linked.

Counterexamples Structures. To provide an even more in-depth analysis of the expressivity of ETNNs beyond the distinguishability of $k$-hop distinct graphs, we also tested them on the counterexamples graphs from (Pozdnyakov et al., ; Joshi et al., ), which test a layer’s ability to create distinguishing fingerprints for local neighborhoods and stress exactly the importance of higher body order of scalarization. In particular, this task evaluates single-layer geometric architectures in distinguishing counterexample graphs that cannot be differentiated using $k$-body scalarization. we refer the reader to (Pozdnyakov et al., ; Joshi et al., ) for further details. We tested 4 different lifts:

- •
Lift 1b: we consider nodes and edges as 0-cells and 1-cells.
- •
Lift 2b: we consider a set of cells composed of the nodes of the original graphs plus one 1-cell
containing all the nodes (equivalent to Lift 1a).
- •
Lift 3b: we consider nodes, edges, and a cell containing all the nodes as 0-cells, 1-cells, and 2-cell, respectively.
- •
Lift 4b: we consider nodes, edges, and paths of length 3 as 0-cells, 1-cells, and 2-cells (this is equivalent to a path complex).

For all the lifts, we use the up/down incidences $\mathcal{N}_{I,\uparrow}$/$\mathcal{N}_{I,\downarrow}$ and adjacencies $\mathcal{N}_{A,\uparrow}$/$\mathcal{N}_{A,\downarrow}$ from (1)-(2 Combinatorial Complexes and Equivariance), respectively, as neighborhood functions, and we test both the invariant and equivariant versions of ETNN. The results are presented in Table 4. As the reader can see, ETNN is able to distinguish all the counterexamples but the 4-body (chiral).

Figure: (a)
Refer to caption: extracted/5899612/Figures/2_1_lift_1.png

Figure: (a)
Refer to caption: extracted/5899612/Figures/4_1_lift_3.png

Figure: (a)
Refer to caption: extracted/5899612/Figures/3_1_lift_2.png

###### Proposition 1 .

###### Proof.

###### Proposition 2 .

###### Proof.

###### Remark 5 .

###### Remark 6 .

## Appendix G Complexity Analysis

Given the flexibility in the definition of cells and neighborhood functions, the complexity of an ETNN layer can vary. To provide a reference complexity, let us consider a CC with $|\mathcal{X}|$ cells, and the up/down incidences and adjacencies from (1)-(2 Combinatorial Complexes and Equivariance) as neighborhood functions.
A cell $x\in\mathcal{X}$ has then $|\mathcal{N}_{I,\uparrow}(x)|+|\mathcal{N}_{I,\downarrow}(x)|$ incident cells and there are $|\mathcal{N}_{A,\uparrow}(x)|+|\mathcal{N}_{A,\downarrow}(x)|=\binom{|\mathcal
{N}_{I,\uparrow}(x)|+|\mathcal{N}_{I,\downarrow}(x)|}{2}$ up/down adjacencies between them. Therefore, each node $z\in\mathcal{S}$ has $|\mathcal{N}_{A,\downarrow}(z)|=\binom{|\mathcal{N}_{I,\uparrow}(z)|}{2}$ adjacent nodes. Then, an ETNN layer as in (6)-(7) has a computational complexity given by

$$ $\displaystyle\Theta\left(\underbrace{\sum_{x\in\mathcal{X}}\left(\left(| \mathcal{N}_{I,\uparrow}(x)|+\mathcal{N}_{I,\downarrow}(x)|\right)+\binom{| \mathcal{N}_{I,\uparrow}(x)|+|\mathcal{N}_{I,\downarrow}(x)|}{2}\right)}_{ \textrm{$\mathbf{h}_{x}$ update from \eqref{eq:eq_message_passing}}}+ \underbrace{\sum_{z\in\mathcal{S}}\binom{|\mathcal{N}_{I,\uparrow}(z)|}{2}}_{ \textrm{$\mathbf{x}_{z}$ update from \eqref{eq:eq_update_pos}}}\right)$ $\displaystyle=\Theta\left(\sum_{x\in\mathcal{X}}\left(\left(|\mathcal{N}_{I, \uparrow}(x)|+\mathcal{N}_{I,\downarrow}(x)|\right)+\left(|\mathcal{N}_{A, \uparrow}(x)|+|\mathcal{N}_{A,\downarrow}(x)|\right)\right)+\sum_{z\in\mathcal {S}}|\mathcal{N}_{A,\uparrow}(z)|\right)$ $\displaystyle=\Theta\left(\sum_{x\in\mathcal{X}}\left(|\mathcal{N}_{A,\uparrow }(x)|+|\mathcal{N}_{A,\downarrow}(x)|\right)\right).$ (29) $$

**Table 5: Average runtime per epoch for different ETNN configurations**
|  | Avg. runtime (seconds/epoch) |
| --- | --- |
| EGNN-graph-W | 60.34 |
| ETNN | 193.16 |
| EMPSN* | 251.3 |

**Table 6: Average memory usage for different ETNN configurations**
|  | Avg. memory consumption (GB) |
| --- | --- |
| EGNN-graph-W | 13.63 |
| ETNN | 12.05 |
| EMPSN* | 27.7 |

If we consider the connectivity of the CC to be sufficiently sparse, which is usually the case, then the neighborhoods’ cardinalities can be absorbed in the bound. Therefore, the overall complexity will be linear in the number of the cells $\Theta\left(|\mathcal{X}|\right)$. Clearly, in the worst case, each cell could be connected to all the other cells, resulting in a quadratic cost in the number of cells. In practice, with our implementation, both the model’s forward and backward pass scale linearly with the number of neighborhood functions, as they are processed in a loop. In Table 5, we compare the runtimes of EGNN-graph-W, EMPSN* (i.e., EMPSN reproduced with our codebase as an instance of ETNN), and an instance of ETNN using the complete Molecular CC (configs. 35 of Tables 11-12). We decided to use this configuration of ETNN because is one of the most computationally intensive. However, better learning performance can be obtained with even (much) lighter configurations, as shown in Tables 11-12. In this case, ETNN uses exactly 3 times as many neighborhoods as in EGNN-graph-W, as the runtimes confirm. A different implementation could parallelize the processing, making the runtime constant w.r.t. the number of neighborhoods. Most importantly, Table 5 provides the reader with a quantitative metric of the significantly reduced computational burden of ETNN w.r.t. EMPSN (Eijkelboom et al., ), as described in the main body. To further highlight this gain, in Table 6 we show the memory usage of the same architectures, proving that ETNN needs less than half memory w.r.t. EMPSN.

## Appendix H Additional References on Real-World Combinatorial Topological Spaces

### H.1 Molecular Data

Molecular graphs. One of the most renowned representations of molecules is graphs. In this case, atoms are the nodes, and bonds among them are the edges(Gilmer et al., ; Gasteiger et al., ; Jiang et al., ). Message passing networks on molecular graphs have been proven to be effective on several tasks. However, graphs cannot explicitly model higher-order, multiway interactions among the atoms. For instance, graphs cannot explicitly consider the information coming from the rings, i.e., the induced cycles of the molecular graphs. To represent different atomic relations, efforts focused on learning representations for small molecules and larger biomolecules (i.e., proteins) via multi-view or relational graphs(Ma et al., ; Zhang et al., ; Sestak et al., ; Li et al., ).

Molecular cell complexes. To overcome the above limitation, cell complexes (CW complexes), being a generalization of simplicial complexes (Eijkelboom et al., ; Bodnar et al., ), have been employed (Bodnar et al., ). Molecular CW complexes can be seen as "hierarchical augmented graphs" in which atoms are 0 0-cells (nodes), bonds are $1$-cells (edges), and rings are $2$-cells (induced cycles). Message passing networks on molecular CW complexes have shown superior performance w.r.t. models on molecular graphs (Giusti et al., ; Bodnar et al., ). However, CW complexes cannot model higher-order interactions among the atoms and bonds that are not described by a cycle in the underlying molecular graph. For instance, molecular CW complexes cannot explicitly consider the information coming from the functional groups of the molecule, i.e., specific motifs (not necessarily cycles) of the underlying molecular graphs.

Molecular hypergraphs. Hypergraphs (HGs) can model arbitrary higher-order interactions due to the flexibility of hyperedges. HGs and message passing networks on HGs have been employed for modeling and learning from molecules (Chen & Schwaller, ; Park et al., ). In molecular HGs, atoms are nodes, and bonds, rings, functional groups, etc, are hyperedges. However, HGs cannot explicitly consider any hierarchy in the hyperedges, thus putting pairwise and higher-order interactions of any kind on the same level.

### H.2 Geospatial Data

Spatial graphs and CTS The single resolution version of the spatial setting in 4 Real-World Combinatorial Complexes has been traditionally handled using proximity graphs, i.e., geometric graphs whose connectivity is dictated by some notion of spatial closeness. In the multiresolution case, the most common approach is to aggregate all the data at a common denominator of spatial resolution. However, substantial information is clearly lost in this process. Some GNN methods for expanding the receptive field of GNNs in spatial data have been proposed. For instance, through edge removal during training to address over-smoothing and over-fitting (Rong et al., ), and the introduction of skip connections (He et al., ; Li et al., ). Yu et al. (Yu et al., ) introduced ST-UNet, a heuristic graph pooling method for coarsened graph construction. Note that these methods do not admit multi-level data, i.e., features available in each level of a hierarchy. Hierarchical GNNs (HGNNs) (Diehl et al., ; Lee et al., ; Ying et al., ) create multi-level graphs balancing the expansion of receptive fields and preservation of local features. HGNNs can admit multi-level features. A highly relevant application is due to Li et al. (Li et al., ), who utilized a geography-based HGNN for modeling geographic information systems. Importantly, this work does not consider the use of geometric (coordinate information) directly, and therefore, it does not explore equivariant architectures. Other related work includes Wu et al. (Wu et al., ), who proposed HRNR for learning road segment representations. Zhang et al. (Zhang et al., ) combined HGNNs with semi-supervised learning for citywide parking availability prediction. It is worth noticing that the constructions of such HGNNs are highly reliant on the specifics of the application, while Geospatial CCs offer a more generally applicable formalism. Motivated by topological approaches, some works used hypergraphs (Jia et al., ), whose limitation have been explained in the previous subsection. Some Topological Data Analysis (TDA) work leveraged simplicial complexes (Bajardi et al., ; Lo & Park, ; Stolz et al., ; Feng & Porter, ; Hickok et al., ). However, TDL should not be confused with TDA, which is a data-driven approach to describing the mathematical properties and “shape" of a topological space. Instead, TDL focuses on modeling relations and data distributions defined over such topological spaces (Barbarossa & Sardellitti, ).

## Appendix I Experimental Setup

### I.1 Molecular Property Prediction

In this section, we present all dataset information, data featurization, and model parameter configuration for the molecular property prediction task.

#### I.1.1 Dataset Statistics

QM9 dataset consists of over $130,000$ molecules with $12$ property prediction targets. See Table 7 for a detailed list of the $12$ molecular properties, as well as statistics over the number of molecules, atoms, bonds, rings, and functional groups.

**Table 7: Statistics of the QM9 Dataset**
| Property | Description | Units |
| --- | --- | --- |
| $\alpha$ | Electronic spatial extent | bohr^3 |
| $\Delta\epsilon$ | Gap between $\epsilon_{HOMO}$ and $\epsilon_{LUMO}$ | meV |
| $\epsilon_{HOMO}$ | Energy of the highest occupied molecular orbital | meV |
| $\epsilon_{LUMO}$ | Energy of the lowest unoccupied molecular orbital | meV |
| $\mu$ | Dipole moment | D |
| $C_{v}$ | Heat capacity at 298.15 K | cal/mol K |
| $G$ | Free energy at 298.15 K | meV |
| $H$ | Enthalpy at 298.15 K | meV |
| $U$ | Internal energy at 298.15 K | meV |
| $U_{0}$ | Internal energy at 0 K | meV |
| $ZPVE$ | Zero-point vibrational energy | meV |
| Number of molecules |  | 133,885 |
| Average number of atoms |  | 18 nodes |
| Average number of bonds |  | 19 |
| Average number of rings |  | 1.6 |
| Average number of functional groups |  | 1.2 |

#### I.1.2 Cell Features

For the molecular prediction task, we considered different peculiar feature vectors for each rank, that can be easily integrated thanks to the ETNN structure.

- 1.
0-cells features (atom features):
•
Atom Types: QM9 molecular samples consist of $5$ atoms: hydrogen (H), carbon (C), nitrogen (N), oxygen (O), and fluorine (F). We used one-hot encoding for the atom types.
•
Functions over atom types: Following Cormorant’s methodology(Anderson et al., ), we computed the atomic fraction (current atomic number divided by the maximum atomic number), multiplied the atomic number with the atomic fraction, and computed the squared quantities.
- 2.
1-cells features (bond features):
•
Bond Types: We can meet $4$ different bond types in QM9 molecules: single, double, triple, aromatic. We used one-hot encoding for the bond types.
•
Conjugation: A binary feature telling whether the bond is part of a conjugated system, affecting electronic and optical properties.
•
Ring Membership: binary feature telling whether the bond is part of a ring, affecting electronic and structural characteristics of the molecule.
- 3.
2-cells features (ring and functional group features):
Ring features
•
Ring Size: Number of atoms that form the ring.
•
Aromaticity: A binary feature telling whether the ring is aromatic.
•
Has Heteroatom: A binary feature telling whether the ring contains a heteroatom, which is an atom other than carbon, such as nitrogen, oxygen, or sulfur. Heteroatoms can introduce different electronic effects and reactivity patterns within the ring structure.
•
Saturation: A binary feature telling whether the ring is saturated, meaning all the bonds between carbon atoms are single bonds.
Functional Group features
•
Conjugation: A binary feature telling whether the functional group is part of a conjugated system, which involves alternating double and single bonds.
•
Acidity: The ability of the functional group to donate protons ($H+$ ions) in a solution. Possible values: ["neutral", "high", "basic", "weakly acidic"].
•
Hydrophobicity: Tendency of the functional group to repel water. $3$ Possible values: [hydrophilic, "moderate", "hydrophobic"]
•
Electrophilicity: Tendency of the functional group to accept electrons during a chemical reaction. High electrophilicity means the functional group readily accepts electrons, influencing its reactivity with nucleophiles. $3$ Possible values: ["low", "moderate", "high"].
•
Nucleophilicity: Tendency of the functional group to donate electrons during a chemical reaction. High nucleophilicity means the functional group readily donates electrons, affecting its reactivity with electrophiles. $3$ possible values: ["low", "moderate", "high"].
•
Polarity: Distribution of electron density within the functional group. High polarity can significantly affect solubility, reactivity, and interactions with other molecules. $3$ possible values: ["low", "moderate", "high"].

#### I.1.3 Parameter Configuration

Next, we present the ETNN’s hyperparameters we used for the molecular property prediction tasks. Table 8 shows the list of the hyperparameters of the optimizer, as well of our model. Specifically:

- •
Following EGNN’s experimentation setup, we use $1e$-$3$ as the initial learning rate for the properties $\Delta\epsilon,\epsilon_{\text{HOMO}},\epsilon_{\text{LUMO}}$, and $5e$-$4$ for the rest of the properties.
- •
We calibrate the number of hidden units (i.e., layer width) among $\{70,104,128,182\}$ for each configuration, so that we maintain roughly the same size of parameter space ($\approx$ 1.5M). The motivation for this choice is to provide the fairest comparison possible. see Appendix J for further motivation and details.
- •
All the reported configurations were run for $1000$ epochs, in agreement with the original EGNN(Satorras et al., ).
- •
We used the same train/validation/test splits as introduced in EGNN(Satorras et al., ). See Appendix J for further motivation and details.
- •
For all the configurations we normalize the geometric invariants via batch normalization. That includes the normalization of pairwise distances, Hausdorff distances, and volumes. Moreover, we perform gradient clipping to a maximum norm of gradients equal to $1.0$.

**Table 8: Hyperparameters for ETNN Model in the Molecular Task**
| Hyperparameter | Value |
| --- | --- |
| Optimizer | Adam |
| Initial Learning Rate | [$5*10^{-4}$, $10^{-3}$] |
| Learning Rate Scheduler | Cosine Annealing |
| Weight Decay | 1e-5 |
| Batch Size | 96 |
| Epochs | $[100,200,350,1000]$ |
| Model |  |
| Number of Message Passing Layers | [4, 7, 10] |
| Hidden Units per Layer | $[70,104,128,182]$ |
| Activation Function | SiLU |
| Invariant Normalization | [True, False] |
| Gradient Clipping | [True, False] |

### I.2 Hyperlocal Air Pollution Downscaling

#### I.2.1 Dataset Statistics

We create a geospatial CC using the neighborhood functions described by 4 Real-World Combinatorial Complexes. The hyperlocal air measurements points (target) are taken from (Wang et al., ). The targets are aggregated at the desired resolution of $0.0002^{\circ}\sim 22m$. After removing roads and tracts that do not contain any measurements, the resulting dataset contains 3,946 point measurements, 550 roads, and 151 census tracts, corresponding to 0-, 1- and 2-cells, respectively.

#### I.2.2 Cell Features

The learning targets are the fine-resolution PM 2.5 from (Wei et al., ) as described in the main text. As such, this is a semisupervised node regression task. We now describe the features of each rank. For the 0-cells, we generate point-level features using the osmnx Python package (Boeing, ), calculating metrics such as density and distance to points of interest within a buffer radius. We remove roads and tracts that do not contain any measurements. We consider only points that correspond to weekdays between 9 and 5 pm. For 1-cells, we use the 2021 annual average daily traffic information (New York City Department of Transportation, ). For 2-cells we generate land-use features from the Primary Land Use Tax Lot Output (PLUTO)(New York City Department of City Planning, ) and previous coarse-resolution estimates of PM 2.5 at the census-tract level are taken from Wei et al. (Wei et al., ). Values are monthly averages over the last quarter of 2020, projected to the census tract level using bilinear interpolation from their native 1 km resolution. Below is a list of the employed features. Table 9 summarizes the data sources.

- 1.
0-cells features (points; OpenStreetMap (OSMnx) and Wang et al.):
•
Intersection density: Using the osmnx Python package, we calculate the density of intersections within a 100m buffer radius (Boeing, ).
•
Nearest school distance: Calculated using osmnx to determine the distance to the nearest school (Boeing, ).
•
Restaurant density: Using osmnx, we calculate the density of restaurants within a 300m radius (Boeing, ).
•
Road density: Derived from osmnx by calculating the density of roads within a buffer radius (Boeing, ).
•
Time of day: We include the median time of measurements by Wang et al. (Wang et al., ) for each of the points in a $0.0002^{\circ}\times 0.0002^{\circ}$ area.
- 2.
1-cells features (road/polylines; AADT, NYC, Dept. of Transportation):
•
AADT 2021: Annual Average Daily Traffic data for 2021 (New York City Department of Transportation, ).
•
PCT 2021: Percentage of trucks traffic (New York City Department of Transportation, ).
- 3.
2-cells features (census tracts):
- PLUTO, NYC Dept. of Planning (New York City Department of City Planning, )
•
Assessment total
•
Building area
•
Commercial area (% area)
•
Factory area (% area)
•
Mixed residential/commercial (% tax lots)
•
Multi-family elevator (% tax lots)
•
One/two family (% tax lots)
•
Open space/outdoor recreation (% tax lots)
•
Parking/vacant (% tax lots)
•
Public facilities/institutions (% tax lots)
•
Transportation/utility (% tax lots)
•
Lot area (sq. km)
•
Lot depth (m)
•
Office area (% area)
•
Residential area (% area)
•
Retail area (% area)
Other sources
•
Coarse $\textrm{PM}_{2.5}$ predictor
(Di et al., ): Coarse-resolution monthly estimates of mean $\textrm{PM}_{2.5}$ by census tract.

**Table 9: Data sources of the Geospatial Air Pollution Downscaling Benchmark**
| Source | Citation | LICENSE |
| --- | --- | --- |
| Wang et al. | (Wang et al., ) | CC 4.0 |
| PLUTO, NYC Dept. of Planning | (New York City Department of City Planning, ) | Open Data |
| AADT, NYC Dept. of Transportation | (New York City Department of Transportation, ) | Open Data |
| OpenStreetMap (OSMnx) | (Boeing, ) | Open Data |
| Wei et al. | (Wei et al., ) | CC 4.0 |

#### I.2.3 Parameter Configuration

The training and evaluation scheme is similar to the molecular case. For training, we use the Huber loss which provides additional robustness to outliers (Huber, ). The evaluation metric is obtained similarly to the molecular task. Since the data consists of a single graph, we use a standard procedure for semisupervised tasks on spatial data and generate training, validation, and test masks. To do so while being mindful of spatial leaks and correlations, we randomly select 70% of census tracts for training and split the remaining 30% equally in test and validation tracts. Since points in the dataset only belong to a single census tract, we can use these splits to partition points in their corresponding sets, ensuring that neighboring points belong, on average, to the same split. While all the nodes (training/validation/test) are given as inputs to the model, the loss function for backpropagation is only computed from the error in the training nodes, as in any standard semisupervised setting (Kipf & Welling, ).

The evaluation metric is the root-mean-squared error (RMSE) in the test split from the best model of each baseline. The base model is determined by the lowest RMSE in the validation RMSE at any point during training from a sweep over the hyperparameters described in Table 10.
The test split only reports the final metric to ensure validity. The EGNN baseline used the same architecture but no lower/upper adjacencies. It also has higher-order features concatenated with point-level features. The multi-layer perception and linear baselines are evaluated similarly. The MLP baseline is implemented with 5 layers to match the number of hidden layers in the ETNN and graph-based baselines.

##### Computational Resources

All experiments that produced the reported configurations in both tasks were conducted using Nvidia A100-SXM4-40GB GPUs. Each system has a capacity of 80GB RAM, 40GB GPU memory, and $6$ CPUs per unit.

**Table 10: Hyperparameters for ETNN Model in the Air Pollution Downscaling Task**
| Hyperparameter | Value |
| --- | --- |
| Optimizer | Adam |
| Initial Learning Rate | [$10^{-3}$, $10^{-2}$] |
| Learning Rate Scheduler | Cosine Annealing |
| Weight Decay | 1e-4 |
| Batch Size | 1 |
| Epochs | $500$ |
| Model |  |
| Number of Message Passing Layers | 4 |
| Hidden Units per Layer | $[4,32]$ |
| Activation Function | SiLU |
| Invariant Normalization | False |
| Gradient Clipping | True |
| Dropout | [0.025, 0.25] |

## Appendix J Ablation Study and Reproducibility for Molecular Property Prediction

In this section, we present the whole set of experiments we carried out to obtain our results in Table 1, i.e., all the different configurations of ETNN we tested. Moreover, we want to further specify some details that are useful to fairly evaluate our results and ensure reproducibility.

Reproducibility. When we started to set up the experiments, we found a poor reproducibility level for the molecular prediction task on QM9 in the literature. In particular, practically every model of the ones listed in Table 1 uses different data splits. The two closest models to ETNN are EGNN (Satorras et al., ) and EMPSN (Eijkelboom et al., ), which are also particular cases of our framework. We were not able to reproduce the results of EMPSN (and this is also the reason we did not include it in Table 1), while we were able to reproduce the EGNN results using the original codebase. For all of the reasons above, we decided to employ the same data splits as EGNN(Satorras et al., ). To guarantee a fair comparison with EMPSN, however, in this appendix we also report the results of EMPSN from the original paper, EMPSN reproduced in TopNets (Verma et al., ), and EMPSN reproduced with our codebase (as described in the main body). Again, and as explained in Appendix C.1, EGNN can be directly derived from ETNN. Before exploring different ETNN configurations, we aligned our training and architecture parameters with EGNN. We ensured that our ETNN code, configured as an EGNN, could reproduce all the considered properties of the QM9 dataset reported in the original EGNN paper. Moreover, we trained our models for 1000 epochs as in EGNN and EMPSN. Finally, to ensure further consistency, we also trained a configuration of ETNN, dubbed ETNN-graph-W, that is again EGNN derived from our codebase but with a higher number of hidden units, chosen such that the number of parameters of ETNN-graph-W matches the average number of parameters of the other ETNN configurations (1.5M, while the original EGNN has 748k).

Results. In Tables 11- 12, we present the full list of the ETNN configurations we tested, representing also an exhaustive ablation study on all the main components of our architectures. We also add the results of TopNet, presented in the contemporary work (Verma et al., ). We grouped the relevant model hyperparameters of ETNN into the following five groups:

- •
Definition of the Molecular CC. We study the performance of ETNN when bonds, rings, and functional groups, are included or not as cells in the complex. In Tables 11- 12, this is described in the first argument, Lifters. The values of the argument indicate what are the considered cells, e.g. A+B+F+R means that atoms, bonds, functional groups, and rings are used as cells. The rank is assigned in increasing order as described in Section 5 Experiments.
- •
Adjacencies. We study the performance of ETNN when different adjacencies are employed in the collection of neighborhood functions $\mathcal{CN}$. In Tables 11-12, this is described in the second argument, Adjacencies. In particular, we test the usage of $\mathcal{N}_{A,\textrm{max}}$ as in (13) in conjunction with the up/down adjacencies $\mathcal{N}_{A,\uparrow}$ and $\mathcal{N}_{A,\downarrow}$ from (2 Combinatorial Complexes and Equivariance). In Tables 11- 12, Adjacencies = max means that only the max adjacency from (13) is used, while Adjacencies = max,$\uparrow$,$\downarrow$ means that both the max adjacency from (13) and the up/down adjacencies from (2 Combinatorial Complexes and Equivariance) are used (effectively resulting in a heterogeneous setting as described in the body of the paper).
- •
Incidences. We study the performance of ETNN when different incidences are employed in the collection of neighborhood functions $\mathcal{CN}$.In Tables 11- 12, this is described in the third argument, Incidences. Testing only the he up/down incidences $\mathcal{N}_{I,\uparrow}$ and $\mathcal{N}_{I,\downarrow}$ from (2 Combinatorial Complexes and Equivariance) would always result in configurations in which cells of rank $k$ communicate only with cells of ranks $k-1$ and $k+1$. For this reason, we also test the $k$-up/down incidences, defined as:
$\displaystyle\mathcal{N}_{I,\uparrow_{k}}(x)=\{y\in\mathcal{X}|\mathrm{rk}(y)=
\mathrm{rk}(x)+k,x\subset y\},$
$\displaystyle\mathcal{N}_{I,\downarrow_{k}}(x)=\{y\in\mathcal{X}|\mathrm{rk}(y
)=\mathrm{rk}(x)-k,y\subset x\}.$
(30)
In this way, cells can communicate in a more flexible way across ranks. In Tables  11- 12, Incidences = all means that the $k$-up/down incidences from (• ‣ J) are used for all the possible $k$, i.e., each rank communicates with all the other ranks; Incidences = 1 means that the up/down incidences from (1) (equivalently, $k=1$ in (• ‣ J)) are used; Incidences = 0 means that no incidences are used, i.e., there are no inter-rank message exchanges and the embeddings are aggregated only in the readout layer.
- •
Higher-order Features. We study the performance of ETNN when higher-order features (not only atom features) are employed. In Tables 11- 12, this is described in the fourth binary argument, HO Features. If HO Features = 1, higher-order features (bonds, ring and/or functional groups based on the complex) are fed to the model.
- •
Virtual Cell. We study the performance of ETNN when the virtual cell is employed. In Tables 11- 12, this is described in the fifth binary argument, Virtual Cell. If Virtual Cell = 1, the virtual cell is included in the complex.

###### Remark 7 .

In all the configurations, we don’t allow the virtual cell to exchange messages or being included in the readout layer. The reason is that we mostly use it in the experiments to obtain a fully connected computational graph on each rank when it is used together with the max adjacency from (13). However, it remains an interesting object, as it is i) a straightforward way to include whole molecule-level features in ETNN by assigning them as features to the virtual cell, and ii) useful to improve expressiveness and generality of ETNN, as shown in Appendix F. Therefore, when Adjacencies = max and Virtual Cell=1, the resulting configuration is equivalent to not including the virtual cell in the complex and using the neighborhood function
$\mathcal{N}_{A,\textrm{all}}(x)=\{y\in\mathcal{X}|\mathrm{rk}(y)=\mathrm{rk}(x)\}$.

To provide the reader with some usage examples of ETNN employing the above hyperparameter grouping (that we follow in our codebase), EGNN as in the original paper (Satorras et al., ) can be obtained setting Lifters = A, Adjacencies = max, Incidences = 0, HO Features = 0, Virtual Cell = 1; an instance of EMPSN (Eijkelboom et al., ) can be obtained setting Lifters = A+B+R, Adjacencies = $\uparrow$, $\downarrow$, Incidences = 1, HO Features = 1, Virtual Cell = 0, and considering only rings of length up to three; an equivariant version of (Bodnar et al., ) can be obtained setting Lifters = A+B+R, Adjacencies = $\uparrow$, $\downarrow$, Incidences = 1, HO Features = 1, Virtual Cell = 0, and considering rings of arbitrary length.

**Table 11: Mean Absolute Error for the first six molecular property predictions in QM9.**
| Task | $\alpha$ | $\Delta\varepsilon$ | $\varepsilon_{\mathrm{HOMO}}$ | $\varepsilon_{\mathrm{LUMO}}$ | $\mu$ | $C_{\nu}$ |
| --- | --- | --- | --- | --- | --- | --- |
| Units | bohr^3 | meV | meV | meV | D | cal/mol K |
| NMP | .092 | 69 | 43 | 38 | .030 | .040 |
| Schnet | .235 | 63 | 41 | 34 | .033 | .033 |
| Cormorant | .085 | 61 | 34 | 38 | .038 | .026 |
| L1Net | .088 | 68 | 46 | 35 | .043 | .031 |
| LieConv | .084 | 49 | 30 | 25 | .032 | .038 |
| DimeNet++* | .044 | 33 | 25 | 20 | .030 | .023 |
| TFN | .223 | 58 | 40 | 38 | .064 | .101 |
| SE(3)-Tr. | .142 | 53 | 35 | 33 | .051 | .054 |
| EGNN | .071 | 48 | 29 | 25 | .029 | .031 |
| EMPSN(Eijkelboom et al., ) | .066 | 37 | 25 | 20 | .023 | .024 |
| EMPSN* | .061 | 42 | 29 | 25 | .030 | .028 |
| EMPSN(Verma et al., ) | .066 | 51 | 32 | 25 | .031 | .027 |
| TopNet (VC, discrete)(Verma et al., ) | .083 | 47 | 37 | 24 | .035 | .032 |
| TopNet (VC, continuous)(Verma et al., ) | .075 | 49 | 36 | 27 | .030 | .035 |
| TopNet (RePHINE, discrete)(Verma et al., ) | .072 | 57 | 33 | 28 | .029 | .028 |
| TopNet (RePHINE, continuous)(Verma et al., ) | .070 | 50 | 35 | 25 | .032 | .030 |
| ETNN (Configurations) |  |  |  |  |  |  |
| [ Lifters | Adjacencies | Incidences | HO Features | Virtual Cell ] |  |  |  |  |  |  |
| 1: [A+B | max | 0 | 1 | 1] | .074 | 46 | 30 | 24 | .03 | .032 |
| 2: [A+B+F | max | 0 | 1 | 1] | .078 | 50 | 32 | 25 | .029 | .032 |
| 3: [A+B+R | max | 0 | 1 | 1] | .079 | 50 | 34 | 25 | .029 | .033 |
| 4: [A+B+F +R | max | 0 | 1 | 1] | .077 | 54 | 31 | 27 | .032 | .033 |
| 5: [A+B | max | 0 | 0 | 1] | .062 | 48 | 27 | 25 | .03 | .033 |
| 6: [A+B | max | all | 1 | 1] | .07 | 47 | 26 | 23 | .023 | .03 |
| 7: [A+B+F | max | all | 1 | 1] | .072 | 47 | 27 | 23 | .024 | .032 |
| 8: [A+B+R | max | all | 1 | 1] | .071 | 49 | 26 | 23 | .023 | .032 |
| 9: [A+B+F+R | max | all | 1 | 1] | .069 | 46 | 26 | 23 | .025 | .033 |
| 10: [A+B+F+R | max, $\uparrow$,$\downarrow$ | all | 1 | 0] | .157 | 72 | 45 | 46 | .31 | .053 |
| 11: [A+B | max | 0 | 1 | 0] | .149 | 69 | 45 | 45 | .298 | .048 |
| 12: [A | max | 0 | 0 | 1] (ETNN-graph) | .07 | 45 | 28 | 23 | .029 | .035 |
| 13: [A | max | 0 | 0 | 1] (ETNN-graph-W) | .07 | 49 | 28 | 25 | .03 | .036 |
| 14: [A+B+F+R | max | all | 1 | 0] | .13 | 66 | 44 | 42 | .285 | .046 |
| 15: [A+B+R | max | all | 1 | 1] (triangles only, EMPSN-like) | .071 | 46 | 27 | 24 | .022 | .032 |
| 16: [A+B+F+R | max | all | 1 | 1] (no geometric features, CCMPN-like) | .191 | 85 | 62 | 59 | .339 | .078 |
| 17: [A+B+F+R | max | all | 1 | 0] (no geometric features, CCMPN-like) | .184 | 86 | 61 | 59 | .351 | .075 |
| 18: [A+B | max | 1 | 1 | 1] | .069 | 47 | 26 | 22 | .024 | .03 |
| 19: [A+B+F | max | 1 | 1 | 1] | .067 | 44 | 27 | 23 | .023 | .032 |
| 20: [A+B+R | max | 1 | 1 | 1] | .069 | 46 | 26 | 24 | .023 | .033 |
| 21: [A+B+F+R | max | 1 | 1 | 1] | .075 | 45 | 27 | 23 | .024 | .034 |
| 23: [A+B+F+R | max | 1 | 1 | 0] | .139 | 68 | 41 | 43 | .281 | .046 |
| 25: [A+B+F+R | max | 1 | 1 | 1] (no geometric features, CCMPN-like) | .193 | 90 | 60 | 63 | .359 | .075 |
| 26: [A+B | max, $\uparrow$,$\downarrow$ | 0 | 1 | 1] | .187 | 85 | 61 | 58 | .342 | .074 |
| 27: [A+B+F | max, $\uparrow$,$\downarrow$ | 0 | 1 | 1] | 1.179 | 438 | 282 | 333 | .772 | .678 |
| 28: [A+B+R | max, $\uparrow$,$\downarrow$ | 0 | 1 | 1] (equivariant CWN-like) | 1.143 | 352 | 262 | 262 | .713 | .628 |
| 29: [A+B | max, $\uparrow$,$\downarrow$ | 0 | 0 | 1] | 1.038 | 332 | 237 | 284 | .721 | .548 |
| 30: [A+B | max, $\uparrow$,$\downarrow$ | 0 | 0 | 1] | .984 | 267 | 209 | 229 | .657 | .496 |
| 31: [A+B | max, $\uparrow$,$\downarrow$ | 1 | 0 | 1] | 1.224 | 410 | 251 | 307 | .712 | .848 |
| 32: [A+B+F | max, $\uparrow$,$\downarrow$ | 1 | 0 | 1] | .141 | 74 | 50 | 48 | .312 | .049 |
| 33: [A+B+R | max, $\uparrow$,$\downarrow$ | 1 | 0 | 1] | .143 | 75 | 48 | 47 | .337 | .048 |
| 34: [A+B+F+R | max, $\uparrow$,$\downarrow$ | 1 | 0 | 1] | .129 | 69 | 44 | 45 | .297 | .046 |
| 35: [A+B+F+R | max, $\uparrow$,$\downarrow$ | 1 | 0 | 0] | .134 | 71 | 46 | 45 | .302 | .046 |
| 36: [A+B+R | max, $\uparrow$,$\downarrow$ | 1 | 0 | 0] (triangles only, heterogeneous EMPSN-like) | .132 | 71 | 45 | 45 | .309 | .045 |
| 37: [A+B+R | max, $\uparrow$,$\downarrow$ | 1 | 1 | 0] (no geometric features, CWN-like) | .145 | 75 | 49 | 49 | .324 | .049 |
| 38: [A+B+F+R | max, $\uparrow$,$\downarrow$ | 1 | 0 | 0] (no geometric features, CCMPN-like) | .184 | 90 | 62 | 62 | .371 | .077 |
| 39: [A+B+F+R | max, $\uparrow$,$\downarrow$ | 1 | 0 | 0] (no geometric features, CCMPN-like) | .19 | 92 | 62 | 60 | .36 | .076 |
| Best ETNN Configuration |  |  |  |  |  |  |
| ETNN-* | .062 | 45 | 26 | 22 | .022 | .030 |
| \hdashlineImprovement over EGNN | -13% | -6% | -10% | -12% | -26% | -3% |

**Table 12: Mean Absolute Error for the last five molecular property predictions in QM9.**
| Task | $G$ | $H$ |  | $U$ | $U_{0}$ | ZPVE |
| --- | --- | --- | --- | --- | --- | --- |
| Units | meV | meV |  | meV | meV | meV |
| NMP | 19 | 17 |  | 20 | 20 | 1.50 |
| Schnet | 14 | 14 |  | 19 | 14 | 1.70 |
| Cormorant | 20 | 21 |  | 21 | 22 | 2.03 |
| L1Net | 14 | 14 |  | 14 | 13 | 1.56 |
| LieConv | 22 | 24 |  | 19 | 19 | 2.28 |
| DimeNet++* | 8 | 7 |  | 6 | 6 | 1.21 |
| TFN | - | - |  | - | - | - |
| SE(3)-Tr. | - | - |  | - | - | - |
| EGNN | 12 | 12 |  | 12 | 11 | 1.55 |
| EMPSN(Eijkelboom et al., ) | 6 | 9 |  | 7 | 10 | 1.37 |
| EMPSN* | 11 | 11 |  | 10 | 10 | 1.46 |
| EMPSN(Verma et al., ) | - | - |  | - | - | 1.44 |
| TopNet (VC, discrete)(Verma et al., ) | - | - |  | - | - | 1.45 |
| TopNet (VC, continuous)(Verma et al., ) | - | - |  | - | - | 1.43 |
| TopNet (RePHINE, discrete)(Verma et al., ) | - | - |  | - | - | 1.38 |
| TopNet (RePHINE, continuous)(Verma et al., ) | - | - |  | - | - | 1.37 |
| ETNN (Configurations) |  |  |  |  |  |  |
| [ Lifters | Adjacencies | Incidences | HO Features | Virtual Cell ] |  |  |  |  |  |  |
| 1: [A+B | max | 0 | 1 | 1] | 15 | 13 |  | 14 | 13 | 1.75 |
| 2: [A+B+F | max | 0 | 1 | 1] | 14 | 12 |  | 12 | 12 | 1.733 |
| 3: [A+B+R | max | 0 | 1 | 1] | 15 | 15 |  | 14 | 14 | 1.763 |
| 4: [A+B+F +R | max | 0 | 1 | 1] | 15 | 15 |  | 15 | 15 | 1.749 |
| 5: [A+B | max | 0 | 0 | 1] | 13 | 12 |  | 12 | 12 | 1.589 |
| 6: [A+B | max | all | 1 | 1] | 11 | 12 |  | 12 | 11 | 1.648 |
| 7: [A+B+F | max | all | 1 | 1] | 12 | 12 |  | 12 | 12 | 1.66 |
| 8: [A+B+R | max | all | 1 | 1] | 11 | 13 |  | 11 | 13 | 1.668 |
| 9: [A+B+F+R | max | all | 1 | 1] | 12 | 12 |  | 13 | 11 | 1.742 |
| 11: [A+B | max | 0 | 1 | 0] | 26 | 27 |  | 27 | 27 | 2.275 |
| 12: [A | max | 0 | 0 | 1] (ETNN-graph) | 15 | 15 |  | 14 | 16 | 1.658 |
| 13: [A | max | 0 | 0 | 1] (ETNN-graph-W) | 14 | 14 |  | 14 | 13 | 1.624 |
| 14: [A+B+F+R | max | all | 1 | 0] | 23 | 24 |  | 24 | 24 | 2.278 |
| 15: [A+B+R | max | all | 1 | 1] (triangles only, EMPSN-like) | 12 | 12 |  | 11 | 12 | 1.703 |
| 16: [A+B+F+R | max | all | 1 | 1] (no geometric features, CCMPN-like) | 32 | 34 |  | 35 | 32 | 3.268 |
| 17: [A+B+F+R | max | all | 1 | 0] (no geometric features, CCMPN-like) | 32 | 34 |  | 34 | 34 | 3.202 |
| 18: [A+B | max | 1 | 1 | 1] | 12 | 12 |  | 11 | 13 | 1.65 |
| 19: [A+B+F | max | 1 | 1 | 1] | 13 | 14 |  | 12 | 13 | 1.644 |
| 20: [A+B+R | max | 1 | 1 | 1] | 12 | 13 |  | 12 | 11 | 1.714 |
| 21: [A+B+F+R | max | 1 | 1 | 1] | 12 | 12 |  | 13 | 11 | 1.694 |
| 23: [A+B+F+R | max | 1 | 1 | 0] | 24 | 24 |  | 24 | 24 | 2.183 |
| 25: [A+B+F+R | max | 1 | 1 | 1] (no geometric features, CCMPN-like) | 32 | 35 |  | 35 | 34 | 3.361 |
| 26: [A+B+F+R | max | 1 | 1 | 0] | 32 | 35 |  | 34 | 34 | 3.183 |
| 27: [A+B | max, $\uparrow$,$\downarrow$ | 0 | 1 | 1] | 427 | 432 |  | 434 | 431 | 17.279 |
| 28: [A+B+R | max, $\uparrow$,$\downarrow$ | 0 | 1 | 1] (equivariant CWN-like) | 367 | 371 |  | 376 | 373 | 16.298 |
| 29: [A+B+R | max, $\uparrow$,$\downarrow$ | 0 | 1 | 1] | 302 | 302 |  | 302 | 302 | 13.019 |
| 30: [A+B | max, $\uparrow$,$\downarrow$ | 0 | 0 | 1] | 236 | 238 |  | 237 | 236 | 12.175 |
| 31: [A+B | max, $\uparrow$,$\downarrow$ | 0 | 0 | 1] | 477 | 491 |  | 485 | 484 | 25.588 |
| 32: [A+B | max, $\uparrow$,$\downarrow$ | 1 | 0 | 1] | 29 | 30 |  | 29 | 30 | 2.369 |
| 33: [A+B+F | max, $\uparrow$,$\downarrow$ | 1 | 0 | 1] | 28 | 29 |  | 28 | 30 | 2.227 |
| 34: [A+B+R | max, $\uparrow$,$\downarrow$ | 1 | 0 | 1] | 23 | 24 |  | 25 | 23 | 2.165 |
| 35: [A+B+F+R | max, $\uparrow$,$\downarrow$ | 1 | 0 | 1] | 25 | 25 |  | 25 | 24 | 2.178 |
| 36: [A+B+R | max, $\uparrow$,$\downarrow$ | 1 | 0 | 0] (triangles only, heterogeneous EMPSN-like) | 25 | 26 |  | 25 | 25 | 2.215 |
| 37: [A+B+R | max, $\uparrow$,$\downarrow$ | 1 | 0 | 0] (no geometric features, CWN-like) | 27 | 28 |  | 27 | 27 | 2.222 |
| 38: [A+B+R | max, $\uparrow$,$\downarrow$ | 1 | 1 | 0] (no geometric features, CCMPN-like) | 34 | 34 |  | 34 | 34 | 3.154 |
| 39: [A+B+F+R | max, $\uparrow$,$\downarrow$ | 1 | 0 | 0] (no geometric features, CCMPN-like) | 32 | 33 |  | 33 | 32 | 3.148 |
| Best ETNN Configuration |  |  |  |  |  |  |
| ETNN-* | 11 | 12 |  | 11 | 11 | 1.589 |
| \hdashlineImprovement over EGNN | -8% | 0% |  | -8% | 0% | 2.5% |

As the reader can notice from Tables  11- 12, several higher-order ETNN configurations reach near-SotA results and significant improvements over the reference EGNNs baselines, i.e., the original EGNN (Satorras et al., ) in the upper part of the tables, and its implementations ETNN-graph and ETNN-graph-W obtained through our codebase. ETNN-graph is the same exact architecture as the original EGNN but trained with gradient clipping and geometric invariants normalization. As mentioned at the beginning of this section, we made sure that ETNN-graph without gradient clipping and normalization almost exactly reproduces the results of the original EGNN. We report EGNN-graph because it is sometimes able to reach better results than the original EGNN on some targets. Moreover, some configurations that we test are also instances of other architectures like EMPSN (Eijkelboom et al., ) or CWN (Bodnar et al., ), further highlighting the versatility of our framework. Notably, ETNN consistent outperforms TopNet(Verma et al., ) too. Overall, these results validate our TDL approach and show that the flexibility of CCs is necessary.

###### Remark 7 .

## Appendix K Architecture Description

In this section, we describe the implementation of E(n) Equivariant Topological Neural Networks (ETNNs). The model architecture is kept as similar to EGNN as possible so as to not introduce alternative explanations for differences in model performance. The main components of ETNNs are as follows:

Model Initialization:

- •
Invariant Normalizer: Normalizes invariant features dynamically using batch normalization with no learnable parameters. Each rank is normalized separately.
- •
Feature Embedding: Embeds input features into a hidden representation. Input features of each rank are embedded with a separate featurizer.
- •
Equivariant Message Passing (EMP) Layers: Repeatedly updates the hidden representations of each cell using messages from neighboring cells. For details, see below.
- •
Pre-Pool Layer: Updates the hidden representation of each cell using distinct MLPs for each rank.
- •
Post-Pool Layer: MLP applied after the global pooling operation to produce the final output.

Forward Pass:

- 1.
Input Handling:
•
Retrieves pre-computed non-geometric cell features, adjacency relationships, and positional information.
- 2.
Initial Feature Computation:
•
Computes node and membership features for each rank. Cells with higher rank may either use their own non-geometric features, the mean feature vector of the nodes the cell is composed of, a membership vector denoting the lifter that generated the cell, or any combination of these.
•
Embeds the chosen non-geometric features into a common hidden representation.
- 3.
Geometric Invariant Computation:
•
Computes E(n) invariant geometric features using the absolute positions of nodes and the specified compute_invariants function.
•
Optionally normalizes the computed invariant features.
- 4.
Message Passing:
•
Applies EMP layers to update cells’ hidden represntations via message passing. For details on message passing, see below.
- 5.
Readout and Pooling:
•
Applies pre-pooling MLPs to the hidden representations of cells. Each rank is processed by a different MLP.
•
If the task is graph-level prediction, hidden representations of cells of same rank are added together to arrive at rank representations. These representations are concatenated and a post-pooling MLP is applied to generate the final output.

Equivariant Message Passing (EMP) Layers:
The computation of the EMP layer takes place in the following steps:

- 1.
Computation of individual messages:
•
For each cell, a message is computed for each of its parents. This message is computed by an adjacency-specific MLP that takes as input the hidden representations of the sending and receiving cells, and the E(n) invariant geometric features derived from the absolute positions of the nodes they are composed of. A separate MLP is defined for each rank pair.
- 2.
Adjacency-wise aggregation of computed messages:
•
For any given cell, the set of messages it received from its neighbors are grouped according to the rank of the sender. The messages within each group are aggregated using a weighted sum of the messages, where the weights are the nonnegative, scalar outputs of an adjacency-specific MLP that takes as input a message.
- 3.
Cell representation update:
•
Finally, an update is computed by feeding the concatenation of the adjacency-wise aggregated messages to a rank-specific update MLP. This update is added to the cell’s hidden representation.
- 4.
(Optional) position update:
•
Optionally, node positions are updated as a weighted sum of differences between the position of the node and its neighboring nodes. The weight is a learnable function of the message that was passed to the node from its neighbor.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 4
questions: Experiments

Could you provide more details on how functional groups were defined or identified in the QM7 dataset? Were these pre-annotated in the data or determined through some other process?

It also seems like many of the best results are given in hyper parameter configurations without the use of higher order cells. Can the authors comment on this? Does this detract from the main thrust of the paper, which is the inclusion of higher order cells?

Novelty: TDL vs GDL

Turning my comments into the weaknesses section in a specific question: what do we gain from introducing the notion of a combinatorial complex? As above, (1) the procedures for turning graphs into combinatorial complexes in this paper are not objects commonly studied in topology — i.e. do not correspond to a simplicial or cellular complex — so do not have access to additional mathematical theory and (2) we could seemingly equally well-formulate this paper as an EGNN over a specific procedure for turning the data into a graph﻿ as per Prop. 1. Why not just say this is a specific example of a EGNN using the construction in Prop 1?

I am very open to hear the author’s thoughts/be corrected on this!

Virtual cells

The virtual cell seems like to dramatically increase performance. However, my understanding is that this totally change the connectivity of the underlying data, meaning that
everything is now connected (as per the comment on 389). ﻿What does this say about the actual importance of the graph structure? 

It seems to suggest the ’topology’ of the underlying molecular graph is basically ignored once the virtual cell is included. Why not just define the architecture as a family of fully connected EGNN over the nodes, edges and potentially the faces, given that the hyperparameters including the virtual cell almost always are superior?

On a related note: "it is clear how ETNN naturally handles heterogeneity, e.g., the same pair of bonds could be connected because part of the same functional group (up adjacency) and the virtual cell (max adjacency), but the two messages will be different across the neighborhood”. Unless I’ve missed something, how are we certain that the two types of propagation add to performance, rather than just overcomplicating the situtation?

Miscellaneous

Line 354: In the defintiion of geospatial CC, the mapping function s : X \to T takes cells to points in T, but the geographic space s(X) is now a subset of T rather than an element. Should s be mapping to the powerset of T, rather than T, if indeed it takes each cell to a subspace of T? 

In general, it would better to have a more concrete definition of what is meant by a ‘geospatial combinatorial complex’. I’m also unfamilar with the term polyline. 

Line 375: "This work is the first to explore combinatorial topological modeling of multi-resolution irregular geospatial data..” The claim that this is the first work to explore combinatorial topological modeling of multi-resolution irregular geospatial data seems overstated given existing literature in Topological Data Analysis applied to geospatial data. Could you clarify how your approach differs from or advances beyond these existing works?" https://arxiv.org/pdf/2104.00720, https://www.researchgate.net/publication/366891451_Topological_data_analysis_for_geographical_information_science_using_persistent_homology, https://pubmed.ncbi.nlm.nih.gov/26353267/

Table 2: It would be helpful to have the variance included over the multiple runs

Line 242: "geometric invariants should make use of the underlying topological structure.” The pairwise distance is more of a geometric structure than a topological one.
soundness: 3
strengths: Presentation, clarity and experimental transparency

I found the paper to be written clearly, and the mathematical statements and proofs were precise, well formulated and easy to follow. Further, the authors included a lot of helpful background about topological deep learning, and explained concepts clearly. 

One of the biggest strengths of this paper is the much appreciated transparency around testing in the appendix. This allowed a confident and clear understanding of what the authors actually did to obtain their results. I really appreciated the detailled description of cell features in section I.1.2 and I.2.2. The detailled ablation studies on the were also super helpful to understand and evaluate overall performance.

Novel geospatial task

I really like that the authors introduced a novel geometric prediction task into the literature. In reading the literature for this review, it struck me that many of the benchmarks for TDL were somewhat old and outdated. It seemed appropriate that the task featured integration of data over different dimensional regions (points, lines, cells) in a way that showcased the central feature of the paper — reconciling data with features on subspaces of differing dimension i.e. the ‘heterogeneous interactions’ promised in the abstract. 

Computational

I think one of the main strengths of the paper is the computational benefits, and I would personally focus on this more in the introduction. The tailored lifting of molecular graphs to the higher order CCs dramatically decreases the number of higher order cell, which is a problem that plagues many architectures in learning on simplicial/cellular complexes. The section on computational complexity also demonstrates this in a robust mathematical setting.
summary: This paper introduces an equivariant model within the framework of topological deep learning. The architecture generalizaes the equivariant graph neural network architecture from Santorras at al. from the setting of graphs to message passing over combinatorial complexes. Notably, this architecture allows for message passing with cells that have heterogeneous node feature over differening ranks. 

The authors first introduce the relevant theory of combinatorial complexes, topological deep learning and important noitions of equivariance.  In section 3, they introduce their architecture using the formalism from the previous section. The authors then discuss important theoretical aspects of their model; proving equivariance, comparing expressiveness with traditional equivariant, scalarization-based techniques and discussing computational complexity of their design. 

The authors discuss in section 4 two complementary examples of how to attain combinatorial complexes: firstly in molecular data and secondly in geospatial data. In section 5, the authors use said combinatorial complexes to benchmark their method against common architectures on the QM9 dataset. Secondly, the authors introduce a novel benchmark for predicting airpollution from annoted geospatial data.
weaknesses: TDL vs GDL: novelty as a conceptual framework

For me personally I find it hard to understand the framing of this as a part of an entirely new conceptual field of topological deep learning beyond GNNs, and question the genuine novelty of papers like this. This is a concern I have of the field of TDL more generally, but I hope that the authors may be able to help clarify given their excellent communication skills demonstrated in the paper. 

Unless I’m mistaken, the basic content of proposition (1) is that an ETNN can be reformulated as an EGNN. This means that the main novelty is the clever choice of ‘lifting' the data into a certain graph and some delination of the learning based on ‘rank’. Indeed, even the proof of theorem (1) is basically a straightforward adaption of the corresponding result for EGNNs. I think the need to reformulate everything as an ETNN then show it’s equivalent to some specific EGNN needs more justification.

I still think that the experiments, results and set-up of the paper are interesting, but I personally get the sense that the ’topology’ part — and hence the novelty of these kinds of architectures — is overplayed a little.  One gets the impression from reading the introduction that there is some topological thing deep down somewhere that is making the difference in performance, whereas my feeling is that the inclusion of domain-specific data — functional groups, rings, etc. — along with the design of the graph is doing most of the work. 

On a similar theme, I don’t personally see a strong connection with the ‘lifted’ combinatorial complexes used in this paper and topology in the classical sense. It’s true that cell complexes and simplicial complexes also have the structure of a combinatorial complex (as in Appendix C), but these classical objects have additional connections to topology — i.e. they are stratifications of a genuine topological space, posses a homology theory, etc. 

QM7 Benchmarking comments

I find the approach to testing for this benchmark slightly unfair on other methods. Searching through so many iterations of hyperparameters, it seems inevitable that eventually some parameter set will outperform current methods at least once just on the balance of probabilities rather than model strength. ﻿Surely a fairer test would be run the competing methods along a similarly vast set of hyperparameters? At minimum, this experiment should be run multiple times, with the variance on results included, to test the robustness of these results.

Could the authors elaborate on why they chose to focus only on comparing the ETNN only with EGNN in the bottom line of Table 1? Upon first look, I assumed that ETNN outperformed SotA, but then on closer look saw that Equiformer outperformed ETNN in (almost)( every category. I would recommend highlighting the best performing method in bold in each column so that it’s more immediately clear that the while ETNN is improving on EGNN, it is still behind the SotA methods like DimeNet++ and Equiformer. My personal opinion is that the bottom line should be removed — the fact that it outperforms EGNNs is not surprising considering the fact that EGNNs are somehow a sub-architecture given the results of the expressiveness appendix.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. If I understand correctly, your argument on “heterogeneous interactions” focuses on different relationships between cells. Meanwhile, this property was actually mentioned in prior literature ([3] to name a few). I don’t think it is fair to claim that ETNNs are set up for this characteristic, but more like TDL in general already possesses this property.
2. The paper mentioned that Combinatorial Complex subsumes Path Complex. I believe there are two distinct lines of work regarding complexes arising from paths. One is path-based Combinatorial Complex [3], and another work is a simplified path complex [2] based on path complex [1]. [2] can’t be derived directly from [3] because [2] reserves the sequential information of paths. From my understanding, your work focuses more on path-based combinatorial complexes.
3. A relevant work [4] is not discussed in the paper.
4. It seems to me that $\mathcal{N}_{A, \text{max}}$ isn’t supported by any experiments. It would be better if the authors provide a set of experiments to support this design choice.
5. A minor subjective feedback on writing. I think it is better to have different notations for “containment” and “is a subset of” operation for clarity (Equation 6 and 7).
6. Regarding section I.1.2, it seems that 2-cell features do not encode any geometric features (velocity for example) but only invariant features. I think it is a missing piece to convince the audience that your framework can work with geometric features. Also, suppose that even when we have geometric features at node levels, it is unclear how to lift these features into higher-order spaces.

[1] Grigor’yan, A.A., Lin, Y., Muranov, Y.V. et al. Path Complexes and their Homologies. J Math Sci 248, 564–599 (2020).

[2] Truong, Q., & Chin, P. (2024). Weisfeiler and Lehman Go Paths: Learning Topological Features via Path Complexes. Proceedings of the AAAI Conference on Artificial Intelligence, 38(14).

[3] Hajij, M. et al. Topological Deep Learning: Going Beyond Graph Data (2023).

[4] Li, L. et al. Path Complex Neural Network for Molecular Property Prediction. ICML 2024 Workshop GRaM (2024).
soundness: 2
strengths: The authors add an important piece of work for the Topological Deep Learning (TDL) community as there is not much literature on Equivariant TDL. The work is well-formulated with clear motivations. The theoretical contributions are well-written. The paper is also self-contained and easy to follow, given the substantial explanations from related prior literature. The ablation studies are well-conducted via many synthetic graphs and additional information (hyperparameters, data statistics, etc.) are provided. The novel benchmark based on geospatial information is novel.
summary: The authors propose an equivariant Topological Deep Learning framework that deals with geometric node features. The frameworks can be generalized to many topological domains including simplicial, cell, combinatorial, and path complexes. The authors provide theoretical analysis regarding his design choice as well as expressiveness of the proposed method. Lastly, the authors support his arguments via real-world datasets QM9 and his proposed benchmark - air pollution downscaling benchmark.
weaknesses: Novelty is the key disadvantage of the paper. It seems that the work just extends prior works on graphs to TDL. Even though the theoretical insights are important, yet they are mostly an extension from graphs. Another important weakness is scalability and practicability  of the problem. There are only two real-world datasets evaluated, and in both cases, graph sizes are small. Furthermore, the performance isn’t convincing given there are only minor improvements over the graph counterparts. The performance could be due to extra parameters for higher-order filters. Perhaps an experiment on model comparison with constraints on parameter budgets and an ablation study on higher-order features masked out are needed to prove your arguments. Lastly, even when the framework makes sense, it is unclear how we can obtain geometric features for higher-order cells. Please refer to question 6 for my concern.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: see weaknesses.
soundness: 3
strengths: 1. The paper offers a simple and straightforward way to adapt higher order message passing to respect $O(d)$ symmetries.

2. The experimental section effectively demonstrates the architecture's performance and introduces two new, interesting real-world TDL benchmarks, addressing a need highlighted in a recent position paper [1]. 



[1] Theodore Papamarkou, Tolga Birdal, Michael M Bronstein, Gunnar E Carlsson, Justin Curry, Yue Gao, Mustafa Hajij, Roland Kwitt, Pietro Lio, Paolo Di Lorenzo, et al. Position: Topological deep learning is the new frontier for relational learning. In Forty-first International Conference on Machine Learning, 2024.
summary: This paper extends message passing supported on geometric graphs [3] and geometric simplicial complexes [2] to the more general setting of geometric combinatorial complexes. Theoretically, similar to how [1] establishes that higher-order message passing is more expressive than standard graph message passing, this paper demonstrates that the same holds true in the geometric setting. The paper also highlights the effectiveness of the proposed architecture through two novel real-world applications: (1) property prediction over geometric graphs representing molecules, where higher-order cells consist of rings and active groups, and (2)  a regression task over geospatial combinatorial complexes. The proposed architecture outperforms previous geometric graph architectures on these tasks.


[1] Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Lio, Guido F Montufar, and Michael Bronstein. Weisfeiler and lehman go cellular: Cw networks. Advances in neural information processing systems, 34:2625–2640, 2021.

[2]  Floor Eijkelboom, Rob Hesselink, and Erik J Bekkers. E (n) equivariant message passing simplicial networks. In International Conference on Machine Learning, pages 9071–9081. PMLR, 2023.

[3] Vıctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pages 9323–9332. PMLR, 2021.
weaknesses: 1. The novelty of the proposed architecture and the theoretical section is somewhat limited. The architecture closely resembles those presented in [4] and [2]. Additionally, the theoretical contributions feel somewhat straightforward, offering limited new insights.

2. Building on the previous comment about the theoretical section, the paper lacks an analysis of how the choice of invariant function (see Equation (6)) affects the architecture's expressivity. It would be valuable to examine whether simpler, computationally efficient invariant functions could result in architectures that are  as expressive as architectures which use more complex alternatives, thereby guiding the choice of which invariant function to use in practice. Additionally, it would have been insightful to analyze whether there exist any natural geometric invariant functions that  geometric graph models are unable to compute while the proposed model succeeds in doing so.

3. The end of Proposition 2 states "In most of the cases, an ETNN is strictly more powerful than an EGNN". I tried finding the proof to this in appendix F and had a hard time. I think the authors refer to proposition 2 in appendix F but I'm not sure. A clearer framing of this result in the appendix, and perhaps an illustrative example or plot in the main body, would improve readability and support this claim.

4. The paper refers to the architecture proposed in [2] for geometric simplicial complexes. A comparison of the expressive power of ETNN using different lifts compared to this architecture would be interesting.

5. The paper [3] benchmarks higher-order message passing on several geometric benchmarks, using data augmentation to address $O(d)$ symmetries. An empirical comparison to this approach would provide valuable insight.

[1] Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Lio, Guido F Montufar, and Michael Bronstein. Weisfeiler and lehman go cellular: Cw networks. Advances in neural information processing systems, 34:2625–2640, 2021.

[2]  Floor Eijkelboom, Rob Hesselink, and Erik J Bekkers. E (n) equivariant message passing simplicial networks. In International Conference on Machine Learning, pages 9071–9081. PMLR, 2023.

[3] Mustafa Hajij, Ghada Zamzmi, Theodore Papamarkou, Nina Miolane, Aldo Guzm´an-S´aenz, Karthikeyan Natesan Ramamurthy, Tolga Birdal, Tamal K Dey, Soham Mukherjee, Shreyas N Samaga, et al. Topological deep learning: Going beyond graph data. arXiv preprint arXiv:2206.00606, 2022.

[4] Vıctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pages 9323–9332. PMLR, 2021.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - Can the authors clarify the motivation for using "k-hop distinct" graphs as the primary measure of expressivity? Is it possible extend the expressivity results to CCs? E.g. is it possible to define a notion of "k-hop distinctness" for geometric combinatorial complexes and analyze ETNN's ability to distinguish between such complexes?
- How does ETNN's expressivity compare to that of simplicial complex networks in distinguishing non-isomorphic geometric graphs/complexes?
- In proposition 2, can you specify the conditions under which ETNNs are provably more expressive than EGNNs? 
- Could you include standard deviations for the air pollution benchmark and QM9 results to assess statistical significance?
soundness: 3
strengths: - **Framework and Theoretical Contributions:**
  - The ETNN architecture introduced in the paper is a novel framework for $\mathrm{E}(n)$-equivariant processing of a wide class of topological data objects.
  - The framework elegantly unifies and generalizes existing architectures -- e.g. EGNN [1] and EMPSN [2].
  - The paper theoretically proves expressivity improvements over $\mathrm{E}(n)$-equivariant graph methods.

- **Empirical Results:**
  - ETNN variants achieve clear performance gains over standard $\mathrm{E}(n)$-equivariant graph methods for QM9 molecular property prediction tasks; ETNNs also improves upon EMPSNs while using less memory and having a faster runtime.

- **Applications and Dataset Contributions:**
  - The paper introduces a principled approach to modeling irregular multi-resolution geospatial data using combinatorial complexes.
  - The *air pollution downscaling* benchmark introduced in the paper is a new dataset for benchmarking TDL architectures. Additionally, the construction and analysis of geospatial combinatorial complexes is in itself an interesting contribution of the paper.

[1] Satorras et al. "E(n) equivariant graph neural networks", ICML 2021.

[2] Eijkelboom et al. "E(n) equivariant message passing simplicial networks", ICML 2023.
summary: **Method:** The paper introduces a framework, termed ETNN, for processing combinatorial complexes with geometric cell features in an $\mathrm{E}(n)$-equivariant way (explicitly introduces a method for $0$-cell geometric features and explains how to generalize to geometric features of cells of arbitrary rank). ETNNs are equivariant to the action of both $\mathrm{Sym}(\mathcal{X})$ (cell renaming) and $\mathrm{E}(n)$ isometries acting on the geometric features. The architecture builds on higher-order message-passing over CCs and has both invariant and equivariant versions. ETNNs generalizes previous work on $\mathrm{E}(n)$-equivariant graph neural networks, and $\mathrm{E}(n)$-equivariant cellular complex networks.

**Theory:** The expressivity of the proposed methods is evaluated based on its ability to distinguish $k$-hop distinct geometric graphs. The paper proves that in *most cases* ETNN is strictly more expressive than baseline geometric graph methods (i.e. message-passing architectures on geometric graphs that do not operate on higher-order cells).

**Experiments:** 
- **QM9:** The first application considered in the paper is QM9 molecular property prediction, where molecules are represented as combinatorial complexes with geometric features. ETNN variants demonstrate clear performance increase over standard geometric graph methods.
- **Air Pollution Benchmark:** The second application introduces a new benchmark for air pollution prediction. CCs are constructed from point measurements (0-cells), road measurements (1-cells), and census data (2-cells). 

**Notes.**
- Typo in equation (13); should be "$x \subset z$, and $y \subset z$".
weaknesses: - **Novelty:** While the proposed method generalizes previous work to general combinatorial complexes, the architectural changes are incremental.
  
- **Theoretical analysis:**
  - The expressivity analysis focuses solely on distinguishing geometric *graphs*, despite defining expressive power in terms of separating more general non-isomorphic *CTSs*.
  - Proposition 2's statement is imprecise: the claim of improved expressivity "in most cases" lacks formal qualification, and the relationship between the expressivity gap and choice of lifting method needs clearer formulation and concrete characterization. A clearer restatement would be helpful.
  - No formal comparison to expressivity of other TDL methods (e.g. equivariant simplicial networks).

- **Empirical evaluation:**
   - Results lack statistical significance analysis (no standard deviations reported).
  - On the air pollution benchmark, improvements over the MLP baseline are modest (~1.5% RMSE reduction) and their statistical significance cannot be assessed due to unreported standard deviations.

Meta-review: This paper extends TNN to ETNN, similar to the extension of GNN to EGNN. There have been very intensive discussions among the authors and reviewers, and the authors put a lot of efforts during the process. Eventually, I believe all the major issues have been resolved. I agree with some of the reviewers that the work is not entirely novel given the similarity with extension of GNN to EGNN. In this sense, I tend to think this is a borderline paper. On the other hand, given the many empirical comparisons and insights of add equivariance to existing methods, I tend to recommend an accept.

Decision: Accept (Poster)