MANTRA: The Manifold Triangulations Assemblage

## Abstract

Abstract The rising interest in leveraging higher-order interactions present in complex systems has led to a surge in more expressive models exploiting high-order structures in the data,
especially in topological deep learning (TDL), which designs neural networks on high-order domains such as simplicial complexes. However, progress in this field is hindered by the scarcity of datasets for benchmarking these architectures. To address this gap, we introduce MANTRA, the first large-scale, diverse, and intrinsically high-order dataset for benchmarking high-order models, comprising over 43,000 and 249,000 triangulations of surfaces and three-dimensional manifolds, respectively. With MANTRA, we assess several graph- and simplicial complex-based models on three topological classification tasks. We demonstrate that while simplicial complex-based neural networks generally outperform their graph-based counterparts in capturing simple topological invariants, they also struggle, suggesting a rethink of TDL. Thus, MANTRA serves as a benchmark for assessing and advancing topological methods, leading the way for more effective high-order models.

## 1 Introduction

Success in machine learning is commonly measured by a model’s ability to
solve tasks on benchmark datasets. While researchers typically devote a
large amount of time to build their models, less time is devoted to data
and its curation. As a consequence, *graph learning* is facing some
issues in terms of reproducibility and wrong assumptions, which serve as
obstructions to progress. An example of this was recently observed while
analyzing long-range features: additional hyperparameter tuning resolves
performance differences between message-passing (MP) graph neural networks on
one side and graph transformers on the other (Tönshoff et al., ).
In a similar vein, earlier work pointed out the relevance of strong
baselines, highlighting the fact that *structural* information
is not exploited equally by all models (Errica et al., ). Recently,
new analyses even showed that for some benchmark datasets, as well
as their associated tasks, graph information may be detrimental for
the overall predictive performance (Bechler-Speicher et al., ).

These troubling trends concerning data are accompanied by increased
interest in leveraging higher-order structures in data, with new models,
usually called *topological models*, extending graph-learning
concepts to *simplicial complexes*, i.e., generalizations of graphs
that incorporate higher-order relations, going beyond the dyadic
relations captured by graphs  (Alain et al., ; Bodnar et al., ; Maggs et al., ; Ramamurthy et al., ; Röell & Rieck, ; Yang et al., ).
Some topological models already incorporate state-of-the-art mechanisms for learning such as message-passing (Gilmer et al., ) or transformer layers (Ballester et al., ), but adapted to high-order domains, sometimes outperforming their original counterparts in graph datasets.
However, as pointed out in a recent position paper (Papamarkou et al., ), there is
a dire need for “higher-order datasets,” i.e., datasets that contain non-trivial higher-order structures.
Indeed, the scarcity of such datasets impedes the development of
reliable benchmarks for assessing
(i) the utility of higher-order structures present in data, and
(ii) the performance of the new models that leverage them,
thus potentially eroding trust in topological models among the broader
deep learning community.
To the best of our knowledge, the only publicly-available high-order
dataset is the “Torus” dataset proposed
in Eitan et al. (),
which consists of a small number of unions of tori triangulations.
However, due to the nature of the dataset, the only varying topological
property among the samples is the number of connected
components of each union, making hard to assess the true capacity of the
models to learn and exploit higher-order structures.
The lack of higher-order datasets is also remarked upon in a recent
benchmarking paper for topological
models (Telyatnikov et al., ),
which restricted itself to existing graph datasets that were subjected
to a variety of *topological liftings*, i.e., methods for endowing
graph datasets with higher-order structures (Bernárdez et al., ; Jonsson, ).
However, it remains unclear whether standard graph neural network architectures
can also learn and take advantage of the information provided by the topological
liftings, as they are solely based on the graph structure.

#### Contributions.

To address this problem, we present MANTRA,
manifold
triangulations
assemblage, which constitutes the first instance
of a large, diverse, and intrinsically high-order dataset, comprising
triangulations of combinatorial $2$-manifolds and
$3$-manifolds.
Along with the data, we provide a list of potential tasks, as well as
a preliminary assessment of the performance of existing methods, both
graph-based and high-order-based, on the dataset. We focus on a subset of tasks concerned with the classification of simplicial complexes according to some topological labels, where we can interpret the success of a model as its effectiveness in extracting high-order topological information. However, these tasks are by no means exhaustive, and the generality offered by MANTRA encourages the emergence of more demanding tasks.
A noteworthy aspect of MANTRA is the conspicuous *absence* of any intrinsic
vertex or edge features such as coordinates or signals.
We argue that this absence renders tasks more topological, as models can only rely on topology, instead of non-topological information contained in features.
Moreover, as manifold triangulations are directly related to the
topological structure of the underlying manifold, we study to which
extent higher-order models are *invariant* to triangulation transformations that preserve the topological structure of the
associated manifold.

## 2 Dataset specification

Figure: Figure 1: Geometric realizations of some manifold triangulations included in MANTRA. The precise coordinates of vertices in Euclidean space are not geometrically significant; what matters is the topology of the resulting polyhedra. Hence, MANTRA is a *purely combinatorial dataset*.
Refer to caption: extracted/5898454/figures/img/random_node_features.png

MANTRA contains $43{,}138$ and $249{,}015$ simplicial complexes corresponding to
triangulations of closed connected two- and three-dimensional manifolds, respectively, with varying number of
vertices.
Manifolds have many applications: the configuration space of a robotic arm can be seen as a manifold (e.g., a torus or hyperbolic space, see Jaquier et al. ()); 3D shapes in geometry processing are triangulated manifolds (Crane, ); physical fields in climate forecasting naturally live on a sphere (Bonev et al., ), and the manifold hypothesis argues that high-dimensional data often lies in lower-dimensional manifolds (Fefferman et al., ).
Throughout the text, we use the term *surface* to refer to a two-dimensional manifold. A *triangulation* of a manifold $M$
is a pair consisting of a simplicial complex K and a
homeomorphism between $M$ and the geometric realization of K.
For brevity, we use the term triangulation to refer exclusively to the simplicial complex K.
See Appendix A.3 for precise definitions and further information.

**Table 1: Number of triangulations by manifold dimension and number of vertices of the triangulation, including total sum of triangulations per dimension.**
|  | Dimension |  |
| --- | --- | --- |
| # vertices | 2 | 3 |
| 4 | 1 | 0 |
| 5 | 1 | 1 |
| 6 | 3 | 2 |
| 7 | 9 | 5 |
| 8 | 43 | 39 |
| 9 | 655 | 1,297 |
| 10 | 42,426 | 249,015 |
| Total | 43,138 | 250,359 |

Triangulations of surfaces and 3-manifolds encode high-order topological information that cannot be inferred solely from their underlying graphs. Indeed, there exist non-homeomorphic surfaces with identical graph structures, as demonstrated by Lawrencenko & Negami (). Figure 1 contains examples of geometric realizations of MANTRA triangulations.
Table 1 contains the
distribution of triangulations in terms of their number of vertices.
Each triangulation contains a set of labels based on its dimension. Common labels
are the number of vertices of the triangulation, the first three Betti numbers
$\beta_{0}$, $\beta_{1}$, $\beta_{2}$, and torsion in homology with integer coefficients. Definitions of these concepts are given in Appendix A.2.
For surfaces, labels also contain orientability and genus. For triangulations of a Klein bottle $K$, a real projective plane $\mathds{R}P^{2}$, a $2$-dimensional sphere $S^{2}$, or a torus $T^{2}$, the homeomorphism type is included explicitly as a surface label. For $3$-manifolds, labels additionally specify the top Betti number $\beta_{3}$ and the homeomorphism type, which can be a $3$-sphere $S^{3}$, a product $S^{2}\times S^{1}$ of a $2$-sphere and a circle, or a Möbius-like $S^{2}$-bundle along $S^{1}$, denoted by $S^{2}\mathbin{\stackrel{{\scriptstyle\sim}}{{\smash{\times}\rule{0.0pt}{3.8749
5pt}}}}S^{1}$. An exploration of the distributions of labels is made in Appendix A.5.

We make the dataset available in two formats: the *raw version*
and the *PyTorch Geometric processed version* at the GitHub repository [https://anonymous.4open.science/r/mantra-D60C](https://anonymous.4open.science/r/mantra-D60C). The raw version is available
as a pair of compressed files 2_manifolds.json.gz and
3_manifolds.json.gz
containing each of them a JSON list with the triangulations of the corresponding
dimension. Each object of the JSON list contains a set of the following fields, depending
on the dimension of the associated triangulation:

- •
id (required, str): This attribute refers to the original ID of
the triangulation as used by  when compiling the triangulations.
This facilitates comparisons to the original dataset if necessary.
- •
triangulation (list of list of int): A
doubly-nested list of the facets of the triangulation.
- •
n_vertices (int): The number of vertices in the triangulation.
- •
name (str): Homeomorphism type of the triangulation.
Possible values are '', 'Klein bottle', 'RP^2',
'S^2', 'T^2' for surfaces, where '' indicates that
the explicit homeomorphism type is not available. For three-dimensional manifolds,
possible values are 'S^2 twist S^1', 'S^2 x S^1',
'S^3',
- •
betti_numbers (list of int): A list of the
Betti numbers of the triangulation, computed using $R=\mathds{Z}$, i.e., integer coefficients.
- •
torsion_coefficients (required, list of str): A
list of the torsion subgroups of the triangulation. Possible values are
'', 'Z_2', where an empty string '' indicates
that no torsion is present in that dimension.
- •
genus (int): For surfaces,
contains the genus of the triangulation.
- •
orientable (bool): For surfaces, specifies if the
triangulation is orientable or not.

The PyTorch Geometric (Fey & Lenssen, , PyG) version is available
as a Python
package that can be installed using the command pip install mantra-dataset.
Package documentation is available at
[https://anonymous.4open.science/r/mantra-D60C](https://anonymous.4open.science/r/mantra-D60C).
Each example of the dataset is implemented as a PyG Data object, containing the same attributes as JSON objects in the raw version.
The main difference with the data in the raw version is that numerical values are stored
as PyTorch tensors. Both formats, raw and processed, are versioned using the Semantic Versioning 2.0.0 convention () and will be made available via Zenodo, thus ensuring reproducibility and clear tracking of the dataset evolution. To decouple software implementation from actual data, we allow to load any dataset version from the PyTorch Geometric package, regardless of the installed version.

## 3 Experiments

Sections 3.1 and 3.2 presents the comprehensive experimental design for MANTRA, outlining the key scientific questions addressed. Section 3.3 provides a detailed analysis of the experimental results.

### 3.1 Main experiments

In this section, we demonstrate MANTRA’s effectiveness as a comprehensive benchmark for
higher-order models.(^†^†The code for the experiments can be found in the repository [https://anonymous.4open.science/r/mantra-benchmarks-2500/README.md](https://anonymous.4open.science/r/mantra-benchmarks-2500/README.md).)
Leveraging the extensive set of labels and triangulations available, our experiments are
designed to address the following critical research questions:

- Q1
To what extent are higher-order models needed to perform inference tasks on
high-order domains like simplicial complexes? Are graph-based models enough to successfully capture the full set of combinatorial properties present in the data?
- Q2
Do current neural networks, both graph- and simplicial complex-based, capture topological
properties in data? Are they able to predict basic topological invariants
such as Betti numbers of simplicial complexes?
- Q3
How invariant are state-of-the-art models to transformations that preserve topological properties of data?

The difference between Q1 and Q2, Q3 is subtle. Combinatorial information is
related to the structure of the data, in our case, simplicial complexes,
while topological information is related to
properties that are invariant under *topological transformations* of the data. For example, in
prediction tasks involving molecules, we expect combinatorial information to be more relevant
than topological features, since the structure of a molecule is crucial in predicting properties of the molecule. Of course,
both types of information are intertwined: to properly compute topological properties of data, we need to consider
its combinatorial structure, as explained in Appendices A.2 and A.3.
To answer the above questions, we benchmarked nine models: five graph-based models
from the PyTorch Geometric library (Fey & Lenssen, ), using only zero- and one-dimensional
simplices of complexes,
and four simplicial complex-based models from the TopoModelX library (Hajij et al., ),
using the full set of simplicial complexes
in different tasks per manifold dimension:

- T1
Predicting the Betti numbers $\beta_{i}$ for triangulated surfaces and $3$-dimensional manifolds.
- T2
Predicting the homeomorphism type of
triangulated surfaces.
- T3
Predicting orientability of
triangulated surfaces.

To address the high proportion of surfaces without explicitly assigned homeomorphism type, we duplicated the experiments on both the full set of surfaces and the subset of surfaces with known type.

#### Models.

The graph-based models benchmarked are the Multi-Layer Perceptron (MLP),
the Graph Convolutional Network (Kipf & Welling, , GCN),
the Graph Attention Network (Veličković et al., , GAT), the Graph
Transformer (Shi et al., , TransfConv),
and the Topology Adaptive Graph Convolutional Network (Du et al., , TAG), while the
simplicial complex-based benchmarked models are the Simplicial Attention Network (Giusti et al., , SAN), and three convolution-based simplicial neural networks introduced in Yang et al. (),
Yang & Isufi (), and Wu et al. (), denoted by
SCCN, SCCNN, and SCN, respectively.
These last ones, SCCN, SCCNN, and SCN, were the simplicial complex-based networks benchmarked in Telyatnikov et al. ().

#### Features.

All nine models assume that simplicial complexes are equipped with feature vectors
on top of a subset of the simplices. The feature vectors
for graph-based models are either:
(1) scalars randomly generated,
(2) degrees of each vertex,
(3) degree one-hot encodings of each vertex.
For simplicial complex-based models, the feature vectors are either:
(1) eight-dimensional vectors generated randomly,
(2) upper degrees of each simplex of dimensions lower than the dimension of the simplicial complex and lower degrees for simplices of the same dimension as the simplicial complex.
The upper degree of each simplex $\sigma$ of dimension $k$ is the number of $k$-simplices $\tau\neq\sigma$
that share a $(k+1)$-simplex $\omega$ such that $\sigma,\tau\in\omega$.
Similarly, the lower degree of each simplex $\sigma$ of dimension $k$ is the number of $k$-simplices $\tau\neq\sigma$ that share a $(k-1)$-simplex $\omega$ such that $\omega\in\sigma,\tau$.

#### Training details.

In total, our experiments span 184 training results across various tasks, feature generation, and models.
To ensure fairness, all configurations use the same learning rate of 0.01 and
the same number of epochs of 6; we observe that graph-based models already overfit after a single epoch, though.
Hyperparameters for graph-based models were mostly extracted from the default
examples from PyTorch Geometric, while hyperparameters for
simplicial-complex based models were set to values similar to the ones from the
TopoBenchmarkX paper (Telyatnikov et al., ). Hyperparameter details can be found in Appendix B.
To mitigate the effects of training randomness, we re-ran each experiment five times and considered both the best and the mean (together with standard deviation) performance obtained across these runs
for each model and initialisation of features.
Due to the high imbalance in the datasets
for most labels, we performed stratified train/validation/test splits for each task individually, with 60/20/20 percentage of the data for each split, respectively.
Splits were generated using the same random seed for each run, ensuring that the same splits are used
across all configurations. All models were trained using the Adam optimizer.

#### Loss and metric functions.

Each task (T1, T2, T3) was treated as
a classification task during testing.
We report the area under the ROC curve (AUROC) (Bradley, ) as performance metric, which is standard for imbalanced classification problems, on all tasks except for predicting $\beta_{0}$, where we report accuracy due to the fact that we only have the label $1$, as all our triangulations correspond to connected manifolds.
For both the homeomorphism type and orientability tasks, we train the models using
the standard cross-entropy loss for classification problems.
We also experimented with weighting the cross-entropy loss to penalize
mispredictions in under-represented classes more heavily, but we did not obtain
improvements. To avoid increasing the computational complexity of our experiments,
we chose not to implement more involved methods for handling the class imbalances
and leave this issue for future work.
For Betti number prediction, we approached training as a multivariate regression task,
since Betti numbers
can theoretically be arbitrarily large.
Our loss function in this case was the mean squared error, and the Betti number prediction
was obtained by rounding the model outputs to the nearest integer.

### 3.2 Barycentric subdivision experiments

The previous experiments try to answer Q1 and Q2: if performances are good for simplicial complex-based models, but not for graph-based ones, then we can conclude that
higher-order models are needed to perform inference tasks on domains with high-order and topological
information. By contrast, if performances are good for graph-based models
then we can conclude that graph models are enough to capture the full set of combinatorial and
topological features present in MANTRA’s dataset, questioning the need for higher-order models.
However, Q3 is more subtle. Although it is closely related to Q2, Q3 emphasizes the *invariance* of the models to transformations that preserve the topological properties of the input data, a desirable property for TDL models known as remeshing symmetry (Papamarkou et al., ).
For example, if a model is well-trained with a dataset containing only triangulations up to a certain number of
vertices, we can expect the model to perform correct predictions in new examples that also have at most
the maximum number of vertices seen in the training dataset. However, what happens if we try to predict from a *refinement* of a manifold triangulation?
For instance, barycentric subdivisions increase the (combinatorial) distances between the original vertices in a triangulation, and this can be harmful for networks relying on the MP algorithm,
since distances determine how many layers are needed to propagate information from one vertex to another. In fact, Horn et al. () showed that MP-based graph neural networks with a small number of layers struggled to obtain good performances on synthetic datasets where the number of cycles and connected components played a crucial role.

To answer Q3, we performed an additional evaluation of the models trained on surface tasks with known homeomorphism type for
the experiments described in Section 3.1. Particularly, for each task,
we evaluated the performance of the trained models on a dataset obtained by performing one barycentric
subdivision on each triangulation in the original test dataset, and then we compared the performances of the models on
both datasets, original and subdivided. We did not analyze barycentric subdivisions of $3$-dimensional manifolds
due to computational constraints.

### 3.3 Analysis

;

**Table 2: Predictive performance of graph- and simplicial complex-based models on surface and $3$-manifold tasks. Results for the full set of surfaces ($2$-$\mathcal{M}^{0}$), for the set of surfaces with known homeomorphism type ($2$-$\mathcal{M}^{0}_{H}$), and for the full set of three-manifolds ($3$-$\mathcal{M}^{0}$) are reported. Additionally, performance metrics for the barycentric subdivision of the test set on the models trained on $2$-$\mathcal{M}^{1}_{H}$ are included; see Section 3.2 for details. For each family of models, $\mathcal{G}$ (graph-based) and $\mathcal{T}$ (simplicial complex-based), we report the mean and standard deviation of the maximum performance achieved across five runs by each combination of feature vector initialization and model contained in the family. The tasks reported are prediction of $\beta_{0}$, $\beta_{1}$, $\beta_{2}$, $\beta_{3}$, prediction of the homeomorphism type, and prediction of orientability. For all tasks except for prediction of $\beta_{0}$, we report the AUROC metric. For $\beta_{0}$, we report accuracy. Metrics are multiplied by $100$ and rounded to the second decimal for a better visualization. Best average result among both families for each task is in bold.**
|  |  | Accuracy ($\uparrow$) | AUROC ($\uparrow$) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Model<br>family | $\beta_{0}$ | $\beta_{1}$ | $\beta_{2}$ | $\beta_{3}$ | Homeomorphism<br>type | Orientability |
| 2-$\mathcal{M}^{0}$ | $\mathcal{G}$ | $100.00$ p m 00.00 | $50.06$ p m 0.09 | $50.00$ p m 0.00 |  | $46.65$ p m 0.50 | $50.00$ p m 0.00 |
| $\mathcal{T}$ | $49.67$ p m 38.69 | $69.25$ p m 14.86 | $64.11$ p m 10.23 |  | $68.01$ p m 12.37 | $56.89$ p m 5.70 |  |
| 2-$\mathcal{M}_{H}^{0}$ | $\mathcal{G}$ | $100.00$ p m 0.00 | $21.43$ p m 0.01 | $50.00$ p m 0.00 |  | $50.55$ p m 0.78 | $50.00$ p m 0.00 |
| $\mathcal{T}$ | $29.18$ p m 31.81 | $25.29$ p m 3.01 | $52.69$ p m 1.26 |  | $69.15$ p m 8.86 | $52.15$ p m 1.41 |  |
| 2-$\mathcal{M}_{H}^{1}$ | $\mathcal{G}$ | $47.19$ p m 49.41 | $21.53$ p m 0.07 | $50.00$ p m 0.00 |  | $49.32$ p m 4.05 | $50.10$ p m 0.36 |
| $\mathcal{T}$ | $6.01$ p m 10.68 | $24.04$ p m 2.02 | $51.39$ p m 1.31 |  | $57.49$ p m 6.51 | $50.62$ p m 0.65 |  |
| 3-$\mathcal{M}^{0}$ | $\mathcal{G}$ | $100.00$ p m 0.00 | $24.08$ p m 0.00 | $12.07$ p m 0.00 | $15.47$ p m 0.00 |  |  |
| $\mathcal{T}$ | $55.47$ p m 45.96 | $30.05$ p m 6.61 | $13.63$ p m 2.63 | $18.56$ p m 3.74 |  |  |  |

Our analysis reports *aggregated results*
and focuses primarily on the comparison between graph-based models ($\mathcal{G}$)
and simplicial complex-based models ($\mathcal{T}$).
Comprehensive results are available in Appendix C. Table 2 presents the mean and standard deviation of the maximum performance achieved by each combination of feature vector initialization and model type across the $5$ runs of each task for both graph-based ($\mathcal{G}$) and simplicial complex-based ($\mathcal{T}$) model families, including performances on the barycentric subdivisions of the test triangulations for each experiment run in the set of surfaces with known homeomorphism type, as described in Section 3.2.
Notably, our experiments suggest that high-order MP-based
models are *not invariant* relative to topological transformations and therefore cannot be considered topological in the strictest sense of the term. Weaknesses in the MP-based models are not a recent phenomenon, as highlighted by oversmoothing (Li et al., ) and oversquashing (Alon & Yahav, ; Topping et al., ), and the MP paradigm has required numerous fixes since its existence (inlcuding, but not limited to, virtual nodes, feature augmentation, and graph lifting). More recently, Eitan et al. () argued that, in many cases, higher-order MP-based models cannot distinguish combinatorial objects based on simple topological properties, and has devised another MP variant to compensate for this.

#### Graph-based ( 𝒢 𝒢 \mathcal{G} caligraphic_G ) vs. simplicial complex-based ( 𝒯 𝒯 \mathcal{T} caligraphic_T ) models.

Table 2
together with the full results of Appendix C show that simplicial complex-based models consistently obtain better performances predicting non-trivial topological properties of triangulated manifolds, meaning $\beta_{1}$, $\beta_{2}$, $\beta_{3}$, orientability, and homeomorphism type. Counterintuitively, we note that graph models *always* correctly detect the connectivity of triangulations in $2$-$\mathcal{M}^{0}$, $2$-$\mathcal{M}_{H}^{0}$, and $3$-dimensional manifolds, thus predicting $\beta_{0}$ exactly, while topological models consistently fail to predict connectivity, except for the SCCN architecture in our experiments. The fact that high-order message passing networks cannot accurately predict connectivity was also found, and theoretically proved, in
(Eitan et al., , Proposition 4.3).
Moreover, although simplicial complex-based models obtain better results overall, these are far (!) from being highly accurate, with averages below 70 for all tasks and with a high performance variance across the models. Nonetheless, the best performances obtained by specific simplicial complex-based models, as described in the full results of Appendix C, are promising, achieving excellent AUROC results in some tasks, such as homeomorphism type prediction for the full set of surfaces, where the SCCN model obtained an AUROC of 89 for its best run, and Betti number prediction on the full set of surfaces, where SCCN and SCN obtained AUROCs of 96 and 80 respectively for predicting the first and second Betti numbers.
Overall, the results suggest that higher-order models are indeed necessary to capture topological and high-order characteristics of data, although several current models are not yet able to do so effectively, partially answering questions Q1 and Q2. Such results were expected, given that one-dimensional structures are insufficient, in principle, to fully characterize the topology of two- or three-dimensional triangulated manifolds, as stated at the beginning of Section 2.
However, it is plausible that graph-based networks can accurately classify approximately 50% of homeomorphism types of surfaces, since the underlying graph of a triangulation determines the Euler characteristic, which in turn defines the homeomorphism type up to orientability (see Appendix A.4).

#### Orientability.

Predicting orientability turns out to be the most difficult task for simplicial complex-based models, and generally difficult for graph-based models. Recall that orientability can be determined from the Betti numbers $\beta_{2}$ and $\beta_{3}$ for surfaces and $3$-manifolds, respectively. One could then argue that predicting these two Betti numbers is precisely as difficult as predicting orientability as a binary classification problem. However, we observed that the simplicial complex-based models are able to predict $\beta_{2}$ with a higher accuracy than orientability, while for graph-based models both metrics are on a par. Our hypothesis is that, forcing to learn the whole set of Betti numbers at the same time encourages simplicial complex models to learn topological properties contained in the triangulations, while predicting orientability as a binary classification problem does not. If this hypothesis is true, graph-based models might not be able to effectively capture subtle topological information contained in data, due to the similar results predicting $\beta_{2}$ and $\beta_{3}$ and orientability, supporting the claim that higher-order models are needed to leverage high-order information.
Both the results and our hypothesis encourage the use of auxiliary learning tasks (Liu et al., ) for high-order models by forcing the network to predict the whole set of topological labels together with the real target, as this seems to help a model learn how to efficiently use the topological information contained in the input data.
We consider this a promising direction for future work, either to generate topological regularization terms or to propose new forms of unsupervised pre-training procedures for higher-order models, as many MANTRA labels can be computed directly from the input simplicial complex using deterministic algorithms.

#### Barycentric subdivisions.

Table 2 shows that the performance of all models dramatically decreases when subdividing the triangulations of the test dataset, clearly indicating that the models are not learning the invariance of topological properties with respect to transformations that leave topological properties invariant. This is a crucial property that any model dealing with topological domains should have, as real data is often highly variable in terms of combinatorial information and representation, but not in terms of their topology.
This phenomenon is particularly evident in mesh datasets, where combinatorial structure varies with resolution. In fact, Papamarkou et al. () pose the capacity of TDL models to capture this invariance, denoted *remeshing symmetry*, as one of the reasons for using topological deep learning models. Our preliminary experimental results challenge this claim, opening the door to a new line of research based on the invariance of input transformations that leave topological properties of the input data unaltered.

#### Limitations.

Although our results challenge the efficiency of state-of-the-art high-order models to predict topological properties of data and open the door to exciting new research avenues, they must be interpreted with care. For example, we mostly tested message-passing networks in our experiments, leaving aside interesting proposals such as topological transformers (Ballester et al., ), high-order state-space models (Montagna et al., ), cellular or combinatorial complex networks (Bodnar et al., ; Hajij et al., ), topological Gaussian processes (Alain et al., ; Yang et al., ) or equivariant high-order neural networks (Battiloro et al., ). Due to computational limitations, training procedures were limited to $6$ epochs, model hyperparameters were not necessarily selected optimally, and barycentric subdivisions experiments were limited to one subdivision.
A significant computational bottleneck arose from the TopoModelX implementations of simplicial complex-based models, which processed data between $\sim 5$ and $\sim 24$ times slower, depending on the dataset and pairs of models, than their graph counterparts as observed in Table 7, highlighting the need for more efficient implementations of TDL methods.
Despite these limitations, we believe that each of the three stated questions should be investigated individually, with a broader set of experiments and ablations to be fully answered.

## 4 Conclusion

We proposed MANTRA, a higher-order dataset of manifold triangulations that is
(i) *diverse*, containing triangulations of surfaces and
three-dimensional manifolds with different
topological invariants and homeomorphism types,
(ii) *large*, with over $43{,}000$ triangulations of surfaces and $249{,}000$
triangulations of three-dimensional manifolds, and
(iii) *naturally higher-order*, as the triangulations are directly
related to the topological structure of the underlying manifold.
Using MANTRA, we observed that existing models, both graph-based and
higher-order-based, struggle to learn topological properties
of triangulations, such as the orientability of two-dimensional manifolds, which was the hardest topological property to predict for surface triangulations, suggesting that new approaches
are needed to leverage higher-order structure associated with the topological information in the dataset. However, we also saw that current high-order models outperform graph-based models in our benchmarks, substantiating the promises
of this new trend of higher-order machine-learning models.
Regarding invariance, we observed that barycentric subdivision deeply affects
the performance of the models, suggesting that current state-of-the-art models are not
invariant to transformations that preserve the topological structure of data, opening
an interesting research direction for future work. This is potentially related to the usage of
the message-passing paradigm, which is known to be sensitive to distances between
simplices in simplicial complexes. Another interesting research direction for barycentric subdivisions is their application as inputs to graph neural networks. The induced graph of a barycentric subdivision represents each simplex of the original complex as a vertex, with edges encoding face relationships on the original complex. This structure provides an effective representation of simplicial complexes
for graph-based neural architectures, potentially facilitating the processing of
higher-order topological information.
We hope that MANTRA will serve as a benchmark for the development of new models leveraging higher-order and topological structures in data, and as a reference for the development of new higher-order datasets.

## Reproducibility

We make the dataset and benchmark code available at

| [https://anonymous.4open.science/r/mantra-D60C/mantra/](https://anonymous.4open.science/r/mantra-D60C/mantra/) |
| --- |
| [https://anonymous.4open.science/r/mantra-benchmarks-2500/README.md](https://anonymous.4open.science/r/mantra-benchmarks-2500/README.md). |

These repositories contain
(i) the raw and processed datasets,
(ii) and the benchmark code to reproduce the results found in this paper.
The dataset and associated Python package are versioned using Semantic Versioning 2.0.0, ensuring reproducibility and clear tracking of dataset evolution. Additionally, the Python package allows the retrieval of any version of the dataset, decoupling data loading implementation and actual data. Detailed hyperparameter settings can be found in Appendix B of the paper. Step by step instructions on how to set up and execute the benchmark experiments are attached in the README file of the repository. Docker images and workflow, together with package dependencies are included to ensure a unique environment across different machine configurations. Finally, random seeds were used to split the datasets in each run.

## Acknowledgments

This paper is dedicated to the memory of *Frank H. Lutz* (1968–2023), who started the initial collection of all
triangulations as one of his many research endeavors.

RB, SE, and CC were supported by the Ministry of Science and Innovation of Spain through
projects PID2019-105093GB-I00, PID2020-117971GB-C22, and
PID2022-136436NB-I00. RB was additionally supported by the Ministry of Universities of Spain through the
FPU contract FPU21/00968, by the Departament de Recerca i Universitats de
la Generalitat de Catalunya (2021 SGR 00697), and by ICREA under the
ICREA Acadèmia programme. MA was supported by a Mathematical
Sciences Doctoral Training Partnership held by Prof.
Helen Wilson, funded by the Engineering and Physical
Sciences Research Council (EPSRC), under Project
Reference EP/W523835/1.

## Appendix A Appendix

### A.1 Simplicial complexes

A *simplicial complex* K
is a family of non-empty finite sets such that, if $\sigma\in\text{K}$ and $\tau\subseteq\sigma$, then $\tau\in\text{K}$. Each $\sigma\in\text{K}$
is called a *simplex* of K, and $\sigma$ is called a *$d$-dimensional face* or a *$d$-face* of K if its cardinality is $d+1$. The 0 0-faces of K are called *vertices* and the $1$-faces are called *edges*. We denote by $\text{K}^{d}$ the set of $d$-faces of K, and define the *dimension* of K as the largest $d$ for which $\text{K}^{d}$ is
non-empty. A simplicial complex of dimension $1$ is called a *graph*.

A *geometric realization* of a simplicial complex K is the union of a collection of affine simplices $\Delta_{\sigma}$ in a Euclidean space $\mathbb{R}^{n}$ for some $n\geq 1$, one for each simplex $\sigma\in\text{K}$, where $\sigma$ is mapped bijectively to the vertices of $\Delta_{\sigma}$, and two affine simplices $\Delta_{\sigma}$ and $\Delta_{\tau}$ share a face corresponding to $\sigma\cap\tau$ whenever this intersection is non-empty. Any two geometric realizations of a simplicial complex K are homeomorphic through a face-preserving map.

The *barycentric subdivision* of a simplicial complex
K is the simplicial complex $\text{Sd}(\text{K})$
obtained by setting its $d$-dimensional faces to be sequences of strict inclusions
$\sigma_{0}\subset\sigma_{1}\subset\cdots\subset\sigma_{d}$
of simplices of K. It then follows that K and $\text{Sd}(\text{K})$ have homeomorphic geometric realizations (Nanda, , Proposition 1.13).

### A.2 Simplicial homology and Betti numbers

Simplicial homology of a simplicial complex K equipped with an order on its set of vertices is defined as follows (Munkres, , § 34). Let $R$ be any commutative ring with unit (including the ring of integers $\mathds{Z}$ or any field). The *chain complex* of K with coefficients
in $R$ is
a sequence of $R$-modules
$(C_{n}(\text{K}))_{n\in\mathds{Z}}$
whose elements are formal sums of $n$-simplices of K
with coefficients in $R$, i.e.,

$$ $C_{n}(\text{K})=\left\{\textstyle\sum\nolimits_{\sigma\in\text{K}^{n}}a_{ \sigma}\sigma\mid a_{\sigma}\in R\right\},$ $$

linked by *boundary homomorphisms*
$\partial_{n}\colon C_{n}(\text{K})\to C_{n-1}(\text{K})$
for all $n\in\mathds{Z}$, given by

$$ $\partial_{n}\left(\textstyle\sum\nolimits_{\sigma\in\text{K}^{n}}a_{\sigma} \sigma\right)=\textstyle\sum\nolimits_{\sigma\in\text{K}^{n}}a_{\sigma} \partial_{n}(\sigma),\qquad\partial_{n}(\sigma)=\sum\nolimits_{i=0}^{n}\,(-1)^ {i}(\sigma\smallsetminus\{v_{i}\}),$ $$

if $v_{0},\dots,v_{n}$ are the ordered vertices of $\sigma$. The main property of the boundary homomorphisms is that
$\partial_{n}\circ\partial_{n+1}=0$ for all $n$, implying that
$\text{Im}(\partial_{n+1})\subseteq\text{Ker}(\partial_{n})$ for all $n$.
This yields *homology $R$-modules*, defined as quotients
$H_{n}(\text{K})=\text{Ker}(\partial_{n})/\text{Im}(\partial_{n+1})$ for all $n$.

If K is a finite simplicial complex and $R=\mathds{Z}$, then $H_{n}(\text{K})$ is a finitely generated abelian group and therefore it decomposes as a direct sum

$$ $H_{n}(\text{K})\cong\mathds{Z}^{\beta_{n}}\oplus\mathds{Z}_{q_{1}}\oplus\cdots \oplus\mathds{Z}_{q_{t}},$ $$

where $\beta_{n}$ is the *$n$-th Betti number* of K, while $q_{1},\ldots,q_{t}$ are prime powers.
The sum $\mathds{Z}_{q_{1}}\oplus\cdots\oplus\mathds{Z}_{q_{t}}$ is the *torsion* subgroup of $H_{n}(\text{K})$. Examples of Betti numbers are provided in
Figure 2.
The $n$-th Betti number of a simplicial complex K
counts the number of linearly independent $n$-dimensional cavities in a geometric realization of K. In low dimensions, $\beta_{0}$ is equal to the number of connected components, and $\beta_{1}$ counts the number of linearly independent loops that are not boundaries of any $2$-dimensional region.

Figure: Figure 2: From left to right, four simplicial complexes $\text{K}_{1}$, $\text{K}_{2}$, $\text{K}_{3}$, and $\text{K}_{4}$ with their respective Betti numbers $\beta_{0}$, $\beta_{1}$, and $\beta_{2}$. Here $\text{K}_{1}$ is a solid tetrahedron with $\beta_{0}=1$, $\beta_{1}=0$, and $\beta_{2}=0$, since $\text{K}_{1}$ has only one connected component, no unfilled cycles, and no empty cavity enclosed by $2$-faces; $\text{K}_{2}$ is a hollow tetrahedron with $\beta_{0}=1$, $\beta_{1}=0$, and $\beta_{2}=1$ (the difference with $\text{K}_{1}$ is that the triangles of $\text{K}_{2}$ enclose a cavity); $\text{K}_{3}$ is the underlying graph, with $\beta_{0}=1$, $\beta_{1}=3$, and $\beta_{2}=0$, since there is no cavity and there are three linearly independent cycles; $\text{K}_{4}$ consists of four vertices and has $\beta_{0}=4$, $\beta_{1}=0$, and $\beta_{2}=0$, since there are four connected components and no cycles nor cavities.

### A.3 Triangulated manifolds

An *$n$-dimensional manifold* is a second-countable Hausdorff topological space $M$ such that every point of $M$ is contained in some open set, called a *chart*, equipped with a homeomorphism into an open subset of a Euclidean space $\mathds{R}^{n}$ (Munkres, , § 36). This definition does not include manifolds with boundary, which are not considered in this article. A manifold is called *closed* if its underlying topological space is compact.

A collection of charts covering a manifold $M$ is an *atlas* of $M$. A manifold $M$ is called *orientable* if it admits an atlas with compatible orientations in its charts. For a closed $n$-dimensional manifold $M$, orientability is determined by its $n$-th Betti number $\beta_{n}$, which is nonzero if and only if $M$ is orientable.

A *triangulation* of a manifold $M$ is a simplicial complex whose geometric realization is homeomorphic to $M$.
Radó () proved that every surface admits a triangulation (which can be chosen to be finite if the surface is compact), and that any two such triangulations admit a common refinement. Moise () proved that the same facts are true for $3$-dimensional manifolds. For dimensions greater than $3$, however, there are examples of manifolds that cannot be triangulated.

### A.4 Classification

Closed connected surfaces can be classified, up to homeomorphism, as given by the following list:
(i) the two-dimensional sphere $S^{2}$;
(ii) a connected sum of tori $T^{2}$;
(iii) a connected sum of projective planes $\mathds{R}P^{2}$.
The *genus* of a surface $M$ is defined as zero if $M\cong S^{2}$ and equal to $g$ if $M$ is a connected sum of $g$ tori or $g$ projective planes.
Thus the homeomorphism type of $M$ is determined by its orientability and genus.

The *Euler characteristic* of a finite triangulation of a manifold $M$ is the alternating sum of the numbers of simplices of each dimension. It does not depend on the choice of a triangulation, and it is equal to the alternating sum of the Betti numbers of $M$ (Hatcher, ). The Euler characteristic of a closed connected surface $M$ of genus $g$ is equal to $2-2g$ if $M$ is orientable and $2-g$ if $M$ is not orientable.

The underlying graph of a finite triangulation of a closed surface $M$ determines the Euler characteristic $v-e+t$. This is due to the fact that, in any triangulation of $M$, each edge bounds precisely two triangles, so $3t=2e$. Therefore, the underlying graph of a triangulation of a closed surface $M$ determines the homeomorphism type of $M$ up to orientability. As shown in Lawrencenko & Negami (), the torus and the Klein bottle admit triangulations with the same underlying graph.

For manifolds of dimension greater than $2$, classification up to homeomorphism is so far unfeasible.
In dimension $3$, the geometrization theorem (Morgan & Tian, ) describes all possible geometries of prime components of closed $3$-manifolds. The Euler characteristic does not carry any information about the homeomorphism type in dimension $3$, since if $M$ is any odd-dimensional closed manifold then $\chi(M)=0$ by Poincaré duality (Hatcher, , 3.37). However, the underlying graph of a finite triangulation of a closed $3$-manifold determines the number $t$ of triangles and the number $f$ of $3$-faces, since $4f=2t$ and $v-e+t-f=0$.

### A.5 Distribution of labels

Tables 3,
4,  5,
and 6 contain statistical information about the distribution of labels in the dataset.

**Table 3: Distribution of Betti numbers $\beta_{i}$ for triangulations of manifolds. Percentages are rounded to the nearest integer, and are computed for each pair of manifold dimension ($2$ or $3$) and Betti number. Manif. dim. stands for manifold dimension.**
|  | Manif. dim. | 0 | 1 | 2 | 3 | 4 | 5 | 6 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\beta_{0}$ | 2 | - | 43,138 | - | - | - | - | - |
|  |  | (100%) |  |  |  |  |  |  |
|  | 3 | - | 250,359 | - | - | - | - | - |
|  |  |  | (100%) |  |  |  |  |  |
| $\beta_{1}$ | 2 | 1,670 | 4,655 | 14,146 | 13,694 | 7,917 | 1,022 | 34 |
|  | (4%) | (11%) | (33%) | (32%) | (18%) | (2%) | (0%) |  |
|  | 3 | 249,225 | 1,134 | 0 | 0 | 0 | 0 | 0 |
|  |  | (100%) | (0%) | (0%) | (0%) | (0%) | (0%) | (0%) |
| $\beta_{2}$ | 2 | 39,718 | 3,420 | - | - | - | - | - |
|  | (92%) | (8%) |  |  |  |  |  |  |
|  | 3 | 249,841 | 518 | - | - | - | - | - |
|  |  | (100%) | (0%) |  |  |  |  |  |
| $\beta_{3}$ | 2 | - | - | - | - | - | - | - |
| 3 | 616 | 249,743 | - | - | - | - | - |  |
|  |  | (0%) | (100%) |  |  |  |  |  |

**Table 4: Distribution of torsion subgroups for triangulations of manifolds. Percentages are rounded to the nearest integer, and are computed for each pair of manifold dimension and homological degree. Manif. dim. stands for manifold dimension.**
|  | $H_{0}$ | $H_{1}$ | $H_{2}$ | $H_{3}$ |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Manif. dim. | 0 0 | $\mathbb{Z}_{2}$ | 0 0 | $\mathbb{Z}_{2}$ | 0 0 | 0 0 |
| 2 | 43,138 | 39,718 | 3,420 | 0 | 43,138 | - |
|  | (100%) | (92%) | (8%) | (0%) | (100%) |  |
| 3 | 250,359 | 0 | 250,359 | 616 | 249,743 | 250,359 |
|  | (100%) | (0%) | (100%) | (0%) | (100%) | (100%) |

**Table 5: Distribution of genus for triangulations of surfaces. Percentages are rounded to the nearest integer. Manif. dim. stands for manifold dimension.**
| Manif. dim. | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 2 | 306 | 3,593 | 5,520 | 11,937 | 13,694 | 7,052 | 1,022 | 14 |
|  | (1%) | (8%) | (13%) | (28%) | (32%) | (16%) | (2%) | (0%) |

**Table 6: Distribution of homeomorphism types for triangulations of manifolds. Percentages are rounded to the nearest integer, and are computed for each manifold dimension. Manif. dim. stands for manifold dimension. Surfaces classified as “Other” do not have explicitly homeomorphism type assigned.**
| Manif. dim. | $S^{2}$ | $\mathbb{R}P^{2}$ | $T^{2}$ | $K$ | $S^{3}$ | $S^{2}\times S^{1}$ | $S^{2}\tilde{\times}S^{1}$ | Other |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 2 | 612 | 2,728 | 4,458 | 9,310 | - | - | - | 69,168 |
|  | (1%) | (3%) | (5%) | (11%) |  |  |  | (80%) |
| 3 | - | - | - | - | 249,225 | 518 | 616 | 0 |
|  |  |  |  |  | (100%) | (0%) | (0%) | (0%) |

## Appendix B Hyperparameter details

- •
GAT
–
Hidden channels: 64
–
Number of heads: 4
–
Hidden layers: 4
–
Readout: Mean
–
Dropout last linear layer: 0.5
–
Activation last layer: Identity
- •
GCN
–
Hidden channels: 64
–
Hidden layers: 4
–
Readout: Mean
–
Dropout last linear layer: 0.5
–
Activation last layer: Identity
- •
MLP
–
Hidden neurons: 64
–
Hidden layers: 4
–
Readout: Mean
–
Dropout last linear layer: 0.0
–
Activation last layer: Identity
- •
TAG
–
Hidden channels: 64
–
Hidden layers: 4
–
Readout: Mean
–
Dropout last linear layer: 0.5
–
Activation last layer: Identity
- •
TransfConv
–
Hidden channels: 64
–
Hidden layers: 4
–
Readout: Mean
–
Dropout last linear layer: 0.5
–
Activation last layer: Identity

- •
SAN
–
Hidden channels: 64
–
Hidden layers: 1
–
$n$-filters : 2
–
Order harmonic : 5
–
Epsilon harmonic : 1e-1
–
Readout: Sum of sums per dimension
- •
SCCN
–
Hidden channels: 64
–
Hidden layers: 2
–
Maximum rank : 2
–
Aggregation activation function : sigmoid
–
Readout: Sum of sums per dimension
- •
SCCNN
–
Hidden channels: 64
–
Hidden layers: 2
–
Order of convolutions: 1
–
Order of simplicial complexes: 1
–
Readout: Sum of sums per dimension
- •
SCN
–
Hidden channels per dimension: Same as input
–
Hidden layers: 2
–
Readout: Sum of sums per dimension

More information on the meaning of specific hyperparameters can be found in the PyTorch geometric and TopoModelX implementations.

## Appendix C Additional experimental details

**Table 7: Mean and standard deviation of training iterations processed per second ($\uparrow$), as measured by PyTorch Lightning (Falcon & The PyTorch Lightning team, ), across all experiments for each model and dataset.**
|  | Model |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Dataset | MLP | GAT | TRANSCONV | TAG | GCN |
| $2-\mathcal{M}^{0}$ | $9.72$ p m 4.54 | $9.41$ p m 4.02 | $9.31$ p m 3.79 | $9.41$ p m 3.53 | $9.45$ p m 3.79 |
| $2-\mathcal{M}^{0}_{H}$ | $12.39$ p m 13.22 | $11.33$ p m 10.6 | $11.71$ p m 10.72 | $11.46$ p m 10.66 | $12.1$ p m 11.43 |
| $3-\mathcal{M}^{0}$ | $8.59$ p m 5.4 | $8.12$ p m 3.83 | $7.57$ p m 3.34 | $7.63$ p m 3.86 | $8.18$ p m 4.27 |
|  | SAN | SCN | SCCN | SCCNN |  |
| $2-\mathcal{M}^{0}$ | $0.65$ p m 0.97 | $0.83$ p m 2.63 | $0.85$ p m 3.06 | $0.73$ p m 1.93 |  |
| $2-\mathcal{M}^{0}_{H}$ | $1.21$ p m 2.93 | $1.72$ p m 6.99 | $1.89$ p m 7.9 | $1.67$ p m 5.79 |  |
| $3-\mathcal{M}^{0}$ | $0.35$ p m 0.24 | $0.39$ p m 0.47 | $0.38$ p m 0.4 | $0.39$ p m 0.76 |  |

Table 7 reports the mean and standard deviation of training iterations processed per second for each model and dataset. Appendix C compares the predictive performance of models across different feature vector initialization methods for the three surface tasks, Betti numbers, homeomorphism type, and orientability prediction, on the full set of surfaces. Tables 15, 17, 16, 21, 22, 19, 18, 20, 12, 13, 14, 9, 10 and 11 report the full set of experimental results.

#### Feature vector initialization analysis

We observe different behaviours for the two families of models, graph-based and simplicial complex-based. For the graph models, random initialization works slightly better or equal than the degree features. On the other hand, for the simplicial complex models, degree initializations consistently outperform their random counterparts on average. Degrees for vertices coincide for both families of models, suggesting that degrees high-order degrees contains more useful information than their dimension zero counterpart to predict topological properties, supporting the need for models that leverage high-order information of the input. Having signal contained in features can make sense if the task in question requires additional information. For example, molecules are more than just combinatorial or topological objects: the types of atoms and the nature of bonds are important for predicting their properties. However, in purely topological tasks, such as predicting topological invariants, the need to enforce topological information into features raises the question: do MP-based models correctly capture topological properties in the first place? Still, standard deviations in the aggregated data for simplicial complex-based models is large, and better ablation is needed to fully understand the differences in initializations and the expressivity of high-order degrees in the context of topological prediction tasks.

random_versus_degree./results/random_versus_degree.csv

**Table 8: Predictive performance of graph- and simplicial complex-based models on the tasks for the full set of surfaces. Results aggregated by the feature vector initialization type and family of models. For each initialization type, random and degree, and family of models, $\mathcal{G}$ (graph-based) and $\mathcal{T}$ (simplicial complex-based), we report the mean and standard deviation of the maximum performance achieved across the $5$ runs by each combination of model contained in the family initialized with the corresponding initialization type. The tasks reported are the prediction of $\beta_{0}$, $\beta_{1}$, $\beta_{2}$, the prediction of the homeomorphism type, and the prediction of orientability. For all tasks except for the prediction of $\beta_{0}$, we report the AUROC metric. For $\beta_{0}$, we report accuracy. Homeo. type and acc. stand for homeomorphism type and accuracy, respectively. Metrics are multiplied by $100$ and rounded to the second decimal for a better visualization. Best average result between random and degree initialization is in bold for each family and task.**
| Target (Metric) | Transform | Family | Performances |  |  |
| --- | --- | --- | --- | --- | --- |
| \DTLforeachrandom_versus_degree\Dataset=Dataset,<br>\Model=Family,<br>\Transform=Transform,<br>\PredictionTarget=Prediction Target (Metric),<br>\Value=Value,<br>\ValueOne=ValueOne,<br>\ValueTwo=ValueTwo\DTLifnullorempty\PredictionTarget\PredictionTarget | \Transform | \Model | \Value | \ValueOne | \ValueTwo\DTLifoddrow<br>\DTLiflastrow |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. I'm not sure if you mention this in the paper but how are the initial labels and triangulations generated for each surface and three-dimensional manifold?

2. For a given surface, how much computational time is required for computing the fields mentioned Section 2?

3. Would MANTRA be useful for developing models for topological reconstruction (to recover incomplete surfaces or perform noise correction)? If so, how?
soundness: 3
strengths: 1. Extensive evaluation of currently available graph and simplicial complex-based models on MANTRA.

2. Provides a foundation for developing and benchmarking advanced TDL methods.

3. Insight into the ability of higher-order models to be invariant to triangulation transformations.
summary: This paper introduces MANTRA, a large-scale, diverse, and high-order dataset of triangulations for benchmarking models in the topological deep learning space. The authors also assess graph and simplicial complex-based models on tasks like betti number estimation, homeomorphism type, and orientability detection. The introduction section is written well with enough definitions/context about various topological properties. The dataset specification includes all properties that are being computed for each object. Furthermore, nine different neural nets are evaluated on the classification task.

Overall, this paper could be a good contribution to topological deep learning, especially for developing advanced TDL methods and benchmarking them. However, the limiting factor is the evaluations (just MP networks) as they do not give a full picture of the impact of the dataset.
weaknesses: 1. Evaluations are mostly focused on MP networks. It would help get better overall picture of the impact the dataset has by evaluating on other architectures like equivariant high-order neural nets and topological transformers.

2. The variance across results is surprising and I am not sure if running for more epochs/more hyperparameter tuning will address this.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - **Dataset Generation**
  - How were the manifolds and triangulations generated?
  - What considerations did you make, if any, in order to encourage topological diversity in the dataset?

- **Experimental Results**
  - For the standard deviation results in Table 2, do they represent variation within a single model or across the entire class of models?

- **Computational Issues**
  - The authors cite computational constraints for limiting some aspect of the experiments, can the authors elaborate on the nature of these limitations given that all triangulations have 10 or fewer vertices?
soundness: 3
strengths: - MANTRA is a large dataset specifically designed for topological property prediction, filling an important a gap in the TDL literature. Existing datasets are limited in scope, size, topological diversity and often rely on artificial topological lifting of graph data.
- The paper presents systematic analysis of graph methods vs. topological methods on fundamental topological property prediction tasks, establishing important baselines for future research in TDL.
- TDL models' ability to capture topological properties (**Q2** in the paper) is a fundamental question in TDL that remained under-explored to this point. Empirical results presented in the paper suggest that TDL models are better, compared to graph based methods, at learning topological properties from data.
- The question of invariance to topology preserving transformations (**Q3**) is also under-explored (albeit less clearly motivated). This paper demonstrate TDL models' sensitivity to remeshing by barycentric subdivisions -- a simple topology preserving transformation.
- The paper is clearly organized, with well defined questions, answered in a methodical manner. 
- The authors ensure reproducibility by providing well-documented code repos, versioned datasets, and multiple data format options (raw and PyTorch Geometric), making it easy to build upon this work.
summary: The paper introduces MANTRA, a new dataset designed for benchmarking topological deep learning (TDL) models. MANTRA consists of triangulations of two- and three-dimensional manifolds, labeled with various topological invariants such as Betti numbers, Homology torsion, homomorphism type (four options for 2D manifold and five options for 3D manifolds), and more. Using this dataset, the authors benchmark standard graph methods and TDL models on three types of tasks: Betti number prediction (2D + 3D), homeomorphism type prediction (2D), and orientability prediction (2D). Their results suggest that while TDL models are better at predicting higher order topological properties, they struggle with invariant to topology preserving transformations. The latter is demonstrated through additional experiments where barycentric subdivision -- a form of remeshing that preserves topological properties -- is performed on the test set.
weaknesses: - **Topological diversity**
  - Other than the genus, first Betti number, and homeomorphism type for 2D manifolds, the distribution of topological invariants is limited and unbalanced.
    - For 3D manifolds:
       - Most Betti numbers, homeomorphism types, and torsion subgroups have only one or two possible values.
       - When two values do exist, one value represents over 99% of the cases.
  - All triangulations are limited to 10 or fewer vertices.

- **Limited model coverage**
  - The only TDL baselines included in the evaluation are simplicial complex models, excluding:
      - Cellular complex methods (e.g. [1, 2]).
      - Combinatorial complex methods (e.g. [3]).
      - Hypergraph methods (e.g. [4, 5]).
  - For both TDL and graph based methods, the evaluation only includes message-passing based architectures, excluding non-message passing graph neural networks (e.g. [6, 7, 8, 9]) and attention based TDL models (e.g. [3, 10]).
  - It would be informative to report the best performing model for each class in the main text.

- **Training methodology**
  - Models were trained for only 6 epochs, hyperparameters were not optimized. The can possibly lead to underestimation of the fully optimized performance of some of the architectures, limiting the scope of the conclusions that can be drawn from the experiments

---

[1] Bodnar et al. "Weisfeiler and Lehman Go Cellular: CW Networks", NeurIPS 2021.

[2] Hajij et al. "Cell complex neural networks", Topological Data Analysis and Beyond Workshop NeurIPS 2020.

[3] Hajij et al. "Topological Deep Learning: Going Beyond Graph Data", arXiv:2206.00606

[4] Feng et al. "Hypergraph neural networks", AAAI 2019.

[5] Kim et al. "Equivariant hypergraph neural networks." ECCV 2022.

[6] Maron et al. "Provably powerful graph networks." NeurIPS 2019.

[7] Abboud et al. "The surprising power of graph neural networks with random node initialization", arXiv:2010.01179.

[8] Bouritsas et al. "Improving graph neural network expressivity via subgraph isomorphism counting", IEEE PAMI 2022.

[9] Zhang et al. "A complete expressiveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests", ICML 2023.

[10] Battiloro et al. "Generalized simplicial attention neural networks", arXiv:2309.02138.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. about claims in the paper: 1) (058-060) To the best of our knowledge, the only publicly-available high-order dataset is the “Torus” dataset proposed in Eitan et al.(2024), which consists of a small number of unions of tori triangulations. 2) (069-071) The authors claimed this is the __first__ instance of a large, diverse, and intrinsically high-order dataset, comprising triangulations of combinatorial 2- and 3-manifolds. 

  - Authors refer to datasets that contain nontrivial higher-order structures as higher-order datasets. It's worthwhile to point out that such structure and the datasets have been long- and well-studied by network science community, as well as numerical modeling, and there has been numerous efforts in building and modeling higher-order topological structures in various networks. However, there are way more such datasets as far as I'm aware of. 
    - In computation biology and social networks, such datasets with higher-order structures appear very often. 
    - in numerical modeling, computational geometry, 2D and 3D manifold triangulations are almost everywhere and they easily grow to the order of millions. 
    - In topological deep learning community, such datasets have been applied by different papers as well, e.g., 1) SaNN: Simple Yet Powerful Simplicial-aware Neural Networks; 2) Unsupervised Parameter-free Simplicial Representation Learning with Scattering Transforms. 

  - As a reviewer, I really shouldn't provide a more complete list of such literatures. I believe this paper could benefit from a more thorough literature review. Note that _there is no such dataset_ does not work as a strong motivation. Moreover, this work is hardly the __first__  dataset involved with high-order topological structures.

2. about the experiment tasks: as a researcher in TDA, I would like to see how graph- and simplicial complex-based models compare to the TDA methods for the investigated three tasks in this paper. Again, this is necessarily needed for being a datasets and benchmarks paper.

3. I do not agree with authors *define* the lower/upper degree of a simplex as the number of simplices which share a face/coface with it. This number is merely the number of neighboring simplices, but not the degree for simplices. The _degree_ of a simplex is the number of faces/cofaces it has. Please refer to _Spectra of combinatorial Laplace operators on simplicial  complexes_ and _RANDOM WALKS ON SIMPLICIAL COMPLEXES AND THE NORMALIZED HODGE 1-LAPLACIAN_ for such definitions. 

4. Confusing experimental setup: 
   1. Randomly generated scalars are used as feature vectors for graph-based models and randomly generated eight-dim vectors are used as feature vectors for simplicial complex-based models? 
      - Questions: What is the motivation of using randomly generated values as input features? Any useful information do they provide to the models or tasks? 

   2. In predicting 0-betti numbers, you only have label 1. When seeing results on predicting $\beta_0$, I have the following questions:
      -  If graph-based models have 100% accuracy, how come simplicial-complex based models can be poor not nearly 50%? My question comes from the fact that: simplicial complex based models are __generalizations__ of graph-based models. If GAT/GCN has an accuracy of 100%, then SAN/SCCN, which are really extensions to higher-order simplices, should be able to achieve the same accuracy _when using the same output_ (Note that here I assume authors considered using the same type of output for predictions, e.g., when using node representations of GCN for predicting $\beta_0$, one should also use node representation of SCCN for predicting $\beta_0$). 
      -  On the other hand, most of the models implemented in this paper are performing representation learning, there is a need for a classifier or a regressor to predict the labels. Here it matters how a classifier/regressor, which takes the learned representation and computes the needed output, is defined. Yet, I couldn't find any details on this, but only __readout__ layers of different models, which are either mean, or sum of all embeddings. I wonder if authors simply used the readout to predict topological features. It doesn't make much sense to apply a direct readout to predict different topological features. 
   3. what is the dataset $2-\mathcal{M}_H^1$? I tried to check the appendix but couldn't find it. If I guessed correctly, it is a refined version of $2-\mathcal{M}_H^0$. But how is this refinement performed? 
   4. I appreciate the authors'effort in investigating the topological invariance of the models. This however is not well-analyzed in the paper, yet, the authors made the claim that high-order MP-based models are not invarinat relative to topological transformations ... in line 284. 
      1. First of all, how is topological transformation defined? Is it only refinement of the triangulation? 
      2. A proper topological predictor should be considered, yet this is unclear from the paper. 
      3. I do not really see message-passing models are implemented. By that, I mean Bodnar et al., 2021a. 
  

other minor suggestions:
- line 150: It seems incorrect to say something was degraded. I'm not an English major, but I think it should be "someone was degraded" or "something degraded". Please ignore it if I'm wrong. 
- Please figure out a way to present Tables 9-22 without rotating them 90 degrees. Though I tried reading them, they are still not easy for readers.
soundness: 2
strengths: The research question is interesting. It is novel to investigate the performance of graph and high-order network models on predicting topological features.
summary: The paper aims to benchmark the performance of graph-based and simplicial complex-based models on a newly-introduced 2-/3- manifold triangulation dataset. The paper focused on topological feature prediction tasks, including predicting Betti numbers, homeomorphism type and orientability.
Overall, the research question is interesting but the paper requires more work to be considered for publication. Please see my detailed comments and questions below.
weaknesses: - There are various works on estimating Betti numbers, but the authors did not mention them. They should provide as solid baselines. For example:
  - Estimating Betti Numbers Using Deep Learning
  - A (simple) classical algorithm for estimating Betti numbers
- The experiments are rather unclear, for example, details on how the architectures are not provided. As a paper in datasets and benchmarks, this is necessary and as a reviewer, I'm interested in knowing how the architectures are designed and if they are reasonable. 
  - Some experimental setup choices are not well-justified. For example, why randomly generated scalars are used as input features for graph-based models and randomly generated eight-dim vectors are used as feature vectors for simplicial complex-based models?
- Authors made a few claims which are not well-supported by either the experiments or literature. 

Note: Please see my detailed questions below related to these points.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - What is your view on the scope of the dataset in terms of the vertices and dimensionality? I would suggest adding a short discussion on this in the limitations. 
- I am not sure if I agree with the statement in line 204: "To ensure fairness, all configurations use the same learning rate of 0.01". Different models may work best with different learning rates, which likely requires cumbersome hyperparameter tuning. The authors acknowledge this as a limitation, but I would reconsider the phrasing of this sentence -- is this really a "fair" comparison?
soundness: 4
strengths: - The paper is very well presented in terms of structure, rigor, and backing statements with relevant references or evidence.
- As far as I can tell, good conventions are followed to make the dataset easily available and usable.
- The experimental protocol is overall solid and the drawn conclusions are relevant and adequate.
- Limitations are discussed.
summary: This work proposes a synthetic benchmark dataset for topological deep learning. Contrary to graphs representing pairwise interactions, this dataset focuses on higher-order interactions. It is generated by considering the simplicial complexes of 2 and 3-dimensional manifolds, resulting in structures with up to 10 vertices. Samples are labeled with topological descriptors, such as Betti numbers, torsion, orientability, genus, or homeomorphisms.

After introducing the dataset, several experiments are carried out to compare the performance of five graph and four topological learning models on topological prediction tasks. The main findings are that topological models perform better than graph models on these tasks, however, topological models are far from having a strong performance on these tasks.
weaknesses: I have two minor remarks:
- The scope of the dataset is limited in dimension (order of interactions) and number of vertices, as well as being purely synthetic. While these are not bad aspects in their own right and this certainly is justified given that TDL is in its early days as it allows to study basic properties of models (e.g., remeshing invariance), the benchmark could potentially quickly lose its relevance.
- As the authors acknowledge, several non-message passing TDL architectures were not included in the experiments. As such, one of the most intriguing interpretations of the results is somewhat diminished ("suggesting that new approaches are needed to leverage higher-order structure associated with the topological information in the dataset").

Meta-review: The paper proposes MANTRA, mainly a synthetic new benchmark focusing on evaluating methods in topological deep learning. Most reviewers were on the positive side, although there is quite a big disagreement with a 'accept' and a 'reject' suggestion. Basically, the point of disagreement is to what extent the paper presents a truly new data setting and benchmark, given that similar and harder problems have been extensively studied and researched in graphics, specifically CGAL (computational geometry algorithms library). The counterargument is that these libraries are indeed addressing a similar problem, although it is definitely non-trivial for researchers who come from ML and are not domain experts to work with these libraries. This is certainly a logical point. 

Another point of concern is whether having a dataset like MANTRA is sufficient, given that it is a synthetic dataset, and given that there is a plethora of real-world problems that could be considered as well.

I believe both sides have merit and I am definitely sensitive to the argument that it is important that ML methods stay close to real-world and practical applications. To decide on this, I note that since this is a dataset+benchmark paper, ignoring the 'magnitude' of the recommendations and only focusing on the 'sign' (positive vs negative review), most reviewers agree it is a useful dataset. Combined with the fact that as it was commented by the reviewers, the papers addresses a big gap in TDL, I recommend acceptance.

Decision: Accept (Poster)