MANTRA: The Manifold Triangulations Assemblage

Authors: Rubén Ballester, Ernst Röell, Daniel Bin Schmid, Mathieu Alain, Sergio Escalera, Carles Casacuberta, Bastian Rieck
Primary area: datasets and benchmarks
Keywords: simplicial complex, topological deep learning, high-order, high-order dataset, simplicial complex learning

Abstract:
The rising interest in leveraging higher-order interactions present in complex systems has
led to a surge in more expressive models exploiting higher-order structures in the data,
especially in topological deep learning (TDL), which designs neural networks on higher-order domains such as simplicial complexes. However, progress in this field is hindered
by the scarcity of datasets for benchmarking these architectures. To address this gap, we
introduce MANTRA, the first large-scale, diverse, and intrinsically higher-order dataset for
benchmarking higher-order models, comprising over 43,000 and 250,000 triangulations
of surfaces and three-dimensional manifolds, respectively. With MANTRA, we assess
several graph- and simplicial complex-based models on three topological classification
tasks. We demonstrate that while simplicial complex-based neural networks generally
outperform their graph-based counterparts in capturing simple topological invariants, they
also struggle, suggesting a rethink of TDL. Thus, MANTRA serves as a benchmark for
assessing and advancing topological methods, paving the way towards more effective
higher-order models.

Decision: accept (poster)
Reviewer scores: 6, 6, 3, 8

Reviews and discussion:
<thread>
creation date:
1730131160905

thread_type:
official_review

<thread_object>
creator: reviewer_chkw

<content>
summary:
This work proposes a synthetic benchmark dataset for topological deep learning. Contrary to graphs representing pairwise interactions, this dataset focuses on higher-order interactions. It is generated by considering the simplicial complexes of 2 and 3-dimensional manifolds, resulting in structures with up to 10 vertices. Samples are labeled with topological descriptors, such as Betti numbers, torsion, orientability, genus, or homeomorphisms.

After introducing the dataset, several experiments are carried out to compare the performance of five graph and four topological learning models on topological prediction tasks. The main findings are that topological models perform better than graph models on these tasks, however, topological models are far from having a strong performance on these tasks.

strengths:
- The paper is very well presented in terms of structure, rigor, and backing statements with relevant references or evidence.
- As far as I can tell, good conventions are followed to make the dataset easily available and usable.
- The experimental protocol is overall solid and the drawn conclusions are relevant and adequate.
- Limitations are discussed.

weaknesses:
I have two minor remarks:
- The scope of the dataset is limited in dimension (order of interactions) and number of vertices, as well as being purely synthetic. While these are not bad aspects in their own right and this certainly is justified given that TDL is in its early days as it allows to study basic properties of models (e.g., remeshing invariance), the benchmark could potentially quickly lose its relevance.
- As the authors acknowledge, several non-message passing TDL architectures were not included in the experiments. As such, one of the most intriguing interpretations of the results is somewhat diminished ("suggesting that new approaches are needed to leverage higher-order structure associated with the topological information in the dataset").

questions:
- What is your view on the scope of the dataset in terms of the vertices and dimensionality? I would suggest adding a short discussion on this in the limitations. 
- I am not sure if I agree with the statement in line 204: "To ensure fairness, all configurations use the same learning rate of 0.01". Different models may work best with different learning rates, which likely requires cumbersome hyperparameter tuning. The authors acknowledge this as a limitation, but I would reconsider the phrasing of this sentence -- is this really a "fair" comparison?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730310223571

thread_type:
official_review

<thread_object>
creator: reviewer_1p1z

<content>
summary:
The paper aims to benchmark the performance of graph-based and simplicial complex-based models on a newly-introduced 2-/3- manifold triangulation dataset. The paper focused on topological feature prediction tasks, including predicting Betti numbers, homeomorphism type and orientability.
Overall, the research question is interesting but the paper requires more work to be considered for publication. Please see my detailed comments and questions below.

strengths:
The research question is interesting. It is novel to investigate the performance of graph and high-order network models on predicting topological features.

weaknesses:
- There are various works on estimating Betti numbers, but the authors did not mention them. They should provide as solid baselines. For example:
  - Estimating Betti Numbers Using Deep Learning
  - A (simple) classical algorithm for estimating Betti numbers
- The experiments are rather unclear, for example, details on how the architectures are not provided. As a paper in datasets and benchmarks, this is necessary and as a reviewer, I'm interested in knowing how the architectures are designed and if they are reasonable. 
  - Some experimental setup choices are not well-justified. For example, why randomly generated scalars are used as input features for graph-based models and randomly generated eight-dim vectors are used as feature vectors for simplicial complex-based models?
- Authors made a few claims which are not well-supported by either the experiments or literature. 

Note: Please see my detailed questions below related to these points.

questions:
1. about claims in the paper: 1) (058-060) To the best of our knowledge, the only publicly-available high-order dataset is the “Torus” dataset proposed in Eitan et al.(2024), which consists of a small number of unions of tori triangulations. 2) (069-071) The authors claimed this is the __first__ instance of a large, diverse, and intrinsically high-order dataset, comprising triangulations of combinatorial 2- and 3-manifolds. 

  - Authors refer to datasets that contain nontrivial higher-order structures as higher-order datasets. It's worthwhile to point out that such structure and the datasets have been long- and well-studied by network science community, as well as numerical modeling, and there has been numerous efforts in building and modeling higher-order topological structures in various networks. However, there are way more such datasets as far as I'm aware of. 
    - In computation biology and social networks, such datasets with higher-order structures appear very often. 
    - in numerical modeling, computational geometry, 2D and 3D manifold triangulations are almost everywhere and they easily grow to the order of millions. 
    - In topological deep learning community, such datasets have been applied by different papers as well, e.g., 1) SaNN: Simple Yet Powerful Simplicial-aware Neural Networks; 2) Unsupervised Parameter-free Simplicial Representation Learning with Scattering Transforms. 

  - As a reviewer, I really shouldn't provide a more complete list of such literatures. I believe this paper could benefit from a more thorough literature review. Note that _there is no such dataset_ does not work as a strong motivation. Moreover, this work is hardly the __first__  dataset involved with high-order topological structures.

2. about the experiment tasks: as a researcher in TDA, I would like to see how graph- and simplicial complex-based models compare to the TDA methods for the investigated three tasks in this paper. Again, this is necessarily needed for being a datasets and benchmarks paper.

3. I do not agree with authors *define* the lower/upper degree of a simplex as the number of simplices which share a face/coface with it. This number is merely the number of neighboring simplices, but not the degree for simplices. The _degree_ of a simplex is the number of faces/cofaces it has. Please refer to _Spectra of combinatorial Laplace operators on simplicial  complexes_ and _RANDOM WALKS ON SIMPLICIAL COMPLEXES AND THE NORMALIZED HODGE 1-LAPLACIAN_ for such definitions. 

4. Confusing experimental setup: 
   1. Randomly generated scalars are used as feature vectors for graph-based models and randomly generated eight-dim vectors are used as feature vectors for simplicial complex-based models? 
      - Questions: What is the motivation of using randomly generated values as input features? Any useful information do they provide to the models or tasks? 

   2. In predicting 0-betti numbers, you only have label 1. When seeing results on predicting $\beta_0$, I have the following questions:
      -  If graph-based models have 100% accuracy, how come simplicial-complex based models can be poor not nearly 50%? My question comes from the fact that: simplicial complex based models are __generalizations__ of graph-based models. If GAT/GCN has an accuracy of 100%, then SAN/SCCN, which are really extensions to higher-order simplices, should be able to achieve the same accuracy _when using the same output_ (Note that here I assume authors considered using the same type of output for predictions, e.g., when using node representations of GCN for predicting $\beta_0$, one should also use node representation of SCCN for predicting $\beta_0$). 
      -  On the other hand, most of the models implemented in this paper are performing representation learning, there is a need for a classifier or a regressor to predict the labels. Here it matters how a classifier/regressor, which takes the learned representation and computes the needed output, is defined. Yet, I couldn't find any details on this, but only __readout__ layers of different models, which are either mean, or sum of all embeddings. I wonder if authors simply used the readout to predict topological features. It doesn't make much sense to apply a direct readout to predict different topological features. 
   3. what is the dataset $2-\mathcal{M}_H^1$? I tried to check the appendix but couldn't find it. If I guessed correctly, it is a refined version of $2-\mathcal{M}_H^0$. But how is this refinement performed? 
   4. I appreciate the authors'effort in investigating the topological invariance of the models. This however is not well-analyzed in the paper, yet, the authors made the claim that high-order MP-based models are not invarinat relative to topological transformations ... in line 284. 
      1. First of all, how is topological transformation defined? Is it only refinement of the triangulation? 
      2. A proper topological predictor should be considered, yet this is unclear from the paper. 
      3. I do not really see message-passing models are implemented. By that, I mean Bodnar et al., 2021a. 
  

other minor suggestions:
- line 150: It seems incorrect to say something was degraded. I'm not an English major, but I think it should be "someone was degraded" or "something degraded". Please ignore it if I'm wrong. 
- Please figure out a way to present Tables 9-22 without rotating them 90 degrees. Though I tried reading them, they are still not easy for readers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730472469336

thread_type:
official_review

<thread_object>
creator: reviewer_fkvz

<content>
summary:
The paper introduces MANTRA, a new dataset designed for benchmarking topological deep learning (TDL) models. MANTRA consists of triangulations of two- and three-dimensional manifolds, labeled with various topological invariants such as Betti numbers, Homology torsion, homomorphism type (four options for 2D manifold and five options for 3D manifolds), and more. Using this dataset, the authors benchmark standard graph methods and TDL models on three types of tasks: Betti number prediction (2D + 3D), homeomorphism type prediction (2D), and orientability prediction (2D). Their results suggest that while TDL models are better at predicting higher order topological properties, they struggle with invariant to topology preserving transformations. The latter is demonstrated through additional experiments where barycentric subdivision -- a form of remeshing that preserves topological properties -- is performed on the test set.

strengths:
- MANTRA is a large dataset specifically designed for topological property prediction, filling an important a gap in the TDL literature. Existing datasets are limited in scope, size, topological diversity and often rely on artificial topological lifting of graph data.
- The paper presents systematic analysis of graph methods vs. topological methods on fundamental topological property prediction tasks, establishing important baselines for future research in TDL.
- TDL models' ability to capture topological properties (**Q2** in the paper) is a fundamental question in TDL that remained under-explored to this point. Empirical results presented in the paper suggest that TDL models are better, compared to graph based methods, at learning topological properties from data.
- The question of invariance to topology preserving transformations (**Q3**) is also under-explored (albeit less clearly motivated). This paper demonstrate TDL models' sensitivity to remeshing by barycentric subdivisions -- a simple topology preserving transformation.
- The paper is clearly organized, with well defined questions, answered in a methodical manner. 
- The authors ensure reproducibility by providing well-documented code repos, versioned datasets, and multiple data format options (raw and PyTorch Geometric), making it easy to build upon this work.

weaknesses:
- **Topological diversity**
  - Other than the genus, first Betti number, and homeomorphism type for 2D manifolds, the distribution of topological invariants is limited and unbalanced.
    - For 3D manifolds:
       - Most Betti numbers, homeomorphism types, and torsion subgroups have only one or two possible values.
       - When two values do exist, one value represents over 99% of the cases.
  - All triangulations are limited to 10 or fewer vertices.

- **Limited model coverage**
  - The only TDL baselines included in the evaluation are simplicial complex models, excluding:
      - Cellular complex methods (e.g. [1, 2]).
      - Combinatorial complex methods (e.g. [3]).
      - Hypergraph methods (e.g. [4, 5]).
  - For both TDL and graph based methods, the evaluation only includes message-passing based architectures, excluding non-message passing graph neural networks (e.g. [6, 7, 8, 9]) and attention based TDL models (e.g. [3, 10]).
  - It would be informative to report the best performing model for each class in the main text.

- **Training methodology**
  - Models were trained for only 6 epochs, hyperparameters were not optimized. The can possibly lead to underestimation of the fully optimized performance of some of the architectures, limiting the scope of the conclusions that can be drawn from the experiments

---

[1] Bodnar et al. "Weisfeiler and Lehman Go Cellular: CW Networks", NeurIPS 2021.

[2] Hajij et al. "Cell complex neural networks", Topological Data Analysis and Beyond Workshop NeurIPS 2020.

[3] Hajij et al. "Topological Deep Learning: Going Beyond Graph Data", arXiv:2206.00606

[4] Feng et al. "Hypergraph neural networks", AAAI 2019.

[5] Kim et al. "Equivariant hypergraph neural networks." ECCV 2022.

[6] Maron et al. "Provably powerful graph networks." NeurIPS 2019.

[7] Abboud et al. "The surprising power of graph neural networks with random node initialization", arXiv:2010.01179.

[8] Bouritsas et al. "Improving graph neural network expressivity via subgraph isomorphism counting", IEEE PAMI 2022.

[9] Zhang et al. "A complete expressiveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests", ICML 2023.

[10] Battiloro et al. "Generalized simplicial attention neural networks", arXiv:2309.02138.

questions:
- **Dataset Generation**
  - How were the manifolds and triangulations generated?
  - What considerations did you make, if any, in order to encourage topological diversity in the dataset?

- **Experimental Results**
  - For the standard deviation results in Table 2, do they represent variation within a single model or across the entire class of models?

- **Computational Issues**
  - The authors cite computational constraints for limiting some aspect of the experiments, can the authors elaborate on the nature of these limitations given that all triangulations have 10 or fewer vertices?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667069467

thread_type:
official_review

<thread_object>
creator: reviewer_r6xh

<content>
summary:
This paper introduces MANTRA, a large-scale, diverse, and high-order dataset of triangulations for benchmarking models in the topological deep learning space. The authors also assess graph and simplicial complex-based models on tasks like betti number estimation, homeomorphism type, and orientability detection. The introduction section is written well with enough definitions/context about various topological properties. The dataset specification includes all properties that are being computed for each object. Furthermore, nine different neural nets are evaluated on the classification task.

Overall, this paper could be a good contribution to topological deep learning, especially for developing advanced TDL methods and benchmarking them. However, the limiting factor is the evaluations (just MP networks) as they do not give a full picture of the impact of the dataset.

strengths:
1. Extensive evaluation of currently available graph and simplicial complex-based models on MANTRA.

2. Provides a foundation for developing and benchmarking advanced TDL methods.

3. Insight into the ability of higher-order models to be invariant to triangulation transformations.

weaknesses:
1. Evaluations are mostly focused on MP networks. It would help get better overall picture of the impact the dataset has by evaluating on other architectures like equivariant high-order neural nets and topological transformers.

2. The variance across results is surprising and I am not sure if running for more epochs/more hyperparameter tuning will address this.

questions:
1. I'm not sure if you mention this in the paper but how are the initial labels and triangulations generated for each surface and three-dimensional manifold?

2. For a given surface, how much computational time is required for computing the fields mentioned Section 2?

3. Would MANTRA be useful for developing models for topological reconstruction (to recover incomplete surfaces or perform noise correction)? If so, how?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732211083638

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers, find attached the results for the surface experiments without the barycentric subdivision. As soon as we have completed the experiments for the 3D datasets, we will update our manuscript with the complete and updated results. The models implemented are:

Cell MP (cellular message passing) [1]
Cellular Transformer (topological transformer) [2]
DECT (differentiable Euler characteristic transforms) [3].

Hyperparameter configurations follow the same patterns as the ones we used for the rest of simplicial models. For the Cell MP, we use a final sum readout, 10 layers, and a hidden size of 64.  For the Cellular Transformer, we use the Hodge Laplacian Eigenvectors defined at [2], a tensor diagram where dimension i attends to dimensions i-1 and i+1, and, global mean pooling, a hidden size of 64, 8 heads, and two layers. For the DECT model, we follow [3] and use 32 directions with a resolution of 32 and hence summarize each simplical complex in a 32x32 image. For the classification and prediction we use an MLP with 64 hidden neurons and 3 hidden layers. 

Betti numbers:

| Dataset | Transform | Betti 0 Accuracy | | | Betti 1 AUROC | | | Betti 2 AUROC | | |
|----------------|-------------------------|------------------------------|------------------------------|------------------------------|---------------------------|---------------------------|---------------------------|---------------------------------|---------------------------------|---------------------------------|
| | | Cell MP | Cell Transformer | DECT | Cell MP | Cell Transformer | DECT | Cell MP | Cell Transformer | DECT |
| All | Degree Transform | 0.46 ± 0.50 | 1.00 ± 0.00 | 1.00 ± 0.00 | 0.62 ± 0.07 | 0.93 ± 0.01 | 0.50 ± 0.00 | 0.49 ± 0.06 | 0.55 ± 0.00 | 0.50 ± 0.00 |
| All | Degree Transform OneHot | | | 1.00 ± 0.00 | | | 0.50 ± 0.00 | | | 0.50 ± 0.00 |
| All | Random Node Features | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 0.84 ± 0.00 | 0.66 ± 0.02 | 0.50 ± 0.00 | 0.52 ± 0.02 | 0.53 ± 0.01 | 0.50 ± 0.00 |
| Known type | Degree Transform | 0.05 ± 0.09 | 1.00 0.00 | 1.00 ± 0.00 | 0.23 ± 0.01 | 0.27 ± 0.01 | 0.21 ± 0.00 | 0.52 ± 0.04 | 0.52 ± 0.03 | 0.50 ± 0.00 |
| Known type | Degree Transform OneHot | | | 1.00 ± 0.00 | | | 0.21 ± 0.00 | | | 0.50 ± 0.00 |
| Known type | Random Node Features | 0.98 ± 0.00 | 1.00 0.00 | 1.00 ± 0.00 | 0.29 ± 0.01 | 0.21 ± 0.00 | 0.21 ± 0.00 | 0.51 ± 0.02 | 0.50 ± 0.00 | 0.50 ± 0.00 |

Homeomorphism type:

| Dataset        | Transform               | AUROC             |                         |                   |
|----------------|-------------------------|-------------------|-------------------------|-------------------|
|                |                         | Cell MP           | Cell Transformer        | DECT              |
| All        | Degree Transform        | 0.85       ± 0.11 | 0.91             ± 0.01 | 0.45       ± 0.00 |
| All        | Degree Transform OneHot |                   |                         | 0.45       ± 0.00 |
| All        | Random Node Features    | 0.89       ± 0.01 | 0.69             ± 0.15 | 0.45       ± 0.00 |
| Known type | Degree Transform        | 0.63       ± 0.14 | 0.83             ± 0.01 | 0.50       ± 0.00 |
| Known type | Degree Transform OneHot |                   |                         | 0.50       ± 0.00 |
| Known type | Random Node Features    | 0.82       ± 0.00 | 0.50             ± 0.02 | 0.48       ± 0.01 |

Orientability:

| Dataset        | Transform               | AUROC        |                  |             |
|----------------|-------------------------|--------------|------------------|-------------|
|                |                         | Cell MP      | Cell Transformer | DECT        |
| All        | Degree Transform        | 0.65  ± 0.07 | 0.55 ± 0.00      | 0.50 ± 0.00 |
| All        | Degree Transform OneHot |              |                  | 0.50 ± 0.00 |
| All        | Random Node Features    | 0.55 ± 0.00  | 0.50 ± 0.00      | 0.50 ± 0.00 |
| Known type | Degree Transform        | 0.51  ± 0.01 | 0.52 ± 0.03      | 0.50 ± 0.00 |
| Known type | Degree Transform OneHot |              |                  | 0.50 ± 0.00 |
| Known type | Random Node Features    | 0.50  ± 0.01 | 0.50 ± 0.00      | 0.50 ± 0.00 |


[1] Bodnar et al. "Weisfeiler and Lehman Go Cellular: CW Networks", NeurIPS 2021.

[2] Ballester, Rubén, et al. "Attending to Topological Spaces: The Cellular Transformer." arXiv preprint arXiv:2405.14094 (2024).

[3 Röell, Ernst, and Bastian Rieck. "Differentiable Euler Characteristic Transforms for Shape Classification." The Twelfth International Conference on Learning Representations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732211128698

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the results

comment:
**Summary of experiments:**

*Betti numbers:*

For the Betti numbers, the scores of the DECT model are comparable to the graph-based models.
Regarding the cellular MP and topological  transformer architectures, both obtain close to on-par results to the best models on the respective tasks. For the first Betti number, the cellular MP and transformer architectures obtain better performances than all the networks except for the best simplicial one, SCCN. For the second betti number, they pose scores slightly better than the worst simplicial model but lack behind the best simplicial model by about 8% in average.

*Homeomorphism type:*

For the full set of surfaces, the cell transformer and the cellular message passing network outperform (0.91 ± 0.01 and 0.89 ± 0.01, respectively) the previous best AUROC of SCCN (0.85 ± 0.08). The DECT model obtain worse values than simplicial models, in general, having results more similar to the ones of the graph networks.

For the set of surfaces with homeomorphism type assigned the same happens: the AUROC metrics are 0.83 ± 0.01 and 0.82 ± 0.00, respectively for cellular transformer and message passing architectures, and the best previous score was (0.80± 00) for SCCN.

*Orientability:*

For the full set of surfaces, the cellular MP outperforms the previous best models (0.65± 0.07 against 0.55 ± 0.01 and 0.55± 0.09 for SCCN and SCCNN, respectively). The cellular transformer matches the same performance (0.55 ± 0.00).

For the set of surfaces with homeomorphism type assigned, both, cellular transformers and cellular message passing networks obtain comparable performances to most most of the other networks, except for SCCN, where they are slightly worse (0.52± 0.03 and 0.51 ± 0.01 for transformer and cell MP against 0.54 ± 0.01 for SCCN, respectively).

**Conclusion:**

Although we saw that for the Betti numbers' experiments the new models did not perform better than the older ones, we also saw that the cellular MP and topological transformer architectures obtained slightly better results than the older models for most homeomorphism type and orientability experiments, being comparable in the ones they were not better. Still, the differences are minimal and thus we believe they do not change our conclusions for the analysis on the MANTRA surfaces.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes MANTRA, mainly a synthetic new benchmark focusing on evaluating methods in topological deep learning. Most reviewers were on the positive side, although there is quite a big disagreement with a 'accept' and a 'reject' suggestion. Basically, the point of disagreement is to what extent the paper presents a truly new data setting and benchmark, given that similar and harder problems have been extensively studied and researched in graphics, specifically CGAL (computational geometry algorithms library). The counterargument is that these libraries are indeed addressing a similar problem, although it is definitely non-trivial for researchers who come from ML and are not domain experts to work with these libraries. This is certainly a logical point. 

Another point of concern is whether having a dataset like MANTRA is sufficient, given that it is a synthetic dataset, and given that there is a plethora of real-world problems that could be considered as well.

I believe both sides have merit and I am definitely sensitive to the argument that it is important that ML methods stay close to real-world and practical applications. To decide on this, I note that since this is a dataset+benchmark paper, ignoring the 'magnitude' of the recommendations and only focusing on the 'sign' (positive vs negative review), most reviewers agree it is a useful dataset. Combined with the fact that as it was commented by the reviewers, the papers addresses a big gap in TDL, I recommend acceptance.