Title: TopoSRL: Topology preserving self-supervised Simplicial Representation Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Simplicial representation learning

Self-supervised learning

Message passing simplicial networks
Abstract: In this paper, we introduce $\texttt{TopoSRL}$, a novel self-supervised learning (SSL) method for simplicial complexes to effectively capture higher-order interactions and preserve topology in the learned representations. $\texttt{TopoSRL}$ addresses the limitations of existing graph-based SSL methods that typically concentrate on pairwise relationships, neglecting long-range dependencies crucial to capture topological information. We propose a new simplicial augmentation technique that generates two views of the simplicial complex that enriches the representations while being efficient. Next, we propose a new simplicial contrastive loss function that contrasts the generated simplices to preserve local and global information present in the simplicial complexes. Extensive experimental results demonstrate the superior performance of $\texttt{TopoSRL}$ compared to state-of-the-art graph SSL techniques and supervised simplicial neural models across various datasets corroborating the efficacy of $\texttt{TopoSRL}$ in processing simplicial complex data in a self-supervised setting.

TopoSRL : Topology Preserving Self-Supervised Simplicial Representation Learning

Abstract

In this paper, we introduce TopoSRL , a novel self-supervised learning (SSL) method for simplicial complexes to effectively capture higher-order interactions and preserve topology in the learned representations. TopoSRL addresses the limitations of existing graph-based SSL methods that typically concentrate on pairwise relationships, neglecting long-range dependencies crucial to capturing topological information. We propose a new simplicial augmentation technique that generates two views of the simplicial complex that enriches the representations while being efficient. Next, we propose a new simplicial contrastive loss function that contrasts the generated simplices to preserve local and global information present in the simplicial complexes. Extensive experimental results demonstrate the superior performance of TopoSRL compared to state-of-the-art graph SSL techniques and supervised simplicial neural models across various datasets corroborating the efficacy of TopoSRL in processing simplicial complex data in a self-supervised setting.

1 Introduction

Simplicial complexes are mathematical structures that explicitly capture higher-order relationships between entities (as nodes) using simplicies of different orders (as edges, triangles, and so on). There has been a growing interest in developing simplicial representation learning models, such as simplicial neural networks (SNN) [1-6], as simplicial complexes are generally more expressive than graphs, which only capture pairwise relations. SNNs incorporate topological information available in simplicial complexes while learning representations of simplicies of different orders, which are useful for various downstream tasks such as node, graph, higher-order link, and trajectory prediction tasks [1, 4-6]. However, a significant challenge in learning representations for simplicial complexes using existing SNN models is the need for task-specific labels required for training. Acquiring meaningful labels for real-world, high-dimensional, and complex data is difficult due to intricate structures, multiple valid labeling schemes, or privacy and ethical concerns.

Self-supervised learning (SSL) schemes learn expressive and powerful representations without requiring labeled data. Specifically, the main goal in SSL is to model an encoder, which is learned using an objective function and unlabelled training data. This paper proposes an SSL method for simplicial complex data that preserves topological and geometric information while learning representations. Although no existing studies focus on SSL for simplicial complex data, a closely related field of SSL for graph data has been extensively studied. SSL for simplicial complex data is an important generalization as every simplicial complex inherently includes an underlying graph, making SSL on graphs a specialized subset of SSL on simplicial complexes.

The general idea behind SSL on graphs is to augment a graph to create two views of the available graph and then maximize the mutual information (MI) between the augmented graphs. So, the research focus thus far has been on designing augmentation techniques and objective functions that maximize MI. However, existing methods of SSL on graphs require complex augmentation [7], negative hirenmadhu,

sampling algorithms [8], or require components to empirically avoid degenerative solutions [9-12]. This leads to a more complex neural model, hindering their direct extension for SSL on simplicial complexes. For instance, deep graph infomax (DGI) [9] maximizes MI between a node and its subgraph by learning two MLPs: one for the subgraph readout and another as a discriminator function classifying if a node exists in the given subgraph or not. Extending DGI to simplicial complexes would necessitate training k +1 discriminators for a simplicial complex of order k (see Section 3 for the definition of the order of a simplicial complex), drastically increasing the training complexity. Similarly, graph contrastive representation learning (GRACE) [8] and graph contrastive representation learning with adaptive augmentation (GCA) [7] need the selection of effective negative samples and require additional storage by selecting all the other nodes in the graph as negative samples. Directly extending these methods to simplicial complexes would lead to prohibitive computational complexity going up to O (2 N ) for a simplicial complex with N nodes. To mitigate the need for negative samples or additional networks, recent approaches like bootstrapped graph representation learning (BGRL) [11], SelfGNN [12], and canonical correlation analysis inspired self-supervised learning on graphs (CCA-SSG) [13] attempt to learn representations by contrasting corresponding node pairs in the two augmented graphs. Self-supervised masked graph autoencoders (GraphMAE) [14] is an alternative approach that uses a reconstruction loss rather than the commonly used contrastive loss in the graph-based SSL methods. However, these methods focus only on contrasting or reconstructing local information and do not account for global long-distance information available in the network.

Motivated by the aforementioned limitations of SSL methods on graphs, we introduce TopoSRL , a self-supervised learning pipeline for simplicial complex data that preserves topology information in the representation space. Preserving topology information is crucial because it allows the learned representations to capture higher-order interactions and relationships unique to simplicial complexes. TopoSRL comprises an intuitive technique to generate stochastically augmented views of a simplicial complex. We introduce a new contrastive loss function to preserve topology information of simplicial complexes in the geometric space to learn more expressive representations. In sum, our major contributions in TopoSRL are as follows:

Simplicial augmentation: We introduce a simple and effective augmentation technique for simplicial complexes that captures the inherent relationships between simplices. In particular, the augmentation method stochastically removes closed simplicies and adds open simplicies guided by the topological structure of a simplicial complex, where a closed simplex is a simplex that is part of the simplicial complex, while an open simplex is a simplex that itself is not a part of the simplicial complex, but all of its subsets are. This procedure is computationally efficient and leads to superior performance than randomly adding simplicies without accounting for the topological structure of a simplicial complex.

Simplicial contrastive loss: We propose a loss function for SSL on simplicial complexes that contrasts pairs of corresponding simplicies as well as considers the relational distance between pairs of simplicies and their augmented counterparts, where the relational distance refers to a measure of dissimilarity between pairs of simplices. We also provide theoretical evidence that the proposed loss function implicitly maximizes MI between a simplex and its neighborhood within the same augmented simplex and the other augmented simplex, helping the model capture the inherent structures and patterns in the data, thereby improving the model's performance and adaptability to various tasks without the requirement of training additional components such as MI estimators.

We conduct experiments demonstrating our proposed method on downstream tasks such as node classification, simplicial closure, graph classification, and trajectory prediction. We also highlight the effectiveness of TopoSRL in learning expressive representations for simplicial complexes with the proposed augmentation technique as opposed to the random augmentation technique. Experiments show that without any complex architectures or expensive augmentation techniques, our method outperforms existing state-of-the-art graph representation learning methods while being competitive with supervised simplicial representation learning methods.

2 Related works

This section briefly discusses a few popular SNN models, one of which can be used in TopoSRL . The first step of TopoSRL 's learning phase is simplicial augmentation. Hence, we also discuss existing

augmentation techniques used in graph SSL and how they can not be extended (due to computationally or empirical limitations) for simplicial complexes. Then, we move on to discussing SSL on graphs, a specialized version of SSL on simplicial complexes.

Simplicial representation learning. SNN models learn representations of simplices of different orders (e.g., nodes, edges, triangles, and so on) in a simplicial complex [2, 3], and are based on an extension of graph convolutions to convolutions over simplicial complexes. Simplicial attention networks (SAN) [1] extends graph attention networks (GAT) [15] for simplicial complexes. Message passing simplicial networks (MPSN) [5] proposes a framework to design more expressive SNN models related to the so-called simplicial Weisfeiler-Lehman isomorphism test. TopoSRL is free from the choice of a specific SNN encoder, and any one of the SSN models can be used as an encoder.

Graph augmentation. Most of the graph SSL methods described in Section 1, namely, GRACE, GCA, and BGRL, use an augmentation technique that includes two steps: (i) add and remove edges at random to generate two graphs from an input graph, and (ii) randomly mask dimensions of initial node features. GCA [7] proposes an adaptive data augmentation technique that models the edge-removal probability differently for each edge in the graph based on the importance of the edge. However, this method requires more expensive augmentations to attain peak performance [11]. Applying an identical augmentation technique to simplicial complexes necessitates modeling removal probabilities for each simplex. This increases the augmentation process's complexity and mandates the development of innovative metrics for measuring simplex importance. Other than this, most methods use uniform probability to add and remove edges. Although adding and removing simplicies uniformly at random is a naive and simple extension, we empirically show that TopoSRL performs considerably better than random augmentation. In contrast, TopoSRL uses a simpler augmentation technique that only requires the addition of open simplicies, which are inherently rich in information as discussed later in Section 4.1, and is very easy to implement.

SSL on graphs. Contrastive learning methods for images have recently been adapted for graphs. This includes DGI [9], which contrasts node-local patches with global graph representations was inspired by Deep InfoMax [16]. GMI [17] maximizes a concept of graphical mutual information inspired by MINE [18], enabling a more granular contrastive loss than DGI. GRACE [8] and its derivatives, such as GCA [7], which rely on more complex data adaptive enhancements, have adapted the SimCLR [19] algorithm for graphs. GraphCL [20] also adapts SimCLR to learn graph-level representations with a contrastive objective. Multi-view graph representation learning (MVGRL) [10] extends contrastive multi-view coding to graphs. All these methods suffer from considerable computational complexities because they rely heavily on negative samples. BGRL [11] and SelfGNN [12] extend Bootstrap you own latent (BYOL) [21], which uses different online and target encoders, wherein the target encoder is updated as an exponential moving average of the online encoder while the online encoder is updated by optimizing a loss function. The difference between BGRL and SelfGNN is that BGRL uses different online and target encoders, but SelfGNN utilizes the same encoder as online and target. Lastly, CCA-SSG[13] uses CCA-based loss. BGRL, SelfGNN, and CCA-SSG are methods free from negative samples and incur lower computational complexity than previous methods. Inspired by these graph SSL methods, TopoSRL aims at a negative sampling-free approach focusing on preserving topology and implicitly maximizing MI.

3 Background

A simplicial complex X is a collection of a finite number of simplicies. A simplex of order k (or, a k -simplex) is a ( k +1) -cardinality subset of the vertex set V so that if a simplex σ k ∈ X then all the non-empty subsets of σ k also belong to the simplicial complex X . The order, K , of a simplicial complex X is the order of the maximally ordered simplex in the simplicial complex.

Suppose σ k , τ k and ρ k are some k -simplices in a simplicial complex. A k -simplex σ k has the following neighbors in the simplicial complex, namely, lower-adjacent neighbors L ( σ k ) = { τ k | ρ k -1 ⊂ σ k ∧ ρ k -1 ⊂ τ k } , upper-adjacent neighbors U ( σ k ) = { τ k | σ k ⊂ ρ k +1 ∧ τ k ⊂ ρ k +1 } , boundary B ( σ k ) = { τ k -1 | τ k -1 ⊂ σ k } , and co-boundary C ( σ k ) = { τ k +1 | σ k ⊂ τ k +1 } .

An open k -simplex is defined as a k -simplex σ k such that all its boundaries B ( σ k ) ⊂ X , but σ k / ∈ X . In other words, if all the boundaries of a simplex are present in the simplicial complex, but the simplex itself is not part of the simplicial complex, it is considered an open simplex.

Shampoo

Detergent

Sanitizer.

→Z（1)=E（x(1）,H(1）)

Diapers

Wipes

Shampoo

>Bread

view

C(Z(1),z(2),w)

optimize

Detergent

Pacifier*

Jam

(X,H)

view 2

→Z（2)=E（x(2)，H(2)

Simplicial contrastive loss

Diapers

Pacifier*

Wipes

Jam

>Bread

view 2

Shampoo

Sanitizer

Detergent

I

1

sadea

Wipes

（x(2),H(2))

>Bread

Simplicial augmentation

Encoders

Pacifier*

Jam

(a)

(b)

A generic SNN model has the following aggregation steps

and a combining step

where t is the layer index of the SNN model, ϕ n is the transformation function with adjacency of type n (We use one layer MLP), h t σ k is the representation of the simplex σ k at layer t with h 0 σ k being the initial embedding. We write the final representations of all the simplices in the simplicial complex X obtained from such an SNN model with L layers compactly as

where θ denotes the set of learnable parameters and H is the initial feature matrix of all the simplices in X . Although the above SNN model is based on MPSN, different SNN variants can be derived from the above model by choosing different AGGREGATE and COMBINE operators and which neighborhoods to aggregate.

4 The proposed TopoSRL pipeline

The main aim of the proposed approach is to learn an SNN encoder function E θ ( · ) to compute expressive representations for simplices in a simplicial complex X in a self-supervised manner. Towards this end, we first compute two augmented views of a simplicial complex X , namely, X (1) and X (2) (as described later on Section 4.1). Then we learn the representations of the two simplicial complexes through the encoder E θ ( · ) as Z (1) = E θ ( X (1) , H (1) ) and Z (2) = E θ ( X (2) , H (2) ) , where Z ( i ) = { Z ( i ) 1 , Z ( i ) 2 , . . . , Z ( i ) K } is the set embedding matrices and Z ( i ) k ∈ R N ( i ) k × D being the D -dimensional embedding matrix of all the k -simplicies in the simplicial complex X ( i ) and N ( i ) k being the total number of k -simplicies in X ( i ) . We contrast these representations and learn E θ ( · ) by optimizing the simplicial contrastive loss (developed in Section 4.2). An illustration of the TopoSRL pipeline is shown in Figure 1(a).

4.1 Simplicial augmentation

We introduce a novel stochastic augmentation technique for simplicial complexes to generate two simplicial complexes X (1) and X (2) from X by adding open simplices and removing closed simplices at random via Bernoulli sampling. To begin with, we compute 1 all the open simplices in X . Next,

Algorithm 1 Simplicial complex augmentation

we draw a Bernoulli random variable for each open simplex to determine whether to include the open simplex in X ( i ) for i = 1 , 2 . Similarly, we draw another Bernoulli random variable for each k -simplex ( k > 1 ) in X to determine whether to retain the closed simplex in X ( i ) for i = 1 , 2 . This procedure is summarized as Algorithm 1.

The proposed augmentation method preserves the inherent structure and relationships within the simplicial complex as illustrated in Figure 1(b). For example, consider the walmart-trips simplicial complex dataset 2 , wherein a set of items purchased together by a customer in one trip to Walmart is a simplex. Now, if a customer purchases {peanut butter and jam} , {bread and jam} , and {peanut butter and bread} on three separate occasions, it is highly probable that the customer will buy {peanut butter, jam, bread} together on a subsequent trip. Adding the simplex {peanut butter, jam, bread} as in view 2 of Figure 1(b) will represent an instance of the simplicial complex if the data were sampled at a later point in time, and hence it still retains and provides better information to contrast compared to adding random simplices.

In contrast, in a random sampling method, where we add simplices at random, we need to include all the subsets of the simplex that is to be added, which costs O (2 k ) for each k -order simplex for k ∈ { 2 , . . . K } . As opposed to this, in the proposed simplicial augmentation technique, since we are adding an open simplex, all the subsets of this simplex are present in the simplicial complex, making the additional cost only O (1) and thereby making it computationally more efficient.

4.2 Simplicial contrastive loss

The proposed method aims to learn simplex representations by capturing topological information in higher-order simplices and their distant neighbors. In contrast, most of the existing self-supervised graph representation learning approaches primarily focus on contrasting corresponding nodes in augmented graphs and do not preserve topological information as the objective function does not focus on the relational distance between pairs of two nodes that are not neighbors.

Cost matrices. We define the following cost matrices to compute the TopoSRL loss function: the intra-view cost matrix , denoted by C ( i ) k ∈ R N ( i ) k × N ( i ) k for i = 1 , 2 , measures the distance between representations of two simplices in the simplicial complex X ( i ) and the inter-view cost matrix , denoted by C (1 , 2) k ∈ R N (1) k × N (2) k , measures the distance between the k -simplex in the two views. The intra-view cost matrix is used to minimize the relational distance between pairs of simplices from two augmented simplicial complexes. Specifically, the ( p, q ) th entry of C ( i ) k , is defined as [ C ( i ) k ] p,q = ∥ [ Z ( i ) k ] p -[ Z ( i ) k ] q ∥ 2 2 , where [ Z ( i ) k ] p is the representation of the k -simplex p in X ( i ) . On the other hand, the inter-view cost matrix is used to minimize the distance between the representation of a simplex and an aggregate representation of a sub-simplicial complex surrounding

this simplex in the other augmented simplicial complex. The ( p, q ) th entry of C (1 , 2) k is defined as [ C (1 , 2) k ] p,q = ∥ [ Z (1) k ] p -[ Z (2) k ] q ∥ 2 2 .

Weight matrix. We also construct a weight matrix W k ∈ R N (1) k × N (2) k for each k -simplex to capture the relation of a simplex σ (1) k ∈ X (1) and σ (2) k ∈ X (2) , which will be used to calculate the aggregate representation of sub-simplicial complexes by assigning different importance weights that depend on how many hops σ (2) k is away from σ (1) k in the simplicial complex X (1) . We assign the entries of W k as follows. If σ (2) k = σ (1) k , we assign a higher value η 0 to [ W k ] σ (1) k ,σ (2) k . If σ (2) k ∈ L ( σ (1) ) ∪U ( σ (1) ) , i.e., if σ (2) k is in one-hop neighborhood of σ (1) k in X (1) , then we assign it a lower value η 1 . Similarly, if σ (2) k is in the two-hop neighborhood of σ (1) k in X (1) , we assign a value of η 2 , such that η 0 > η 1 > η 2 and then we apply row-wise softmax to W k to normalize it. By construction, W k is a row-stochastic matrix.

Loss function. The proposed simplicial contrastive loss is a convex combination of two terms, namely, the sub-simplicial complex loss L sub and the relative simplicial complex loss L rel . The term L sub measures the cumulative (over all the simplicies) weighted distance between the representation of a k -simplex σ i k ∈ X (1) and k -simplices σ i ′ k ∈ X (2) with appropriate weights in the matrix W k , and is given by

The term L rel calculates the relational distance between pairs of simplices and is given by

For instance, consider two pairs of simplices ( σ i k , σ j k ) and ( σ i ′ k , σ j ′ k ) such that σ i k , σ j k ∈ X (1) k , σ i ′ k , σ j ′ k ∈ X (2) k , [ W k ] σ i k ,σ i ′ k > 0 , and [ W k ] σ j k ,σ j ′ k > 0 , that is, ( σ i k , σ i ′ k ) and ( σ j k , σ j ′ k ) are distant neighbors (not necessarily one-hop). Hence, minimizing L rel minimizes the squared difference of the distance between the representations of ( σ i k , σ j k ) and ( σ i ′ k , σ j ′ k ) , which leads to similar pairs of simplices from two distinct simplicial complexes maintaining an equal distance. In other words, if σ j k is m -hop away from σ i k and σ j ′ k is m -hop away from σ i ′ k , then minimizing L rel will reduce the difference between the distance of the pairs and preserve the m -hop information present in X . Finally, the overall loss is

where α ∈ [0 , 1] is a tunable parameter. The overall loss preserves the topological properties as adjacent simplices will be embedded more closely in the Euclidean space because of the L sub term, capturing the local information. The difference in distance of contrasting simplex pairs will be minimized by minimizing L rel , capturing the global information.

We end this section with two interesting theoretical results 3 about the simplicial contrastive loss function and an overview of the TopoSRL pipeline in Algorithm 2. In the next proposition, we show that L sub preserves the local information in a simplicial complex.

Proposition 1. Minimizing L sub is equivalent to jointly minimizing a lower bound on the distance between the representation of a simplex σ i k ∈ X (1) and the aggregate representation of σ i ′ k ∈ X (2) that is adjacent to σ i k and the distance between the representation of σ i ′ k ∈ X (2) and the aggregate of representations of σ i ′ k ∈ X (2) that is adjacent to σ i k ∈ X (1) .

Most graph SLL methods maximize MI between augmented graphs using MI estimators [9]. In the next theorem, we show that optimizing the proposed loss function is also related to optimizing MI between representations of simplicies in one augmented simplicial complex and representations of

Algorithm 2 TopoSRL

its neighbors in the other augmented simplicial complex and representations of simplices and their neighborhoods within the same augmented simplicial complex conditioned on the input data.

Suppose X represents the random variable corresponding to the input data, X k represents the k -simplices in X , and X ( i ) represents an augmented view of X , sharing the same sample space as X , and the input simplicial complex data X ∼ X . TopoSRL is designed to learn representations, denoted as Z for the input data and Z ( i ) for its augmentation. We also denote H ( A ) and I ( A,B ) as the entropy of the random variable A and the MI between the random variables A and B , respectively.

Theorem 1. Minimizing the expected loss L sub (expectation is with respect to the random variable X ) is equivalent to maximizing the MI between Z ( i ) k and X k , i.e.,

for k = 0 , 1 , . . . , K , where L sub ,k is the k th summand in (1) , I ( Z ( i ) k , X k ) is the MI between the representations of the augmented k -simplices Z ( i ) k and the k -simplicies in the originial data X k .

The above theorem shows that in expectation, minimizing the term L sub is equivalent to minimizing the variance between representations in the augmented simplicial complexes, which leads to minimization of the conditional entropy H ( Z ( i ) k | X K ) , implying the maximization of MI as I ( Z ( i ) k , X k ) = H ( Z ( i ) k ) -H ( Z ( i ) k | X k ) . The proof also shows that minimizing the simplicial contrastive loss implicitly maximizes MI between simplicies from one augmented simplicial complex and representations of its neighbors in the other augmented simplicial complex conditioned on the input and representations of simplices and their neighborhoods within the same augmented simplicial complex. This result is along the lines of earlier methods like DGI [9] and InfoGraph [22]. However, TopoSRL does not require additional components in DGI or InfoGraph for MI maximization.

5 Experiments

We follow the standard setting of self-supervised learning methods. Firstly, we train the encoder with the proposed simplicial contrastive loss. Next, we freeze the encoders' model parameters, extract representations for all the simplicies, and train a classifier for the following two downstream tasks. The code is available at https://github.com/HirenMadhu/TopoSRL .

Downstream tasks. We focus on node classification , simplicial closure , trajectory prediction and graph classification tasks. Node classification focuses on predicting labels for 0 -simplicies (aka nodes) in a given simplicial complex. We perform node classification on the following publicly available datasets 4 , namely, contact-primary-school , contact-high-school , senate-bills . We use classification accuracy as the performance metric. In simplicial closure, the aim is to predict the closure of open simplicies in a time series of simplicial complex data. We perform simplicial

closure on email-Eu , email-Enron , contact-high-school datasets. We first split the data across time on these temporal datasets and then train the encoder on the first 80% of the data. The last 20% is used for inference. Since the class distribution is very skewed, we use F1-macro to evaluate the performance. Trajectory prediction focuses on predicting the next node in a trajectory, given a sequence of nodes. We perform experiments on two datasets, namely, Ocean and the Synthetic dataset generated using the method described in [6]. We use accuracy as the metric for comparison. In graph classification, the focus is on learning representations for the whole graph and classifying them. Clique lifting is used to convert a graph to a simplicial complex and then use an SNN model to extract representations for the whole graph. We evaluate TopoSRL 's performance in graph classification task on PROTEINS , NCI1 , REDDIT-B , REDDIT-M and IMDB-B datasets from the TUDatasets [23] repository.

Baselines. We compare TopoSRL against several supervised techniques employing architectures such as SAN [1], SCNN [3], and MPSN [5]. This comparison builds confidence in the expressive capabilities of TopoSRL and solidifies its usefulness in learning simplex representations in settings with less-labeled or unlabeled data. Additionally, we conduct experiments involving various graph neural network models, including graph convolutional network (GCN) [24], GAT[15], GIN [25], and GraphSage[26]. Furthermore, we also compare our method with the current state-of-the-art graph SSL methods CCA-SSG[13], BGRL [11], GraphMAE [14] and GCA [7]. This comparison also clarifies that in the SSL setting, the proposed simplicial SSL method (i.e., TopoSRL ) performs better than graph SSL methods, as demonstrated by the existing supervised simplicial representation learning methods. For TopoSRL , we use a 3-layer MPSN as the encoder network E θ ( · ) . Details about the experimental setup, hyperparameters, and results with other SNN encoder models are available in the supplementary material. We follow the standard practice where all the results are averaged over ten different seeds, and one run is performed for each seed.

Results and discussion. Our results demonstrate that TopoSRL consistently outperforms state-ofthe-art graph SSL methods. In Table 1, we see that TopoSRL surpasses supervised methods on graphs and state-of-the-art graph SSL techniques while being competitive with supervised approaches for the node classification task. TopoSRL has a superior performance compared to MPSN on the contact-high-school dataset while being competitive with MPSN on the other two datasets. As can be seen in experiments reported in the supplementary material, contact-primary-school , being a denser dataset, does not benefit significantly from minimizing both the terms, but minimizing only L rel (i.e., α = 1 ) improves performance.

Table 2 showcases experimental results on simplicial closure datasets. On this task, we see that the less parameterized SCNN model outperforms both TopoSRL and MPSN on the email-Enron and contact-primary-school datasets, while TopoSRL performs better than MPSN and is competitive overall. These results can be attributed to the skewed nature of simplicial closure datasets, where models with large parameters like MPSN and SAN are prone to overfitting labels. Although TopoSRL

focuses on preserving topology rather than concentrating on a specific downstream task, it has improved performance compared to these models.

Table 3 presents the results on trajectory prediction. As we can see, the results indicate that TopoSRL outperforms SCNN and ScoNe when applied to the Ocean dataset. When tested on the synthetic dataset, TopoSRL demonstrates performance on par with a supervised SCNN. These results further highlight the expressive representation capabilities of TopoSRL on oriented simplicial complexes and its use cases in practical applications. As we can see in Table 4, TopoSRL performs on par with supervised graph baselines and simplicial baselines. Further, TopoSRL outperforms or performs on par with graph SSL baselines, showing the advantages of TopoSRL on a standard graph dataset with clique lifting compared to graph SSL methods.

To further validate the efficacy of our augmentation technique, we conduct an ablation study, comparing TopoSRL with and without the proposed augmentation strategy. The results indicate a significant performance improvement when incorporating the proposed simplicial augmentation, underlining its crucial role in capturing long-range dependencies and higher-order interactions. Table 5 shows that our augmentation technique results in considerable performance gains over random augmentation methods. We observe that only adding open simplices and not removing closed simplices is a better augmentation technique than random augmentation, but its performance deteriorates compared to the proposed augmentation technique. This occurs because the contrastive objective function is more effective when there is more information to contrast. Removing closed simplices allows the contrastive loss and encoder to contrast more information, resulting in higher performance instead of only adding open simplices. Additionally, we have observed that incorporating higher-order information from simplices with more than three vertices leads to reduced performance. This reduction is due to the limited presence of higher-order simplices in these datasets, which causes overparameterization of

the model and a slight decline in performance. More experimental results comparing different SNN encoders and different values of alpha are reported in the supplementary material.

To test the ability of TopoSRL to learn expressive representations, we perform the following experiment: 1) Use the pre-trained TopoSRL encoders to extract representations and 2) Use only a partially labeled (e.g., 20% train and 80% test, 40% train and 60% test, etc.) data to train a logistic regression classifier for node classification task in the contact-high-school dataset. MPSN and GCN are trained with cross-entropy loss on the train set. Since the weights for the TopoSRL encoder trained without labels have been saved, no new encoders were trained to produce the results. As we can see in Table 6, with an increase in the size of the train set, the performance increases across all the methods. Furthermore, TopoSRL has a significantly improved performance of about 5% over supervised MPSN in the 20-80 and 40-60 split. This provides empirical evidence about the expressive capabilities of TopoSRL and its efficacy with less-labeled data. Hence, TopoSRL would be preferable over standard supervised models in the less-labeled data setting.

6 Conclusions

We have introduced a novel framework for SSL on simplicial complexes by leveraging topological properties underlying a simplicial complex. In particular, we have proposed two key components: a stochastic simplicial augmentation method and a simplicial contrastive loss function, which collectively provide a mechanism for learning representations that retain local and global topological information in the simplicial complex. The proposed simplicial augmentation method offers the advantage of generating topologically consistent views of the original simplicial complex, allowing the model to learn a rich set of features efficiently. The contrastive loss function comprises two primary terms: the sub-simplicial complex loss and the relative simplicial complex loss. The former focuses on minimizing the distance between the representation of a simplex and its adjacent simplices in the augmented complex, thereby preserving local information. The latter aims to maintain relative spatial relationships present in the original simplicial complex, thereby capturing global information. We have also theoretically proved that the proposed loss function is related to the MI objective function, which is commonly used in SSL on graphs. Our model outperforms state-of-the-art graphs SSL methods on a variety of datasets. We believe this work lays a solid foundation for further exploration of SSL methods for simplicial complexes, e.g., via alternative augmentation methods, contrastive loss functions, scalable models, and new applications, potentially opening up new avenues for topological data analysis.

References

Christopher Wei Jin Goh, Cristian Bodnar, and Pietro Liò. Simplicial Attention Networks . 2022. DOI: 10.48550/ARXIV.2204.09455 . URL: https://arxiv.org/abs/2204.09455 .

Stefania Ebli, Michaël Defferrard, and Gard Spreemann. 'Simplicial neural networks'. In: arXiv preprint arXiv:2010.03633 (2020).

Maosheng Yang, Elvin Isufi, and Geert Leus. 'Simplicial convolutional neural networks'. In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE. 2022, pp. 8847-8851.

Eric Bunch et al. 'Simplicial 2-complex convolutional neural nets'. In: arXiv preprint arXiv:2012.06010 (2020).

Cristian Bodnar et al. 'Weisfeiler and lehman go topological: Message passing simplicial networks'. In: International Conference on Machine Learning . PMLR. 2021, pp. 1026-1037.

T. Mitchell Roddenberry, Nicholas Glaze, and Santiago Segarra. 'Principled Simplicial Neural Networks for Trajectory Prediction'. In: Proceedings of the 38th International Conference on Machine Learning . Ed. by Marina Meila and Tong Zhang. Vol. 139. Proceedings of Machine Learning Research. PMLR, 2021, pp. 9020-9029. URL: https://proceedings.mlr. press/v139/roddenberry21a.html .

Yanqiao Zhu et al. 'Graph contrastive learning with adaptive augmentation'. In: Proceedings of the Web Conference 2021 . 2021, pp. 2069-2080.

Yanqiao Zhu et al. 'Deep Graph Contrastive Representation Learning'. In: CoRR abs/2006.04131 (2020). arXiv: 2006.04131 . URL: https://arxiv.org/abs/2006.04131 .

Petar Velickovic et al. 'Deep graph infomax.' In: ICLR (Poster) 2.3 (2019), p. 4.

Kaveh Hassani and Amir Hosein Khasahmadi. 'Contrastive multi-view representation learning on graphs'. In: International conference on machine learning . PMLR. 2020, pp. 4116-4126.

Shantanu Thakoor et al. 'Large-scale representation learning on graphs via bootstrapping'. In: arXiv preprint arXiv:2102.06514 (2021).

Zekarias T. Kefato and Sarunas Girdzijauskas. 'Self-supervised Graph Neural Networks without explicit negative sampling'. In: CoRR abs/2103.14958 (2021). arXiv: 2103.14958 . URL: https://arxiv.org/abs/2103.14958 .

Hengrui Zhang et al. 'From canonical correlation analysis to self-supervised graph neural networks'. In: Thirty-Fifth Conference on Neural Information Processing Systems . 2021.

Zhenyu Hou et al. 'Graphmae: Self-supervised masked graph autoencoders'. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 2022, pp. 594-604.

Petar Veliˇ ckovi´ c et al. Graph Attention Networks . 2018. arXiv: 1710.10903 [stat.ML] .

R Devon Hjelm et al. Learning deep representations by mutual information estimation and maximization . 2019. arXiv: 1808.06670 [stat.ML] .

Zhen Peng et al. 'Graph Representation Learning via Graphical Mutual Information Maximization'. In: Proceedings of The Web Conference 2020 . WWW '20. Taipei, Taiwan, 2020, 259-270. ISBN: 9781450370233. DOI: 10.1145/3366423.3380112 . URL: https://doi. org/10.1145/3366423.3380112 .

Mohamed Ishmael Belghazi et al. 'Mutual Information Neural Estimation'. In: Proceedings of the 35th International Conference on Machine Learning . Ed. by Jennifer Dy and Andreas Krause. Vol. 80. Proceedings of Machine Learning Research. PMLR, 2018, pp. 531-540. URL: https://proceedings.mlr.press/v80/belghazi18a.html .

Ting Chen et al. 'A Simple Framework for Contrastive Learning of Visual Representations'. In: Proceedings of the 37th International Conference on Machine Learning . Ed. by Hal Daumé III and Aarti Singh. Vol. 119. Proceedings of Machine Learning Research. PMLR, 2020, pp. 1597-1607. URL: https://proceedings.mlr.press/v119/chen20j.html .

Yuning You et al. 'Graph contrastive learning with augmentations'. In: Advances in neural information processing systems 33 (2020), pp. 5812-5823.

Jean-Bastien Grill et al. 'Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning'. In: CoRR abs/2006.07733 (2020). arXiv: 2006.07733 . URL: https://arxiv. org/abs/2006.07733 .

Fan-Yun Sun et al. 'Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization'. In: arXiv preprint arXiv:1908.01000 (2019).

Christopher Morris et al. 'TUDataset: A collection of benchmark datasets for learning with graphs'. In: ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020) . 2020. arXiv: 2007.08663 . URL: www.graphlearning.io .

Thomas N. Kipf and Max Welling. 'Semi-Supervised Classification with Graph Convolutional Networks'. In: International Conference on Learning Representations . 2017. URL: https: //openreview.net/forum?id=SJU4ayYgl .

Keyulu Xu et al. 'How powerful are graph neural networks?' In: arXiv preprint arXiv:1810.00826 (2018).

William L. Hamilton, Rex Ying, and Jure Leskovec. 'Inductive Representation Learning on Large Graphs'. In: CoRR abs/1706.02216 (2017). arXiv: 1706.02216 . URL: http: //arxiv.org/abs/1706.02216 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new self-supervised learning (SSL) method called TopoSRL for simplicial complexes. It claims to overcome the limitations of existing graph-based SSL methods by capturing higher-order interactions and preserving topology in learned representations. The authors propose a simplicial augmentation technique to generate enriched representations. They also propose a simplicial contrastive loss function to preserve local and global information in simplicial complexes. The experimental results allegedly demonstrate that TopoSRL outperforms state-of-the-art graph SSL techniques and supervised simplicial neural models on various datasets.\n\nSTRENGTHS:\n- The proposed method offers an interesting and intuitive approach.\n- It addresses the capture of higher-order interactions and preserves topology in learned representations.\n- The simplicial augmentation technique enriches representations.\n- The proposed simplicial contrastive loss function preserves local and global information in simplicial complexes.\n\nWEAKNESSES:\n- The paper could improve its presentation and writing to enhance accessibility for general readers.\n- The organization of the method section can be refined to provide clearer explanations and reduce overflow of mathematical expressions.\n- Important definitions are placed in footnotes, which is not ideal.\n- The effectiveness of the proposed method in practice is not adequately demonstrated.\n- The choice of datasets used may not fully reflect the performance potential of the proposed method.\n\nTo enhance accessibility for general readers, improvements can be made in the presentation and writing style of the paper. Specifically, organizing the method section more effectively, providing clearer explanations, and reducing the overflow of mathematical expressions would greatly improve understanding. Placing important definitions within the main text instead of footnotes would also aid comprehension.\n\nWhile the proposed method holds promise, the paper would benefit from further demonstration of its effectiveness in practical applications. The choice of datasets used in the experiments may not fully reflect the performance potential of the proposed method. Exploring a wider range of datasets, including those with a higher number of 3-simplices and higher-dimensional simplices, would help evaluate the method more comprehensively.\n\nIn terms of demonstrating the method's effectiveness, the results of the experiments are unclear. On one hand, the proposed method appears to improve the node-classification task, but it seems that 0-simplicies can already be readily incorporated into graphs. On the other hand, the authors mention that the method does not display outstanding performance in the experiment on simplicial closure datasets because it is not designed for a specific downstream task. This raises the question of what would be the proper downstream task that can truly showcase the benefits of the proposed method.\n\nTherefore, it would be valuable to investigate the proper downstream task that can best showcase the advantages of the proposed method. Additionally, understanding how the method addresses the computational burden associated with higher-order simplices would provide valuable insights into its scalability.\n\nQUESTIONS:\n- It would be valuable to explore the proper downstream task that can showcase the true benefits of the proposed method.\n- Additional information on the number of 3-simplices and higher-dimensional simplices in the datasets would help assess the effectiveness of the proposed method.\n- It would be beneficial to understand how the method addresses the computational burden arising from higher-order simplices.\n\nLIMITATIONS:\n- The datasets used in the paper lack higher-order simplices, leading to overparameterization.\n- The effectiveness of the proposed method may not be fully showcased without datasets containing a sufficient number of higher-order simplices.\n- Addressing computational issues associated with higher-order simplices is important.\n- If datasets with an adequate number of higher-order simplices are not readily available, it may be challenging for the proposed method to demonstrate its effectiveness in real-world data without access to extremely large datasets.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel self-supervised learning (SSL) method for simplicial complexes. Advanced results are achieved by introducing appropriate augumentation methods and contrastive loss, taking into account the topological properties of the simplicial complex. It also compares the proposed method with the supervised and SSL methods, including the conventional method for graphs, and shows the effectiveness of the proposed method.\n\nSTRENGTHS:\nThis paper proposes a novel SSL framework for simplicial complexes. The key to SSL, data augmentation and contrastive loss, are analogous to general methods, but they are non-trivial because they require consideration of data-specific characteristics, so it is commendable that a specific framework has been developed.\n\nWEAKNESSES:\nThe concern is what effect the proposed method is intended to have. The experiments section includes a comparison of graph analysis methods and simplical complexes analysis methods and a comparison of supervised and SSL. Some of the references [1-6] include those where the effect is formally certified, but the target seems to be those that are treated as simplical complexes(e.g. polygons with holes). In other words, it is an object that cannot be processed as a graph.\nTherefore, the comparison of the graph method with the simplical complexes method reiterates the superiority of the simplical complexes method, but since the data sets used are not common in graph analysis and are few in number, there is a strong suspicion that the superior data sets have been selected. On the other hand, if the subject is to propose SSL methods for simplical complexes, the main focus is on comparisons within simplical complexes. However, since supervised and SSL cannot be simply compared, it should be clarified how the effect should be recognized.\n\nQUESTIONS:\nPlease give me your opinion on what you stated in Weakness.\n\nLIMITATIONS:\nThe authors do not explicitly address Limitation. The conditions of applicability should be clarified.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new algorithm for self-supervised learning for simplicial complex-based networks.\nThe paper proposes a new simplicial contrastive loss function that contrasts the generated simplices to preserve local and global information present in the simplicial complexes. \nExperiments show the utility of the proposed method. In particular, it outperforms another SSL method for simplicial complexes. Also, its quality is on par with supervised counterparts.\n\nSTRENGTHS:\nThe paper belongs to a new field: topological deep learning, which is a generalization of graph neural networks to more complex structures (simplicial complexes, cell complexes, hypergraphs, etc). There are not so many papers dedicated to this field. Literature review is adequate.\nExperiments show that the proposed SSL method outperforms another (SOTA) SSL method. Confidence intervals are provided. Theoretical results regarding the proposed losses are provided. The quality is almost on par with supervised methods. \nhe language of the manuscript is clear.\n\nWEAKNESSES:\nLearning with simplical complexes is a quite niche research area. So, the impact of the paper is limited.\nAlso, main ideas like using augmented complexes is similar to such augmentation for graphs [1] and the contrastive loss is also quite popular.\nPossible applications of simplicial networks must be discussed in more details.\n\n[1] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning\n429 on graphs. In ICML, pages 4116–4126, 2020.\n\nQUESTIONS:\n1. Can you please explain the Fig.2, right. What is the structure of the dataset? What is the prediction target?\n2. The inter-view cost matrix is used to minimize the distance between the representation of a simplex and an aggregate representation of a sub-simplicial complex surrounding this simplex in the other augmented simplicial complex.\nWhy this value should be minimized?\n3. Why you haven't compared with other SSL baselines in Table 3?\n\nLIMITATIONS:\nAuthors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This work designs an SSL pipeline coined as TopoSRL for simplicial complex data, and proposes a new contrastive loss function to preserve topology information to learn more expressive representations.

STRENGTHS:
1. The motivation is meaningful. 
2. The experiments are fairly comprehensive.

WEAKNESSES:
1. In Table 1 and  Table  3, the baselines are all supervised settings, i.e., lack of comparison with other self-supervised algorithms.
2. The visualization results in Figure 2 do not adequately reflect the differences in clustering ability among the algorithms. Can authors give some numerical evaluations such as AMI and ARI?

QUESTIONS:
In lines 131-132,  the statements about lower-adjacent neighbors and upper-adjacent neighbors are not clear enough.  Can authors give some examples like Figure 1?
In lines 137-138, What does the \phi mean? And the statements like "∀τk+1 ∈ C(σk)" need to move outside.

LIMITATIONS:
N/A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes a Self-Supervised Learning (SSL) approach for simplicial complexes, extending graph SSL to higher-order structures. While recent works exist that extend and adapt standard graph representation learning techniques to simplicial complexes, none consider SSL. In particular, the authors propose to employ a contrastive learning procedure that involves two steps: first, for each complex, two artificial simplicial complexes are created with a data augmentation technique that randomly removes closed simplices and randomly adds open ones to the complex. Then, the complexes are contrasted with an ad hoc contrastive loss that has two goals: the first is to minimize the distance of each k-simplex across the two augmented views, and the second is to minimize the distance of a $k$-simplex and the aggregated representation of a sub-simplicial complex surrounding it in the other view. The per-simplex distances are simply given by the squared euclidean distance. The authors then prove that the proposed contrastive loss implicitly maximizes the mutual information between a simplex and its neighborhood within the same augmented simplex and the mutual information between a simplex and its neighbors in the other augmented complex. The approach is tested over node classification and simplicial closure for a set of social network datasets, and shows competitive performance wrt a set of supervised baselines both acting on graphs and simplicial complexes. The work is also shown to outperform an existing graph SSL baseline.\n\nSTRENGTHS:\n- The paper is well written and easy to understand, providing the reader the necessary background and explaining the contribution in simple terms.\n- The work is the first to study the problem of self-supervised learning on simplicial complexes, noting that existing graph SSL methods may not be directly adapted. Given the recent raise of interest in the field, this work may well be the first of many.\n- The approach makes intuitive sense, is simple and has theoretical justification. Being model agnostic, the approach can (and has been) used on any simplicial complex encoder.\n- The experimental evidence seems to indicate that, on the considered tasks, the approach is surprisingly competitive with supervised baselines.\n\nWEAKNESSES:\n- Part of the reason why there are no works on self-supervised learning on simplicial complexes is that data distributions that can be naturally designed as simplicial complexes are rare, and the work indeed only considers social network datasets that are only seen in works of the same niche. For this reason, it would be interesting to see whether the approach still shows benefits on a standard graph dataset converted to complex with lifting technique (e.g. using clique complexes). Indeed, main works in the field, e.g. [1], usually also compare on established graph datasets such as TUDatasets. \n- The method only presents a single graph SSL baseline, assessing it to be the state of the art. In general, it would still be desirable to have several baselines for two reasons: i) the considered SOTA model could prove to be better only in the datasets, tasks or evaluation setting considered in the original paper, and ii) the comparison with less-performing baselines would still be valuable to the experimenter as well as the reader.\n- It is not immediately clear how the supervised graph-learning baselines were chosen, and why no chosen network is at least 1-WL expressive. It would be good to at least report the results for GIN [2].\n- The method and insights may not be of immediate interest to the broader NeurIPS community as they are specifically tailored to simplicial complexes, and would probably be of greater interest in a graph-specific venue. While the venue also accepts domain-specific works, these usually have more general insight that can be appreciated by practitioners outside the niche. \n- Much of the utility of self-supervised learning comes when it can be used for pretraining an encoder on a large dataset that can be later fine-tuned for a downstream task. It would be great to see if this is the case for the proposed approach, as otherwise it's not immediately clear why it would be advisable wrt to standard supervised models.\n\nOverall, I am more than happy to raise my score if my concerns are addressed.\n\n**References**: \n\n[1] Bodnar, C., Frasca, F., Wang, Y., Otter, N., Montufar, G. F., Lio, P., & Bronstein, M. (2021, July). Weisfeiler and lehman go topological: Message passing simplicial networks. In International Conference on Machine Learning (pp. 1026-1037). PMLR.\n\n[2] Xu, K., Hu, W., Leskovec, J., & Jegelka, S. (2018). How powerful are graph neural networks?. arXiv preprint arXiv:1810.00826.\n\nQUESTIONS:\n- Table 2: to be called an ablation study, shouldn’t it also consider the case in which the closed simplices are not removed while the open simplices are still added?\n- Are the results averaged over different seeds? how many runs were performed?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}