SlowGuess/NeurIPS_2023_OCR huggingface dataset:
TopoSRL: Topology Preserving Self-Supervised Simplicial Representation Learning
Hiren Madhu and Sundeep Prabhakar Chepuri
Indian Insitute of Science
hirenmadhu, spchepuri@iisc.ac.in
In this paper, we introduce TopoSRL, a novel self-supervised learning (SSL) method for simplicial complexes to effectively capture higher-order interactions and preserve topology in the learned representations. TopoSRL addresses the limitations of existing graph-based SSL methods that typically concentrate on pairwise relationships, neglecting long-range dependencies crucial to capturing topological information. We propose a new simplicial augmentation technique that generates two views of the simplicial complex that enriches the representations while being efficient. Next, we propose a new simplicial contrastive loss function that contrasts the generated simplices to preserve local and global information present in the simplicial complexes. Extensive experimental results demonstrate the superior performance of TopoSRL compared to state-of-the-art graph SSL techniques and supervised simplicial neural models across various datasets corroborating the efficacy of TopoSRL in processing simplicial complex data in a self-supervised setting.
Simplicial complexes are mathematical structures that explicitly capture higher-order relationships between entities (as nodes) using simplices of different orders (as edges, triangles, and so on). There has been a growing interest in developing simplicial representation learning models, such as simplicial neural networks (SNN) [1-6], as simplicial complexes are generally more expressive than graphs, which only capture pairwise relations. SNNs incorporate topological information available in simplicial complexes while learning representations of simplices of different orders, which are useful for various downstream tasks such as node, graph, higher-order link, and trajectory prediction tasks [1, 4-6]. However, a significant challenge in learning representations for simplicial complexes using existing SNN models is the need for task-specific labels required for training. Acquiring meaningful labels for real-world, high-dimensional, and complex data is difficult due to intricate structures, multiple valid labeling schemes, or privacy and ethical concerns.
Self-supervised learning (SSL) schemes learn expressive and powerful representations without requiring labeled data. Specifically, the main goal in SSL is to model an encoder, which is learned using an objective function and unlabeled training data. This paper proposes an SSL method for simplicial complex data that preserves topological and geometric information while learning representations. Although no existing studies focus on SSL for simplicial complex data, a closely related field of SSL for graph data has been extensively studied. SSL for simplicial complex data is an important generalization as every simplicial complex inherently includes an underlying graph, making SSL on graphs a specialized subset of SSL on simplicial complexes.
The general idea behind SSL on graphs is to augment a graph to create two views of the available graph and then maximize the mutual information (MI) between the augmented graphs. So, the research focus thus far has been on designing augmentation techniques and objective functions that maximize MI. However, existing methods of SSL on graphs require complex augmentation [7], negative
sampling algorithms [8], or require components to empirically avoid degenerative solutions [9-12]. This leads to a more complex neural model, hindering their direct extension for SSL on simplicial complexes. For instance, deep graph infomax (DGI) [9] maximizes MI between a node and its subgraph by learning two MLPs: one for the subgraph readout and another as a discriminator function classifying if a node exists in the given subgraph or not. Extending DGI to simplicial complexes would necessitate training  $k + 1$  discriminators for a simplicial complex of order  $k$  (see Section 3 for the definition of the order of a simplicial complex), drastically increasing the training complexity. Similarly, graph contrastive representation learning (GRACE) [8] and graph contrastive representation learning with adaptive augmentation (GCA) [7] need the selection of effective negative samples and require additional storage by selecting all the other nodes in the graph as negative samples. Directly extending these methods to simplicial complexes would lead to prohibitive computational complexity going up to  $\mathcal{O}(2^{N})$  for a simplicial complex with  $N$  nodes. To mitigate the need for negative samples or additional networks, recent approaches like bootstrapped graph representation learning (BGRL) [11], SelfGNN [12], and canonical correlation analysis inspired self-supervised learning on graphs (CCA-SSG) [13] attempt to learn representations by contrasting corresponding node pairs in the two augmented graphs. Self-supervised masked graph autoencoders (GraphMAE) [14] is an alternative approach that uses a reconstruction loss rather than the commonly used contrastive loss in the graph-based SSL methods. However, these methods focus only on contrasting or reconstructing local information and do not account for global long-distance information available in the network.
Motivated by the aforementioned limitations of SSL methods on graphs, we introduce TopoSRL, a self-supervised learning pipeline for simplicial complex data that preserves topology information in the representation space. Preserving topology information is crucial because it allows the learned representations to capture higher-order interactions and relationships unique to simplicial complexes. TopoSRL comprises an intuitive technique to generate stochastically augmented views of a simplicial complex. We introduce a new contrastive loss function to preserve topology information of simplicial complexes in the geometric space to learn more expressive representations. In sum, our major contributions in TopoSRL are as follows:
- Simplicial augmentation: We introduce a simple and effective augmentation technique for simplicial complexes that captures the inherent relationships between simplices. In particular, the augmentation method stochastically removes closed simplices and adds open simplices guided by the topological structure of a simplicial complex, where a closed simplex is a simplex that is part of the simplicial complex, while an open simplex is a simplex that itself is not a part of the simplicial complex, but all of its subsets are. This procedure is computationally efficient and leads to superior performance than randomly adding simplices without accounting for the topological structure of a simplicial complex.
- Simplicial contrastive loss: We propose a loss function for SSL on simplicial complexes that contrasts pairs of corresponding simplices as well as considers the relational distance between pairs of simplices and their augmented counterparts, where the relational distance refers to a measure of dissimilarity between pairs of simplices. We also provide theoretical evidence that the proposed loss function implicitly maximizes MI between a simplex and its neighborhood within the same augmented simplex and the other augmented simplex, helping the model capture the inherent structures and patterns in the data, thereby improving the model's performance and adaptability to various tasks without the requirement of training additional components such as MI estimators.
We conduct experiments demonstrating our proposed method on downstream tasks such as node classification, simplicial closure, graph classification, and trajectory prediction. We also highlight the effectiveness of TopoSRL in learning expressive representations for simplicial complexes with the proposed augmentation technique as opposed to the random augmentation technique. Experiments show that without any complex architectures or expensive augmentation techniques, our method outperforms existing state-of-the-art graph representation learning methods while being competitive with supervised simplicial representation learning methods.
This section briefly discusses a few popular SNN models, one of which can be used in TopoSRL. The first step of TopoSRL's learning phase is simplicial augmentation. Hence, we also discuss existing
augmentation techniques used in graph SSL and how they can not be extended (due to computationally or empirical limitations) for simplicial complexes. Then, we move on to discussing SSL on graphs, a specialized version of SSL on simplicial complexes.
Simplicial representation learning. SNN models learn representations of simplices of different orders (e.g., nodes, edges, triangles, and so on) in a simplicial complex [2, 3], and are based on an extension of graph convolutions to convolutions over simplicial complexes. Simplicial attention networks (SAN) [1] extends graph attention networks (GAT) [15] for simplicial complexes. Message passing simplicial networks (MPSN) [5] proposes a framework to design more expressive SNN models related to the so-called simplicial Weisfeiler-Lehman isomorphism test. TopoSRL is free from the choice of a specific SNN encoder, and any one of the SSN models can be used as an encoder.
Graph augmentation. Most of the graph SSL methods described in Section 1, namely, GRACE, GCA, and BGRL, use an augmentation technique that includes two steps: (i) add and remove edges at random to generate two graphs from an input graph, and (ii) randomly mask dimensions of initial node features. GCA [7] proposes an adaptive data augmentation technique that models the edge-removal probability differently for each edge in the graph based on the importance of the edge. However, this method requires more expensive augmentations to attain peak performance [11]. Applying an identical augmentation technique to simplicial complexes necessitates modeling removal probabilities for each simplex. This increases the augmentation process's complexity and mandates the development of innovative metrics for measuring simplex importance. Other than this, most methods use uniform probability to add and remove edges. Although adding and removing simplices uniformly at random is a naive and simple extension, we empirically show that TopoSRL performs considerably better than random augmentation. In contrast, TopoSRL uses a simpler augmentation technique that only requires the addition of open simplices, which are inherently rich in information as discussed later in Section 4.1, and is very easy to implement.
SSL on graphs. Contrastive learning methods for images have recently been adapted for graphs. This includes DGI [9], which contrasts node-local patches with global graph representations was inspired by Deep InfoMax [16]. GMI [17] maximizes a concept of graphical mutual information inspired by MINE [18], enabling a more granular contrastive loss than DGI. GRACE [8] and its derivatives, such as GCA [7], which rely on more complex data adaptive enhancements, have adapted the SimCLR [19] algorithm for graphs. GraphCL [20] also adapts SimCLR to learn graph-level representations with a contrastive objective. Multi-view graph representation learning (MVGRL) [10] extends contrastive multi-view coding to graphs. All these methods suffer from considerable computational complexities because they rely heavily on negative samples. BGRL [11] and SelfGNN [12] extend Bootstrap you own latent (BYOL) [21], which uses different online and target encoders, wherein the target encoder is updated as an exponential moving average of the online encoder while the online encoder is updated by optimizing a loss function. The difference between BGRL and SelfGNN is that BGRL uses different online and target encoders, but SelfGNN utilizes the same encoder as online and target. Lastly, CCA-SSG[13] uses CCA-based loss. BGRL, SelfGNN, and CCA-SSG are methods free from negative samples and incur lower computational complexity than previous methods. Inspired by these graph SSL methods, TopoSRL aims at a negative sampling-free approach focusing on preserving topology and implicitly maximizing MI.
A simplicial complex  $\mathcal{X}$  is a collection of a finite number of simplices. A simplex of order  $k$  (or, a  $k$ -simplex) is a  $(k + 1)$ -cardinality subset of the vertex set  $\mathcal{V}$  so that if a simplex  $\sigma_k \in \mathcal{X}$  then all the non-empty subsets of  $\sigma_k$  also belong to the simplicial complex  $\mathcal{X}$ . The order,  $K$ , of a simplicial complex  $\mathcal{X}$  is the order of the maximally ordered simplex in the simplicial complex.
Suppose  $\sigma_{k}$ ,  $\tau_{k}$  and  $\rho_{k}$  are some  $k$ -simplices in a simplicial complex. A  $k$ -simplex  $\sigma_{k}$  has the following neighbors in the simplicial complex, namely, lower-adjacent neighbors  $\mathcal{L}(\sigma_k) = \{\tau_k|\rho_{k - 1}\subset \sigma_k\wedge \rho_{k - 1}\subset \tau_k\}$ , upper-adjacent neighbors  $\mathcal{U}(\sigma_k) = \{\tau_k|\sigma_k\subset \rho_{k + 1}\wedge \tau_k\subset \rho_{k + 1}\}$ , boundary  $\mathcal{B}(\sigma_k) = \{\tau_{k - 1}|\tau_{k - 1}\subset \sigma_k\}$ , and co-boundary  $\mathcal{C}(\sigma_k) = \{\tau_{k + 1}|\sigma_k\subset \tau_{k + 1}\}$ .
An open  $k$ -simplex is defined as a  $k$ -simplex  $\sigma_k$  such that all its boundaries  $B(\sigma_k) \subset \mathcal{X}$ , but  $\sigma_k \notin \mathcal{X}$ . In other words, if all the boundaries of a simplex are present in the simplicial complex, but the simplex itself is not part of the simplicial complex, it is considered an open simplex.
Figure 1: (a) The TopoSRL pipeline. Here,  $\mathbf{W}$  is a weight matrix that emphasizes the relations between simplices in the two augmented simplicial complexes. (b) Simplicial augmentation example.
A generic SNN model has the following aggregation steps
$$
m _ {B} ^ {t + 1} \left(\sigma_ {k}\right) = \mathbf {A G G R E G A T E} \left(\phi_ {B} \left(h _ {\sigma_ {k}} ^ {t}, h _ {\tau_ {k - 1}} ^ {t}, \forall \tau_ {k - 1} \in \mathcal {B} \left(\sigma_ {k}\right)\right) \right.
$$
$$
m _ {C} ^ {t + 1} \left(\sigma_ {k}\right) = \operatorname {A G G R E G A T E} \left(\phi_ {C} \left(h _ {\sigma_ {k}} ^ {t}, h _ {\tau_ {k + 1}} ^ {t}, \forall \tau_ {k + 1} \in \mathcal {C} \left(\sigma_ {k}\right)\right)\right)
$$
$$
m _ {L} ^ {t + 1} \left(\sigma_ {k}\right) = \mathsf {A G G R E G A T E} \left(\phi_ {L} \left(h _ {\sigma_ {k}} ^ {t}, h _ {\tau_ {k}} ^ {t}, \forall \tau_ {k} \in \mathcal {L} \left(\sigma_ {k}\right)\right)\right)
$$
$$
m _ {U} ^ {t + 1} \left(\sigma_ {k}\right) = \operatorname {A G G R E G A T E} \left(\phi_ {U} \left(h _ {\sigma_ {k}} ^ {t}, h _ {\tau_ {k}} ^ {t}, \forall \tau_ {k} \in \mathcal {U} \left(\sigma_ {k}\right)\right)\right)
$$
and a combining step
$$
h _ {\sigma_ {k}} ^ {t + 1} = \operatorname {C o m b i n e} \left(h _ {\sigma_ {k}} ^ {t}, m _ {B} ^ {t + 1} \left(\sigma_ {k}\right), m _ {C} ^ {t + 1} \left(\sigma_ {k}\right), m _ {L} ^ {t + 1} \left(\sigma_ {k}\right), m _ {U} ^ {t + 1} \left(\sigma_ {k}\right)\right),
$$
where  $t$  is the layer index of the SNN model,  $\phi_{n}$  is the transformation function with adjacency of type  $n$  (We use one layer MLP),  $h_{\sigma_k}^t$  is the representation of the simplex  $\sigma_{k}$  at layer  $t$  with  $h_{\sigma_k}^0$  being the initial embedding. We write the final representations of all the simplices in the simplicial complex  $\mathcal{X}$  obtained from such an SNN model with  $L$  layers compactly as
$$
\mathbf {Z} = \mathcal {E} _ {\theta} (\mathcal {X}, \mathbf {H}),
$$
where  $\theta$  denotes the set of learnable parameters and  $\mathbf{H}$  is the initial feature matrix of all the simplices in  $\mathcal{X}$ . Although the above SNN model is based on MPSN, different SNN variants can be derived from the above model by choosing different AGGREGATE and COMBINE operators and which neighborhoods to aggregate.
4 The proposed TopoSRL pipeline
The main aim of the proposed approach is to learn an SNN encoder function  $\mathcal{E}_{\theta}(\cdot)$  to compute expressive representations for simplices in a simplicial complex  $\mathcal{X}$  in a self-supervised manner. Towards this end, we first compute two augmented views of a simplicial complex  $\mathcal{X}$ , namely,  $\mathcal{X}^{(1)}$  and  $\mathcal{X}^{(2)}$  (as described later on Section 4.1). Then we learn the representations of the two simplicial complexes through the encoder  $\mathcal{E}_{\theta}(\cdot)$  as  $\mathbf{Z}^{(1)} = \mathcal{E}_{\theta}(\mathcal{X}^{(1)},\mathbf{H}^{(1)})$  and  $\mathbf{Z}^{(2)} = \mathcal{E}_{\theta}(\mathcal{X}^{(2)},\mathbf{H}^{(2)})$ , where  $\mathbf{Z}^{(i)} = \{\mathbf{Z}_1^{(i)},\mathbf{Z}_2^{(i)},\dots,\mathbf{Z}_K^{(i)}\}$  is the set embedding matrices and  $\mathbf{Z}_k^{(i)}\in \mathbb{R}^{N_k^{(i)}\times D}$  being the  $D$ -dimensional embedding matrix of all the  $k$ -simplices in the simplicial complex  $\mathcal{X}^{(i)}$  and  $N_{k}^{(i)}$  being the total number of  $k$ -simplices in  $\mathcal{X}^{(i)}$ . We contrast these representations and learn  $\mathcal{E}_{\theta}(\cdot)$  by optimizing the simplicial contrastive loss (developed in Section 4.2). An illustration of the TopoSRL pipeline is shown in Figure 1(a).
4.1 Simplicial augmentation
We introduce a novel stochastic augmentation technique for simplicial complexes to generate two simplicial complexes  $\mathcal{X}^{(1)}$  and  $\mathcal{X}^{(2)}$  from  $\mathcal{X}$  by adding open simplices and removing closed simplices at random via Bernoulli sampling. To begin with, we compute all the open simplices in  $\mathcal{X}$ . Next,
1Identifying open simplices of order  $k + 1$  in a simplicial complex  $\mathcal{X}$  costs  $\mathcal{O}(N_k^2)$ , where  $N_{k}$  is the total number of  $k$ -simplices. We only compute it once outside the training process.
Algorithm 1 Simplicial complex augmentation
1: procedure SIMPLICIALAUGMENTATION  $(\mathcal{X},\rho)$   
2:  $\mathcal{X}_{\mathrm{open}}\gets 0$  openSimplices  $(\mathcal{X})$  ▷ Compute open  $k$  -simplices in  $\mathcal{X}$  with  $k\geq 2$   
3: for  $i = 1$  and 2 do  
4:  $\mathcal{X}^{(i)}\gets \mathcal{X}$   
5: for each  $\sigma_{k}\in \mathcal{X}$  with  $k > 1$  do  
6: Draw  $r\sim$  Bernoulli  $(\rho)$   
7: if  $r = 1$  then ▷remove a closed simplex  
8:  $\mathcal{X}^{(i)}\gets \{\mathcal{X}^{(i)}\setminus \sigma_k\}$   
9: for each  $\sigma \in \mathcal{X}_{\mathrm{open}}$  do  
10: Draw  $a\sim$  Bernoulli  $(\rho)$   
11: if  $a = 1$  then ▷add a open simplex  
12:  $\mathcal{X}^{(i)}\gets \{\mathcal{X}^{(i)}\cup \sigma \}$   
13: return  $\mathcal{X}^{(1)}$  and  $\mathcal{X}^{(2)}$
we draw a Bernoulli random variable for each open simplex to determine whether to include the open simplex in  $\mathcal{X}^{(i)}$  for  $i = 1,2$ . Similarly, we draw another Bernoulli random variable for each  $k$ -simplex ( $k > 1$ ) in  $\mathcal{X}$  to determine whether to retain the closed simplex in  $\mathcal{X}^{(i)}$  for  $i = 1,2$ . This procedure is summarized as Algorithm 1.
The proposed augmentation method preserves the inherent structure and relationships within the simplicial complex as illustrated in Figure 1(b). For example, consider the Walmart-trips simplicial complex dataset<sup>2</sup>, wherein a set of items purchased together by a customer in one trip to Walmart is a simplex. Now, if a customer purchases {peanut butter and jam}, {bread and jam}, and {peanut butter and bread} on three separate occasions, it is highly probable that the customer will buy {peanut butter, jam, bread} together on a subsequent trip. Adding the simplex {peanut butter, jam, bread} as in view 2 of Figure 1(b) will represent an instance of the simplicial complex if the data were sampled at a later point in time, and hence it still retains and provides better information to contrast compared to adding random simplices.
In contrast, in a random sampling method, where we add simplices at random, we need to include all the subsets of the simplex that is to be added, which costs  $\mathcal{O}(2^k)$  for each  $k$ -order simplex for  $k \in \{2, \ldots, K\}$ . As opposed to this, in the proposed simplicial augmentation technique, since we are adding an open simplex, all the subsets of this simplex are present in the simplicial complex, making the additional cost only  $\mathcal{O}(1)$  and thereby making it computationally more efficient.
4.2 Simplicial contrastive loss
The proposed method aims to learn simplex representations by capturing topological information in higher-order simplices and their distant neighbors. In contrast, most of the existing self-supervised graph representation learning approaches primarily focus on contrasting corresponding nodes in augmented graphs and do not preserve topological information as the objective function does not focus on the relational distance between pairs of two nodes that are not neighbors.
Cost matrices. We define the following cost matrices to compute the TopoSRL loss function: the intra-view cost matrix, denoted by  $\mathbf{C}_k^{(i)}\in \mathbf{R}^{N_k^{(i)}\times N_k^{(i)}}$  for  $i = 1,2$ , measures the distance between representations of two simplices in the simplicial complex  $\mathcal{X}^{(i)}$  and the inter-view cost matrix, denoted by  $\mathbf{C}_k^{(1,2)}\in \mathbf{R}^{N_k^{(1)}\times N_k^{(2)}}$ , measures the distance between the  $k$ -simplex in the two views. The intra-view cost matrix is used to minimize the relational distance between pairs of simplices from two augmented simplicial complexes. Specifically, the  $(p,q)$ th entry of  $\mathbf{C}_k^{(i)}$ , is defined as  $[\mathbf{C}_k^{(i)}]_{p,q} = \| [\mathbf{Z}_k^{(i)}]_p - [\mathbf{Z}_k^{(i)}]_q\| _2^2$ , where  $[\mathbf{Z}_k^{(i)}]_p$  is the representation of the  $k$ -simplex  $p$  in  $\mathcal{X}^{(i)}$ . On the other hand, the inter-view cost matrix is used to minimize the distance between the representation of a simplex and an aggregate representation of a sub-simplicial complex surrounding
$^{2}$ https://www.cs.cornell.edu/\~arb/data/walmart-trips/
this simplex in the other augmented simplicial complex. The  $(p,q)$ th entry of  $\mathbf{C}_k^{(1,2)}$  is defined as  $[\mathbf{C}_k^{(1,2)}]_{p,q} = \left\| [\mathbf{Z}_k^{(1)}]_p - [\mathbf{Z}_k^{(2)}]_q\right\| _2^2$ .
Weight matrix. We also construct a weight matrix  $\mathbf{W}_k \in \mathbb{R}^{N_k^{(1)} \times N_k^{(2)}}$  for each  $k$ -simplex to capture the relation of a simplex  $\sigma_k^{(1)} \in \mathcal{X}^{(1)}$  and  $\sigma_k^{(2)} \in \mathcal{X}^{(2)}$ , which will be used to calculate the aggregate representation of sub-simplicial complexes by assigning different importance weights that depend on how many hops  $\sigma_k^{(2)}$  is away from  $\sigma_k^{(1)}$  in the simplicial complex  $\mathcal{X}^{(1)}$ . We assign the entries of  $\mathbf{W}_k$  as follows. If  $\sigma_k^{(2)} = \sigma_k^{(1)}$ , we assign a higher value  $\eta_0$  to  $[\mathbf{W}_k]_{\sigma_k^{(1)},\sigma_k^{(2)}}$ . If  $\sigma_k^{(2)} \in \mathcal{L}(\sigma^{(1)}) \cup \mathcal{U}(\sigma^{(1)})$ , i.e., if  $\sigma_k^{(2)}$  is in one-hop neighborhood of  $\sigma_k^{(1)}$  in  $\mathcal{X}^{(1)}$ , then we assign it a lower value  $\eta_1$ . Similarly, if  $\sigma_k^{(2)}$  is in the two-hop neighborhood of  $\sigma_k^{(1)}$  in  $\mathcal{X}^{(1)}$ , we assign a value of  $\eta_2$ , such that  $\eta_0 > \eta_1 > \eta_2$  and then we apply row-wise softmax to  $\mathbf{W}_k$  to normalize it. By construction,  $\mathbf{W}_k$  is a row-stochastic matrix.
Loss function. The proposed simplicial contrastive loss is a convex combination of two terms, namely, the sub-simplicial complex loss  $\mathcal{L}_{\mathrm{sub}}$  and the relative simplicial complex loss  $\mathcal{L}_{\mathrm{rel}}$ . The term  $\mathcal{L}_{\mathrm{sub}}$  measures the cumulative (over all the simplices) weighted distance between the representation of a  $k$ -simplex  $\sigma_k^i \in \mathcal{X}^{(1)}$  and  $k$ -simplices  $\sigma_k^{i'} \in \mathcal{X}^{(2)}$  with appropriate weights in the matrix  $\mathbf{W}_k$ , and is given by