SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {\mathrm {s u b}} = \sum_ {k = 0} ^ {K} \sum_ {i = 1} ^ {N _ {k} ^ {(1)}} \sum_ {j = 1} ^ {N _ {k} ^ {(2)}} \left[ \mathbf {C} _ {k} ^ {(1, 2)} \right] _ {i, j} \left[ \mathbf {W} _ {k} \right] _ {i, j}. \tag {1}
$$
The term  $\mathcal{L}_{\mathrm{rel}}$  calculates the relational distance between pairs of simplices and is given by
$$
\mathcal {L} _ {\text {r e l}} = \sum_ {k = 0} ^ {K} \sum_ {i, j = 1} ^ {N _ {k} ^ {(1)}} \sum_ {i ^ {\prime}, j ^ {\prime} = 1} ^ {N _ {k} ^ {(2)}} \left(\left[ \mathbf {C} _ {k} ^ {(1)} \right] _ {i, j} - \left[ \mathbf {C} _ {k} ^ {(2)} \right] _ {i ^ {\prime}, j ^ {\prime}}\right) ^ {2} \left[ \mathbf {W} _ {k} \right] _ {i, i ^ {\prime}} \left[ \mathbf {W} _ {k} \right] _ {j, j ^ {\prime}}. \tag {2}
$$
For instance, consider two pairs of simplices  $(\sigma_k^i,\sigma_k^j)$  and  $(\sigma_k^{i'},\sigma_k^{j'})$  such that  $\sigma_k^i,\sigma_k^j\in \mathcal{X}_k^{(1)}$ $\sigma_{k}^{i^{\prime}},\sigma_{k}^{j^{\prime}}\in \mathcal{X}_{k}^{(2)},[\mathbf{W}_{k}]_{\sigma_{k}^{i},\sigma_{k}^{i^{\prime}}} > 0$  , and  $[\mathbf{W}_k]_{\sigma_k^j,\sigma_k^{j'}} > 0$  , that is,  $(\sigma_k^i,\sigma_k^{i'})$  and  $(\sigma_k^j,\sigma_k^{j'})$  are distant neighbors (not necessarily one-hop). Hence, minimizing  $\mathcal{L}_{\mathrm{rel}}$  minimizes the squared difference of the distance between the representations of  $(\sigma_k^i,\sigma_k^j)$  and  $(\sigma_k^{i'},\sigma_k^{j'})$  , which leads to similar pairs of simplices from two distinct simplicial complexes maintaining an equal distance. In other words, if  $\sigma_k^j$  is  $m$  -hop away from  $\sigma_k^i$  and  $\sigma_k^{j'}$  is  $m$  -hop away from  $\sigma_k^{i'}$  , then minimizing  $\mathcal{L}_{\mathrm{rel}}$  will reduce the difference between the distance of the pairs and preserve the  $m$  -hop information present in  $\mathcal{X}$  Finally, the overall loss is
$$
\mathcal {L} = \alpha \mathcal {L} _ {\text {s u b}} + (1 - \alpha) \mathcal {L} _ {\text {r e l}},
$$
where  $\alpha \in [0,1]$  is a tunable parameter. The overall loss preserves the topological properties as adjacent simplices will be embedded more closely in the Euclidean space because of the  $\mathcal{L}_{\mathrm{sub}}$  term, capturing the local information. The difference in distance of contrasting simplex pairs will be minimized by minimizing  $\mathcal{L}_{\mathrm{rel}}$ , capturing the global information.
We end this section with two interesting theoretical results about the simplicial contrastive loss function and an overview of the TopoSRL pipeline in Algorithm 2. In the next proposition, we show that  $\mathcal{L}_{\mathrm{sub}}$  preserves the local information in a simplicial complex.
Proposition 1. Minimizing  $\mathcal{L}_{\mathrm{sub}}$  is equivalent to jointly minimizing a lower bound on the distance between the representation of a simplex  $\sigma_k^i \in \mathcal{X}^{(1)}$  and the aggregate representation of  $\sigma_k^{i'} \in \mathcal{X}^{(2)}$  that is adjacent to  $\sigma_k^i$  and the distance between the representation of  $\sigma_k^{i'} \in \mathcal{X}^{(2)}$  and the aggregate of representations of  $\sigma_k^{i'} \in \mathcal{X}^{(2)}$  that is adjacent to  $\sigma_k^i \in \mathcal{X}^{(1)}$ .
Most graph SLL methods maximize MI between augmented graphs using MI estimators [9]. In the next theorem, we show that optimizing the proposed loss function is also related to optimizing MI between representations of simplices in one augmented simplicial complex and representations of
Proofs are available in the supplementary material.
Algorithm 2 TopoSRL
1: procedure TOPOSRL  $(\mathcal{X},\rho ,\kappa ,\mathbf{H})$ $\triangleright \kappa$  is the maximum order of the simplices of interest   
2: Initialize  $\theta$    
3:  $e\gets 1$    
4: while  $e\leq \#$  epochs do   
5:  $\mathcal{X}^{(1)},\mathcal{X}^{(2)}\gets$  SimplicialAugmentation  $(\mathcal{X},\rho)$    
6: Calculate  $\mathbf{W}_k$  for  $k = 0,\dots ,\kappa$    
7:  $\mathbf{Z}^{(1)}\gets \mathcal{E}_{\theta}(\mathcal{X}^{(1)},\mathbf{H}^{(1)})$  and  $\mathbf{Z}^{(2)}\gets \mathcal{E}_{\theta}(\mathcal{X}^{(2)},\mathbf{H}^{(2)})$    
8: Calculate  $\mathbf{C}_k^{(1)},\mathbf{C}_k^{(2)},\mathbf{C}_k^{(12)}$  for  $k = 0,\dots ,\kappa$    
9: Minimize  $\mathcal{L} = \alpha \mathcal{L}_{\mathrm{sub}} + (1 - \alpha)\mathcal{L}_{\mathrm{rel}}$    
10:  $e = e + 1$    
11:  $\mathbf{Z}\gets \mathcal{E}_{\theta}(\mathcal{X},\mathbf{H})$    
12: return Z
its neighbors in the other augmented simplicial complex and representations of simplices and their neighborhoods within the same augmented simplicial complex conditioned on the input data.
Suppose  $X$  represents the random variable corresponding to the input data,  $X_{k}$  represents the  $k$ -simplices in  $X$ , and  $X^{(i)}$  represents an augmented view of  $X$ , sharing the same sample space as  $X$ , and the input simplicial complex data  $\mathcal{X} \sim X$ . TopoSRL is designed to learn representations, denoted as  $\mathbf{Z}$  for the input data and  $\mathbf{Z}^{(i)}$  for its augmentation. We also denote  $H(A)$  and  $I(A,B)$  as the entropy of the random variable  $A$  and the MI between the random variables  $A$  and  $B$ , respectively.
Theorem 1. Minimizing the expected loss  $\mathcal{L}_{\mathrm{sub}}$  (expectation is with respect to the random variable  $X$ ) is equivalent to maximizing the MI between  $\mathbf{Z}_k^{(i)}$  and  $X_{k}$ , i.e.,
$$
\underset {\theta} {\operatorname {m i n i m i z e}} \quad \mathcal {L} _ {\text {s u b}, k} \equiv \underset {\theta} {\operatorname {m a x i m i z e}} \quad I \left(\mathbf {Z} _ {k} ^ {(i)}, X _ {k}\right), \tag {3}
$$
for  $k = 0,1,\ldots ,K$ , where  $\mathcal{L}_{\mathrm{sub},k}$  is the  $k$ th summand in (1),  $I(\mathbf{Z}_k^{(i)},X_k)$  is the  $MI$  between the representations of the augmented  $k$ -simplices  $\mathbf{Z}_k^{(i)}$  and the  $k$ -simplices in the original data  $X_{k}$ .
The above theorem shows that in expectation, minimizing the term  $\mathcal{L}_{\mathrm{sub}}$  is equivalent to minimizing the variance between representations in the augmented simplicial complexes, which leads to minimization of the conditional entropy  $H(\mathbf{Z}_k^{(i)}|X_K)$ , implying the maximization of MI as  $I(\mathbf{Z}_k^{(i)},X_k) = H(\mathbf{Z}_k^{(i)}) - H(\mathbf{Z}_k^{(i)}|X_k)$ . The proof also shows that minimizing the simplicial contrastive loss implicitly maximizes MI between simplices from one augmented simplicial complex and representations of its neighbors in the other augmented simplicial complex conditioned on the input and representations of simplices and their neighborhoods within the same augmented simplicial complex. This result is along the lines of earlier methods like DGI [9] and InfoGraph [22]. However, TopoSRL does not require additional components in DGI or InfoGraph for MI maximization.
We follow the standard setting of self-supervised learning methods. Firstly, we train the encoder with the proposed simplicial contrastive loss. Next, we freeze the encoders' model parameters, extract representations for all the simplices, and train a classifier for the following two downstream tasks. The code is available at https://github.com/HirenMadhu/TopoSRL.
Downstream tasks. We focus on node classification, simplicial closure, trajectory prediction and graph classification tasks. Node classification focuses on predicting labels for 0-simplices (aka nodes) in a given simplicial complex. We perform node classification on the following publicly available datasets<sup>4</sup>, namely, contact-primary-school, contact-high-school, senate-bills. We use classification accuracy as the performance metric. In simplicial closure, the aim is to predict the closure of open simplices in a time series of simplicial complex data. We perform simplicial
<sup>4</sup>Dataset details are presented in the supplementary material. Datasets are available at https://www.cs.cornell.edu/~arb/data/
Method	Type	high-school	primary-school	senate-bills
GCN	S	0.4±0.04	0.30±0.04	0.67±0.06
GraphSage	S	0.27±0.05	0.37±0.05	0.54±0.03
GIN	S	0.18±0.04	0.16±0.02	0.53±0.04
GAT	S	0.34±0.05	0.19±0.06	0.5±0.04
CCA-SSG	SSL	0.68±0.16	0.14±0.07	0.62±0.04
GCA	SSL	0.18±0.08	0.12±0.05	0.5±0.0
BGRL	SSL	0.11±0.01	0.09±0.01	0.5±0.0
GraphMAE	SSL	0.78±0.05	0.2±0.02	0.57±0.01
SAN	S	0.86±0.04	0.29±0.06	0.53 ± 0.09
SCNN	S	0.81 ± 0.01	0.67±0.04	0.615 ± 0.05
MPSN	S	0.89 ± 0.01	0.79 ± 0.06	0.75 ± 0.05
TopoSRL	SSL	0.92 ± 0.05	0.61 ± 0.05	0.72±0.06
Table 1: Node classification accuracies on simplicial complex datasets; S stands for supervised setting, and SSL stands for self-supervised setting. (Best accuracy is bold and second best accuracy is underline)
closure on email-Eu, email-Enron, contact-high-school datasets. We first split the data across time on these temporal datasets and then train the encoder on the first  $80\%$  of the data. The last  $20\%$  is used for inference. Since the class distribution is very skewed, we use F1-macro to evaluate the performance. Trajectory prediction focuses on predicting the next node in a trajectory, given a sequence of nodes. We perform experiments on two datasets, namely, Ocean and the Synthetic dataset generated using the method described in [6]. We use accuracy as the metric for comparison. In graph classification, the focus is on learning representations for the whole graph and classifying them. Clique lifting is used to convert a graph to a simplicial complex and then use an SNN model to extract representations for the whole graph. We evaluate TopoSRL's performance in graph classification task on PROTEINS, NCI1, REDDIT-B, REDDIT-M and IMDB-B datasets from the TUDatasets [23] repository.
Baselines. We compare TopoSRL against several supervised techniques employing architectures such as SAN [1], SCNN [3], and MPSN [5]. This comparison builds confidence in the expressive capabilities of TopoSRL and solidifies its usefulness in learning simplex representations in settings with less-labeled or unlabeled data. Additionally, we conduct experiments involving various graph neural network models, including graph convolutional network (GCN) [24], GAT[15], GIN [25], and GraphSage[26]. Furthermore, we also compare our method with the current state-of-the-art graph SSL methods CCA-SSG[13], BGRL [11], GraphMAE [14] and GCA [7]. This comparison also clarifies that in the SSL setting, the proposed simplicial SSL method (i.e., TopoSRL) performs better than graph SSL methods, as demonstrated by the existing supervised simplicial representation learning methods. For TopoSRL, we use a 3-layer MPSN as the encoder network  $\mathcal{E}_{\theta}(\cdot)$ . Details about the experimental setup, hyperparameters, and results with other SNN encoder models are available in the supplementary material. We follow the standard practice where all the results are averaged over ten different seeds, and one run is performed for each seed.
Results and discussion. Our results demonstrate that TopoSRL consistently outperforms state-of-the-art graph SSL methods. In Table 1, we see that TopoSRL surpasses supervised methods on graphs and state-of-the-art graph SSL techniques while being competitive with supervised approaches for the node classification task. TopoSRL has a superior performance compared to MPSN on the contact-high-school dataset while being competitive with MPSN on the other two datasets. As can be seen in experiments reported in the supplementary material, contact-primary-school, being a denser dataset, does not benefit significantly from minimizing both the terms, but minimizing only  $\mathcal{L}_{\mathrm{rel}}$  (i.e.,  $\alpha = 1$ ) improves performance.
Table 2 showcases experimental results on simplicial closure datasets. On this task, we see that the less parameterized SCNN model outperforms both TopoSRL and MPSN on the email-Enron and contact-primary-school datasets, while TopoSRL performs better than MPSN and is competitive overall. These results can be attributed to the skewed nature of simplicial closure datasets, where models with large parameters like MPSN and SAN are prone to overfitting labels. Although TopoSRL
Method	Type	email-Enron	contact-primary-school	contact-high-school
SAN	S	0.57±0.12	0.39±0.03	0.3±0.07
SCNN	S	0.61±0.08	0.49±0.09	0.41±0.11
MPSN	S	0.43±0.07	0.43±0.05	0.47±0.20
TopoSRL	SSL	0.59±0.05	0.46±0.01	0.43±0.0
Table 2: Simplicial closure performance using F1 scores; S stands for supervised setting and SSL stands for self-supervised setting.
Method	Type	Ocean	Synthetic
Projection	S	27.15±0.0	52.0±0.0
SCoNe	S	30.0±0.6	55.4±1.1
SCNN	S	28.5±0.6	50.5±1.0
TopoSRL	SSL	42.0±3.0	50.0±1.0
focuses on preserving topology rather than concentrating on a specific downstream task, it has improved performance compared to these models.
Table 3 presents the results on trajectory prediction. As we can see, the results indicate that TopoSRL outperforms SCNN and ScoNe when applied to the 0cean dataset. When tested on the synthetic dataset, TopoSRL demonstrates performance on par with a supervised SCNN. These results further highlight the expressive representation capabilities of TopoSRL on oriented simplicial complexes and its use cases in practical applications. As we can see in Table 4, TopoSRL performs on par with supervised graph baselines and simplicial baselines. Further, TopoSRL outperforms or performs on par with graph SSL baselines, showing the advantages of TopoSRL on a standard graph dataset with clique lifting compared to graph SSL methods.
Table 3: Trajectory prediction performance using accuracy scores; S stands for supervised setting, and SSL stands for self-supervised setting.
Method	Type	PROTEINS	NCI1	REDDIT-B	REDDIT-M	IMDB-B
GCN	S	0.58	0.53	0.71	0.49	0.69
GraphSage	S	0.61	0.48	0.69	0.5	0.68
GIN	S	0.61	0.702	0.73	0.57	0.71
GAT	S	0.57	0.45	0.53	0.5	0.6
CCA-SSG	SSL	0.64	0.67	0.72	0.58	0.675
BGRL	SSL	0.62	0.63	0.72	0.61	0.66
SAN	S	0.64	0.702	0.71	0.59	0.72
SCNN	S	0.61	0.370	0.69	0.57	0.67
MPSN	S	0.63	0.702	0.71	0.59	0.67
TopoSRL	SSL	0.75	0.700	0.72	0.606	0.695
Table 4: Graph classification accuracies on TUDatasets; S stands for supervised setting, and SSL stands for self-supervised setting.
To further validate the efficacy of our augmentation technique, we conduct an ablation study, comparing TopoSRL with and without the proposed augmentation strategy. The results indicate a significant performance improvement when incorporating the proposed simplicial augmentation, underlining its crucial role in capturing long-range dependencies and higher-order interactions. Table 5 shows that our augmentation technique results in considerable performance gains over random augmentation methods. We observe that only adding open simplices and not removing closed simplices is a better augmentation technique than random augmentation, but its performance deteriorates compared to the proposed augmentation technique. This occurs because the contrastive objective function is more effective when there is more information to contrast. Removing closed simplices allows the contrastive loss and encoder to contrast more information, resulting in higher performance instead of only adding open simplices. Additionally, we have observed that incorporating higher-order information from simplices with more than three vertices leads to reduced performance. This reduction is due to the limited presence of higher-order simplices in these datasets, which causes overparameterization of
Dataset	Order	R	OO	O
contact-high-school	3	0.62±0.05	0.81±0.04	0.92±0.05
contact-high-school	4	0.6±0.05	0.78±0.03	0.85±0.06
contact-primary-school	3	0.36±0.06	0.51±0.04	0.61±0.05
contact-primary-school	4	0.4±0.09	0.44±0.08	0.52±0.06
senate-bills	3	0.52±0.07	0.66±0.05	0.72±0.06
the model and a slight decline in performance. More experimental results comparing different SNN encoders and different values of alpha are reported in the supplementary material.
To test the ability of TopoSRL to learn expressive representations, we perform the following experiment: 1) Use the pre-trained TopoSRL encoders to extract representations and 2) Use only a partially labeled (e.g.,  $20\%$  train and  $80\%$  test,  $40\%$  train and  $60\%$  test, etc.) data to train a logistic regression classifier for node classification task in the contact-high-school dataset. MPSN and GCN are trained with cross-entropy loss on the train set. Since the weights for the TopoSRL encoder trained without labels have been saved, no new encoders were trained to produce the results. As we can see in Table 6, with an increase in the size of the train set, the performance increases across all the methods. Furthermore, TopoSRL has a significantly improved performance of about  $5\%$  over supervised MPSN in the 20-80 and 40-60 split. This provides empirical evidence about the expressive capabilities of TopoSRL and its efficacy with less-labeled data. Hence, TopoSRL would be preferable over standard supervised models in the less-labeled data setting.
Table 5: Performace on node classification using random augmentation; R stands for random augmentation, OO stands for adding only open simplices augmentation, and O stands for augmentation with open simplices using Algorithm 1.
Method	Type	20-80	40-60	60-40	80-20
GCN	S	0.31±0.04	0.34±0.03	0.36±0.04	0.4±0.04
CCA-SSG	SSL	0.39±0.04	0.45±0.04	0.53±0.04	0.68±0.16
BGRL	SSL	0.48±0.00	0.49±0.00	0.48±0.00	0.51±0.00
SCNN	S	0.71±0.02	0.74±0.02	0.81±0.04	0.86±0.04
MPSN	S	0.74±0.02	0.80±0.02	0.86±0.01	0.89±0.01
TopoSRL	SSL	0.79±0.02	0.84±0.01	0.86±0.02	0.92±0.05
Table 6: Node classification accuracies on contact-high-school; S stands for supervised setting, and SSL stands for self-supervised setting, 20-80 refers to  ${20}\%$  data for training,80 percent for test.
We have introduced a novel framework for SSL on simplicial complexes by leveraging topological properties underlying a simplicial complex. In particular, we have proposed two key components: a stochastic simplicial augmentation method and a simplicial contrastive loss function, which collectively provide a mechanism for learning representations that retain local and global topological information in the simplicial complex. The proposed simplicial augmentation method offers the advantage of generating topologically consistent views of the original simplicial complex, allowing the model to learn a rich set of features efficiently. The contrastive loss function comprises two primary terms: the sub-simplicial complex loss and the relative simplicial complex loss. The former focuses on minimizing the distance between the representation of a simplex and its adjacent simplices in the augmented complex, thereby preserving local information. The latter aims to maintain relative spatial relationships present in the original simplicial complex, thereby capturing global information. We have also theoretically proved that the proposed loss function is related to the MI objective function, which is commonly used in SSL on graphs. Our model outperforms state-of-the-art graphs SSL methods on a variety of datasets. We believe this work lays a solid foundation for further exploration of SSL methods for simplicial complexes, e.g., via alternative augmentation methods, contrastive loss functions, scalable models, and new applications, potentially opening up new avenues for topological data analysis.
[22] Fan-Yun Sun et al. "Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization". In: arXiv preprint arXiv:1908.01000 (2019).
[23] Christopher Morris et al. "TUDataset: A collection of benchmark datasets for learning with graphs". In: ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+2020). 2020. arXiv: 2007.08663. URL: www.graphlearning.io.
[24] Thomas N. Kipf and Max Welling. "Semi-Supervised Classification with Graph Convolutional Networks". In: International Conference on Learning Representations. 2017. URL: https://openreview.net/forum?id=SJU4ayYgl.
[25] Keyulu Xu et al. "How powerful are graph neural networks?" In: arXiv preprint arXiv:1810.00826 (2018).
[26] William L. Hamilton, Rex Ying, and Jure Leskovec. "Inductive Representation Learning on Large Graphs". In: CoRR abs/1706.02216 (2017). arXiv: 1706.02216. URL: http://arxiv.org/abs/1706.02216.