Learning topology-preserving data representations
Authors: Ilya Trofimov
Daniil Cherniavskii
Eduard Tulchinskii
Nikita Balabin
Evgeny Burnaev
Serguei Barannikov
Venue: ICLR 2023 poster (2023)
Keywords: representation learning
dimensionality reduction
topological data analysis

Abstract: We propose a method for learning topology-preserving data representations (dimensionality reduction). The method aims to provide topological similarity between the data manifold and its latent representation via enforcing the similarity in topological features (clusters,  loops, 2D voids, etc.) and their localization. 
The core of the method is the minimization of the Representation Topology Divergence (RTD) between original high-dimensional data and low-dimensional representation in latent space. RTD minimization provides closeness in topological features with strong theoretical guarantees. 
We develop a scheme for RTD differentiation and apply it as a loss term for the autoencoder. The proposed method "RTD-AE" better preserves the global structure and topology of the data manifold than state-of-the-art competitors as measured by linear correlation, triplet distance ranking accuracy, and Wasserstein distance between persistence barcodes.

Published as a conference paper at ICLR 2023
LEARNING TOPOLOGY-PRESERVING DATA REPRESEN- TATIONS
Iyya T Daniil Cherniavskij Eduard Tulchinskij..., Nikita Balabin',
Evgeny Burnaev":14, Serguei Barannikoy
Skolkovo Institute of Science and Technology; RCNSS, Université Paris Cité; HHuwwi Noah's Ark lab;
Artificial Intelligence Research Institute (AIRI)
ABSTRACT
We propose method for learning topology-preeeeeeeooo preserving data representations (dimen- sionality reduction). The method aims to provide topological similarity between the data manifold and its latent representation via enforcing the similarity in topo- logical features (clusters, loops, 2D voids, etc.) and their localization. The core of the method is the minimization of the Representation Topology Divergence (RTD) between original high-dimensional data and low-dimensional representation in latent space. RTD minimization provides closeness in topological features with strong theoretical guarantees. We develop scheme for RTD differentiation and apply it as loss term for the autoencoder. The proposed method "RTD-AE" better preserves the global structure and topology of the data manifold than state-of-the- art competitors as measured by linear correlation, triplet distance ranking accuracy, and Wasserstein distance between persistence barcodes.
INTRODUCTION
Dimensionality reduction useful tool for data visualization, preprocessing, and exploratory data analysis. Clearly, immersion of high-dimensional data into 2D or 3D space is impossible without distortions which vary for popular methods. Dimensionality reduction methods can be broadly classified into global and local methods. Classical global methods (PCA, MDS) tend to preserve the global structure of a manifold. However, in many practical applications, produced visualizations non-informative since they don't capture complex non-linear structures. Local methods (UMAP (McInnes et al.) 2018), PaCMAP (Wang et al.) 202T). t-SNE (Van der Maaten & Hinton 2008), Laplacian Eigenmaps (Belkin & Niyogi (201)). ISOMAP (Tenenbaum al.) 20000) focus on preserving neighborhood data and local structure with the cost of sacrificing the global structure. The most popular methods like t-SNE and UMAP are a good choice for inferring cluster structures but often fail to describe correctly the data manifold's 's topology. t-SNE and UMAP have hyperparameters influencing representations neighborhood size taken into account. Different values of hyperparameters lead to significantly different visualizations and neither of them is the "canonical" one that correctly represents high-dimensional data.
We take different perspective on dimensionality reduction. We propose the approach based on Topological Data Analysis (TDA). Topological Data Analysis (Barannikov) 1994, Zomorodian] |000]] Chazal & Michel 2017) is field devoted to the numerical description of multi-scale topological properties of data distributions by analyzing point clouds sampled from them. TDA methods naturally capture properties of data manifolds multiple distance scales and arguably a good trade-off between local and global approaches.
The state-of-the-art TDA approach of this kind is TopoAE Mooor et al. 2020). However, it has several weaknesses: 1) the loss term is not continuous 2) the nullity of the loss term is only necessary but not sufficient condition for the coincidence of topology, measured by persistence barcodes, see more details in Appendix J
"Equal senior contribution. Correspondence: i-tr@ yande.neex.ru

---

Published as a conference paper at ICLR 2023
(b) Autoencoder
(c) RTD-AE
(d) I-SNE
(e) UMAP
(I) TopoAE
(a) Orig. 3D data
Figure 1: Dimensionaliity reduction (3D + 2D) the "Mammoth" dataset. The proposed RTD-AE
method better captures both global and local structure.
In our paper, we suggest using the Representation Topology Divergence (RTD) (Barannikov et al. 2022) to produce topology-awar dimensionality reduction. RTD measures the topological discrep- ancy between two point clouds with one-to-one correspondence between clouds and enjoys nice theoretical properties (Section (.2). The major obstacle to incorporate RTD into deep learning is its differentiation. There exist approaches to the differentiation of barcodes, generic barcodes-basee functions with respect to deformations of filtration (Carriére et al. 2021) and to TDA differentiation
in special cases (Hofer et al. 2019 Poulenard et al. 2018).
In this paper, we make the following contributions:
1. We develop an approach for RTD differentiation. Topological metrics are difficult to differentiate; the differentiability of RTD and its implementation on GPU is valuable step forward the TDA context which opens novel possibilities in topological optimizations:
We propose a new method for topology-aware dimensionality reduction: an autoencoder enhanced with the differentiable RTD loss: "RTD-AE", Minimization of RTD loss between real and latent spaces forces closeness in topological features and their localization with strong theoretical guarantees;
3. By doing computational experiments, we show that the proposed RTD-AE outperforms state-of- the-art methods of dimensionality reduction and the vanilla autoencoder in terms of preserving the global structure and topology of a data manifold; we measure it by the linear correlation, the triplet distance ranking accuracy, Wasserstein distance between persistence barcodes, and RTD. In some cases, the proposed RTD-AE produces more faithful and visually appealing low-dimensional embeddings than state-of-ther -ft---rrrtttttttt algorithms. We release the RTD-AE source code.
2 RELATED WORK
Various dimensionality reduction methods have been proposed to obtain 2D/3D visualization of high-dimensional data (Tenenbaum et al. 2000 Belkin & Niyogi 2001 Van der Maaten & Hinton 2008: McInnees et al. 2018). Natural science researchers often use dimensionality reduction methods
2019 Kobak & Berens 2019 Karloy et al. 2019: Andronov et al. 2021 Szubert et 2019). The main problem with these methods is inevitable distortions (Chari et al. 2021 Batson et al., 2021 Wang al. 2021) and incoherent results for different hyperparameters. These distortions can largely affect global representation structure such as inter-cluster relationships and pairwise distances. As the interpretation of these quantities in some domain such as physics or biology can lead to incorrect conclusions, it is of high importance to preserve them as much as possible. UMAP and t-SNE visualizations are frequently sporadic and cannot be considered as "canonical" representation of high-dimensional data. often overlooked issue is the initialization which significantly contributes to the performance of dimensionality reduction methods (Kobak & Linderman] 1021 Wang et al.) 2021). Damnich & Hamprecht (2021) revealed that the UMAP's true loss function is different from the purported from its theory because of negative sampling. There is number of works that try to tackle the distortion problem and preserve as much inter-data relationships as possible. Authors of PHATE (Moon al. (209) and ivis (Szubert et al.] 2019) claim that their methods are able to capture local as well as global features, but provide no theoretical guarantees for this. (Wagner et al. 2021
for exploratory data analysis even to focus further experiments (Becht al.
githu..com/dannhhem7///11 AF
2

---

Published as a conference paper at ICLR 2023
propose DIPOLE, approach to dimensionality reduction combining techniques of metric geometry and distributed persistent homology.
From broader view, deep representation learning is also dedicated to obtaining low low-dimensional representation of data. Autoencoder (Hinton & Salakbutdinov (006) and Variational Autoencoder (Kingma & Welling, 2013) are mostly used to learn representations of objects useful for solving downstream tasks or data generation. They are not designed for data visualization and fail to preserve simultaneously local and global structure on 2D/3D spaces. Though, their parametric nature makes them scalable and applicable to large datasets, which is why they are used in methods such as parametric UMAP (Sainburg et al. 202I) and ivis (Szubert al. 2019) and ours.
Moor et al. (020) proposed TopoAE, including an additional loss for the autoencoder to preserve topological structures of the input space in latent representations. The topological similarity is achieved by retaining similarity in the multi-scale connectivity information. Our approach has stronger theoretical foundation and outperforms TopoAE in computational experiments.
An approach for differentiation of persistent homology-bate bassed functions was proposed by Carriére jtt al. (202T). Leygonie et al. (2021) systematizes different approaches to regularisation of persistence diagrams function and defines notions of differentiability for maps to and from the space of persistence barcodes. [uoo et al. (02T) proposed topology-preserving dimensionality reduction method based on graph autoencoder. Kim et al. (2020) proposed differentiable topological layer for general deep
learning models based on persistence landscapes.
PRELIMINARIES
3.1 TOPOLOGICAL DATA ANALYSIS, PERSISTENT HOMOLOGY
Topology is often considered to describe the "shape of data", that is, multi-scale properties of the datasets. Topological information was generally recognized be important for various data analysis problems. In the perspective of the commonly assumed manifold hypothesis (Goodfellow et al. 2016), datasets are concentrated near low-dimensional manifolds located high-dimensional ambient spaces. The standard direction is to study topological features of the underlying manifold. The common approach is to cover the manifold via simplices. Given the threshold a, we take sets of the points from the dataset X which are pairwise closer than a. The family of such sets is called the Vietoris-Rips simplicial complex. For further convenience, we introduce the fully-connected weighted graph G whose vertices are the points from X and whose edges have weights given by the distances between the points. Then, the Vietoris- simplicial complex is defined as:
VR. (9)= = {i......, } im E Vert(g) m,. S a}
where maj the distance between points, Vert(g) (1...., | is the vertices set of the graph G.
For each (G). we define the vector space Ck. which consists of formal linear combinations of all k-dimensional simplices from (G) with modulo 2 arithmetic. The boundary operator Oe C2 + Ck-1 maps every simplex to the sum of its facets. One can show that On o 0z-1 0 and the chain complex can be created:
+ CR+I DEg' C₁ Og Ck-1 .....
The quotient vector space H₁ = H2 are called homology classes. The dimension By dim (Hmm(H)) is called the k-th Betti number and it approximates the number of basic topological features of the manifold represented by the point cloud X.
is called the k-th homology group, elements of
The immediate problem here is the selection of appropriate a which is not known beforehand. The standard solution is to analyze all a > O. Obviously, if on S 02 .... S amm then VR., (9) S VRa,(G) .... C VR (9); the nested sequence is called the filtration. The evolution of cycles across the nested family of simplicial complexes Sa, is canonically decomposed into "birth" and "death" of basic topological features, that basic feature appears in (S.) a specific threshold ag and disappears at a specific threshold Be. 3. - a₂ describes the "lifespan" " or persistence of the homology class. The set of the corresponding intervals [ac, 3c] for the basic homology classes from H₁ is called the persistence barcode; the whole theory is dubbed the persistent homology (Chazal & Michel 2017 Barannikoy] 1994 Zomorodian 200T
3

---

Published as a conference paper at ICLR 2023
3.2 REPRESENTATION TOPOLOGY DIVERGENCE (RTD)
The classic persistent homology is dedicated to the analysis of a single point cloud X. Recently, Representation Topology Divergence (RTD) (Barannikov et al. 2022) was proposed to measure the dissimilarity in the multi-scale topology between two point clouds X, of equal size N with one-to- one correspondence between clouds. Let g 9 be graphs with weights on edges equal pairwise distances of X, X. To provide the comparison, the auxiliary graph Gm,, with doubled set of vertices and edge weights matrix m(w, 2)), see details in Appendix [B] is created. The persistence barcode of the graph Śu, is called the --Cross-Barcodle and it tracks the differences in the multi-scale topology of the two point clouds by comparing their a-neighborhood graphs for all
R-Cross-Bareode, Here we give a simple example of an R-Cross- Barcode, see also (Cherniavskii et al. 2022). Suppose we have two point clouds A and B, of seven points each, with distances between points as shown in the top row of Figure 2 Consider the R- R-Cross-Be paooss--Brcodee44 (A, B). it consists of 4 intervals (the bottom row of the figure). The 4 intervals describe the topological discrepancies between connected components of o-neighborhood graphs of A and B.
Distonce matri for gloud A
Distance maztix for goud B
0.96
0.91
0.94
U15
488
406
14
012
0.66
0.0
a 0.5
= 0.53
0.54
0.57
0.61
a 0.62
0.83
0.86
Figure 2: A graphical representation of an R-Cross-Barcode, (A, B) for the point clouds A and B. The pairwise distance matrices for A and B are shown in the top raw. Edges present in the a-neighborhood graphs for B but not for A are colored in red. Edges present in the a-neighborhood graph for are colored in grey. The timeline for distinguishing the two graphs is shown. The appearance-disapperreeee disappearance process is illustrated by the underlying bars, connecting the corresponding thresholds
of topological features
An interval is opened, i.e. a topological discrepancy appears, at threshold a = w when in the union of a-neighborrhod graph of A and B, two vertex sets C₁ and C2 disjoint at smaller thresholds, are joined into one connected component by the edge (uv) from B. This interval is closed at threshold wae when the two vertex sets C₁ and C2 are joined into one connected component in the a a-neighborhood graph of A.
For example, discrepancy appears at the threshold a 0.53 when the vertex sets (4} and {3, 6, 7} are joined into one connected component in the union of neighborhood graphs of A and B by the edge (4, 7). We identify the "death" of this R-Cross-Barcode Barcode feature at a 0.57, when these two sets are joined into one connected component in the neighborhood graph of cloud A (via the edge (4, 7) in Figure becoming grey).
By definition, RX,, X) is the sum of intervals' lengths in the (Cross-Barcodey and measures its closeness to an empty set.
Propusition (Barannikov et al. (0222). If -RDD X) = for all 2 1, then the barcodes of the weighted graphs n and 9D are the same in any degree. Moreover. in this case the topological features are located in the same places: the inclusions VR,(G"v C VR, (gmin((w) ()) induce homology isomorphisms for any threshold a.
VR.. "g") C (Rm
The Proposition[ is a strong basis for topology comparison and optimization. Given a fixed data representation X, how to find X lying in a different space, and having a topology similar to X, in particular, similar persistence barcodes? Proposition states that it is sufficient minimize
4

---

Published as a conference paper at ICLR 2023
E RTD,(X,,) (XD(((X In most of our experiments we minimized RTD; (X, (X,X) (X, X). RTD; can be calculated faster than RTD2+, also RTD2+ are often close to zero. To simplify notation, we denote RTD(X, X) 1/2(DDD, (X, X) + RXD;
RTD,
Comparison with TopoAE loss. TopoAE (Moor al. 2020) is the state-of-the-art algorithm for appoloyy-preservin dimensionality reduction. The TopoAE topological loss is based on comparison of minimum spanning trees in X and X spaces. However, it has several weak spots. First, when the TopoAE loss is zero there is no guarantee that persistence barcodes of X and X coincide. Second, the TopoAE loss can be discontinuous rather standard situations, see Appendix
! At the same time, RTD loss is continuous, and its nullity guarantees the coincidence of persistence barcodes X and X. (Proposition 2).
The continuity of the RTD loss follows from the stability of the
Proposition 2. (a) For any quadruple of edge weights sets Wij, Wig, Vig. Viy on G: d
(w, ú) ; Cross--arcodeer (u, ū)) S max(maxx)yg - weyl max wayl)
(b) For any pair of edge weights sets Way Wij on 9:
|R- Me-Crros--arcodae (w. (B)lB max Wigl-
(c) The expectation for the bottleneck distance between R- R-Cros--Burcodex(uu w) and R-Cross-Barrodey Barcode (w', :) where Wij = w(xt, Tz), w/j = ('4,,, Iz), Waj = w(x, Tz): w, w', i is a triple of metrics on a measure space (X, u). and X = {_I,.... In). 24 E X a sample from (X, p)), is upper bounded by Gromov- Wasserstein distance between w and w':
Jxx...x xx...xx
d LB((CCrosy) Cross-Barcodes(tt w), C.CCrs-Barccoeellee W))d S (W,,WWW,,n w').
(d) The expectation for the bottleneck norm of -Cross-aarcoder (w, w) for two weighted graphs with edge weights Wij = Iz), Way = w(xs, 2.)), where t is pair of metrics on a measure space (). and X {z1...., Enl. 24 X is a sample from (X, p). upper bounded by Gromov-Wone Wasserstent distance between w and w:
J xxx...x
| R-Cross-Barcoder (w, lddu S n
The proof's are given in Appendix K
4 METHOD
4.1
DIFFRENNIATION OF RTD
Heconstruction Loss
We propose use RTD as loss in neural networks. Here we describe our approach to RTD differentiation. Denote by Σg the set of all k simplices in the Vietoris-Rae Rips complex of the graph Gw,,, and by Te the set of all intervals in the (X, X). Fix (an arbitrary) strict order Tи-
X
rec
Z
There exists a function f& U(b, (b;, - Σ& that maps by (or de) to simplex whose appearance leads "birth" (or "death"") of the corresponding homological class. Let
RTD (proposed)
Figure 3: RTD Autoencoder
m, max mi.j i.jEn
denote the function of my equal to the filtration value which the simplex joins the filtration.
Since (X,X) (X,x) 1, we obtain the following equation for the subgradient ab.
Od,
i (X,
Σ [(f(dd)) o) Σ
I(fa(bb)) = o).
Om,
iET
SETE
Here, for any a no more than one term has non-zero indicator. Then
a (X,
i RTD X) 3m.
0ms,j
Om,
0m
GE
5

---

Published as a conference paper at ICLR 2023
The only thing that is left is to obtain subgradients of RTD(X,,))) (an arbitrary) element Ma,j of matrix m. There are 4 possible scenarios:
(X, X)) by points from X and X
Consider
1. 1,j N, in other words mijj is from the upper-left quadrant of Its length is constant and thus 4i amiid 0X1 ax.
0m,,.
2. S N < j. in other words ma,j is from the upper-right quadrant of m. Its length is computed as Euclidean distance and thus 0ms,1 ax,
Xx-xxl (similar for XN-)).
X. X,-X,-N
3. j S < i, similar to the previous case.
4. N < i.j. in other words m4, is from the bottom-right quadrant of m. Here we have subgradients like
0miy X;- K X--N I(W;-N5-N < ØG-NJ-Ni 0X,-N || -X,--y-Nlly Xy-Nll2
Similar for Xj-N. XI-N and X--N.
Subgradients RIDIXX.X) and RID(X.X) can be derived from the beforementioned using the chain ax: rule and the formula of full (sub)gradient. Now we are able to minimize RTD(X, X)) by methods of (sub)gradient optimization. We discuss some possible tricks for improving RTD differentiation in Appendix
0X;
4.2 RTD AUTOENCODER
Given the data X {-r) 24 Rd. in high-dimensional space, goal is find the repre- sentation in low-dimensional space Z = {zi}. 2; E RR. For the visualization purposes, = 2, 3. Our idea is find representation topological properties of the point clouds, as much as possible. The straightforward approach is to solve ming RTD(X, Z), where the optimizatiin is performed over n vectors 2; R", in the flavor similar to UMAP and t-SNE. This approach is workable albeit very time-consuming and could be applied only to small datasets, see Appendix A practical solution to learn representations via the encoder network E(w,æ:) X + Z, see Figure
which preserves persistence burcodes, that is, multi-scale
Algorithm. Initially, we train the autoencoder for E, epochs with the reconstruction loss +||x x only. Then, we train for E2 epochs with the loss - RRTDXXX Z). Both losses are calculated on mini-batches. The two-step procedure speedups training since calculating RTD(X, Z) for the untrained network takes much time.
UMAP
E-SNE
TopoAE (Moor et.al.)
RTD-AE
Spheres
PaCMAP
Ivis
PHATE
PCA
Spheres
Figure 4: Results dimensionality reduction to 3D-space

---

Published as a conference paper at ICLR 2023
5 EXPERIMENTS
In computational experiments, we perform dimensionality reduction to high-dimensional and 2D/3D space for ease of visualization. We compare original data with latent representations by (1) linear correlation of pairwise distances, (2) Wasserstein distance (W.D.) between Hu persistence barcodes (Chazal & Michel, 2017). (3) triplet distance ranking accuracy (Wagg et al. (021) (4) RTD. All of tailored to evaluate how the manifold's global structure and topology preserved. We note that RTD, as quality measure, provides more precise comparison of topology than the W.D. between Ho persistence barcodes. First, RTD takes into account the localization of topological features, while W.D. does not. Second, W.D. is invariant to permutations of points, but we are interested in comparison between original data and latent representation where natural one-to-one correspondence holds.
the quality measures
We compare the proposed RTD-AE with t-SNE Van der Maaten & Hinton 2008). UMAP (Melnnes
et al. 201 8), TopoAE (Moor et al. 2020), vanilla autoencoder (AH), PHATE (Moon et al.) 2019). Ivis (Szubert Drozdov] 2019). PacMAP (Wang et [ The complete description of all the used datasets can be found in Appendix
See hyperparameters in Appendix H
5.1 SYNTHETIC DATASETS
We start with the synthetic dataset "Spheres": eleven 100D spheres the 101D space, any two of those do not intersect and one of the spheres contains all other inside. For the visualization, perform dimensionality reduction to 3D space. Figure[4shows the results: RTD-AE is the best one preserving the nestedness for the "Spheres" dataset. Also, RTD-AE outperforms other methods by quality measures, see [ We were unable to run MDS on "Spheres" dataset because it was too large for that method. See more results Appendix M
5.2 REAL WORLD DATASETS
We performed experiments with a number of real-world datasets: MNIST (LeCun et al. 1998),
F-MNIST (Xiao al. 2017), COIL-20 (Nene et al. 1996). scRNA mice Yuan al. 2017 ScRNA melanoma (Tirosh et al. 2016) with latent dimension of 16 and 2, see Tables 2 S The choice of scRNA datasets was motivated by the increased importance dimensionality reduction methods in natural sciences, was previously mentioned. RTD-AE is consistently better than competitors; moreover, the gap metrics for the latent dimension 16 larger than such for the latent dimension 2 (see Appendix D For the latent dimension 2, RTD-AE is the first the second one among the methods by the quality measures (see Table 5 Figure 7 in Appendix D). We conclude that the
proposed RTD-AE does good job in preserving g global structure of data manifolds.
PPATT execution take too much time and results are no presented for many datasets.
Table 1: Quality of data manifold global structure preservation at projection from 101D into 3D space.
Quality measure
Dataset
Method
L. C.
W. D. Hp
A.
RTD
Spheres 3D
t-SNE
0.087
47.89 2.59
0.206 0.01
37.32
+ 1.44
UMAP
0.049
48.31 + 1.83
0.313 + 0.03
44.70
+ 1.47
PaCMAP
0.394
46.48
1.61
0.156:0.00
45.88
+ 1.51
PHATE
0.302
48.78 + 1.65
0.207 0.02
44.05
1.42
PCA
0.155
47.15 1.89
0.174 0.02
38.96 + 1.25
MDS
N.A.
N.A.
N.A.
N.A.
Ivis
0.257
46.32
2.04
0.130 0.01
41.15 + 1.28
AE
0.441
45.07
2.27
0.333 0.02
39.64 1.45
TopoAE
0.424
45.89
+ 2.35
0.02 0.274 +: 0.02
38.49
+ 1.59
RTD-AE
0.633
45.02 2.69
0.346 = 0.02
35.80 + 1.63

---

Published as a conference paper at ICLR 2023
Table 2: Quality of data manifold global structure preservation at projection into 16D space.
Quality measure
Dataset
Method
L. C.
W. D. Ho
T. A.
RTD
F-MNIST
UMAP
0.602
592.0 3.9
0.741 0.018
12.31 + 0.44
PaCMAP
0.600
585.9 3.2
0.741 = 0.013
12.72 = 0,48
Ivis
0,582
552.6
0.718 0.014
10.76 0.30
PHATE
0.603
576.4
10.72 0.016
AE
0.879
320.5 = 1.9
0.850
5.12
TopoAE
0.905
190.7 1.2
0.867 0.006
RTD-AE
0.960
181.2 +: 0.8
0.907 + 0.004
3.01 0.13 0.13
MNIST
UMAP
0,427
879.1 5.6
0.625 + 0.016
= 1.77 0.73
PaCMAP
0.410
887.5
0.644 0.012
20.07 + 0.70
Ivis
0.423
712.6
0.668 0.013
12.40 + 0.32
PHATE
0.358
819.5
0.626
15.01
AE
0.773
391.0
7.22 0.14
TopoAE
0,801
367.5 1.9
5.84 0.014
RTD-AE
0.879
329.6 2.6
0.833
4.15 = 0.18
COIL-20
UMAP
0.301
274.7 0.0
0.574 + 0.011
15.99 1 0.52
PaCMAP
0,230
$73.5
0.548 + 0.012
15.18 = 0.35 N.A.
Ivis
N.A.
N.A.
N.A.
PHATE
0.396
0.000
0.575 + 0.014
13.76 + 0.78
AE
0.834
0.0
0.809 0.008
8.35 0.15
TopoAE
0.910
148.0 0.0
0.822 + 0.020
6.90 =
RTD-AE
0.944
88.9 0.0
0.892 0.007
5.78 0.10
ScRNA mice
UMAP
0.560
1141.0 + 0.0
0.712 0.010
21.30 0.17
PaCMAP
0.496
1161.3 0.0
0.674 + 0.016
21.89 + 0.13
Ivis
0.401
1082.6 0.0
0.636 0.007
22.56 # 1.13
PHATE
0.489
1134.6
21.34 0.32
AE
0.710
1009.2
0.788 0.013
20.80 = 0.16
TopoAE
0.634
826.0 + 0.0
0.748 +
15.37 + 0.22
RTD-AE
0.777
932.9 + 0.0
0.802 0.006
17.03 + 0.15
scRNA melanoma
UMAP
0.474
1416.9 9.2
0.682 0.013
20.02 # 0.35
PaCMAP
0.357
1441.8 9.1
0.681 + 0.014
20.53 = 0.36
Ivis
0.465
1168.0 11.4
0.653
16.31 = 0.28
PHATE
0,427
1427.5 9.1
0.687 0.018
20.18 + 0.41
AE
0.458
1345.9 11.3
0.708 0.016
19.50 + 0.37
TopoAE
0.544
973.7 + 11.1
0.709 0.011
13.41 = 0.35
RTD-AE
0.684
769.5 1 11.5
0.728 0.017
10.35 + 0.33
did dimensionality reduction 3D + 2D. Besides good quality measures, RTD-AE produced an appealing 2D visualization: both large-scale (shape) and low-scale (chest bones, toes, tusks) features are preserved.
For the "Mammoth" (Coenen Pearce 2019b) dataset (Figure !
5.3 ANALYSIS OF DISTORTIONS
Next, to study distortions produced by various dimensionality reduction methods we learn transfor- mation from 2D 2D space, see Figure Here, we observe that RTD-AE in general recovers the global structure for all of the datasets. RTD-AE typically does not suffer from the squeezing (or bottleneck) issue, unlike AE, which is noticeable in "Random". "3 Clusters" and "Circle". Whereas t-SNE and UMAP struggle to preserve cluster densities and intercluster distances. RTD-AE manages to that in every case. does not cluster random points together, like t-SNE, Finally, the overall shape of representations produced by RTD-AE is consistent, it does not tear apart close points, which something UMAP does some cases, as shown in the "Circle" dataset. The metrics, presented in the Table in Appendix also confirm the statements above. RTD-AE has typically higher pairwise distances linear correlation and triplet accuracy, which accounts for good multi-scale properties, while having lower Wasserstein distance between persistence barcodes.
8

---

Other columns: results of Published as a conference paper at ICLR 2023
Original data
UNAP
E-SNE
PaCMAP
PHATE
his
AE
RoooE [Moor et.al.
RTD
Creil
Random
Figure 5: Results on synthetic 2D data. First column: original data. dimensionality reduction methods.
5.4 LIMITATIONS AND COMPUTATIONAL COMPLEXITY
The main source of complexity is RTD computation. For the batch size b, object dimensionality d and latent dimensionality k., the complexity O (d + k)) operations since all the pairwise distances should be calculated. The R-C involved. However, the computation is often quite fast for batch sizes S 256 since the boundary matrix is typically sparse for real datasets. The selection of simplices whose addition leads to "birth" or "death" " of the corresponding homological class doesn't 't take extra time. For RTD calculation and differentiation, we used GPU GPU-optimized software. As calculation relies heavily the batch size, the training time of RTD-AE ranges from 1.5x the time of the basic autoencoder at batch size to 4-6x the time in case of batch 512. For COIL--00, the it took 10 minutes to train basic AE and ~22 minutes for RTD-AF. Overall, the computation of a R-Cross-Barcood takes a similar time as in the previous step even on datasets of big dimensionality.
computation is at worst cubic in the number of simplices
5.5 DISCUSSION
Experimental results show that RTD-AE better preserves the data manifold global structure than its competitors. The most interesting comparison is with TopoAE, the state-of-the-ant which uses an alternative topology-preserve Wasserstein distances between persistence barcodes. Tables show that RTD-AE is better than TopoAE. RTD minimization has stronger theoretical foundation than the loss from TopoAE (see Section 3.2
loss. The measures of interest for topology comparison are the
CONCLUSIONS
In this paper, have proposed an approach for sionality reduction). The topological similarity between data points in original and latent spaces is achieved by minimizing the Representation Topology Divergence (RTD) between original data and latent representations. Our approach is theoretically sound: RTD=0 means that persistence barcodes of any degree coincide and the topological features are located in the same places. We proposed how to make RTD differentiable and implemented it as an additional loss to the autoencoder, constructing ATD-auutencoder (RTD-AE). Computational experiments show that the proposed RTD-AE better preserves the global structure of the data manifold (as measured by linear correlation, triplet distance ranking accuracy, Wasserstein distance between persistence barcodes) than popular methods t-SNE and UMAP. Also, we achieve higher topological similarity than the alternative TopoAE method. Of course, the application RTD loss not limited to autoencoders and we expect more deep learning applications involving one-to-one correspondence between points. The main limitation is that calculation of persistence barcodes and RTD, in particular, is computationally demanding. We here another opportunity for further research.
representation learning (dimen-

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The work was supported by the Analytical center under the RF Government (subsidy agreement W00000D73032115500000 Grant No. 70-2021-00145 02. 02.11.2021)
REPRODUCIBILITY STATEMENT
To provide reproducibility, we release the source code of the proposed RTD-AF, see section | for hyperparameters see Appendix H For other methods, we used either official implementations or implementations from scikit-learn with default hyperparameters. We used public datasets (see Section S Appendix L. We generated several synthetic datasets and made the generating code available.
REFERENCES
Mikhail Andronov, Maxim V Fedorov, and Sergey Sosmin. Exploring chemical reaction space with reaction difference fingerprints and parametric t-SNE. ACS omega, 6(45):30743-3075,, 2021.
Serguei Barannikov. The framed Morse complex and its invariants. Advances in Soviet Mathematics, 21:93-115, 1994.
Serguei Barannikov, Ilya Trofimov, Grigorii Sotnikov, Ekaterina Trimbach, Alexander Korotin, Alexander Filippov, and Evgeny Burnaev. Manifold Topology Divergence: a framework for comparing data manifolds. Advances in Neural Information Processing Systems, 34:7294-7305. 2021.
Serguei Barannikov, Ilya Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation Topology Divergence: method for comparing neural network representations. In International Conference Machine Learning (ICML), pp. 1607-1626. PMLR, 2022.
Joshua Batson, C Grace Haaf, Yonatan Kahn, and Daniel A Roberts. Topological obstructions to autoencoding. Journal of High Energy Physics, 2021 2021.
Etienne Becht, Leland Mclnnes, John Healy, Charles- Antoine Dutertre, Immanuel WH Kwok, Lai Guan Ng, Florent Ginhoux. and Evan W Newell. Dimensionality reduction for visualizing single-cell data using UMAP. Nature biotechnology. 37(1 ::33 44, 2019.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. Advances in neural information processing systems, 14, 2001.
Mathieu Carriére, Frédéric Chazal, Marc Glisse, Yuichi Ike, Hariprasad Kannan, and Yuhei Umeda. Optimizing persistent homology based functions. In International Conference on Machine Learn- ing. pp. 1294-1303, PMLR, 2021.
Tara Chari, Joeyta Banerjee, and Lior Pachter. The specious art of single-cell genomics, bioRxiv, 2021.
Frédéric Chazal and Bertrand Michel. An introduction to topological data analysis: fundamental and practical aspects for data scientists. arXiv preprint arXiv:1710.04011 2017.
Daniil Cherniavskii, Eduard Tulchinskii, Vladislav Mikhailov, Irina Proskurina, Laida Kushnareva, Ekaterina Artemova, Serguei Barannikov, Irina Piontkovskaya, Dmitri Piontkovski. and Evgeny Burnaev, Acceptability judgements via examining the topology attention maps. Findings of the Association for Computational Linguistics: EMNLP 2022, 2022.
Andy Coenen and Adam Pearce. Understanding UMAP. URL: https://pair-code github.
io/understanding-umee 2019a.
Andy Coenen and Adam Pearce. Understanding UMAP, mammoth dataset. 2019b. URL https //github.com http://AARrccooeeddeeeeeeeeeeeee httpa///eeeeraaadeoonnnnnnnneeee
Sebastian Damrich and Fred A Hamprecht. On umap's 's true loss function. Advances in Neural
Information Processing Systems, 34:5798-5809, 2021.
10

---

Published as a conference paper at ICLR 2023
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504 507, 2006.
Christoph Hofer, Roland Kwitt, Mare Nicthammer, and Mandar Dixit. Comnctivity-opiimizee representation learning via persistent homology. In International conference machine learning, pp. 2751-2760. PMLR, 2019.
Dmitry S Karlov, Sergey Sosnin, Igor V Tetko, and Maxim V Fedorov. Chemical space exploration guided by deep neural networks. RSC advances, 99):1511-5157, 2019.
Kwangho Kim, Jisu Kim, Manzil Zaheer, Joon Kim, Frédéric Chazal, and Larry Wasserman. Pllay: efficient topological layer based on persistent landscapes. Advances in Neural Information Pro- cessing Systems, 33: 33:15965-13 19955-15977, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 14, 2013.
arXiv preprint
Dmitry Kobak and Philipp Berens, The art of using t-SNE for single-cell transcriptomics. Nuture communications, 10(1):1-14, 2019.
Dmitry Kobak and George C Linderman. Initialization is critical for preserving global data structure in both t-SNE and UMAP. Nature biotechnology, 39(2):156-155) 2021.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324 1998.
Jacob Leygonie, Steve Oudot, and Ulrike Tillmann. framework for differential calculus on persistence barcodes. Foundations of Computational Mathematics, pp. 1-63, 2021.
Zixiang Luo, Chenyu Xu, Zhen Zhang, and Wenfei Jin. A topology-preserving dimensionality reduction method for single-cell ma-seq data using graph autoencoder. Scientific reports, 11(1): 1-8, 2021.
Leland MeInnes, John Healy, and James Melville. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1802.0::00..3444 2018.
Kevin R Moon, David van Dijk, Zheng Wang, Scott Gigante, Daniel Burkhardt, William Chen, Kristina Yim, Antonia van den Elzen, Matthew J Hir, Ronald R Coifman, et al. Visualizing structure and transitions high-dimensional biological data. Nature biotechnology. 37(12): 1482-1492, 2019.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International conference on machine learning, pp. 7045-7054. PMLR, 2020.
Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-100). 1996.
Adrien Poulenard, Primoz Skraba, and Maks Ovsjanikov. Topological function optimization for continuous shape matching. Computer Graphics Forum, 37(5):13-25, 2018. 2018.
Tim Sainburg, Leland McInnes, and Timothy Q Gentner. Parametric UMAP embeddings for representation and semisupervised learning. Neural Computation, 33(1 11:8118-28777 2021.
Benjamin Szubert and Ignat Drozdov. ivis: dimensionality reduction in very large datasets using siamese networks. J. Open Source Softw., 4(40):1596, 2019.
Benjamin Szubert, Jennifer E Cole, Claudia Monaco, and Ignat Drozdov. Siructurepreseevving visualisation of high dimensional single-cell datasets. Scientific reports, 9(1):1-10, 2019.
Joshua Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500) 2000.

---

Published as a conference paper at ICLR 2023
Itay Tirosh, Benjamin Izar, Sanjay M Prakadan, Marc H Wadsworth, Daniel Treacy, John Trombetta, Asaf Rotem, Christopher Rodman, Christine Lian, George Murphy, et al. Dissecting the multi- cellular ecosystem of metastatic melanoma by single-cell ma-seq. Science, 352(6282)1189-196 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
Alexander Wagner, Elchanan Solomon, and Paul Bendich. Improving metric dimensionality reduction with distributed topology. arXiv preprint 10X.:22006.06133 2021.
Yingfan Wang, Haiyang Huang, Cynthia Rudin, and Yaron Shaposhnik. Understanding how dimen- sion reduction tools work: an empirical approach to deciphering t-SNE, UMAP, TriMAP, and PaCMAP for data visualization. Mach. Learn. Res, 22:1-73, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Guo-Cheng Yuan, Long Cai, Michael Elowitz, Tariq Enver, Guoping Fan, Guoji Guo, Rafael Irizarry, Peter Kharchenko, Junhyong Kim, Stuart Orkin, et al. Challenges and emerging directions in single-cell analysis. Genome biology, 18(1):1-8, 2017.
Simon Zhang, Mengbai Xiao, and Hao Wang. GPU-accelerated computation of vietoris-rips persis- tence barcodes. In 36th International Symposium on Computational Geometry. Schloss Dagstuhl- Leibnnz-ZZetrrum für Informatik, 2020.
Afra J. Zomorodian. Computing and comprehending topology: Persistence and hierarchical Morse complexes (P..D.Thesis) University of Illinois at Urbana- Urbana-Champata Champaign, 2001.
12

---

Published as a conference paper at ICLR 2023
A SIMPLICIAL COMPLEXES AND FILTRATIONS
Here we briefly recall basic topologiaal objects mentioned in our paper. Suppose we have full graph of points in some metric space (R, d).
X = {æo, 21.--. Zn
Definition A.1 Any set の C X is (combinatorial) simplex. Its vertices are all points that belong to K. Its dimensiionality is number equal to |o|| 1. Its faces are all proper subsets of a.
Definition A.2 Simplicial complex is a set of simplices such that for every simplex a E it contains all faces of and for every simplices σ1, 02 C their intersection is face of both of them.
Simplicial complexes can be seen as higher-dimensional generalization graphs. There are many ways build simplical complex from of points, but only two important for this work: Vietoris-Rips and Chech complexes.
Definition A.3 Given threshold the Vietoris-Rips (simplicial) complex threshold (denotes as VR, (X)) defined as set set of all simplices a such that Vo,, E a holds d(z;,0))) S a.
Definition A.4 Given a threshold a the Čech (simplicial) complex at threshold a (denotes as Cech, (X)) is defined as set of all simplices a such that all closed balls of radius and with centers in vertices of a have a non non-empty intersection.
Alhough Čech complexes are rarely used in applications of Topological Data Analysis, they are important due the fact that their fundamental topological properties equal to those of the manifold 'behind" X (so-called Nerve theorem, see (Chazal & Michel 2017) for proper explanation).
The Vietoris- Sectoris-Rips complexes 'approximate" Čech complexes:
VR, (X) C (ech,.()) S (X)
Note that the definition of the Vietoris-Rips complex doesn't require (even indirectly) function d(-) to be metric it should only be symmetric and non-negative. And complex of a weighted graph G (V, E). To do so we modify Definition A.3 by replacing X with V and taking d(u,, Uj) the weight of the edge between v; and vy for i & j and d(v,, vz) = 0, Vi.
define the Vietoris-Rii -ettoris-Riis
In the scope of this work we consider only Vietoris-Rips complexes of graphs.
Definition A.5 A filtration of simplicial complex C is nested family of subcomplexes (Ct) where T C R. such that for any t1, 12 E T if < t2 then Cr, C Ct., and C = USET C. The set T may be either finite infinite.
Vietoris-Rips filtration can 'reflect' topology of data set at every scale. Usually data sets are finite so there is finite number of thresholds that give different Vietoris Siposss-iissso complexes and thus finite filtration is enough for it.
B FORMAL DEFINITION OF RTD
The classic persistent homology is dedicated to the analysis of a single point cloud X. Recently, Representation Topology Diver- gence (RTD) (Barannikoy et al. (022) was proposed to measure the dissimilarity in the multi-scale topology between point clouds X, X of equal size N with one-to-one correspondence between clouds.
A
A
Let (G"), (GD) be two Vietoris-Rips simplicial com- plexes, where w, are the distance matrices of X. The idea behind RTD is compare (G") with VR, (Gmin(m.))) )), where (mmin(w,0)) is the graph having weights min(w, w) its edges. By definition, (G") C VR.. (gmin((.))))) VR, (g") VR, (Gméé)(m,.)))
Figure 6: The graph nm. to compare GeD {A1 Ag, As} (A,, Ag)-
and g Dashed edges correspond to zero weights, green edges to blue edges to min(w, w);; edges with weight too are not shown.
To compare (G") with VR, (Gmin(w;,))) the auxiliary graph is constructed with doubled set of vertices gu,u (Figure and weights
13

---

Published as a conference paper at ICLR 2023
on edges given in the simplest case by:
(w+)T min(u) w. d))
W+
where u. is the w matrix with lower-triangular part replaced by -0, see ( (Barannikov et al. 2022 section 2.2) for the general form of the matrix m. The persistence barcode of the weighted graph VR(@u; is called the (-rrss--Brrooue (for Representations' Cross-Barcodey Note that for every two nodes in the graph @ there exists path with edges having zero weights. Thus, the Ho barcode in the R- Cross-Barcode is always empty.
Intuitively, the k-th barcode of VR, (Gw.e) records the k-dimensional topological features that born in (C but not yet born near the same place in (g"), and the (Ic - 1) dimensionnl topological features that dead in VR, (Gmin(u...)) but not yet dead in VR, (9 The R-
(X, XX) records the differences in the multi-scale topology of the two point clouds. The topological features with longer lifespans indicate in general the essential features.
Basic properties of R- Cross-Barcodde,d (X, X) Barannikov et al. 2022 are:
if X = X, then for all l L-Cross--Brcoodee [X, 2;
if all distances within X are zero i.c. all objects are represented by the same point in X, then for all 2 0: R-Cross-Barcode:--1 X) = Barcodex (X) the standard barcode of the point cloud X;
for any value of threshold a, the following sequence of natural linear maps of homology groups
ra H;(VR (g")))
12
HV (C(iin((u 19111,
13i+1
ras Hi (V Ru, (G H.(V (G
rai-1 m, Ho(V (gmin((w,))) 10, (1)
is exact, i.e. for any j the kernel the map v, is the image of the map rj+1- Proposition 3. Given an exact sequence as in 7 with finite- diitte-dimsiinnall filtered complexes Aar Ba Ca, the alternating sums over k of their topological featurr features lifespans satisfy
((-)~((A)) Σ(-1)+((B)) + ((-1)(((C)=
where ()) denotes the sum of bars lengths in Barcodeu (Z), here for simplicity all lifespans are assumed be finite.
Proof. The exact sequence implies that the alternating sums of dimensions of the homology groups satisfy, for any a,
Σ(-)) dim dim Ha(Ao) E(-1) dim (x( Σ 111** dim H²(Ca) = 0
Notice that for any 01 < 022
dim H₂ (Z.z) dim Hz(Z..) #b(Z, a2), #d(Z, (a1, a2), k)
where #b(Z, (a), a2) ] k), respectfully #d(Z, (a,, a2), k), is the number of births, respectfully deaths, of dimension k topologiaal features in thresholds a, an < a S ag Hence
Σ ( (d))
Σ 11-1) #B) o2),
(+b (a1, (3)
14

---

Published as a conference paper at ICLR 2023
Table 3: Quality of data manifold global structure preservation at projection into 16D space.
Quality measure
Dataset
Method
L. C.
W. D. Ho
T.
F-MNIST
PCA
0.977
351.3
0.951 0.005
RTD-AE
0.960
181.2 + 0.8
0.907 + 0.004
MNIST
PCA
0.911
397.4 + 1.3
0.863 + 0.010
RTD-AE
0.879
329.6 + 2.6
0.833 + 0.006
COIL-20
PCA
0.966
196.4 + 0.0
0.933 + 0.004
RTD-AE
0.944
88.9 + 0.0
0.892 0.007
Setting (n = 02 a + e, we get, for any
E (
- + - -
where #b(Z, a, k)), respectfully #d Z, a, k), is the number of births, respectfully deaths, of dimen- sion is topological features in Z at the threshold a. Summing this over all nontrivial filtration steps gives the identity (2).
Proposition
Σ (-11) (w,, [(II))((VR(()")) + Σ
=0
(4)
C LEARNING REPRESENTATIONS IN HIGHER DIMENSIONS
The following table shows results of the experiment with latent dimensions 16, 32, 64 and 128 for the F-MNIST dataset. RTD-AE are consistently better than the competitors.
D REAL WORLD DATASETS, 2D LATENT SPACE
Table 5 and Figure 7 present the results.
E SYNTHETIC DATASETS, 2D LATENT SPACE
Tshlos the results.
F
RTD MINIMIZATION WITHOUT THE AUTOENCODER
Given the set X = (x.),= of n objects in high- seth-dmmensional space E Rd, our goal is to find their representations in low-dimensional space = {x+) 2n E R It possible to solve
min RTD(X, Z)
directly w.r.t vectors E RR in the flavor similar to UMAP and t-SNE, Figures show the results of two experiments with +DD dimensionality reduction. We conclude that dimensionality reduction via RTD optimization better preserves data topology: meridians B) and the nestedness is retained (Figure D9. The optimization took ~ hour. For the experiment with nested spheres, the RTD optimization was warmstarted with the MDS solution.
kept connected (Figure
15

---

Published as a conference paper at ICLR 2023
Table Quality of data manifold global structure preservation at projection into high-dimensional space.
Quality measure
Dataset
Method
L. C.
W. D. Ho
W. D. H1
T. A.
RTD
F-MNIST-I28D
UMAP
0,605
594.2 3.0
20.73 = 0.56
0.739 0.010
0.22
PCA
0.996
107.4.20.0
11.46 + 0.39
0.39
0.981 = 0.002
1.93 =
PaCMAP
0.589
20.90 5.7.9
0.736 0.012
12.94
Ivis
0.521
5..
2.69
0.693 0.011
10.46
PHATE
0.604
3.8
0.52
0.753 0.010
10.61
AE
0.892
3.2
22.88 + 1.14
0.860 0.006
4 0.15
TopoAE
0.954
66.2 2.1
72.02+000
0.902 0.005
2.16
0.14
RTD-AE
0.943
16.9 + 1.8
9,43 = 0.73
0.884 + 0.010
1,41 = 0.09
R-MNIST-64D
UMAP
0.596
590.7 + 4.3
20.27 0.71
0.735 + 0.021
12.38
0.35
PCA
0.992
1 1.9
18.61 0.44
0.970 + 0.003
=.09
PaCMAP
0.510
3.4
21.37
0.731 0.014
1331
Ivis
0.521
26.86 3.3
0.691 0.011
10.34
PHATE
0.586
586.1 3.2
20.78
0.751 0.012
1.36
AE
0.888
2.2
24.78
0.861 = 0.007
4.18
TopoAE
0.938
89.3 + 1.8
15.27 + 0.68
0.889 0.005
2.56
0.13
RTD-AE
0.954
57.0 + 0.6
11.76 + 0.28
0.895 + 0.008
1.48 0.09
R-MNIST-32D
UMAP
0.593
597.1 5.3
20.39 + 0.24
0.741 + 0.013
12.11 + 0.30
PCA
0.986
263.0 2.3
24.76 + 0.97
0.960 0.006
4.47
0.12
PaCMAP
0.585
589.1 1 4.9
0.738 + 0.010
12.61 0.36
Ivis
0.696
559.8 10.8:4000
0.57
0.770 + 0.014
10.14
PHATE
0.599
576.7 3.5
21.79
0.753 +
1..48
AE
0.904
302.2 2.6
26.37
0.870 0.008
5.28
TopoAE
0.942
120.9 2.5
2.5
1..84
0.892 = 0.006
2.49 0.10
RTD-AE
0.963
108.7 1.8
14.03 + 0.90
0.907 + 0.006
1.85 0.06
F-MNIST-I6D
UMAP
0.588
592.2 - 4.0
20.37 + 0.37
0.739 + 0.013
12.31 0.44
PCA
0.977
351.3 1.7
29.15 + 1.08
0.951 0.005
5.91 + 0.19
PaCMAP
0.600
+.2.
+1.94+0.55
0.741 + 0.013
12.72 +
Ivis
0.582
3.5
2
0.718 0.014
10.76
PHATE
0,603
4.4
21.61
0.756 - 0.016 0.016
10.72
AE
0.879
320.5 1 1.9
27.01 + 0.89
0.850 1 0.004
5.12
TopoAE
0.905
190.7 190.7 1.2
25.65 1.06
0.867 = 0.006
3.69
RTD-AE
0.960
181.2 0.8
20.94
0.907 + 0.004
3.01
G ALTERNATIVE RTD VARIANT
RTD relies on the auxiliary graph with doubled set of vertices gu.,0 and weights on edges:
(w+)T min( w. w)
m
w.
An alternative variant of RTD is possible with the following matrix of weights:
max(w, w
max(w, w)+
in the simplest case. Both of them share similar properties and guarantee that RTD(X, 2) when all the pairwise distances in point clouds X and Z are the same. Also both cases RT (x, Z) RTD; X) 0 for The minimization the sum of both variants of RTD leads to richer gradient information. We used this loss in the experiment with the "Mammoth" dataset.
then the persistence diagrams of X and Z coincide.
16

---

Published as a conference paper at ICLR 2023
Table 5: Quality of data manifold global structure preservation for real-world data dimension reduction to 2D.
Quality measure
Dataset
Method
L.
D. Ho
A.
RTD
Mammoth
t-SNE
0.787
21.31 + 0.25
0.830 - 0.011
5,52
0.12
UMAP
0.776
28.64 + 0.25
0.801 + 0.016
6.81 0.25
AR
0.966
21.94
0.935 + 0.005
6.38+0.22
PaCMAP
0.868
21.13 0.21
0.866 0.008
5.91
Ivis
0.737
13.48 + 0.30
0.764 = 0.007
6.24
TopoAE
0.915
21.51 0.22
0.886 0.007
5.16 0.08
RTD-AE
0.972
17.45 = 0.23
0.928 - 0.006
3.87 + 0.07
F-MNIST
L-SNE
0.547
602.9 2.8
0.695 0.011
11.11 + 0.28
UMAP
0.595
616.5 2.8
0.722 = 0.011
11.72 = 0.24
AE
0.762
614.7 3.1
0.736
11.51 0.42
PaCMAP
0.630
612.8+66
0.732 0.010
11.48 + 0.27
Ivis
0.496
609.2
0.694
11.70
PHATE
0,613
608.2 2.7
0,012
11,60 0.22
TopoAE
0.795
599.0
0.827 0.011
11.84 = 0.43
RTD-AE
0.789
600.0 + 3.1
0.807 = 0.011
10.67 = 0.26
MNIST
t-SNE
0.355
890.1 + 6.8
0.611 + 0.016
15.89 + 0.28
UMAP
0.347
905.5 6.7
+.020
16.49 = 0.21
AE
0.415
392.7
0
16.41 = 0.27
PaCMAP
0.310
902.8
0.015
16.41 0.27
Ivis
0.377
887.5 + 6.8
0.630
16.03 + 0.24
PHATE
0.389
899.7 3.4
+
16.21 + 0.29
TopoAE
0.349
891.5 4.6
0.612
15.71 = 0.09
RTD-AE
0.501
885.1 4.9
0.664 0.009
15.79 1 0.38
COIL-20
t-SNE
0,462
273.9 0.0
0.648 + 0.025
12.23 - 0.27
UMAP
0.247
2.0
0.587 0.013
13.72 - 0.35
AE
0.667
2.00
0.750 0.016
11,82 1 0.28
PaCMAP
0,506
2.6
0.670 0.012
12.60 = 0.42
Ivis
N.A.
N.A.
N.A.
N.A. 12.18 + 0.23
PHATE
0.305
+
0.592 + 0.018
13.11 + 0.39
TopoAE
0.465
261.4 +
0.662 + 0.013
RTD-AE
0.769
262.9
0.796 + 0.009
11.51 + 0.19
ScRNA mice
t-SNE
0.634
1151.3 + 0.0
0.749 + 0.010
21.93 - 0.10
UMAP
0.513
1161.3
0.709 0.015
22.26 - 0.09
PCA
0.733
1147.3
0.790
22.05 +: 0.06
AE
0.677
1142.2 0.0
0.778 1778+00000 0.008
21.34
PaCMAP
0.483
1167.1 0.0
22.44 0.015
Ivis
0,254
1146.8
0.009
22.49 0.25
PHATE
0.522
1159.0
0.711
22.09 = 0.07
TopoAE
0.628
1144.2 0.0
+
21.15 + 0.10
RTD-AE
0.780
1142.3 + 0.0
0.797 + 0.010
21.03 + 0.13
ScRNA melanoma
L-NE
0.505
1445.6
0.699 0.019
20.45 + 0.38
UMAP
0.471
445955+33
0.68440.01 0.014
20.87 = 0.43
PCA
0.536
1446.7 3.1
0.722
20.64 + 0.39
AE
0.407
1442.0 +
0.084
20.90 + 0.41
PaCMAP
0.401
1460.9 3.1.
0.017
20.91 + 0.38
Ivis
0.504
1442.5
20.50 0.099
PHATE
0.427
1458.8 3.1
0.689 0.023
20.94 0.41
TopoAE
0.521
1442.9 3.2
0.709 + 0.009
20.23 + 0.38
RTD-AE
0.639
1438.2 3.0
0.747 + 0.009
19.81 + 0.37
17

---

Published as a conference paper at ICLR 2023
UMAP
T-SNE
PaCMAP
PHATE
Ivis
AE
TopoAE (Moor etall.
MNIST
F-MNIST
COI- 20
SORMA, melarone SCRMA mice
Figure Results on real-world data reduction to 2D.
0-
0
-1
L-SNE
(d) UMAP
(b) RTD
(a) Original 3D data
Figure 8: "Meridians the sphere" dataset. Notice the disconnectedness of meridians in (c) and (d).
H HYPERPARAMETERS
In the experiments with projecting to 3D-space we trained model for 100 epochs using Adam optimizer. We initially trained autoencoder for 10 epochs with only the reconstruction loss and learning rate le-4, then continued with RTD. Epochs 11-30 were trained with learning rate le-2, epochs 31-50 with learning le-3 and for epochs all after learning rate le-4 was used. Batch size was 80.
(b) RTD
E-NE
UMAP
(a) Original 3D data
Figure 9: "Nested spheres" dataset.
18

---

Published as a conference paper at ICLR 2023
Table Quality of data manifold global structure preservation on synthetic data.
Quality measure
Dataset
Method
C.
W. D. Ho
W.D. H1
T. A.
RTD
Circle
I-NNE
0.986
1.073
0.079
0.95
0.59
UMAP
0.808
1.823
0.712
0,81
1.46
AE
0.630
1.179
0.744
0.81
1.02
PaCMAP
0.747
2.263
N.A.
0.81
1.61
Ivis
0.990
0.182
N.A.
0.96
0.18
PHATE
0.891
0.871
N.A.
0.88
1.04
TopoAE
0.978
0.220
0,080
0.95
0.19
RTD-AE
0.984
0.105
0.070
0.96
0.07
2 Clusters
I-NNE
0.633
9.122
1.171
0.72
5.18
UMAP
0.542
9.003
0.914
0.84
6.27
AE
0.925
1,654
0.807
0.94
2.03
PaCMAP
0.269
10.41
N.A.
0.64
5.71
Ivis
0.423
7.400
N.A.
0.76
5.58
PHATE
0.281
7.356
N.A.
0.66
4.90
TopoAE
0.719
7.692
0.883
0.87
3.59
RTD-AE
0.999
0.313
0.313
0.96
0.32
3 Clusters
t-SNE
0.751
4.111
0.370
0.81
0.91
UMAP
0.615
2.671
0.280
0.78
0.83
AE
0.907
1.013
0.054
0.93
0.59
PaCMAP
0.778
2.620
N.A.
0.89
0.92
Ivis
0.918
2.511
N.A.
0,82
1.30
PHATE
0.651
1.538
N.A.
0.72
0.99
TopoAE
0.997
0.586
0.054
0.81
0.13
RTD-AE
0.999
0.307
0.028
0.99
0.11
Random
t-SNE
0.981
4.182
1.938
0.95
1.54
UMAP
0.950
0.979
0,622
0.91
0.55
AE
0.700
9.976
2.343
0.75
1.32
PaCMAP
0.982
5.398
N.A.
0.95
2.08
Ivis
0.648
11.49
N.A.
0.75
2.17
PHATE
0.945
6.703
N.A.
0.92
2.12
TopoAE
0.854
3.288
1,367
0.84
0.91
RTD-AE
0.996
0.148
0.389
0.98
0.17
For 2D and high- nigh-dimensional projections, we used fully-connected autoencoders with hyperparame- ters specified the Table 7 The autoencoder was initially trained only with reconstruction loss for some number of epochs, and then the RTD loss kicked in. The learning rate stayed the same for an entire duration training.
For experiments we used NVIDIA TITAN RTX.
I RTD OPTIMIZATION SPEED-UPS
For all computations of MTD--bacooes in this work we used modified version of Ripser++ software (Zhang al. 2020). Modification that we made was intended at decreasing computational time via exploration of the structure of graph gw.ii (see Section 3.2). The idea behind it is to reduce the size of filtered complex by excluding from it the simplices that do not affect the persistence homology.
Here we consider only simplices of dimension at least 1.
We exclude all simplices spanned by vertices from the first half of the vertex set of the graph gw... Those are the vertices corresponding the upper-left quadrant of the graph's 's edge weights matrix
19

---

Published as a conference paper at ICLR 2023
Table 7: Hyperparameters description.
Dataset name
Batch size
I.R
Hidden dim
# layers
Epochs
RTD epoch
Circle
80
10-3
16
3
100
20
Random
80
10-3
16
100
20
2 Clusters
80
10-3
16
3
100
20
3 Clusters
80
10-3
16
3
100
20
Mammoth
256
10-3
32
100
MNIST
256
10-4
512
250
60
F-MNIST
256
10
512
3
250
60
COIL-20
256
10-1
512
3
250
60
scRNA mice
256
10-3
768
250
scRNA melanoma
256
10-3
768
3
250
60
m from section B
All of them have diameters equal to zero. And if any such simplex spawn topological feature, it is immediately killed by another such simplex.
As before, let N be the number of vertices point clouds. Then ga.e has 2N vertices and our modification eliminates (2) out of (2)) simplices of dimension d 1.
In particular, this eliminates around 1/8 of rows and 1/4 of columns (around 1/3 cells in total) from the boundary matrix used for the computation of persistence pairs of dimension 1. On average, comparing to the standard Ripser++ computation, this gives 45% less time for the computation of persistence intervals of dimension 1.
Next, describe some techniques that can improve convergence when RTD is to be minimized without an autoencoder (F).
Usually we perform (sub)gradient descent minimize RTD(X,X) between "movable" cloud and given constant X.
Gradient smoothing. Subgradients computed at each step of this procedure associate each homo- logical class with at most points from X, while topological structures often include much more. Moreover, adjustments w.r.t. them may be inconsistent for nearby points.
To overcome this, we "smooth" gradients by passing to each point averaged gradients its neighbours. Let v(Ah) be the gradient value for X; at step & and U(x(A)) be some neighbourhood of ((()), Then the formula for each step the gradient descent is
x(() x(x(1) d
BV (1-
Σ x EU(
#(x! ((X(")))
Here B E [0: is some parameter.
Minimum bypassing. Suppose want to shorten edge mcNN from bottom-right quadrant of matrix (i.e. 0 X)D(X,8)
0). It may occur that Waj W.s so
Om:=N,J-N
Ōm++N ax,
X; X, Huis w
||, X,112
and gradient descent will stuck here (since X is constant). Thus there may appear a certain threshold below which RTD(X. XX X) can't be minimized in this case. But it can be further minimized if we move points X, and X, close enough to each other wijj < Wi.j-
20

---

Published as a conference paper at ICLR 2023
Table 8: RTD optimization speed-ups
Optimization trics
RTD
Relative value (%)
None
3.09059
100.00%
Gradient Smoothing
2.68406
86.73%
Minimum Bypassing
1.64366
53.07%
Both
1.20738
38.84%
To do it, if 3 RID(X.X)
8m,48.0. 0, compute ax.
8mg,a without indicator, i.e. as X,-X,
This will assure
Wi, decreasing and at certain point will became lower than uis.j-- ||x;-X,lla>
If 0 X
don't change anything, because the discussing effect appears only if minimize a minimum a function and constant.
Dmi+N.J+N 2
We performed an experiment to transform cloud in the shape of the infinity sign by minimizing the RTD between this cloud and ring-shaped cloud. Both clouds had 100 points and we did not use batch optimisation. We performed 100 iterations of gradient descend minimize RTD in each of the following four setups: using none, each or both of Gradient Smoothing and Minimum Bypassing tricks. For each setup also searched for the best learning rate. The Table shows the results after 100 iterations.
J COMPARISON WITH TOPOAE LOSS
Figure 10: Discontinuity of the TopoAE loss. The point cloud X consists of two clusters 21,
and {4} (top). The point cloud X (bottom left) consists of two clusters [1,2] and {3, 4}. (bottom left). The distances within each cluster are of order 10-1 and the distances between the clusters equal to 103 + 10-1, The TopoAE loss is discontinuous because under small perturbation of points, the minimal spanning tree I' may change. When the point 3 moves slightly as indicated, then the minimal spanning tree I', coloured by yellow, changes and the term (wi4 w11)² ~ 10-2 in TopoAE loss is replaced by (W23- 2 ~ 10
The following simple example Figure 10 shows that the TopoAE loss be discontinuous in rather standard situation. The To loss (Moor et al. 2020) is constructed by calculating first the two minimal spanning trees r, for each the graphs G G whose weights are the distances within two point clouds X and X. Then the TopoAE loss is the sum of two terms L TopoAA = + ī. One term is the sum over the set of edges of of: E EnjEEdgoS(T) (Wij w,;) and the other is the E Eiyedgestt ( (Wij w Under small perturbation
analogous over the edges of r: of points, the minimal spanning tree l' may change, e.g. with change of pair of the closest points
21

---

Published as a conference paper at ICLR 2023
from two clusters. But then the corresponding weights i change in general discontinuosly. The point cloud X on Figure 10 consists of two clusters {1, 2, 3) and (4). The point cloud X consists of two clusters {1, } and 43. We set the distances within each cluster to be of order 10 and the distance between the clusters equal 10 + 10-1 When the point moves in X slightly indicated, then the minimal spanning tree r, coloured by yellow, changes and the term (W14 w14)² 10-2 in / is replaced by (W23 "22)² 10%.
Proposition 3. The RTD loss is continuous. The RTD& depends continuously on (X,X).
The proof follows from the stability of the barcode of the filtered complex VR, (R,() with respect to the bottleneck distance under perturbation of the edge weights, see Appendix K
K STABILITY OF 2-Cross-Barcodee AND RTD
Proposition 6. For any perturbations X' of a point cloud X and X' of a point cloud X,
d (E-Crross Crross-Baroode, (X, R.,RRroos-Barcodee (X), (x))
S 2 max(mmax||x; Xill, max|| - xx (5)
where dæ denotes the bottleneck distance.
Proof. By construction, the R-Cross-Band (roos-BarcoodesAAA (X, X) is the k-th persistence barcode of the
weighted graph G with the weights Waj = - x,1 and min( Usy. Way). where tig = ||X; X If max; X XiIl
then luly wayl S 2s for w/y || Xýll. Similarly, J w.y - 22,
where E = max, [|X; - X,ll. It follows that minnw W
min(wa,, T)ny) S. 2 max(=, E). Hence
the filtration of each simplex in VR, changes at most by max(e, E) under the perturbations. Next, it follows from e.g. the description of metamorphoses of canonical forms in (Barannikov
1994) that the birth or the death of each segment in the k-th barcode of Gu, changes under such perturbations at most by 2 m))
The above arguments give also the proof for the following stability result. Proposition 7. For any quadruple of edge weights sets Wais Wij, why th on G:
d Re-Cross-aarceen (w, i)), Caross-Barrodex.. (w', ,"))
<max(max(ul, wigh max mayl) (6)
where dy denotes the bottleneck distance and R- R-Cross-uee Baass--Brrode w) denotes the persistence barcode for the weighted graph gw.s.
Proposition 8. For any pair of edge weights sets Wij. thy:
R-Corss
UR-Cross-Barcodeyg Barcode; (w., (B I max Wiy
if
where ||]| denotes the bottleneck norm.
Proof. Substitute w' ./ into 6
Notice that 7 is analogous to (Barannikov et al. 202T Proposition 1).
Given pair of metrics u, u' on measure space (X, 1), an analogue of Gromov-Wassersee distance
between u and v'
(W, u') = inf / pz(ee())e'(x)) du e,e'xx-12
(8)
where e: X + Z, el X + Z are embeddings to various metric spaces (Z, Pz) that are isometric with respect to v'.
22

---

Published as a conference paper at ICLR 2023
Proposition 9. Given triple of metrics u', à measure space (X, u.) the expectation for the bottleneck distance between the R-Cross R-Cross-Barcodes Barcode (w, w) and the comparing the pairs of weighted graphs associated with sample X {Z1...... In I; E X, with the edge weights Waj = Ug), w/y = (z), Wig = u(Tx, Ry), is upper bounded by the Gromov- Wasserstein distance between u and ul:
Barsss-Brrode w)
Le(R-Crosss Cross-Barccoey (w, i) R- R-Crross-Baut Barsode (w', ú)) dp I SnGW (W, (9)
XX..XX
Proof. It follows from the R-Cross-Barcodd stability (6) that
Jax..xx dB(R- (R-Croos-Barroon (w, w) R-Cross--Baiiie Baross--Brrode,(ee (w', x)) du 8n
max w/y1 du
xxx.xXX 19
For any pair of isometric embeddings e: X Z, X + Z:
|ui w'yl (oz(e(z,), (z)' (z,))) 5
"
$ pzz(e(a)), ((:)) pz pz(e(z)), (x;)) Σ pzz(e(z,),
by the triangle inequality for pz. Therefore
J.h..x
(w, w) R-Cross-Barcce Baroos-Brrcode((u ()] dn S
$ $
n
Σ (2(e(x)). dp = nn p2(e(t), Xx..XXX
Proposition 10. The expectation for the boutleneck norm of (Cross-Barcodee (w, w) for two weighted graphs with edge weights Wij u;);, Wij ū(x;, Kr)), where u, ú is a pair of metrics on a measure space (X, H), and X = (Z1...., ,n x, E X is sample from (X, p), is upper bounded by Gromov- Wasserstein distance between u and
S
|R-Cross-Barcudey (w, u) la du 8n S n CCCWuü)
(10)
Proof. Substitute v' ū. w = into
L DATASETS
The exact size, nature and dimension of the datasets are presented in Table 9 The errors for the synthetic data are not reported as they are zero due to the small sizes of the datasets.
L.1 SYNTHETIC DATA
The "Random" dataset consists of 500 points randomly distributed 2-dimensional unit square. The choice for this dataset was inspired by Coenen Pearce (2019a) and the ability of UMAP to find clusters in noise.
The "Circle" dataset is represented by 100 points randomly distributed on 2D circle. This dataset has simple non-trivial topology.
The "2 Clusters" dataset consists of 200 points, half of which goes to a dense Gaussian cluster, and the other half goes sparse Gaussian cluster with the same mean. It is used test the methods abilities to preserve cluster density.
The "3 Clusters" dataset consists of 3 Gaussian clusters each having 100 points. Two clusters located much closer to each other than the remaining one. We propose to test the preservation of the global structure, i.e. the distances between clusters.
23

---

Published as a conference paper at ICLR 2023
Table 9: Datasets description.
Dataset
Total size
Nature
Dimension
Circle
1 102
Synthetic
2
Random
5 102
Synthetic 2
2 Clusters
2 x 102
Synthetic
2
3 Clusters
3 10
Synthetic
2
Mammoth
50 10
Real
3
F-MNIST
70 x 10
Real
784
COIL-20
1440
Real
16384
scRNA mice
1402
Real
25392
scRNA melanoma
4645
Real
23686
L.2 REAL-WORLD DATASETS
Both MNIST and F-MNIST are typical datasets, consisting of 60000 x8x28 pixel pictures of 10 different numbers and 10 types of clothes accordingly. COIL-20 dataset of pictures of 20 objects taken from 72 different angles spanning 360 degrees. scRNA mice dataset has 1402 single nuclei extracted from hippocampal anatomical sub-regions (DG, CAI, CA2, and CA3), and scRNA melanoma dataset monitors expression of 4645 cells isolated from 19 metastatic melanoma patients (Szubert et al. 2019).
Datasets licences:
Mammoth (Coenen & Pearce 2019b), CC Zero License. Mammuthus primigenius (blumbach), The Smithsonian Institute,
9lmmbach:3441c66:.9999 001--44444-ddd--3fff5633/////
MNIST (LeCun al.
1998). MIT License.
Fashion-MNIST (Xiao et
2017), MIT License.
COIL-220 (Nene et al.
1996).
scRNA mice
Yuuan et al.
2017).
scRNA melanoma Tirosh et al.
2016).
M MORE DETAILS EXPERIMENTS WITH "SPHERES" AND "TORUS"
We performed experiments on dimensionality reduction 3D space evaluate preservation of 3-dimensional structures in data by our method. Experimental setup was outlined in Section
For this task we have used two synthetic datasets.
The "Spheres" dataset consists of 17,250 points randomly distributed on surface of eleven 100-spheres in 101-dimensional space. Any two of those do not intersect and one of the spheres contains all other inside. Similar to "Circle" dataset (Section 5.3) UMAP splits bigger sphere (light grey) into 10 parts and wraps each small sphere into one of them. PacMAP performs similar
it also splits part of bigger sphere into separate sphere. PCA and Ivis preserve the shape of inner spheres only and turn all structure 'inside out'. Both t-SNE and regular AE projects all points onto one sphere without clear separation between clouds. The addition of topological loss, both in TopoΛE and in RTD-AE. preserves the global structure inlaid clusters. However, TopoAE flattens inner clusters into disks, while RTD-AE makes them into (hollow) spheres.
The "Torus" " dataset consists 5,000 points randomly distributed on surface of a 2-torus (T²) immersed into 100-dimensi space. Due such nature of this dataset, PCA and MDS methods
24

---

Published as a conference paper at ICLR 2023
Table 10: Quality of data manifold global structure preservation projection Torus dataset from 100D into 3D space and Spheres dataset from 101D to 2D.
Quality measure
Dataset
Method
L. C.
W. D. Ho
W. D. H1
T. A.
RTD
Torus 2D
t-SNE
0.989
1.021 0.07
0.594 0.05
0,896 + 0.01
1.533 = 0.09
UMAP
0.955
2.052 0.15
0.931 = 0.07
0,931 + 0.07
3.250 0.17
PaCMAP
0.987
1,410 =
0.833 0.08
0.883 0.01
2.114 - 0.08
PHATE
0.873
2.967 = 0.33
1646
4.061
= 0.20
PCA
1.0
0.871
0.24
0.014 0.00
0.999 0.00
0.000
0.00
MDS
1.0
0.880
+ 0.24
0.022 + 0.00
0.999 0.00
0.000
+ 0.00
Ivis
0.844
2.606
0.27
1.086 0.11
0.580 + 0.02
4.073
0.17
AE
0,880
2.023 = 0.30
1,073 = 0.08
0.662 0.02
3,433 = 0.12
TopoAE
0.920
2.616 0.34
1.017 = 0.09
0.696 0.02
2.975 = 0.14
RTD-AE
0.992
0.907 = 0.08
0.109 + 0.01
0.902 0.01
0.148 + 0.01
Spheres 2D
I-SNE
0.018
49.77 = 1.40
0.349 = 0.05
0.166 0.01
44.00 + 1.44
UMAP
0.020
47.55 = 1.33
0.233 = 0.03
0.191
45.41 1.47
PaCMAP
0.342
46.57 = 1.68
0.208 0.02
0.155 0.01
45.56 = 1.46
PHATE
0.040
48.68 = 1.70
0.188 - 0.03
0,201
0.01
45.08 1.93
PCA
0.117
49.58
1.60
0.447 0.05
0.02 0.180
43.01 1.36
MDS
N.A.
N.A.
N.A.
N.A.
N.A.
Ivis
0.280
48.84 + 1.73
0.342 0.05
0.125 0.01
44.21 1.36
AE
0.334
48.31 - 1.74
0.320 0.04
0.124
43.74 + 1.60
TopoAE
0.264
49.94
1.52
0.634 0.06
4245
42.70 1.74
RTD-AE
0.611
48.20 = 1.72
0.538 + 0.05
0.343 + 0.01
41.22 + 1.70
Table 11: Quality of data manifold global structure preservation for projection of COIL.-20 into 3DDspace..
Quality measure
Dataset
Method
L. C.
W. D. Ho
T. A.
COIL-20
t-SNE
0.608
255 + 0.0
0.706 + 0.01
UMAP
0.250
278 0.0
0.574 + 0.012
AE
0.792
253 +
0.803 + 0.009
TopoAE
0.677
0.0
0.740
RTD-AE
0.811
233 133:00 0.0
0.814 0.014
perform on it very well. RTD-AE takes the third place with very similar quality, see Figure II and Table 10
For "Spheres" dataset we have also performed experiments on dimensionality reduction to 2D space. Overall results quite similar to those obtained for 3D case. The behavior of baselines remains essentially the same. The only interesting change is that RTD-AE now projects bigger sphere to ring and puts the projections of smaller spheres into the ring's hollow center. RTD-AE outperforms other methods in terms of linear correlation and triplet accuracy.
All of the representations were generated with default parameters of baseline methods. Results are presented at Figures ("Spheres" to 3D space) and [T ("Spheres" to 2D space and "Torus" to 3D).
N ABLATION STUDY
In this section we investigate the effect of adding RTD loss on the performance the model. We add a hyperparameter responsible for the scale the RTD loss variable: (X,, X) ARTD(X, Z). We our experiments datasets: COIL-20 and Circle. The hyperparameter value ranged
25

---

Published as a conference paper at ICLR 2023
UMAP
t.SNE
AE
TopoAE (Moor et.al.)
RTD AE
Spheres
PaCMAP
PHATE
PCA
Spheres
UMAP
t-SNE
AE
TopoAE (Moor et.al.)
RTD-AE
Torus
PaCMAP
Ivis
PHATE
PCA
MDS
Torus
Figure 11: Results on dimensionality reduction of Spheres to 2D-space and Torus to 3D-space
26

---

Published as a conference paper at ICLR 2023
Circle
COIL-20
1030
1420
13
0.930
0145
W.D.H. Hy
58
0.855
Figure 12: Results of ablation study. The plot depicts the value of the metrics relative its value 1.0, which was used in all previous experiments. We clearly see that the addition of our RTD loss indeed increases linear correlation and triplet accuracy and at the same time decreases W.D.H. Ho At the same time choosing À = 1.0 seems reasonable to us as increasing its value further does not affect on the quality.
PaCMAP
(b) Ivis
Figure 13: Additional dimensionality reduction methods applied to the "Mammoth" dataset
from 10-6 10 For each value of À the procedure 8 times to get the confidence levels of our metrics. The results are depicted at Figure
O MORE DIMENSIONALITY REDUCTION METHODS ON "MAMMOTH" DATASET
See Figure 13
P R- R-CROSS--ARCODES
See Figure 14
Q RECONSTRUCTION LOSS
See Table 12r results.
R HYPERPARAMETERS SEARCH FOR SPHERES DATASET (INTO 2D)
For TopoAE performed hyperparametrs search in accordance with the original paper Moor et al. (022)) and selected best combination according to KL..1 diveereee............
For RTD-AE searched for batch size in [20:250] and in [0.1; 10). Best combination was once again selected w.r.t. KLo,1-diverge diverrgeee.
Results presented in Table 13 For Wasserstein Distance and Triplet Accuracy difference between means is lesser than standard derivations, and due to this, we performed one-tailed Student's 's t-test to verify their relation. According its results, reject the null hypothesis that the mean W.D.
27

---

Published as a conference paper at ICLR 2023
10000
10000
8000
8000
segment
6000
segment
6000
2000
2000
H1
H1
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
(time)
(time)
10000
10000
8000
8000
segment
6000
segment
6000
4000
4000
2000
2000
0
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.00
0.02
0.04
0.06
0.08
0.10
0.12 0.14
(time)
(time)
Figure R-Cross-BBe -arooss- between latent representations and original data points. Top: R-
Zo, X). R-Cross-Barcode( Zo). Bottom: R-Cross-Baroode(2. X). R-Cross- Barcode Z). "Mammoth" dataset, latent representations from RTD-AE, Zo- latent representation from the untrained autoencoder. Intervals in -Cross-Barcodes smaller after training.
TopoAE
RTD-AE
Figure 15: Dimensionality reduction of Spheres dataset to 2D-space after hyperparameter search.
Ho for TopoAE is lower than the mean W.D. Hg for RTD-AE a significance level 0.05. Same result confirming the better performance of RTD-AE obtained for the triplet accuracy.
ON IDENTITY OF INDISCERNIBLES FOR THE TOPOAE LOSS
We compare two point clouds X, X from Figure 16 For these point clouds, RTD(X, 0.207. while the topological part the TopoAE loss cquals 0. The distinguishing topological feature
between X and X the cycle X which born at a 1 and dies at a = /2. --Cross-Barcodee X) depicts this difference.
28

---

Published as a conference paper at ICLR 2023
(a) The point cloud X.
(b) The point cloud X.
2.00
133
0.00
05
0.6
0.0
0.5
(c) The barcode of X.
(d) The barcode of X.
0.7
5.92
2.
(e) The Cross-Barcode; (X, (X,
Figure 16: Two point clouds X, X for which the identity of indiscernibles property doesn't hold for the topological term in the TopoAE loss. The one-to-one correspondence between clouds is depicted by numbers. The minimal spanning trees 1 - 2 - 3 - 4 have edges of identical length for both point clouds. For these point clouds, RTD(X, X) = 0.207, while the topological term of the TopoAE loss cquals 0. The topology of these point clouds is different, in particular they have different barcodes. The distinguishing topological feature between and X is the cycle in X which is born and dies at a v2. The R-Cross-Barcode; (X, X) depicts this difference.
29

---

Published as a conference paper at ICLR 2023
Table 12: Reconstruction loss for when projecting into 16 dimension latent space
Dataset
Method
Reconstruction loss
COIL-20
AE
1.89 x 10-4
TopoAE
3.30 x 10 4
RTD-AE
4.54 10
F-MNIST
AE
2.73 x 10-3
TopoAE
2.84 10-3
RTD-AE
3.17 x 10-3
MNIST
AE
3.78 10-3
TopoAE
3.70 x 10-3
RTD-AE
4.88 x 10-3
scRNA mice
AE
1.31 x 10-3
TopoAE
1.23 x 10-3
RTD-AE
1.32 10-3
scRNA melanoma
AE
1.16 x 10-3
TopoAE
1.11 10-3
RTD-AE
1.15 X 10-3
Table 13: Hyperparameter search for Spheres (into 2D) dataset
Dataset
Method
L.C.
W.D. Ho
T.A.
RTD
Spheres 2D
TopoAE
0.691
43.291 + 1.583
0.3688 0.0165
39.837 1.318
RTD-AE
0.706
42.133 + 1.683
0.3765 + 0.0124
37.286 + 1.393
30


Reviews:
[{"review_id": "TMzTocuZPCU", "reviewer": "Reviewer_f3NA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach for topology-preserving representation learning (dimensionality reduction). The topological similarity between data points in original and latent spaces was obtained by minimizing the Representation Topology Divergence (RTD) between original data and latent representations. They demonstrated how to make RTD differentiable and implemented it as an additional loss to the autoencoder, constructing RTD-AE. According to their computational experiments, the proposed RTD-AE better preserves the global structure of the data manifold than popular methods t-SNE and UMAP. Moreover, higher topological similarity than the alternative TopoAE method was achieved.", "summary_of_the_review": "Overall I liked the idea of the paper and found their contribution very interesting and significant.", "strength_and_weaknesses": "**Strength**\n\nThe main idea of the paper as well as the empirical part are very interesting. \n\n**Weaknesses and Questions**:\n\n1) TDA literature usually includes a theoretical study of the signature's stability when proposing a new topological signature. In terms of noise in the output or weights, how robust is the R-Cross-barcode?\n\n\n2) For a larger dataset, the number of simplices exponentially increases and so the construction of the VR complex would computationally be inefficient. How does your method deal with this problem? \n\n3) A barcode is used to compare point clouds, but its superiority over other proposed metrics is unclear. Some statistical measures can also be used to compute a \"divergence\" or an actual metric distance between point clouds that might perform better. Mahalanobis distance between points, for instance, can be treated as instantiations of random processes. \n\n\n4) Can we consider RTD as a metric?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ei1d1xT7w6", "reviewer": "Reviewer_LAR4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new dimensionality reduction method, named RTD-AE, that attempts to preserve the topological structure of the data when they are compressed to a lower-dimensional space. This is done in a standard autoencoder fashion where the loss function consists of a reconstruction loss and a topology-preserving loss. The closest rival to RTD-AE is TopoAE (Moor et al, 2020) but it comes with a discontinuous topology-preserving loss that does not guarantee identity of indiscernibles (d(x,y) = 0 does not mean x = y). Instead, RTD-AE uses the Representation Topological Divergence (RTD) (Barannikov et al, 2022), which is continuous and guarantees identity of indiscernibles. However, RTD is not C1-smooth so the paper's main contribution is RTD differentiation using subgradients.\n\nExperiments on synthetic datasets and practical datasets MNIST, F-MNIST, COIL-20, scRNA mice and scRNA melanoma, where the proposed method, named RTD-AE, is compared against recent methods like UMAP, t-SNE, a vanilla autoencoder (AE), TopoAE, Ivis, PacMAP and PHATE, show favouring results towards RTD-AE.", "summary_of_the_review": "This is a good, well-rounded paper. My only concern is that for this conference I would expect the contribution to be more theoretical, although I appreciate the possibly large amount of efforts put in place to make RTD differentiation and RTD-AE work. However, the actual results mostly favouring RTD-AE over TopoAE and more distant methods is encouraging. ", "strength_and_weaknesses": "### Weaknesses\n\nThere is not much in terms of theoretical contribution. My impression is that it is mainly about making RTD work with differentiation, since RTD has been shown quite a useful divergence. In contrast, I acknowledge that it can be a very intensive engineering challenge.\n\nThe authors provided optimization tricks in Appendix I, including restricting the simplices of interest to just edges (k=1), gradient smoothing via averaging nearby subgradients, and bypassing plateaus when $w_{i,j} > \\tilde{w}_{i,j}$. Given that topology-based optimization is typically very compute-intensive, often in cubic time, I was hoping to see some further analysis on how each of those tricks contributed to the learning of RTD-AE, in terms of changes in accuracy and/or speed. However, none was offered.\n\n### Strength\n\nAppendix J pointing out why the TopoAE loss is discontinuous is good to have. It would be good to also give an example why the TopoAE loss does not guarantee identity of indiscernibles.\n\nAppendix M shows the effect of adding the RTD loss on linear correlation, triplet accuracy, and Wasserstein distance is. I find it helpful in understanding RTD.\n\nThe results show that RTD-AE outperform TopoAE in almost all settings.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2zbiG6y14rM", "reviewer": "Reviewer_8g5w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes RTD-AE, a topology-based autoencoder that aims to preserve topological information from the data space in latent embeddings. For this, RTD-AE leverages Representation Topology Divergence (RTD), a technique that allows to measure the similarity of VR complexes between two point clouds, here the data space and latent space.\n\nThe paper considers an interesting problem, that may offer some benefits over existing methods. However, there are several small-to-medium red flags that currently let me question the validity of the results and the interpretations followed from them.", "summary_of_the_review": "Overall, the paper proposes an interesting idea, but the execution of the experiments, the clarity of the paper as well as the demonstration of the benefits seem insufficient to me at this point.\n", "strength_and_weaknesses": "Strengths:\n- Preserving the global structure in non-linear dim. red is important and underexplored\n- RTD loss more closely reflects metric properties than e.g. previous TopoAE loss (but is it really a metric, or just identity of indiscernibles?)\n- Many baselines considered.\n\n\nWeaknesses:\n- The paper lacks in clarity. Most strikingly this is evident in Fig 2, which should give a clear and simple high-level overview. However, I found it just confusing.\n- The required two-step training schedule makes the method less likely to be useful in practice. In general, AE-based dim red / visualization techniques are more challenging to apply for end users out of the box (compared to UMAP, t-SNE etc), so adding an additional layer of complexity like two different training schemes makes it less likely that this method will be widely used by the community.\n- Certain experimental results look a bit as if they were optimized to favor the proposed method. For instance, in Fig 1, the proposed method (panel c)) is rotated exactly as the original input data (by chance?), whereas e.g. f) that also looks reasonable is rotated differently making it look worse than it may be. Another reason why certain plots may be unfair to the baselines: You write \"All of the representations were generated with default parameters of baseline methods.\" If this is the case, but the proposed method was tuned in a hyperparameter search, then there is no surprise that on the proposed method shines in several plots. And its also no surprise that certain methods seem to perform poorer than in their original paper (e.g. 2D visualization of Spheres with TopoAE). Another thing raising a minor red flag: The experimental structure / setup follows neatly the TopoAE paper, however certain parameters are chosen differently without explaining why. E.g. 3D visualisation of the Spheres (instead of 2), or 16D latent space instead of 2. Without any explanation, one is tempted to assume that these slightly arbitrary configurations were searched over to find best performing settings.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-GknL0XOzx", "reviewer": "Reviewer_XfmV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new topology-based loss term for dimensionality reduction. The loss term is based on the recently-introduced \"Representation Topology Divergence\" (RTD) and has beneficial theoretical as well as empirical properties. Next to comparing the loss term with existing losses arising in the context of topological machine learning, the paper presents a very detailed suite of experiments that showcase the empirical utility of the method.\n", "summary_of_the_review": "This is a high-quality paper with a substantial contribution to the literature, namely a new topology-based loss term that satisfies advantageous theoretical and empirical properties. Apart from minor issues (see above), I am very happy to endorse this paper for publication. I am confident that the proposed changes can be performed within the revision cycle.", "strength_and_weaknesses": "The paper makes a strong case for a new topology-based loss term. I appreciate the ingenious use of RTD, which seems to be perfectly apt for the task at hand (permitting the comparison of point clouds of the same size, among other things). Having a strong theoretical foundation with representation quality guarantees thus makes this a very strong contribution to the literature.\n\nI have two minor issues with the manuscript in its current form, though:\n\n1. Some details about RTD and its usage are missing; this should be rectified for a revision (please refer to the \"Clarity\" section below for more details).\n2. The experimental setup needs to be strengthened by adding at least one or two \"geometrical\" metrics for the representation quality (please refer to the \"Quality\" section below for more details).\n\nApart from that, there are several minor issues, which I am confident can be addressed using some additional rewriting and rephrasing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster