SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Diffusion maps with the heat kernel Diffusion maps [5] define an embedding with the first  $k$  eigenvectors  $(\phi_i)_i$  of  $\pmb{P}$ , while Laplacian eigenmaps [1] uses the eigenvectors  $(\psi_i)_i$  of  $\pmb{L}$ . In the following, we recall the links between the two methods, and show that a rescaled Laplacian eigenmaps preserves the diffusion distance with the heat kernel  $\pmb{H}_t$ .
Lemma 5.1. Rescaling the Laplacian eigenmaps embedding with  $x_{i} \mapsto (e^{-2t\lambda_{1}}\psi_{1,i},\ldots ,e^{-2t\lambda_{k}}\psi_{k,i})$  preserves the diffusion distance  $DM_{H_t}$ .
Relation to PHATE The potential distance in PHATE (Sec. 3) is defined by comparing the transition probabilities of two  $t$ -steps random walks initialized from different vertices. The transition matrix  $P^t$  mimics the heat propagation on a graph. The heat-geodesic dissimilarity provides a new interpretation of PHATE. In the following proposition, we show how the heat-geodesic relates to the PHATE potential distance with a linear combination of  $t$ -steps random walks.
Proposition 5.2. The PHATE potential distance with the heat kernel  $PH_{H_t}$  can be expressed in terms of the heat-geodesic dissimilarity with  $\sigma = 0$
$$
P H _ {\boldsymbol {H} _ {t}} = (1 / 4 t) ^ {2} \left\| d _ {t} \left(x _ {i}, \cdot\right) - d _ {t} \left(x _ {j}, \cdot\right) \right\| _ {2} ^ {2},
$$
and it is equivalent to a multiscale random walk distance with kernel  $\sum_{k > 0}m_t(k)\pmb{P}^k$ , where  $m_t(k) := t^k e^{-t} / k!$ .
Proof. We present a simplified version of the proof, more details are available in Appendix A. For  $\sigma = 0$ , we have  $d_{t}(x_{i},x_{j}) = -4t\log (\pmb{H}_{t})_{ij}$ , the relation between the PHATE potential and the heat-geodesic follows from the definition
$$
P H _ {\boldsymbol {H} _ {t}} (x _ {i}, x _ {j}) = \sum_ {k} \left(- \log \boldsymbol {H} _ {t} (x _ {i}, x _ {k}) + \log \boldsymbol {H} _ {t} (x _ {j}, x _ {k})\right) ^ {2} = (1 / 4 t) ^ {2} \| d _ {t} (x _ {i}, \cdot) - d _ {t} (x _ {j}, \cdot) \| _ {2} ^ {2}.
$$
Using the heat kernel  $H_{t}$  with the random walk Laplacian  $L_{rw} = Q^{-1}L = I_{n} - Q^{-1}W$  corresponds to a multiscale random walk kernel. We can write  $L_{rw} = S\Lambda S^{-1}$ , where  $S := Q^{-1/2}\Psi$ . Since  $P = I_{n} - L_{rw}$ , we have  $P^{t} = S(I_{n} - \Lambda)^{t}S^{-1}$ . Interestingly, we can relate the eigenvalues of  $H_{t}$  and  $P$  with the Poisson distribution. The probability mass function of a Poisson distribution with mean  $t$  is given by  $m_{t}(k) := t^{k}e^{-t}/k!$ . For  $t \geq 0$ , we have  $e^{-t(1 - \mu)} = \sum_{k \geq 0}m_{t}(k)\mu^{k}$ . With this relationship, we can express  $H_{t}$  as a linear combination of  $P^{t}$  weighted by the Poisson distribution. Indeed, substituting  $\lambda = 1 - \mu$  in yields
$$
\boldsymbol {H} _ {t} = \boldsymbol {S} e ^ {- t \Lambda} \boldsymbol {S} ^ {- 1} = \boldsymbol {S} \sum_ {k = 0} ^ {\infty} m _ {t} (k) \left(\boldsymbol {I} _ {n} - \Lambda\right) ^ {k} \boldsymbol {S} ^ {- 1} = \sum_ {k = 0} ^ {\infty} m _ {t} (k) \boldsymbol {P} ^ {k}.
$$
Remark 5.3. In the previous proposition, the same argument holds for the symmetric Laplacian and the affinity matrix  $\mathbf{A} \coloneqq \mathbf{Q}^{-1/2}\mathbf{W}\mathbf{Q}^{-1/2}$  used in other methods such as diffusion maps [5]. This is valid since we can write  $\mathbf{L}_{sym} = \mathbf{Q}^{-1/2}\Psi\Lambda\Psi^{T}\mathbf{Q}^{-1/2}$ , and  $\mathbf{A} = \mathbf{I}_n - \mathbf{L}_{sym}$ .
Remark 5.4. This proposition shows that, as the denoising parameter  $\rho \to 1$ , Heat Geodesic Embedding interpolates to the PHATE embeddings with a weighted kernel  $\sum_{k=0}^{\infty} m_t(k) P^k$ .
Relation to SNE The heat-geodesic method also relates to SNE [11], and its variation using the Student distribution t-SNE [18]. In SNE, the similarity between points is encoded via transition probabilities  $p_{ij}$ . The objective is to learn an affinity measure  $q$ , that depends on the embedding distances  $\| y_i - y_j\| _2$ , such that it minimizes  $D_{\mathrm{KL}}(p\| q)$ . Intuitively, points that have a strong affinity in the ambient space, should also have a strong affinity in the embedded space. Even though the heat-geodesic minimization is directly on the embedding distances, we can show an equivalent with SNE. In Appendix A, we provide additional comparisons between SNE and our method.
Proposition 5.5. The Heat-Geodesic embedding with  $\sigma = 0$  and squared distances minimization weighted by the heat kernel is equivalent to SNE with the heat kernel affinity in the ambient space, and a Gaussian kernel in the embedded space  $q_{ij} = \exp (-\| y_i - y_j\|^2 /4t)$ .
6 Results
In this section, we show the versatility of our method, showcasing its performance in terms of clustering and preserving the structure of continuous manifolds. We compare the performance of Heat Geodesic Embedding with multiple state-of-the-art baselines on synthetic datasets and real-world datasets. For all models, we perform sample splitting with a 50/50 validation-test split. The validation and test sets each consists of 5 repetitions with different random initializations. The hyper-parameters are selected according to the performance on the validation set. We always report the results on the test set, along with the standard deviations computed over the five repetitions. We use the following methods in our experiments: our Heat Geodesic Embedding, diffusion maps [5], PHATE [22], shortest-path (used in Isomap [31]) which estimates the geodesic distance by computing the shortest path between two nodes in a graph built on the point clouds,  $t$ -SNE [32], UMAP [19], and metric MDS with Euclidean distance. Details about each of these methods, and results for different parameters (graph type, heat approximation, etc.) are given in Appendix C.
Table 1: Pearson and Spearman correlation between the inferred and ground truth distance matrices on the Swiss roll and Tree datasets (higher is better). Best models on average are bolded.
Swiss roll	Tree
Distance	Pearson	Spearman	Pearson	Spearman
Diffusion distance	0.476 ± 0.226	0.478 ± 0.138	0.656 ± 0.054	0.653 ± 0.057
PHATE potential	0.457 ± 0.01	0.404 ± 0.024	0.766 ± 0.023	0.743 ± 0.028
Shortest path	0.497 ± 0.144	0.558 ± 0.134	0.780 ± 0.009	0.757 ± 0.019
Euclidean	0.365 ± 0.006	0.413 ± 0.005	0.735 ± 0.014	0.704 ± 0.033
Heat-geodesic (ours)	0.702 ± 0.086	0.700 ± 0.073	0.822 ± 0.008	0.807 ± 0.016
6.1 Distance matrix comparison
We start by evaluating the ability of the different distances or dissimilarities to recover the ground truth distance matrix of a point cloud. For this task, we use the Swiss roll and Tree datasets, for which the ground truth geodesic distance is known. The Swiss roll dataset consists of data points sampled on a smooth manifold (see Fig. 1). The Tree dataset is created by connecting multiple high-dimensional Brownian motions in a tree-shape structure. In Fig. 1, we present embeddings of both datasets. Our method recovers the underlying geometry, while other methods create artificial clusters or have too much denoising. Because we aim at a faithful relative distance between data points, we compare the methods according to the Pearson and Spearman correlations of the estimated distance matrices with respect to ground truth. Results are displayed in Tab. 1. We observe that Heat Geodesic Embedding typically outperforms previous methods in terms of the correlation with the ground truth distance matrix, confirming the theoretical guarantees provided in Sec. 4 & 2. Additional results such as computation time and correlation for different noise levels are available in Appendix C.
Optimal diffusion time In Section 4, we described a heuristic to automatically choose the diffusion time based on the entropy of  $H_{t}$ . In Fig. 2, we show that the knee-point of  $t \mapsto H(H_{t})$ , corresponds to a high correlation with the ground distance, while yielding a low approximation error of the distance matrix (measured by the Frobenius norm of the difference between  $D$  and the ground truth).
6.2 Preservation of the inherent data structure
A crucial evaluation criteria of manifold learning methods is the ability to capture the inherent structure of the data. For instance, clusters in the data should be visible in the resulting low dimensional representation. Similarly, when the dataset consists of samples taken at different time points, one expects to be able to characterize this temporal evolution in the low dimensional embedding [22]. We thus compare the different embedding methods according to their ability to retain clusters and temporal evolution of the data.
Figure 2: Evolution of the correlation between estimated and ground truth distance matrices in function of the diffusion time  $t$ .
Figure 3: Embeddings of 2000 differentiating cells from embryoid body [22] over 28 days. UMAP and t-SNE do not capture the continuous manifold representing the cells' evolution.
Identifying clusters. We use the PBMC dataset, the Swiss roll, the Tree dataset, MNIST [8], and COIL-20 [23] dataset. The PBMC dataset consists of single-cell gene expressions from 3000 individual peripheral blood mononuclear cells. Cells are naturally clustered by their cell type. For
Figure 4: Embeddings on PBMC using the triplet distance with the heat-geodesic for different regularization parameter  $\rho$ .
the Tree dataset, we use the branches as clusters. For the Swiss roll dataset, we sample data points on the manifold according to a mixture of Gaussians and use the mixture component as the ground truth cluster labels. The MNIST and COIL-20 datasets are clustered by digits or objects but may not respect the manifold hypothesis. For each method, we run k-means on the two-dimensional embedding and compare the resulting cluster assignments with ground truth. Tab. 10 reports the results in terms of homogeneity and adjusted mutual information (aMI). Heat Geodesic Embedding is competitive with PHATE and outperforms t-SNE and UMAP on all metrics except on the MNIST and COIL-20 datasets. Yet, we show in Appendix C that all methods tend to perform equally well when the noise level increases. In Fig. 4, we present the PBMC embeddings of PHATE and HeatGeo, showing that HeatGeo interpolates to PHATE for  $\rho \rightarrow 1$ .
Table 2: Clustering quality metrics for different methods. We report the homogeneity and the adjusted mutual information (aMI). Best models on average are bolded (higher is better).
Swiss roll	Tree	PBMC	MNIST (Non Manifold)	COIL (Non Manifold)
Method	Homogeneity	aMI	Homogeneity	aMI	Homogeneity	aMI	Homogeneity	aMI	Homogeneity	aMI
UMAP	0.810 ± 0.036	0.726 ± 0.045	0.678 ± 0.086	0.681 ± 0.086	0.177 ± 0.037	0.148 ± 0.035	0.851 ± 0.016	0.86 ± 0.015	0.871 ± 0.009	0.826 ± 0.012
t-SNE	0.748 ± 0.067	0.668 ± 0.068	0.706 ± 0.054	0.712 ± 0.055	0.605 ± 0.019	0.544 ± 0.022	0.903 ± 0.003	0.902 ± 0.003	0.907 ± 0.014	0.88 ± 0.02
Isomap	0.806 ± 0.089	0.743 ± 0.065	0.673 ± 0.045	0.691 ± 0.042	0.242 ± 0.007	0.21 ± 0.007	0.742 ± 0.001	0.74 ± 0.001	/	/
Non-Metric MDS	0.003 ± 0.001	0.0 ± 0.001	0.003 ± 0.001	0.001 ± 0.001	0.011 ± 0.003	0.001 ± 0.003	0.019 ± 0.003	0.001 ± 0.004	0.296 ± 0.02	0.005 ± 0.027
PHATE	0.731 ± 0.035	0.652 ± 0.046	0.550 ± 0.042	0.555 ± 0.042	0.798 ± 0.012	0.785 ± 0.01	0.822 ± 0.01	0.835 ± 0.011	0.804 ± 0.017	0.735 ± 0.021
Diffusion Maps	0.643 ± 0.053	0.585 ± 0.051	0.341 ± 0.103	0.358 ± 0.093	0.026 ± 0.001	0.038 ± 0.001	0.556 ± 0.002	0.622 ± 0.002	0.21 ± 0.036	0.142 ± 0.024
HeatGeo (ours)	0.820 ± 0.008	0.740 ± 0.018	0.784 ± 0.051	0.786 ± 0.051	0.734 ± 0.009	0.768 ± 0.017	0.785 ± 0.0	0.829 ± 0.001	0.849 ± 0.016	0.806 ± 0.022
Temporal data representation. For this task, we aim at representing data points from population observed at consecutive points in time. We use single cell gene expression datasets collected across different time points, including the Embryoid Body (EB), IPSC [22], and two from the 2022 NeurIPS multimodal single-cell integration challenge (Cite & Multi). To quantitatively evaluate the quality of the continuous embeddings, we first embed the entire dataset and obfuscate all samples from a particular time point ( $e.g., t = 2$ ). We then estimate the distribution of the missing time point by using displacement interpolation [35] between the adjacent time points ( $e.g., t = 1$  and  $t = 3$ ). We report the Earth Mover Distance (EMD) between the predicted distribution and true distribution. A low EMD suggests that the obfuscated embeddings are naturally located between the previous and later time points, and that the generated embedding captures the temporal evolution of the data adequately. Results are presented in Tab. 3. Heat Geodesic Embedding outperforms other methods on the EB, Multi, and IPSC datasets and is competitive with other approaches on Cite. We show a graphical depiction of the different embeddings for the embryoid (EB) dataset in Fig. 3.
Table 3: EMD between a linear interpolation of two consecutive time points  $t - 1, t + 1$ , and the time points  $t$ . Best models on average are bolded (lower is better).
Method	Cite	EB	Multi	IPSC
Euclidean	0.978 ± 0.069	1.012 ± 0.039	1.212 ± 0.199	1.085 ± 0.234
Isomap	0.978 ± 0.105	0.993 ± 0.062	1.299 ± 0.307	1.026 ± 0.253
Non-metric MDS	0.81 ± 0.012	0.85 ± 0.014	0.806 ± 0.015	1.013 ± 0.067
UMAP	0.791 ± 0.045	0.942 ± 0.053	1.418 ± 0.042	0.866 ± 0.058
t-SNE	0.905 ± 0.034	0.964 ± 0.032	1.208 ± 0.087	1.006 ± 0.026
PHATE	1.032 ± 0.037	1.088 ± 0.012	1.254 ± 0.042	0.955 ± 0.033
Diffusion Maps	0.989 ± 0.080	0.965 ± 0.077	1.227 ± 0.086	0.821 ± 0.039
HeatGeo (ours)	0.890 ± 0.046	0.733 ± 0.036	0.958 ± 0.044	0.365 ± 0.056
The ability to visualize complex high-dimensional data in an interpretable and rigorous way is a crucial tool of scientific discovery. In this work, we took a step in that direction by proposing a general framework for understanding diffusion-based dimensionality reduction methods through the lens of Riemannian geometry. This allowed us to define a novel embedding based on the heat geodesic dissimilarity—a more direct measure of manifold distance. Theoretically, we showed that our methods brings greater versatility than previous approaches and can help gaining insight into popular manifold learning methods such as diffusion maps, PHATE, and SNE. Experimentally, we demonstrated that it also results in better geodesic distance preservation and excels both at clustering and preserving the structure of a continuous manifold. This contrasts with previous methods that are typically only effective at a single of these tasks.
Despite the strong theoretical and empirical properties, our work presents some limitations. For instance, our method is based on a similarity measure, which is a relaxation of a distance metric. Additionally, the Harnack equation suggests that our parameters for the volume correction could be tuned depending on the underlying manifold. We envision that further analysis of this regularization is a fruitful direction for future work.
This research was enabled in part by compute resources provided by Mila (mila.quebec). It was partially funded and supported by ESP Mérète [G.H.], CIFAR AI Chair [G.W.], NSERC Discovery grant 03267 [G.W.], NIH grants (1F30AI157270-01, R01HD100035, R01GM130847, R01GM135929) [G.W.,S.K.], NSF Career grant 2047856 [S.K.], the Chan-Zuckerberg Initiative grants CZF2019-182702 and CZF2019-002440 [S.K.], the Sloan Fellowship FG-2021-15883 [S.K.], and the Novo Nordisk grant GR112933 [S.K.]. The content provided here is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies. The funders had no role in study design, data collection & analysis, decision to publish, or preparation of the manuscript.
[1] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373-1396, 2003.
[2] Jan Niklas Böhm, Philipp Berens, and Dmitry Kobak. Attraction-repulsion spectrum in neighbor embeddings. Journal of Machine Learning Research, 23(95):1-32, 2022.
[3] Erica L Busch, Jessie Huang, Andrew Benz, Tom Wallenstein, Guillaume Lajoie, Guy Wolf, Smita Krishnaswamy, and Nicholas B Turk-Browne. Multi-view manifold learning of human brain-state trajectories. Nature Computational Science, pages 1-14, 2023.
[4] J Douglas Carroll and Phipps Arabie. Multidimensional scaling. Measurement, judgment and decision making, pages 179-250, 1998.
[5] Ronald R. Coifman and Stéphane Lafon. Diffusion maps. Applied and Computational Harmonic Analysis, 21(1):5-30, 2006.
[6] Keenan Crane, Clarisse Weischedel, and Max Wardetzky. Geodesics in heat: A new approach to computing distance based on heat flow. ACM Transactions on Graphics (TOG), 32(5):1-11, 2013.
[7] Michael Defferrard, Lionel Martin, Rodrigo Pena, and Nathanael Perraudin. Pygsp: Graph signal processing in python.
[8] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
[9] Laleh Haghverdi, Maren Böttner, F. Alexander Wolf, Florian Buettner, and Fabian J. Theis. Diffusion pseudotime robustly reconstructs lineage branching. Nature Methods, 13(10):845-848, 2016.
[29] Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multiscale signature based on heat diffusion. In Computer graphics forum, volume 28, pages 1383-1392. Wiley Online Library, 2009.
[30] Yoshio Takane, Forrest W Young, and Jan De Leeuw. Nonmetric individual differences multidimensional scaling: An alternating least squares method with optimal scaling features. Psychometrika, 42:7-67, 1977.
[31] Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319-2323, 2000.
[32] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[33] David Van Dijk, Roshan Sharma, Juozas Nainys, Kristina Yim, Pooja Kathail, Ambrose J Carr, Cassandra Burdziak, Kevin R Moon, Christine L Chaffer, Diwakar Pattabiraman, et al. Recovering gene interactions from single-cell data using data diffusion. Cell, 174(3):716-729, 2018.
[34] Sathamangalam R Srinivasa Varadhan. On the behavior of the fundamental solution of the heat equation with variable coefficients. Communications on Pure and Applied Mathematics, 20(2): 431-455, 1967.
[35] Cédric Villani. Displacement interpolation. Optimal Transport: Old and New, pages 113-162, 2009.
[36] F Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology, 19:1-5, 2018.
[37] Jonathan X Zheng, Samraat Pawar, and Dan FM Goodman. Graph drawing by stochastic gradient descent. IEEE transactions on visualization and computer graphics, 25(9):2738-2748, 2018.
A Theory and algorithm details
A.1 Kernel preserving embeddings
In this section, we attempt to create a generalized framework for dimensionality reduction methods. These methods often have been viewed as disparate or competing but here we show that many of them are related to one another given the right template for methodology comparison. In order to do this, we introduce a general definition suited for distance-preserving dimensionality reduction methods. With this definition, we can cast many dimensionality reduction methods within the same framework, and easily compare them. We recall that the observations in the ambient space are denoted  $x$ , and those in the embedded space are denoted  $y$ . The definition relies on kernel functions  $H_{t}^{x}$ ,  $H_{t}^{y}$  defined respectively on the ambient and embedded spaces and on transformations  $T^{x}$ ,  $T^{y}$  applied to the kernels. We recall that a divergence  $f: \mathbb{R} \times \mathbb{R} \to \mathbb{R}^{+}$  is such that  $f(a, b) = 0$  if and only if  $a = b$  and  $f(a, a + \delta)$  is a positive semi-definite quadratic form for infinitesimal  $\delta$ .
Definition A.1. We define a kernel features preserving embedding as an embedding which minimizes a loss  $L$  between a transformation  $T^x$  of the ambient space kernel  $H_t^x$  and its embedded space counterpart
$$
L := f \left(T ^ {x} \left(H _ {t} ^ {x}\right), T ^ {y} \left(H _ {t ^ {\prime}} ^ {y}\right)\right), \tag {6}
$$
where  $f$  is any  $C^2$  divergence on  $\mathbb{R}_{\geq 0}$
Example 1. We formulate MDS as a kernel feature-preserving embedding. Suppose we want to preserve the Euclidean distance, we have  $H_{t}^{x}(x_{i},x_{j}) = \| x_{i} - x_{j}\|_{2}$ ,  $H_{t}^{y}(y_{i},y_{j}) = \| y_{i} - y_{j}\|_{2}$ ,  $f(a,b) = \| a - b\|_{2}$ , and  $T^x = T^y = I$ .
In the following, we present popular dimensionality reduction methods that are kernel features preserving embeddings. With this definition, we can distinguish between methods that a preserve a kernel via affinities or distances. For the methods considered in this work,  $H_{t}^{x}$  is an affinity kernel, but its construction varies from one method to another. In PHATE and Diffusion maps,  $H_{t}^{x}$  is a random walk  $P$ , while in Heat Geodesic Embedding we use the heat kernel  $H_{t}$ . t-SNE defines  $H_{t}^{x}$  as a symmetrized random walk matrix from a Gaussian kernel, while UMAP uses an unnormalized version. Methods such as PHATE and diffusion maps define a new distance matrix from a kernel in the ambient space and preserve these distances in the embedded space. Other methods like t-SNE and UMAP define similarities from a kernel and aim to preserve these similarities in the ambient space and embedded space via an entropy-based loss. We note the Kullback-Leibler divergence  $D_{\mathrm{KL}}(a,b) = \sum_{ij}a_{ij}\log [a_{ij} / b_{ij}]$ .
Proposition A.2. The embeddings methods HeatGeo, PHATE, Diffusion Maps, SNE, t-SNE, and UMAP are kernel feature-preserving embeddings.
Proof. We assume that the affinity kernel in the ambient space  $H_{t}^{x}$ , is given, to complete the proof we need to define  $f, H_{t}^{y}, T^{x}, T^{y}$  for all methods.
We start with the distance preserving embeddings; HeatGeo, PHATE, and Diffusion Maps. For these methods, the kernel in the embed space is simply  $H_{t}^{y}(y_{i},y_{j}) = \| y_{i} - y_{j}\|_{2}$ , without transformation, i.e.  $T^y = I$ . Since they preserve a distance, the loss is  $f(T^{x}(H_{t}^{x}),T^{y}(H_{t^{\prime}}^{y})) = \| H_{t}^{x} - H_{t^{\prime}}^{y}\|_{2}$ .
In the Heat Geodesic Embedding we apply a transformation on  $H_{t}^{x} = \pmb{H}_{t}$  to define a dissimilarity, hence  $T^{x}(H_{t}^{x}) = -t\log H_{t}^{x}$  (for  $\sigma = 0$ ), where log is applied elementwise.
In PHATE, the potential distance is equivalent to  $(T^{x}(H_{t}^{x}))_{ij} = \| -\log (H_{t}^{x})_{i} + \log (H_{t}^{x})_{j}\|_{2}$ . In Diffusion Maps, the diffusion distance is  $(T^{x}(H_{t}^{x}))_{ij} = \| (H_{t}^{x})_{i} - (H_{t}^{x})_{j}\|_{2}$ .
SNE, t-SNE, and UMAP preserve affinities from a kernel. For these three methods, the loss is a divergence between distributions, namely  $f = D_{\mathrm{KL}}$ . They vary by defining different affinity kernel and transformation in the embedded space. SNE uses the unnormalized kernel  $H_t^y(y_i, y_j) = \exp(-\left(1/t\right)\|y_i - y_j\|_2^2)$ , with  $T^x = T^y = I$ . Whereas, t-SNE uses  $(H_1^y)_{ij} = (1 + \|y_i - y_j\|^2)^{-1}$ ,
Table 4: Overview of kernel preserving methods.
Method	Hyt(yi,yj)	Tx(Hxt)	Ty(Hyt)	f
PHATE	||yi-yj||2	||-log(Hxt)i+log(Hxt)j||2	Hyt	||·||2
Heat Geodesic	||yi-yj||2	-t log(Hxt)ij	Hyt	||·||2
Diffusion Maps	||yi-yj||2	||(Hxt)i-(Hxt)j||2	Hyt	||·||2
SNE	exp(-(1/t)||yi-yj||2)	Hxt	Hyt	DKL
t-SNE	(1+||yi-yj||2)-1	Hxt	Hyt	DKL
UMAP	(1+||yi-yj||2)-1	Hxt	(H1y)ij/(1-(H1y)ij)	DKL
and  $T^x = T^y = I$ . UMAP define a pointwise transformation in the embedded space with  $(H_1^y)_{ij} = (1 + \| y_i - y_j\|^2)^{-1}$ ,  $(T^y(H_t^y))_{ij} = (H_1^y)_{ij} / (1 - (H_1^y)_{ij})$ , and  $T^x = I$ .
We summarize the choice of kernels and functions in Tab. 4
Proof of Proposition 4.3. The effect of the noise on the square distance is  $(d_t(x_i, x_j) + \epsilon)^2 / d(x_i, x_j)^2 = 1 + (2\epsilon d_t(x_i, x_j) + \epsilon^2) / d(x_i, x_j)^2$ . Denoting the perturbed triplet distance by  $\widetilde{D_{\mathrm{T}}}$ , we have