SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guillaume Huguet $^{1*}$  Alexander Tong $^{1*}$  Edward De Brouwer $^{2*}$
Yanlei Zhang $^{1}$  Guy Wolf $^{1}$  Ian Adelstein $^{2\dagger}$  Smita Krishnaswamy $^{2\dagger}$
<sup>1</sup>Université de Montréal; Mila - Quebec AI Institute 2 Yale University
Diffusion-based manifold learning methods have proven useful in representation learning and dimensionality reduction of modern high dimensional, high throughput, noisy datasets. Such datasets are especially present in fields like biology and physics. While it is thought that these methods preserve underlying manifold structure of data by learning a proxy for geodesic distances, no specific theoretical links have been established. Here, we establish such a link via results in Riemannian geometry explicitly connecting heat diffusion to manifold distances. In this process, we also formulate a more general heat kernel based manifold embedding method that we call heat geodesic embeddings. This novel perspective makes clearer the choices available in manifold learning and denoising. Results show that our method outperforms existing state-of-the-art in preserving ground truth manifold distances, and preserving cluster structure in toy datasets. We also showcase our method on single cell RNA-sequencing datasets with both continuum and cluster structure, where our method enables interpolation of withheld timepoints of data. Finally, we show that parameters of our more general method can be configured to give results similar to PHATE (a state-of-the-art diffusion based manifold learning method) as well as SNE (an attraction/repulsion neighborhood based method that forms the basis of t-SNE).
The advent of high throughput and high dimensional data in various fields of science have made dimensionality reduction and visualization techniques an indispensable part of exploratory analysis. Diffusion-based manifold learning methods, based on the data diffusion operator, first defined in [5], have proven especially useful due to their ability to handle noise and density variations while preserving structure. As a result, diffusion-based dimensionality reduction methods, such as PHATE [22], T-PHATE [3], and diffusion maps [5], have emerged as methods for analyzing high throughput noisy data in various situations. While these methods are surmised to learn manifold geodesic distances, no specific theoretical links have been established. Here, we establish such a link by using Varadhan's formula [34] and a parabolic Harnack inequality [17, 25], which relate manifold distances to heat diffusion directly. This lens gives new insight into existing dimensionality reduction methods, including when they preserve geodesics, and suggests a new method for dimensionality reduction to explicitly preserve geodesics, which we call heat geodesic embeddings<sup>3</sup>. Furthermore, based on our understanding of other methods [22, 5], we introduce theoretically justified parameter choices that
†Co-senior authors
<sup>3</sup>https://github.com/KrishnaswamyLab/HeatGeo
allow our method to have greater versatility in terms of distance denoising and emphasis on local versus global distances.
Generally, data diffusion operators are created by first computing distances between datapoints, transforming these distances into affinities by pointwise application of a kernel function (like a Gaussian kernel), and then row normalizing with or without first applying degree normalization into a Markovian diffusion operator  $P$  [5, 9, 14, 21, 33]. The entries of  $P(x,y)$  then contain probabilities of diffusing (or random walk probabilities) from one datapoint to another. Diffusion maps and PHATE use divergences between these diffusion or random walk-based probability distributions  $P(x,\cdot)$  and  $P(y,\cdot)$  to design a diffusion-based distance that may not directly relate to manifold distance. Our framework directly utilizes a heat kernel based distance, and offers a more comprehensive perspective to study these diffusion methods. By configuring parameters in our framework, we show how we can navigate a continuum of embeddings methods from PHATE [22] to Stochastic Neighbor Embedding (SNE) [11].
In summary, our contributions are as follows:
- We define the heat-geodesic dissimilarity based on Varadhan's formula and the two-sided heat kernel bounds.
- Based on this dissimilarity, we present a versatile geodesic-preserving method for dimensionality reduction which we call heat geodesic embedding.
- We establish a relationship between diffusion-based distances and the heat-geodesic dissimilarity.
- We establish connections between our method and popular dimensionality reduction techniques such as PHATE and SNE, shedding light on their geodesic preservation and denoising properties based on modifications of the computed dissimilarity and distance preservation losses.
- We empirically demonstrate the advantages of Heat Geodesic Embedding in preserving manifold geodesic distances in several experiments showcasing more faithful manifold distances in the embedding space, as well as our ability to interpolate data within the manifold.
Figure 1: Embeddings of the Swiss roll (top) and Tree (bottom) datasets for different manifold learning methods. Our HeatGeo method correctly unrolls the Swiss roll while t-SNE and UMAP create undesirable artificial clusters.
First, we introduce fundamental notions that form the basis of our manifold learning methods: Varadhan's formula [34] on a manifold, diffusion processes on graphs, efficient heat kernel approximations, and multidimensional scaling [4, 12, 16].
Varadhan's formula Varadhan's formula is a powerful tool in differential geometry that establishes a connection between the heat kernel and the shortest path (geodesic) distance on a Riemannian manifold. Its versatility has led to widespread applications in machine learning [6, 10, 15, 27-29]. Let  $(M,g)$  be a closed Riemannian manifold, and  $\Delta$  the Laplace-Beltrami operator on  $M$ . The heat kernel  $h_t(x,y)$  on  $M$  is the minimal positive fundamental solution of the heat equation  $\frac{\partial u}{\partial t} = \Delta u$  with initial condition  $h_0(x,y) = \delta_x(y)$ . In a  $d$ -dimensional Euclidean space the heat kernel is  $h_t(x,y) = (4\pi t)^{-d/2} e^{-d(x,y)^2/4t}$  so that  $-4t \log h_t(x,y) = 2dt \log (4\pi t) + d^2(x,y)$  and
we observe the following limiting behavior:
$$
\lim  _ {t \rightarrow 0} - 4 t \log h _ {t} (x, y) = d ^ {2} (x, y). \tag {1}
$$
Varadhan [34] (see also [20]) proved that eq. 1 (now Varadhan's formula) holds more generally on complete Riemannian manifolds  $M$ , where  $d(x,y)$  is the geodesic distance on  $M$ , and the convergence is uniform over compact subsets of  $M$ . A related result for complete Riemannian manifolds that satisfy the parabolic Harnack inequality (which includes convex domains in Euclidean space and Riemannian manifolds with non-negative Ricci curvature) is the two-sided heat kernel bound [25, 17], showing that for any  $\epsilon \in (0,1)$  there exist constants  $c(\epsilon)$  and  $C(\epsilon)$  such that
$$
\frac {c (\epsilon)}{V (x , \sqrt {t})} \exp \left(- \frac {d (x , y) ^ {2}}{4 (1 + \epsilon) t}\right) \leq h _ {t} (x, y) \leq \frac {C (\epsilon)}{V (x , \sqrt {t})} \exp \left(- \frac {d (x , y) ^ {2}}{4 (1 - \epsilon) t}\right), \tag {2}
$$
where  $V(x, \sqrt{t})$  is the volume of a ball or radius  $\sqrt{t}$  centered at  $x$ . We denote this relation by  $h_t(x,y) \simeq V(x, \sqrt{t})^{-1} \exp(-d(x,y)^2 / t)$  and note that it again recovers eq. 1 in the  $t \to 0$  limit, which is unsurprising as Varadhan's result holds more generally. More important for our purposes is that  $h_t(x,y) \simeq V(x, \sqrt{t})^{-1} \exp(-d(x,y)^2 / t)$  holds for  $t > 0$  which will allow us to approximate geodesic distances  $d(x,y)$  from a diffusion based estimation of the heat kernel  $h_t(x,y)$  and the volume  $V(x, \sqrt{t})$ . In appendix C.3, we provide examples using inequality (2).
Graph construction and diffusion Our construction starts by creating a graph from a point cloud dataset  $X$  of size  $n$ . We use a kernel function  $\kappa: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}^+$ , such that the (weighted) adjacency matrix is  $W_{ij} := \kappa(x_i, x_j)$  for all  $x_i, x_j \in X$ . The kernel function could be a Gaussian kernel, or constructed from a nearest neighbor graph. The resulting graph  $\mathcal{G}$  is characterized by the set of nodes (an ordering of the observations), the adjacency matrix, and the set of edges, i.e. pairs of nodes with non-zero weights. The graph Laplacian is an operator acting on signals on  $\mathcal{G}$  such that it mimics the negative of the Laplace operator. The combinatorial graph Laplacian matrix is defined as  $L := Q - W$  and its normalized version as  $L = I_n - Q^{-1/2}WQ^{-1/2}$ , where  $Q$  is a diagonal degree matrix with  $Q_{ii} := \sum_j W_{ij}$ . The Laplacian is symmetric positive semi-definite, and has an eigen-decomposition  $L = \Psi \Lambda \Psi^T$ . Throughout the presentation, we assume that  $Q_{ii} > 0$  for all  $i \in \{1, \dots, n\}$ . The Laplacian allows us to define the heat equation on  $\mathcal{G}$ , with respect to an initial signal  $f_0 \in \mathbb{R}^n$  on  $\mathcal{G}$ :
$$
\frac {\partial}{\partial t} \boldsymbol {f} (t) + \boldsymbol {L} \boldsymbol {f} (t) = \mathbf {0}, s. t. \quad \boldsymbol {f} (0) = \boldsymbol {f} _ {0} \quad t \in \mathbb {R} ^ {+}. \tag {3}
$$
The solution of the above differential equation is obtained with the matrix exponential  $\pmb{f}(t) = e^{-t\pmb{L}}\pmb{f}_0$ , and we define the heat kernel on the graph as  $\pmb{H}_t \coloneqq e^{-t\pmb{L}}$ . By eigendecomposition, we have  $\pmb{H}_t = \Psi e^{-t\Lambda}\Psi^T$ . The matrix  $\pmb{H}_t$  is a diffusion matrix that characterizes how a signal propagates through the graph according to the heat equations.
Other diffusion matrices on graphs have also been investigated in the literature. The transition matrix  $\pmb{P} \coloneqq \pmb{Q}^{-1}\pmb{W}$  characterizing a random walk on the graph is another common diffusion matrix used for manifold learning such as PHATE [22] and diffusion maps [5]. It is a stochastic matrix that converges to a stationary distribution  $\pi_{i} \coloneqq Q_{ii} / \sum_{i}Q_{ii}$ , under mild assumptions.
Fast computation of heat diffusion Exact computation of the (discrete) heat kernel  $H_{t}$  is computationally costly, requiring a full eigendecomposition in  $O(n^{3})$  time. Fortunately, multiple fast approximations have been proposed, including using orthogonal polynomials or the Euler backward methods. In this work, we use Chebyshev polynomials, as they have been shown to converge faster than other polynomials on this problem [13].
Chebyshev polynomials are defined by the recursive relation  $\{T_k\}_{k \in \mathbb{N}}$  with  $T_0(y) = 0$ ,  $T_1(y) = y$  and  $T_k(y) = 2yT_{k-1}(y) - T_{k-2}(y)$  for  $k \geq 2$ . Assuming that the largest eigenvalue is less than two (which holds for the normalized Laplacian), we approximate the heat kernel with the truncated polynomials of order  $K$
$$
\boldsymbol {H} _ {t} \approx p _ {K} (\boldsymbol {L}, t) := \frac {b _ {t , 0}}{2} + \sum_ {k = 1} ^ {K} b _ {t, k} T _ {k} (\boldsymbol {L} - \boldsymbol {I} _ {n}), \tag {4}
$$
where the  $K + 1$  scalar coefficients  $\{b_{t,i}\}$  depend on time and are evaluated with the Bessel function. Computing  $p_K(L,t)f$  requires  $K$  matrix-vector product and  $K + 1$  Bessel function evaluation. The expensive part of the computation are the matrix-vector products, which can be efficient if the Laplacian matrix is sparse. Interestingly, we note that the evaluation of  $T_{k}$  do not depend on the diffusion time. Thus, to compute multiple approximations of the heat kernel  $\{p_K(L,t)\}_{t\in \mathcal{T}}$ , only necessitates reweighting the truncated polynomial  $\{T_k\}_{k\in [1,\dots,K]}$  with the corresponding  $|\mathcal{T}|$  sets of Bessel coefficients. The overall complexity is dominated by the truncated polynomial computation which takes  $O(K(E + n))$  time where  $E$  is the number of non-zero values in  $\pmb{L}$ .
Another possible approximation is using the Euler backward method. It requires solving  $K$  systems of linear equations  $\pmb{f}(t) = (\pmb{I}_n + (t / K)\pmb{L})^{-K}\pmb{f}(0)$ , which can be efficient for sparse matrices using the Cholesky decomposition [10, 28]. We quantify the differences between the heat kernel approximations in Appendix C.
Metric multidimensional scaling Given a dissimilarity function  $d$  between data points, metric multidimensional scaling (MDS) [16] finds an embedding  $\phi$  such that the difference between the given dissimilarity and the Euclidean distance in the embedded space is minimal across all data points. Formally, for a given function  $d:\mathbb{R}^d\times \mathbb{R}^d\to \mathbb{R}^+$ , MDS minimizes the following objective:
$$
L (\boldsymbol {X}) = \left(\sum_ {i j} \left(d \left(x _ {i}, x _ {j}\right) - \left\| \phi \left(x _ {i}\right) - \phi \left(x _ {j}\right) \right\| _ {2}\right) ^ {2}\right) ^ {1 / 2}, \tag {5}
$$
In metric MDS the solution is usually found by the SMACOF algorithm [30], or stochastic gradient descent [37].
We review state-of-the-art embedding methods and contextualize them with respect to Heat Geodesic Embedding. A formal theoretical comparison of all methods is given in Section 5. Given a set of high-dimensional datapoints, the objective of embedding methods is to create a map that embeds the observations in a lower dimensional space, while preserving distances or similarities. Different methods vary by their choice of distance or dissimilarity functions, as shown below.
Diffusion maps In diffusion maps [5], an embedding in  $k$  dimensions is defined via the first  $k$  non-trivial right eigenvectors of the t-steps random walk  $\pmb{P}^t$  weighted by their eigenvalues. The embedding preserves the diffusion distance  $DM_{\pmb{P}}(x_i, x_j) \coloneqq \| (\delta_i \pmb{P}^t - \delta_j \pmb{P}^t)(1 / \pi)\|_2$ , where  $\delta_i$  is a vector such that  $(\delta_i)_j = 1$  if  $j = i$  and 0 otherwise, and  $\pi$  is the stationary distribution of  $\pmb{P}$ . Intuitively,  $DM_{\pmb{P}}(x_i, x_j)$  considers all the  $t$ -steps paths between  $x_i$  and  $x_j$ . A larger diffusion time can be seen as a low frequency graph filter, i.e. keeping only information from the low frequency transitions such as the stationary distributions. For this reason, using diffusion with  $t > 1$  helps denoising the relationship between observations.
PHATE This diffusion-based method preserves the potential distance [22]  $PH_{P} \coloneqq \| -\log \delta_{i}P^{t} + \log \delta_{j}P^{t}\|_{2}$ , and justifies this approach using the log transformation to prevent nearest neighbors from dominating the distances. An alternative approach is suggested using a square root transformation. Part of our contributions is to justify the log transformation from a geometric point of view. The embedding is defined using multidimensional scaling, which we present below.
SNE, t-SNE, UMAP Well-known attraction/repulsion methods such as SNE [11], t-SNE [32], and UMAP [19] define an affinity matrix with entries  $p_{ij}$  in the ambient space, and another affinity matrix with entries  $q_{ij}$  in the embedded space. To define the embedding, a loss between the two affinity matrices is minimized. Specifically, the loss function is  $D_{\mathrm{KL}}(p||q)\coloneqq \sum_{ij}p_{ij}\log p_{ij} / q_{ij}$  in SNE and t-SNE, whereas UMAP is equivalent to adding  $D_{\mathrm{KL}}(1 - p||1 - q)$  for unnormalized densities [2]. While these methods preserve affinities, they do not preserve any types of distances in the embedding.
4 Heat-Geodesic Embedding
In this section, we present our Heat Geodesic Embedding which is summarized in Alg. 1. We start by introducing the heat-geodesic dissimilarity, then present a robust transformation, and a heuristic to choose the optimal diffusion time. Proofs not present in the main text are given in AppendixA.
We consider the discrete case, where we have a set of  $n$  points  $\{x_i\}_{i=1}^n \eqqcolon \mathbf{X}$  in a high dimensional Euclidean space  $x_i \in \mathbb{R}^d$ . From this point cloud, we want to define a map  $\phi: \mathbb{R}^d \to \mathbb{R}^k$  that embeds the observation in a lower dimensional space. An important property of our embedding is that we preserve manifold geodesic distances in a low dimensional space.
Heat-geodesic dissimilarity Inspired by Varadhan's formula and the Harnack inequalities, we defined a heat-geodesic dissimilarity based on heat diffusion on graphs. From observations (data-points) in  $\mathbb{R}^d$ , we define an undirected graph  $\mathcal{G}$ , and compute its heat kernel  $H_{t} = e^{-tL}$ , where  $L$  is the combinatorial or symmetrically normalized graph Laplacian (the heat kernel is thus symmetric). Following the inequality (2), we can rearrange the terms to isolate the geodesic distance, inspired by this observation, we define the following dissimilarity.
Definition 4.1. For a diffusion time  $t > 0$  and tunable parameter  $\sigma > 0$ , we define the heat-geodesic dissimilarity between  $x_{i}, x_{j} \in X$  as
$$
d _ {t} \left(x _ {i}, x _ {j}\right) := \left[ - 4 t \log \left(\boldsymbol {H} _ {t}\right) _ {i j} - \sigma 4 t \log \left(\boldsymbol {V} _ {t}\right) _ {i j} \right] ^ {1 / 2}
$$
where  $\pmb{H}_t$  is the heat kernel on the graph  $\mathcal{G}$ , and  $(\pmb{V}_t)_{ij} \coloneqq 2[(\pmb{H}_t)_{ii} + (\pmb{H}_t)_{jj}]^{-1}$ .
Here the log is applied elementwise, and the term  $-4t\log (H_t)_{ij}$  corresponds to the geodesic approximation when  $t\to 0$  as in Varadhan's formula. In practice one uses a fixed diffusion time  $t > 0$ , so we add a symmetric volume correction term as in the Harnack inequality, ensuring that  $d_{t}(x_{i},x_{j})$  is symmetric. From Sec. 2, we have  $h_t(x,x)\simeq V(x,\sqrt{t})^{-1}$ , and we use the diagonal of  $\pmb{H}_{t}$  to approximate the inverse of the volume. With this volume correction term and  $\sigma = 1$ , the dissimilarity is such that  $d_{t}(x_{i},x_{i}) = 0$  for all  $t > 0$ . When  $\sigma = 0$  or the manifold has uniform volume growth (as in the constant curvature setting) we show that the heat-geodesic dissimilarity is order preserving:
Proposition 4.2. When  $\sigma = 0$  or the manifold has uniform volume growth, i.e.  $(\pmb{H}_t)_{ii} = (\pmb{H}_t)_{jj}$ , and the heat kernel is pointwise monotonically decreasing w.r.t. a norm  $|\cdot|$  in ambient space, we have for triples  $x_i, x_j, x_k \in \pmb{X}$  that  $|x_i - x_j| > |x_i - x_k|$  implies  $d_t(x_i, x_j) > d_t(x_i, x_k)$ , i.e. the heat-geodesic dissimilarity is order preserving.
Proof. When  $\sigma = 0$  or the manifold has uniform volume growth we need only consider the  $-4t\log (H_t)_{ij}$  terms. The assumption of pointwise monotonicity of the heat kernel entails that  $|x_i - x_j| > |x_i - x_k|$  implies  $H_{t}(x_{i},x_{j}) < H_{t}(x_{i},x_{k})$ . We are able to conclude that  $-4t\log H_t(x_i,x_j) > - 4t\log H_t(x_i,x_k)$  and thus  $d_{t}(x_{i},x_{j}) > d_{t}(x_{i},x_{k})$ .
Denoising distances with triplet computations We note that both diffusion maps and PHATE compute a triplet distance between datapoints, i.e., rather than using the direct diffusion probability between datapoints, they use a distance between corresponding rows of a diffusion operator. In particular, diffusion maps use Euclidean distance, and PHATE uses an M-divergence. Empirically, we notice that this step acts as a denoiser for distances. We formalize this observation in the following proposition. We note  $D_{\mathrm{T}}$  the triplet distance. The triplet distance compares the distances relative to other points. Intuitively, this is a denoising step, since the effect of the noise is spread across the entire set of points. For a reference dissimilarity like the heat-geodesic, it is defined as  $D_{\mathrm{T}}(x_i,x_j)\coloneqq \| d_t(x_i,\cdot) - d_t(x_j,\cdot)\| _2$ . For linear perturbations of the form  $d_{t}(x_{i},x_{j}) + \epsilon$ , where  $\epsilon \in \mathbb{R}$ , the effect of  $\epsilon$  on  $D_{\mathrm{T}}(x_i,x_j)$  is less severe than on  $d_{t}(x_{i},x_{j})$ . Our embedding is based on a linear combination between the heat-geodesic dissimilarity and its triplet distance  $(1 - \rho)d_{t} + \rho D_{\mathrm{T}}$ , where  $\rho \in [0,1]$ .
Proposition 4.3. Denote the perturbed triplet distance by  $\widetilde{D_{\mathrm{T}}} (x_i,x_j) = ||\tilde{d}_t(x_i,\cdot) - \tilde{d}_t(x_j,\cdot)||_2$  where  $\tilde{d}_t(x_i,x_j)\coloneqq d_t(x_i,x_j) + \epsilon$  and  $\tilde{d}_t(x_i,x_k)\coloneqq d_t(x_i,x_k)$  for  $k\neq j$ . Then the triplet distance  $D_{\mathrm{T}}$  is robust to perturbations, i.e., for all  $\epsilon >0$
$$
\left(\frac {\widetilde {D _ {\mathrm {T}}} (x _ {i} , x _ {j})}{D _ {\mathrm {T}} (x _ {i} , x _ {j})}\right) ^ {2} \leq \left(\frac {d _ {t} (x _ {i} , x _ {j}) + \epsilon}{d _ {t} (x _ {i} , x _ {j})}\right) ^ {2}.
$$
Optimal diffusion time Varadhan's formula suggests a small value of diffusion time  $t$  to approximate geodesic distance on a manifold. However, in the discrete data setting, geodesics are based on graph constructions, which in turn rely on nearest neighbors. Thus, small  $t$  can lead to disconnected graphs. Additionally, increasing  $t$  can serve as a way of denoising the kernel (which is often computed from noisy data) as it implements a low-pass filter over the eigenvalues, providing the additional advantage of adding noise tolerance. By computing a sequence of heat kernels  $(\pmb{H}_t)_t$  and evaluating their entropy  $H(\pmb{H}_t) \coloneqq -\sum_{ij}(\pmb{H}_t)_{ij}\log (\bar{\pmb{H}}_t)_{ij}$ , we select  $t$  with the knee-point method [26] on the function  $t\mapsto H(H_t)$ . We show in Sec. 6.1 that our heuristic for determining the diffusion time automatically leads to better overall results.
Weighted MDS The loss in MDS (eq.5) is usually defined with uniform weights. Here, we optionally weight the loss by the heat kernel. In Sec. 5, we will show how this modification relates our method to the embedding defined by SNE[11]. For  $x_{i}, x_{j} \in X$ , we minimize  $(H_{t})_{ij}(d_{t}(x_{i}, x_{j}) - \| \phi(x_{i}) - \phi(x_{j}) \|_{2})^{2}$ . This promotes geodesic preservation of local neighbors, since more weights are given to points with higher affinities.
Heat-geodesic embedding To define a lower dimensional embedding of a point cloud  $X$ , we construct a matrix from the heat-geodesic dissimilarity, and then use MDS to create the embedding. Our embedding defines a map  $\phi$  that minimizes  $\left(d_t(x_i,x_j) - \| \phi (x_i) - \phi (x_j)\| _2\right)^2$ , for all  $x_{i},x_{j}\in X$ . Hence, it preserves the heat-geodesic dissimilarity as the loss decreases to zero. In Alg. 1, we present the main steps of our algorithm using the heat-geodesic dissimilarity. A detailed version is presented in Appendix A.
Algorithm 1 Heat Geodesic Embedding
1: Input:  $N \times d$  dataset matrix  $\mathbf{X}$ , denoising parameter  $\rho \in [0,1]$ , Harnack regularization  $\sigma > 0$ , output dimension  $k$ .
2: Returns:  $N \times k$  embedding matrix  $E$ .
3:  $\mathbf{H}_t \gets p_K(\mathbf{L}, t)$
4:  $t\gets$  Kneedle  $\{H(H_t)\} _t$
5:  $\pmb{D} \gets [-4t\log(\pmb{H}_t)_{ij} - \sigma 4t\log(\pmb{V}_t)_{ij}]^{1/2}$
6:  $D\gets (1 - \rho)D + \rho D_{\mathrm{T}}$
7: Return  $\pmb{E} \gets$  MetricMDS  $(\pmb{D}, \| \cdot \|_2, k)$
$\triangleright$  Heat approximation
$\triangleright$  Knee detection e.g. [26]
$\triangleright$  log is applied elementwise
$\triangleright$  Triplet interpolation step
5 Relation to other manifold learning methods
In this section, we elucidate theoretical connections between the Heat Geodesic Embedding and other manifold learning methods. We relate embeddings via the eigenvalues of  $\mathbf{H}_t$  or  $\mathbf{P}^t$  with Laplacian eigenmaps and diffusion maps. We then present the relation between our methods and PHATE and SNE. We provide further analysis in the Appendix A. In particular, we introduce a new definition of kernel preserving embeddings; either via kernel-based distances (diffusion maps, PHATE) or via similarities (e.g. t-SNE, UMAP).