Title: A Heat Diffusion Perspective on Geodesic Preserving Dimensionality Reduction
Venue: NeurIPS 2023
Accepted: yes
Keywords: manifold learning

heat diffusion

geodesic

metric preserving

dimensionality reduction

embedding
Abstract: Diffusion-based manifold learning methods have proven useful in representation learning and dimensionality reduction of modern high dimensional, high throughput, noisy datasets. Such datasets are especially present in fields like biology and physics. While it is thought that these methods preserve underlying manifold structure of data by learning a proxy for geodesic distances, no specific theoretical links have been established. Here, we establish such a link via results in Riemannian geometry explicitly connecting heat diffusion to manifold distances. In this process, we also formulate a more general heat kernel based manifold embedding method that we call heat geodesic embeddings. This novel perspective makes clearer the choices available in manifold learning and denoising. Results show that our method outperforms existing state of the art in preserving ground truth manifold distances,  and preserving cluster structure in toy datasets. We also showcase our method on single cell RNA-sequencing datasets with both continuum and cluster structure, where our method enables interpolation of withheld timepoints of data. Finally, we show that parameters of our more general method can be configured to give results similar to PHATE (a state-of-the-art diffusion based manifold learning method) as well as SNE (an attraction/repulsion neighborhood based method that forms the basis of t-SNE).

A Heat Diffusion Perspective on Geodesic Preserving Dimensionality Reduction

Abstract

Diffusion-based manifold learning methods have proven useful in representation learning and dimensionality reduction of modern high dimensional, high throughput, noisy datasets. Such datasets are especially present in fields like biology and physics. While it is thought that these methods preserve underlying manifold structure of data by learning a proxy for geodesic distances, no specific theoretical links have been established. Here, we establish such a link via results in Riemannian geometry explicitly connecting heat diffusion to manifold distances. In this process, we also formulate a more general heat kernel based manifold embedding method that we call heat geodesic embeddings . This novel perspective makes clearer the choices available in manifold learning and denoising. Results show that our method outperforms existing state-of-the-art in preserving ground truth manifold distances, and preserving cluster structure in toy datasets. We also showcase our method on single cell RNA-sequencing datasets with both continuum and cluster structure, where our method enables interpolation of withheld timepoints of data. Finally, we show that parameters of our more general method can be configured to give results similar to PHATE (a state-of-the-art diffusion based manifold learning method) as well as SNE (an attraction/repulsion neighborhood based method that forms the basis of t-SNE).

1 Introduction

The advent of high throughput and high dimensional data in various fields of science have made dimensionality reduction and visualization techniques an indispensable part of exploratory analysis. Diffusion-based manifold learning methods, based on the data diffusion operator, first defined in [5], have proven especially useful due to their ability to handle noise and density variations while preserving structure. As a result, diffusion-based dimensionality reduction methods, such as PHATE [22], T-PHATE [3], and diffusion maps [5], have emerged as methods for analyzing high throughput noisy data in various situations. While these methods are surmised to learn manifold geodesic distances, no specific theoretical links have been established. Here, we establish such a link by using Varadhan's formula [34] and a parabolic Harnack inequality [17, 25], which relate manifold distances to heat diffusion directly. This lens gives new insight into existing dimensionality reduction methods, including when they preserve geodesics, and suggests a new method for dimensionality reduction to explicitly preserve geodesics, which we call heat geodesic embeddings 3 . Furthermore, based on our understanding of other methods [22, 5], we introduce theoretically justified parameter choices that

allow our method to have greater versatility in terms of distance denoising and emphasis on local versus global distances.

Generally, data diffusion operators are created by first computing distances between datapoints, transforming these distances into affinities by pointwise application of a kernel function (like a Gaussian kernel), and then row normalizing with or without first applying degree normalization into a Markovian diffusion operator P [5, 9, 14, 21, 33]. The entries of P ( x, y ) then contain probabilities of diffusing (or random walk probabilities) from one datapoint to another. Diffusion maps and PHATE use divergences between these diffusion or random walk-based probability distributions P ( x, · ) and P ( y, · ) to design a diffusion-based distance that may not directly relate to manifold distance. Our framework directly utilizes a heat kernel based distance, and offers a more comprehensive perspective to study these diffusion methods. By configuring parameters in our framework, we show how we can navigate a continuum of embeddings methods from PHATE [22] to Stochastic Neighbor Embedding (SNE) [11].

In summary, our contributions are as follows:

We define the heat-geodesic dissimilarity based on Varadhan's formula and the two-sided heat kernel bounds.

Based on this dissimilarity, we present a versatile geodesic-preserving method for dimensionality reduction which we call heat geodesic embedding.

We establish a relationship between diffusion-based distances and the heat-geodesic dissimilarity.

We establish connections between our method and popular dimensionality reduction techniques such as PHATE and SNE, shedding light on their geodesic preservation and denoising properties based on modifications of the computed dissimilarity and distance preservation losses.

We empirically demonstrate the advantages of Heat Geodesic Embedding in preserving manifold geodesic distances in several experiments showcasing more faithful manifold distances in the embedding space, as well as our ability to interpolate data within the manifold.

UMAP

t-SNE

PHATE

Diffusion Maps

HeatGeo

Branches

0

123

4

2 Preliminaries

First, we introduce fundamental notions that form the basis of our manifold learning methods: Varadhan's formula [34] on a manifold, diffusion processes on graphs, efficient heat kernel approximations, and multidimensional scaling [4, 12, 16].

Varadhan's formula Varadhan's formula is a powerful tool in differential geometry that establishes a connection between the heat kernel and the shortest path (geodesic) distance on a Riemannian manifold. Its versatility has led to widespread applications in machine learning [6, 10, 15, 2729]. Let ( M,g ) be a closed Riemannian manifold, and ∆ the Laplace-Beltrami operator on M . The heat kernel h t ( x, y ) on M is the minimal positive fundamental solution of the heat equation ∂u ∂t = ∆ u with initial condition h 0 ( x, y ) = δ x ( y ) . In a d -dimensional Euclidean space the heat kernel is h t ( x, y ) = (4 πt ) -d/ 2 e -d ( x,y ) 2 / 4 t so that -4 t log h t ( x, y ) = 2 dt log(4 πt ) + d 2 ( x, y ) and

we observe the following limiting behavior:

Varadhan [34] (see also [20]) proved that eq. 1 (now Varadhan's formula) holds more generally on complete Riemannian manifolds M , where d ( x, y ) is the geodesic distance on M , and the convergence is uniform over compact subsets of M . A related result for complete Riemannian manifolds that satisfy the parabolic Harnack inequality (which includes convex domains in Euclidean space and Riemannian manifolds with non-negative Ricci curvature) is the two-sided heat kernel bound [25, 17], showing that for any ϵ ∈ (0 , 1) there exist constants c ( ϵ ) and C ( ϵ ) such that

where V ( x, √ t ) is the volume of a ball or radius √ t centered at x. We denote this relation by h t ( x, y ) ≃ V ( x, √ t ) -1 exp( -d ( x, y ) 2 /t ) and note that it again recovers eq. 1 in the t → 0 limit, which is unsurprising as Varadhan's result holds more generally. More important for our purposes is that h t ( x, y ) ≃ V ( x, √ t ) -1 exp( -d ( x, y ) 2 /t ) holds for t > 0 which will allow us to approximate geodesic distances d ( x, y ) from a diffusion based estimation of the heat kernel h t ( x, y ) and the volume V ( x, √ t ) . In appendix C.3, we provide examples using inequality (2).

Graph construction and diffusion Our construction starts by creating a graph from a point cloud dataset X of size n . We use a kernel function κ : R d × R d → R + , such that the (weighted) adjacency matrix is W ij := κ ( x i , x j ) for all x i , x j ∈ X . The kernel function could be a Gaussian kernel, or constructed from a nearest neighbor graph. The resulting graph G is characterized by the set of nodes (an ordering of the observations), the adjacency matrix, and the set of edges, i.e. pairs of nodes with non-zero weights. The graph Laplacian is an operator acting on signals on G such that it mimics the negative of the Laplace operator. The combinatorial graph Laplacian matrix is defined as L := Q -W and its normalized version as L = I n -Q -1 / 2 WQ -1 / 2 , where Q is a diagonal degree matrix with Q ii := ∑ j W ij . The Laplacian is symmetric positive semi-definite, and has an eigen-decomposition L = ΨΛΨ T . Throughout the presentation, we assume that Q ii > 0 for all i ∈ { 1 , . . . , n } . The Laplacian allows us to define the heat equation on G , with respect to an initial signal f 0 ∈ R n on G :

The solution of the above differential equation is obtained with the matrix exponential f ( t ) = e -t L f 0 , and we define the heat kernel on the graph as H t := e -t L . By eigendecomposition, we have H t = Ψ e -t Λ Ψ T . The matrix H t is a diffusion matrix that characterizes how a signal propagate through the graph according to the heat equations.

Other diffusion matrices on graphs have also been investigated in the literature. The transition matrix P := Q -1 W characterizing a random walk on the graph is another common diffusion matrix used for manifold learning such as PHATE [22] and diffusion maps [5]. It is a stochastic matrix that converges to a stationary distribution π i := Q ii / ∑ i Q ii , under mild assumptions.

Fast computation of heat diffusion Exact computation of the (discrete) heat kernel H t is computationally costly, requiring a full eigendecomposition in O ( n 3 ) time. Fortunately, multiple fast approximations have been proposed, including using orthogonal polynomials or the Euler backward methods. In this work, we use Chebyshev polynomials, as they have been shown to converge faster than other polynomials on this problem [13].

Chebyshev polynomials are defined by the recursive relation { T k } k ∈ N with T 0 ( y ) = 0 , T 1 ( y ) = y and T k ( y ) = 2 yT k -1 ( y ) -T k -2 ( y ) for k ≥ 2 . Assuming that the largest eigenvalue is less than two (which holds for the normalized Laplacian), we approximate the heat kernel with the truncated polynomials of order K

where the K +1 scalar coefficients { b t,i } depend on time and are evaluated with the Bessel function. Computing p K ( L , t ) f requires K matrix-vector product and K + 1 Bessel function evaluation. The expensive part of the computation are the matrix-vector products, which can be efficient if the Laplacian matrix is sparse. Interestingly, we note that the evaluation of T k do not depend on the diffusion time. Thus, to compute multiple approximations of the heat kernel { p K ( L , t ) } t ∈T , only necessitates reweighting the truncated polynomial { T k } k ∈ [1 ,...,K ] with the corresponding |T | sets of Bessel coefficients. The overall complexity is dominated by the truncated polynomial computation which takes O ( K ( E + n )) time where E is the number of non-zero values in L .

Another possible approximation is using the Euler backward method. It requires solving K systems of linear equations f ( t ) = ( I n + ( t/K ) L ) -K f (0) , which can be efficient for sparse matrices using the Cholesky decomposition [10, 28]. We quantify the differences between the heat kernel approximations in Appendix C.

Metric multidimensional scaling Given a dissimilarity function d between data points, metric multidimensional scaling (MDS) [16] finds an embedding ϕ such that the difference between the given dissimilarity and the Euclidean distance in the embedded space is minimal across all data points. Formally, for a given function d : R d × R d → R + , MDS minimizes the following objective:

In metric MDS the solution is usually found by the SMACOF algorithm [30], or stochastic gradient descent [37].

3 Related Work

We review state-of-the-art embedding methods and contextualize them with respect to Heat Geodesic Embedding. A formal theoretical comparison of all methods is given in Section 5. Given a set of high-dimensional datapoints, the objective of embedding methods is to create a map that embeds the observations in a lower dimensional space, while preserving distances or similarities. Different methods vary by their choice of distance or dissimilarity functions, as shown below.

Diffusion maps In diffusion maps [5], an embedding in k dimensions is defined via the first k non-trivial right eigenvectors of the t-steps random walk P t weighted by their eigenvalues. The embedding preserves the diffusion distance DM P ( x i , x j ) := ∥ ( δ i P t -δ j P t )(1 / π ) ∥ 2 , where δ i is a vector such that ( δ i ) j = 1 if j = i and 0 otherwise, and π is the stationary distribution of P . Intuitively, DM P ( x i , x j ) considers all the t -steps paths between x i and x j . A larger diffusion time can be seen as a low frequency graph filter, i.e. keeping only information from the low frequency transitions such has the stationary distributions. For this reason, using diffusion with t > 1 helps denoising the relationship between observations.

PHATE This diffusion-based method preserves the potential distance [22] PH P := ∥-log δ i P t + log δ j P t ∥ 2 , and justifies this approach using the log transformation to prevent nearest neighbors from dominating the distances. An alternative approach is suggested using a square root transformation. Part of our contributions is to justify the log transformation from a geometric point of view. The embedding is defined using multidimensional scaling, which we present below.

SNE, t-SNE, UMAP Well-known attraction/repulsion methods such as SNE [11], t-SNE [32], and UMAP [19] define an affinity matrix with entries p ij in the ambient space, and another affinity matrix with entries q ij in the embedded space. To define the embedding, a loss between the two affinity matrices is minimized. Specifically, the loss function is D KL ( p || q ) := ∑ ij p ij log p ij /q ij in SNE and t-SNE, whereas UMAP is equivalent to adding D KL (1 -p || 1 -q ) for unnormalized densities [2]. While these methods preserve affinities, they do not preserve any types of distances in the embedding.

4 Heat-Geodesic Embedding

In this section, we present our Heat Geodesic Embedding which is summarized in Alg. 1. We start by introducing the heat-geodesic dissimilarity, then present a robust transformation, and a heuristic to choose the optimal diffusion time. Proofs not present in the main text are given in AppendixA.

We consider the discrete case, where we have a set of n points { x i } n i =1 =: X in a high dimensional Euclidean space x i ∈ R d . From this point cloud, we want to define a map ϕ : R d → R k that embeds the observation in a lower dimensional space. An important property of our embedding is that we preserve manifold geodesic distances in a low dimensional space.

Heat-geodesic dissimilarity Inspired by Varadhan's formula and the Harnack inequalities, we defined a heat-geodesic dissimilarity based on heat diffusion on graphs. From observations (datapoints) in R d , we define an undirected graph G , and compute its heat kernel H t = e -t L , where L is the combinatorial or symmetrically normalized graph Laplacian (the heat kernel is thus symmetric). Following the inequality (2), we can rearange the terms to isolate the geodesic distance, inspired by this observation, we define the following dissimilarity.

Definition 4.1. For a diffusion time t > 0 and tunable parameter σ > 0 , we define the heat-geodesic dissimilarity between x i , x j ∈ X as

Here the log is applied elementwise, and the term -4 t log( H t ) ij corresponds to the geodesic approximation when t → 0 as in Varadhan's formula. In practice one uses a fixed diffusion time t > 0 , so we add a symmetric volume correction term as in the Harnack inequality, ensuring that d t ( x i , x j ) is symmetric. From Sec. 2, we have h t ( x, x ) ≃ V ( x, √ t ) -1 , and we use the diagonal of H t to approximate the inverse of the volume. With this volume correction term and σ = 1 , the dissimilarity is such that d t ( x i , x i ) = 0 for all t > 0 . When σ = 0 or the manifold has uniform volume growth (as in the constant curvature setting) we show that the heat-geodesic dissimilarity is order preserving:

Proposition 4.2. When σ = 0 or the manifold has uniform volume growth, i.e. ( H t ) ii = ( H t ) jj , and the heat kernel is pointwise monotonically decreasing w.r.t. a norm | · | in ambient space, we have for triples x i , x j , x k ∈ X that | x i -x j | > | x i -x k | implies d t ( x i , x j ) > d t ( x i , x k ) , i.e. the heat-geodesic dissimilarity is order preserving.

Proof. When σ = 0 or the manifold has uniform volume growth we need only consider the -4 t log( H t ) ij terms. The assumption of pointwise monotonicity of the heat kernel entails that | x i -x j | > | x i -x k | implies H t ( x i , x j ) < H t ( x i , x k ) . We are able to conclude that -4 t log H t ( x i , x j ) > -4 t log H t ( x i , x k ) and thus d t ( x i , x j ) > d t ( x i , x k ) .

Denoising distances with triplet computations We note that both diffusion maps and PHATE compute a triplet distance between datapoints, i.e., rather than using the direct diffusion probability between datapoints, they use a distance between corresponding rows of a diffusion operator. In particular, diffusion maps uses Euclidean distance, and PHATE uses an M-divergence. Empirically, we notice that this step acts as a denoiser for distances. We formalize this observation in the following proposition. We note D T the triplet distance. The triplet distance compares the distances relative to other points. Intuitively, this is a denoising step, since the effect of the noise is spread across the entire set of points. For a reference dissimilarity like the heat-geodesic, it is defined as D T ( x i , x j ) := ∥ d t ( x i , · ) -d t ( x j , · ) ∥ 2 . For linear perturbations of the form d t ( x i , x j ) + ϵ , where ϵ ∈ R , the effect of ϵ on D T ( x i , x j ) is less severe than on d t ( x i , x j ) . Our embedding is based on a linear combination between the heat-geodesic dissimilarity and its triplet distance (1 -ρ ) d t + ρD T , where ρ ∈ [0 , 1] .

̸

Proposition 4.3. Denote the perturbed triplet distance by ˜ D T ( x i , x j ) = || ˜ d t ( x i , · ) -˜ d t ( x j , · ) || 2 where ˜ d t ( x i , x j ) := d t ( x i , x j ) + ϵ and ˜ d t ( x i , x k ) := d t ( x i , x k ) for k = j . Then the triplet distance D T is robust to perturbations , i.e., for all ϵ > 0 ,

Optimal diffusion time Varadhan's formula suggests a small value of diffusion time t to approximate geodesic distance on a manifold. However, in the discrete data setting, geodesics are based on graph constructions, which in turn rely on nearest neighbors. Thus, small t can lead to disconnected graphs. Additionally, increasing t can serve as a way of denoising the kernel (which is often computed from noisy data) as it implements a low-pass filter over the eigenvalues, providing the additional advantage of adding noise tolerance. By computing a sequence of heat kernels ( H t ) t and evaluating their entropy H ( H t ) := -∑ ij ( H t ) ij log( H t ) ij , we select t with the knee-point method [26] on the function t ↦→ H ( H t ) . We show in Sec. 6.1 that our heuristic for determining the diffusion time automatically leads to better overall results.

Weighted MDS The loss in MDS (eq.5) is usually defined with uniform weights. Here, we optionally weight the loss by the heat kernel. In Sec. 5, we will show how this modification relates our method to the embedding defined by SNE[11]. For x i , x j ∈ X , we minimize ( H t ) ij ( d t ( x i , x j ) -∥ ϕ ( x i ) -ϕ ( x j ) ∥ 2 ) 2 . This promotes geodesic preservation of local neighbors, since more weights are given to points with higher affinities.

Heat-geodesic embedding To define a lower dimensional embedding of a point cloud X , we construct a matrix from the heat-geodesic dissimilarity, and then use MDS to create the embedding. ( ) 2 .

Our embedding defines a map ϕ that minimizes d t ( x i , x j ) -∥ ϕ ( x i ) -ϕ ( x j ) ∥ 2 , for all x i , x j ∈ X Hence, it preserves the heat-geodesic dissimilarity as the loss decreases to zero. In Alg. 1, we present the main steps of our algorithm using the heat-geodesic dissimilarity. A detailed version is presented in Appendix A.

Algorithm 1 Heat Geodesic Embedding

1: Input: N × d dataset matrix X , denoising parameter ρ ∈ [0 , 1] , Harnack regularization σ > 0 , output dimension k .

2: Returns: N × k embedding matrix E .

3: H t ← p K ( L , t )

4: t ← Kneedle { H ( H t ) } t

5: D ← [ -4 t log( H t ) ij -σ 4 t log( V t ) ij ] 1 / 2

6: D ← (1 -ρ ) D + ρD T

7: Return E ← MetricMDS( D , ∥ · ∥ 2 , k )

5 Relation to other manifold learning methods

In this section, we elucidate theoretical connections between the Heat Geodesic Embedding and other manifold learning methods. We relate embeddings via the eigenvalues of H t or P t with Laplacian eigenmaps and diffusion maps. We then present the relation between our methods and PHATE and SNE. We provide further analysis in the Appendix A. In particular, we introduce a new definition of kernel preserving embeddings; either via kernel-based distances (diffusion maps, PHATE) or via similarities (e.g. t-SNE, UMAP).

Diffusion maps with the heat kernel Diffusion maps [5] define an embedding with the first k eigenvectors ( ϕ i ) i of P , while Laplacian eigenmaps [1] uses the eigenvectors ( ψ i ) i of L . In the following, we recall the links between the two methods, and show that a rescaled Laplacian eigenmaps preserves the diffusion distance with the heat kernel H t .

Lemma5.1. Rescaling the Laplacian eigenmaps embedding with x i ↦→ ( e -2 tλ 1 ψ 1 ,i , . . . , e -2 tλ k ψ k,i ) preserves the diffusion distance DM H t .

Relation to PHATE The potential distance in PHATE (Sec. 3) is defined by comparing the transition probabilities of two t -steps random walks initialized from different vertices. The transition matrix P t mimics the heat propagation on a graph. The heat-geodesic dissimilarity provides a new interpretation of PHATE. In the following proposition, we show how the heat-geodesic relates to the PHATE potential distance with a linear combination of t -steps random walks.

▷

Heat approximation

▷ Knee detection e.g. [26]

▷ log is applied elementwise

▷

Triplet interpolation step

Proposition 5.2. The PHATE potential distance with the heat kernel PH H t can be expressed in terms of the heat-geodesic dissimilarity with σ = 0

and it is equivalent to a multiscale random walk distance with kernel ∑ k> 0 m t ( k ) P k , where m t ( k ) := t k e -t /k ! .

Proof. We present a simplified version of the proof, more details are available in Appendix A. For σ = 0 , we have d t ( x i , x j ) = -4 t log( H t ) ij , the relation between the PHATE potential and the heat-geodesic follows from the definition

Using the heat kernel H t with the random walk Laplacian L rw = Q -1 L = I n -Q -1 W corresponds to a multiscale random walk kernel. We can write L rw = S Λ S -1 , where S := Q -1 / 2 Ψ . Since P = I n -L rw , we have P t = S ( I n -Λ) t S -1 . Interestingly, we can relate the eigenvalues of H t and P with the Poisson distribution. The probability mass function of a Poisson distribution with mean t is given by m t ( k ) := t k e -t /k ! . For t ≥ 0 , we have e -t (1 -µ ) = ∑ k ≥ 0 m t ( k ) µ k . With this relationship, we can express H t as a linear combination of P t weighted by the Poisson distribution. Indeed, substituting λ = 1 -µ in yields

Remark 5.3 . In the previous proposition, the same argument holds for the symmetric Laplacian and the affinity matrix A := Q -1 / 2 WQ -1 / 2 used in other methods such as diffusion maps [5]. This is valid since we can write L sym = Q -1 / 2 ΨΛΨ T Q -1 / 2 , and A = I n -L sym .

Remark 5.4 . This proposition shows that, as the denoising parameter ρ → 1 , Heat Geodesic Embedding interpolates to the PHATE embeddings with a weighted kernel ∑ ∞ k =0 m t ( k ) P k .

Relation to SNE The heat-geodesic method also relates to SNE [11], and its variation using the Student distribution t-SNE [18]. In SNE, the similarity between points is encoded via transition probabilities p ij . The objective is to learn an affinity measure q , that depends on the embedding distances ∥ y i -y j ∥ 2 , such that it minimizes D KL ( p || q ) . Intuitively, points that have a strong affinity in the ambient space, should also have a strong affinity in the embedded space. Even though the heat-geodesic minimization is directly on the embedding distances, we can show an equivalent with SNE. In Appendix A, we provide additional comparisons between SNE and our method.

Proposition 5.5. The Heat-Geodesic embedding with σ = 0 and squared distances minimization weighted by the heat kernel is equivalent to SNE with the heat kernel affinity in the ambient space, and a Gaussian kernel in the embedded space q ij = exp( -∥ y i -y j ∥ 2 / 4 t ) .

6 Results

In this section, we show the versatility of our method, showcasing its performance in terms of clustering and preserving the structure of continuous manifolds. We compare the performance of Heat Geodesic Embedding with multiple state-of-the-art baselines on synthetic datasets and realworld datasets. For all models, we perform sample splitting with a 50/50 validation-test split. The validation and test sets each consists of 5 repetitions with different random initializations. The hyper-parameters are selected according to the performance on the validation set. We always report the results on the test set, along with the standard deviations computed over the five repetitions. We use the following methods in our experiments: our Heat Geodesic Embedding , diffusion maps [5], PHATE [22], shortest-path (used in Isomap [31]) which estimates the geodesic distance by computing the shortest path between two nodes in a graph built on the point clouds, t-SNE [32], UMAP [19], and metric MDS with Euclidean distance. Details about each of these methods, and results for different parameters (graph type, heat approximation, etc.) are given in Appendix C.

6.1 Distance matrix comparison

We start by evaluating the ability of the different distances or dissimilarities to recover the ground truth distance matrix of a point cloud. For this task, we use the Swiss roll and Tree datasets, for which the ground truth geodesic distance is known. The Swiss roll dataset consists of data points sampled on a smooth manifold (see Fig. 1). The Tree dataset is created by connecting multiple high-dimensional Brownian motions in a tree-shape structure. In Fig. 1, we present embeddings of both datasets. Our method recovers the underlying geometry, while other methods create artificial clusters or have too much denoising. Because we aim at a faithful relative distance between data points, we compare the methods according to the Pearson and Spearman correlations of the estimated distance matrices with respect to ground truth. Results are displayed in Tab. 1. We observe that Heat Geodesic Embedding typically outperforms previous methods in terms of the correlation with the ground truth distance matrix, confirming the theoretical guarantees provided in Sec. 4 & 2. Additional results such as computation time and correlation for different noise levels are available in Appendix C.

Optimal diffusion time In Section 4, we described a heuristic to automatically choose the diffusion time based on the entropy of H t . In Fig. 2, we show that the knee-point of t ↦→ H ( H t ) , corresponds to a high correlation with the ground distance, while yielding a low approximation error of the distance matrix (measured by the Frobenius norm of the difference between D and the ground truth).

6.2 Preservation of the inherent data structure

A crucial evaluation criteria of manifold learning methods is the ability to capture the inherent structure of the data. For instance, clusters in the data should be visible in the resulting low dimensional representation. Similarly, when the dataset consists of samples taken at different time points, one expects to be able to characterize this temporal evolution in the low dimensional embedding [22]. We thus compare the different embedding methods according to their ability to retain clusters and temporal evolution of the data.

Impact of the diffusion time on theperformance metrics

1.0

0.9

0.8

0.7

0.6

0.5

Spearman-p

Pearson-p

Normalized I1-

0.4

Normalized Entropy

Automatic Diffusion Time t

0

20

40

60

80

100

Diffusion time t

UMAP

t-SNE

PHATE

Isomap

Diffusion Maps

HeatGeo (ours)

Days

[0,3]

[6,9]

[12.15]

[18,21]

[24,27]

Identifying clusters. We use the PBMC dataset, the Swiss roll, the Tree dataset, MNIST [8], and COIL-20 [23] dataset. The PBMC dataset consists of single-cell gene expressions from 3000 individual peripheral blood mononuclear cells. Cells are naturally clustered by their cell type. For

PHATE

HeatGeop=0

HeatGeop=0.5

HeatGeop=1

Cell types

CD4 T cells

CD14+ Monocytes

B cells

CD8 T cells

NK cells

FCGR3A+ Monocytes

Dendritic cells

Megakaryocytes

the Tree dataset, we use the branches as clusters. For the Swiss roll dataset, we sample data points on the manifold according to a mixture of Gaussians and use the mixture component as the ground truth cluster labels. The MNIST and COIL-20 datasets are clustered by digits or objects but may not respect the manifold hypothesis. For each method, we run k-means on the two-dimensional embedding and compare the resulting cluster assignments with ground truth. Tab. 10 reports the results in terms of homogeneity and adjusted mutual information (aMI). Heat Geodesic Embedding is competitive with PHATE and outperforms t-SNE and UMAP on all metrics except on the MNIST and COIL-20 datasets. Yet, we show in Appendix C that all methods tend to perform equally well when the noise level increases. In Fig. 4, we present the PBMC embeddings of PHATE and HeatGeo, showing that HeatGeo interpolates to PHATE for ρ → 1 .

Temporal data representation. For this task, we aim at representing data points from population observed at consecutive points in time. We use single cell gene expression datasets collected across different time points, including the Embryoid Body (EB), IPSC [22], and two from the 2022 NeurIPS multimodal single-cell integration challenge (Cite & Multi). To quantitatively evaluate the quality of the continuous embeddings, we first embed the entire dataset and obfuscate all samples from a particular time point ( e.g., t = 2 ). We then estimate the distribution of the missing time point by using displacement interpolation [35] between the adjacent time points ( e.g., t = 1 and t = 3 ). We report the Earth Mover Distance (EMD) between the predicted distribution and true distribution. A low EMD suggests that the obfuscated embeddings are naturally located between the previous and later time points, and that the generated embedding captures the temporal evolution of the data adequately. Results are presented in Tab. 3. Heat Geodesic Embedding outperforms other methods on the EB, Multi, and IPSC datasets and is competitive with other approaches on Cite. We show a graphical depiction of the different embeddings for the embryoid (EB) dataset in Fig. 3.

7 Conclusion and Limitations

The ability to visualize complex high-dimensional data in an interpretable and rigorous way is a crucial tool of scientific discovery. In this work, we took a step in that direction by proposing a general framework for understanding diffusion-based dimensionality reduction methods through the lens of Riemannian geometry. This allowed us to define a novel embedding based on the heat geodesic dissimilarity-a more direct measure of manifold distance. Theoretically, we showed that our methods brings greater versatility than previous approaches and can help gaining insight into popular manifold learning methods such as diffusion maps, PHATE, and SNE. Experimentally, we demonstrated that it also results in better geodesic distance preservation and excels both at clustering and preserving the structure of a continuous manifold. This contrasts with previous methods that are typically only effective at a single of these tasks.

Despite the strong theoretical and empirical properties, our work presents some limitations. For instance, our method is based on a similarity measure, which is a relaxation of a distance metric. Additionally, the Harnack equation suggests that our parameters for the volume correction could be tuned depending on the underlying manifold. We envision that further analysis of this regularization is a fruitful direction for future work.

Acknowledgments and Disclosure of Funding

This research was enabled in part by compute resources provided by Mila (mila.quebec). It was partially funded and supported by ESP Mérite [G.H.], CIFAR AI Chair [G.W.], NSERC Discovery grant 03267 [G.W.], NIH grants (1F30AI157270-01, R01HD100035, R01GM130847, R01GM135929) [G.W.,S.K.], NSF Career grant 2047856 [S.K.], the Chan-Zuckerberg Initiative grants CZF2019182702 and CZF2019-002440 [S.K.], the Sloan Fellowship FG-2021-15883 [S.K.], and the Novo Nordisk grant GR112933 [S.K.]. The content provided here is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies. The funders had no role in study design, data collection & analysis, decision to publish, or preparation of the manuscript.

References

Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation , 15(6):1373-1396, 2003.

Jan Niklas Böhm, Philipp Berens, and Dmitry Kobak. Attraction-repulsion spectrum in neighbor embeddings. Journal of Machine Learning Research , 23(95):1-32, 2022.

Erica L Busch, Jessie Huang, Andrew Benz, Tom Wallenstein, Guillaume Lajoie, Guy Wolf, Smita Krishnaswamy, and Nicholas B Turk-Browne. Multi-view manifold learning of human brain-state trajectories. Nature Computational Science , pages 1-14, 2023.

J Douglas Carroll and Phipps Arabie. Multidimensional scaling. Measurement, judgment and decision making , pages 179-250, 1998.

Ronald R. Coifman and Stéphane Lafon. Diffusion maps. Applied and Computational Harmonic Analysis , 21(1):5-30, 2006.

Keenan Crane, Clarisse Weischedel, and Max Wardetzky. Geodesics in heat: A new approach to computing distance based on heat flow. ACM Transactions on Graphics (TOG) , 32(5):1-11, 2013.

Michaël Defferrard, Lionel Martin, Rodrigo Pena, and Nathanaël Perraudin. Pygsp: Graph signal processing in python.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Laleh Haghverdi, Maren Büttner, F. Alexander Wolf, Florian Buettner, and Fabian J. Theis. Diffusion pseudotime robustly reconstructs lineage branching. Nature Methods , 13(10):845-848, 2016.

Matthieu Heitz, Nicolas Bonneel, David Coeurjolly, Marco Cuturi, and Gabriel Peyré. Ground metric learning on graphs. Journal of Mathematical Imaging and Vision , 63:89-107, 2021.

Geoffrey E Hinton and Sam Roweis. Stochastic neighbor embedding. Advances in neural information processing systems , 15, 2002.

Michael C Hout, Megan H Papesh, and Stephen D Goldinger. Multidimensional scaling. Wiley Interdisciplinary Reviews: Cognitive Science , 4(1):93-103, 2013.

Shih-Gu Huang, Ilwoo Lyu, Anqi Qiu, and Moo K Chung. Fast polynomial approximation of heat kernel convolution on manifolds and its application to brain sulcal and gyral graph pattern analysis. IEEE transactions on medical imaging , 39(6):2201-2212, 2020.

Guillaume Huguet, Alexander Tong, Bastian Rieck, Jessie Huang, Manik Kuchroo, Matthew Hirn, Guy Wolf, and Smita Krishnaswamy. Time-inhomogeneous diffusion geometry and topology. arXiv preprint arXiv:2203.14860 , 2022.

Guillaume Huguet, Alexander Tong, María Ramos Zapatero, Guy Wolf, and Smita Krishnaswamy. Geodesic Sinkhorn: optimal transport for high-dimensional datasets. arXiv preprint arXiv:2211.00805 , 2022.

Joseph B Kruskal. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika , 29(1):1-27, 1964.

Peter Li and Shing Tung Yau. On the parabolic kernel of the Schrödinger operator. Acta Mathematica , 156(none):153 - 201, 1986.

George C. Linderman and Stefan Steinerberger. Clustering with t-SNE, provably. arXiv:1706.02582 [cs, stat] , 2017.

Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

Stanislav A Molchanov. Diffusion processes and riemannian geometry. Russian Mathematical Surveys , 30(1):1, 1975.

Kevin R. Moon, Jay S. Stanley, Daniel Burkhardt, David van Dijk, Guy Wolf, and Smita Krishnaswamy. Manifold learning-based methods for analyzing single-cell RNA-sequencing data. Current Opinion in Systems Biology , 7:36-46, 2018.

Kevin R. Moon, David van Dijk, Zheng Wang, Scott Gigante, Daniel B. Burkhardt, William S. Chen, Kristina Yim, Antonia van den Elzen, Matthew J. Hirn, Ronald R. Coifman, Natalia B. Ivanova, Guy Wolf, and Smita Krishnaswamy. Visualizing structure and transitions in highdimensional biological data. Nat Biotechnol , 37(12):1482-1492, 2019.

Samer A. Nene, Shree K. Nayar, and Hiroshi Murase. Columbia object image library (coil20). Technical Report CUCS-005-96, Department of Computer Science, Columbia University, February 1996.

Adam Nowak, Peter Sjögren, and Tomasz Z Szarek. Sharp estimates of the spherical heat kernel. Journal de Mathématiques Pures et Appliquées , 129:23-33, 2019.

Laurent Saloff-Coste. The heat kernel and its estimates. Probabilistic approach to geometry , 57:405-436, 2010.

Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. Finding a" kneedle" in a haystack: Detecting knee points in system behavior. In 2011 31st international conference on distributed computing systems workshops , pages 166-171. IEEE, 2011.

Nicholas Sharp, Yousuf Soliman, and Keenan Crane. The vector heat method. ACMTransactions on Graphics (TOG) , 38(3):1-19, 2019.

Justin Solomon, Fernando De Goes, Gabriel Peyré, Marco Cuturi, Adrian Butscher, Andy Nguyen, Tao Du, and Leonidas Guibas. Convolutional wasserstein distances: Efficient optimal transportation on geometric domains. ACM Transactions on Graphics (ToG) , 34(4):1-11, 2015.

Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multiscale signature based on heat diffusion. In Computer graphics forum , volume 28, pages 1383-1392. Wiley Online Library, 2009.

Yoshio Takane, Forrest W Young, and Jan De Leeuw. Nonmetric individual differences multidimensional scaling: An alternating least squares method with optimal scaling features. Psychometrika , 42:7-67, 1977.

Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science , 290(5500):2319-2323, 2000.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

David Van Dijk, Roshan Sharma, Juozas Nainys, Kristina Yim, Pooja Kathail, Ambrose J Carr, Cassandra Burdziak, Kevin R Moon, Christine L Chaffer, Diwakar Pattabiraman, et al. Recovering gene interactions from single-cell data using data diffusion. Cell , 174(3):716-729, 2018.

Sathamangalam R Srinivasa Varadhan. On the behavior of the fundamental solution of the heat equation with variable coefficients. Communications on Pure and Applied Mathematics , 20(2): 431-455, 1967.

Cédric Villani. Displacement interpolation. Optimal Transport: Old and New , pages 113-162, 2009.

F Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology , 19:1-5, 2018.

Jonathan X Zheng, Samraat Pawar, and Dan FM Goodman. Graph drawing by stochastic gradient descent. IEEE transactions on visualization and computer graphics , 25(9):2738-2748, 2018.

Appendix

A Theory and algorithm details

A.1 Kernel preserving embeddings

In this section, we attempt to create a generalized framework for dimensionality reduction methods. These methods often have been viewed as disparate or competing but here we show that many of them are related to one another given the right template for methodology comparison. In order to do this, we introduce a general definition suited for distance-preserving dimensionality reduction methods. With this definition, we can cast many dimensionality reduction methods within the same framework, and easily compare them. We recall that the observations in the ambient space are denoted x , and those in the embedded space are denoted y . The definition relies on kernel functions H x t , H y t defined respectively on the ambient and embedded spaces and on transformations T x , T y applied to the kernels. We recall that a divergence f : R × R → R + is such that f ( a, b ) = 0 if and only if a = b and f ( a, a + δ ) is a positive semi-definite quadratic form for infinitesimal δ .

Definition A.1. We define a kernel features preserving embedding as an embedding which minimizes a loss L between a transformation T x of the ambient space kernel H x t and its embedded space counterpart

where f is any C 2 divergence on R ≥ 0 .

Example 1 . We formulate MDS as a kernel feature-preserving embedding. Suppose we want to preserve the Euclidean distance, we have H x t ( x i , x j ) = ∥ x i -x j ∥ 2 , H y t ( y i , y j ) = ∥ y i -y j ∥ 2 , f ( a, b ) = ∥ a -b ∥ 2 , and T x = T y = I .

In the following, we present popular dimensionality reduction methods that are kernel features preserving embeddings. With this definition, we can distinguish between methods that a preserve a kernel via affinities or distances. For the methods considered in this work, H x t is an affinity kernel, but its construction varies from one method to another. In PHATE and Diffusion maps, H x t is a random walk P , while in Heat Geodesic Embedding we use the heat kernel H t . t-SNE defines H x t as a symmetrized random walk matrix from a Gaussian kernel, while UMAP uses an unnormalized version. Methods such as PHATE and diffusion maps define a new distance matrix from a kernel in the ambient space and preserve these distances in the embedded space. Other methods like t-SNE and UMAP define similarities from a kernel and aim to preserve these similarities in the ambient space and embedded space via an entropy-based loss. We note the Kullback-Leibler divergence D KL ( a, b ) = ∑ ij a ij log[ a ij /b ij ] .

Proposition A.2. The embeddings methods HeatGeo, PHATE, Diffusion Maps, SNE, t-SNE, and UMAP are kernel feature-preserving embeddings.

Proof. We assume that the affinity kernel in the ambient space H x t , is given, to complete the proof we need to define f, H y t , T x , T y for all methods.

We start with the distance preserving embeddings; HeatGeo, PHATE, and Diffusion Maps. For these methods, the kernel in the embed space is simply H y t ( y i , y j ) = ∥ y i -y j ∥ 2 , without transformation, i.e. T y = I . Since they preserve a distance, the loss is f ( T x ( H x t ) , T y ( H y t ′ )) = ∥ H x t -H y t ′ ∥ 2 .

In the Heat Geodesic Embedding we apply a transformation on H x t = H t to define a dissimilarity, hence T x ( H x t ) = -t log H x t (for σ = 0 ), where log is applied elementwise.

In PHATE, the potential distance is equivalent to ( T x ( H x t )) ij = ∥ -log( H x t ) i +log( H x t ) j ∥ 2 . In Diffusion Maps, the diffusion distance is ( T x ( H x t )) ij = ∥ ( H x t ) i -( H x t ) j ∥ 2 .

SNE, t-SNE, and UMAP preserve affinities from a kernel. For these three methods, the loss is a divergence between distributions, namely f = D KL . They vary by defining different affinity kernel and transformation in the embedded space. SNE uses the unnormalized kernel H y t ( y i , y j ) = exp( -(1 /t ) ∥ y i -y j ∥ 2 2 ) , with T x = T y = I . Whereas, t-SNE uses ( H y 1 ) ij = (1 + ∥ y i -y j ∥ 2 ) -1 ,

and T x = T y = I . UMAP define a pointwise transformation in the embedded space with ( H y 1 ) ij = (1 + ∥ y i -y j ∥ 2 ) -1 , ( T y ( H y t )) ij = ( H y 1 ) ij / (1 -( H y 1 ) ij ) , and T x = I .

We summarize the choice of kernels and functions in Tab. 4

A.2 Proofs

Proposition 4.3. Denote the perturbed triplet distance by ˜ D T ( x i , x j ) = || ˜ d t ( x i , · ) -˜ d t ( x j , · ) || 2 where ˜ d t ( x i , x j ) := d t ( x i , x j ) + ϵ and ˜ d t ( x i , x k ) := d t ( x i , x k ) for k = j . Then the triplet distance D T is robust to perturbations , i.e., for all ϵ > 0 ,

̸

Proof of Proposition 4.3. The effect of the noise on the square distance is ( d t ( x i , x j ) + ϵ ) 2 /d ( x i , x j ) 2 = 1 + (2 ϵd t ( x i , x j ) + ϵ 2 ) /d ( x i , x j ) 2 . Denoting the perturbed triplet distance by ˜ D T , we have

̸

and we have

For ϵ > 0 , this gives

For ϵ < 0 , we have

Thus ϵ ∈ ( -∞ , -2 d t ( x i , x j )) ∪ (0 , ∞ ) . As we require the perturbation factor ϵ << d t ( x i , x j ) , hence we choose ϵ ∈ (0 , ∞ ) .

Lemma5.1. Rescaling the Laplacian eigenmaps embedding with x i ↦→ ( e -2 tλ 1 ψ 1 ,i , . . . , e -2 tλ k ψ k,i ) preserves the diffusion distance DM H t .

Proof of Lemma 5.1. Since the eigendecompotision of H t form an orthonormal basis of R n , and since its first eigenvector is constant, we can write the diffusion distance ∥ δ i H t -δ i H t ∥ 2 2 = ∑ k ≥ 0 e -2 tλ k ( ψ ki -ψ kj ) 2 = ∑ k ≥ 1 e -2 tλ k ( ψ ki -ψ kj ) 2 . In particular, this defines the k dimensional embedding x ↦→ ( e -tλ 1 ψ 1 ( x ) , . . . , e -tλ k ψ k ( x )) .

Proposition 5.2. The PHATE potential distance with the heat kernel PH H t can be expressed in terms of the heat-geodesic dissimilarity with σ = 0

and it is equivalent to a multiscale random walk distance with kernel ∑ k> 0 m t ( k ) P k , where m t ( k ) := t k e -t /k ! .

Proof of Proposition 5.2. For σ = 0 , we have d t ( x i , x j ) = -4 t log( H t ) ij , the relation between the PHATE potential and the heat-geodesic follows from the definition

Using the heat kernel H t with the random walk Laplacian L rw = Q -1 L = I n -Q -1 W corresponds to a multiscale random walk kernel. Recall that we can write L rw in terms of the symmetric Laplacian L rw = Q -1 / 2 L s Q 1 / 2 , meaning that the two matrices are similar, hence admit the same eigenvalues Λ . We also know that L s is diagonalizable, since we can write L s = Q -1 / 2 LQ -1 / 2 = Q -1 / 2 ΨΛΨ T Q -1 / 2 . In particular, we have L rw = S Λ S -1 , where S := Q -1 / 2 Ψ . The random walk matrix can be written as P = I n -R rw , hence its eigenvalues are ( I n -Λ) , and we can write P t = S ( I n -Λ) t S -1 . Similarly, the heat kernel with the random walk Laplacian can be written as H t = S e -t Λ S -1 . Interestingly, we can relate the eigenvalues of H t and P with the Poisson distribution. Note the probability mass function of a Poisson as m t ( k ) := t k e -t /k ! , for t ≥ 0 , we have

We note that (7) is the probability generating function of a Poisson distribution with parameter t , i.e. E [ µ X ] , where X ∼ Poisson ( t ) . With this relationship, we can express H t as a linear combination of P t weighted by the Poisson distribution. Indeed, substituting λ = 1 -µ in (7) links the eigenvalues of H t and P . We write the heat kernel as a linear combination of random walks weighted by the Poisson distribution, we have

Proposition 5.5. The Heat-Geodesic embedding with σ = 0 and squared distances minimization weighted by the heat kernel is equivalent to SNE with the heat kernel affinity in the ambient space, and a Gaussian kernel in the embedded space q ij = exp( -∥ y i -y j ∥ 2 / 4 t ) .

Proof of Proposition 5.5. The squared MDS weighted by the heat kernel corresponds to

If there exists an embedding that attain a zero loss, then it is the same as ∑ ij h t ( x i , x j )(log h t ( x i , x j ) -log exp( -∥ y i -y j ∥ 2 /t ) = D KL ( h t || q ) .

A.3 Algorithm details

We present a detailed version of the Heat Geodesic Embedding algorithm in Alg.2.

For the knee-point detection we use the Kneedle algorithm [26]. It identifies a knee-point as a point where the curvature decreases maximally between points (using finite differences). We summarize the four main steps of the algorithm for a function f ( x ) , and we refer to [26] for additional details.

Algorithm 2 Heat Geodesic Embedding

Smoothing with a spline to preserve the shape of the function.

Normalize the values, so the algorithm does not depend on the magnitude of the observations.

Computing the set of finite differences for x and y := f ( x ) , e.g. y d i := f ( x i ) -x i .

Evaluating local maxima of the difference curve y d i , and select the knee-point using a threshold based on the average difference between consecutive x .

B Experiments and datasets details

Our experiments compare our approach with multiple state-of-the-art baselines for synthetic datasets (for which the true geodesic distance is known) and real-world datasets. For all models, we perform sample splitting with a 50/50 validation-test split. The validation and test sets each consist of 5 repetitions with different random initializations. The hyper-parameters are selected according to the performance on the validation set. We always report the results on the test set, along with the standard deviations computed over the five repetitions. We use the following state-of-the-art methods in our experiments: our Heat Geodesic Embedding, diffusion maps [5], PHATE [22], Heat-PHATE (a variation of PHATE using the Heat Kernel), Rand-Geo (a variation of Heat Geodesic Embedding where we use the random walk kernel), Shortest-path which estimates the geodesic distance by computing the shortest path between two nodes in a graph built on the point clouds, t-SNE [32], and UMAP [19].

B.1 Datasets

We consider two synthetic datasets, the well known Swiss roll and the tree datasets. The exact geodesic distance can be computed for these datasets. We additionally consider real-world datasets: PBMC, IPSC [22], EB [22], and two from the from the 2022 NeurIPS multimodal single-cell integration challenge 4 .

B.1.1 Swiss Roll

The Swiss roll dataset consists of data points samples on a smooth manifold inspired by shape of the famous alpine pastry. In its simplest form, it is a 2-dimensional surface embedded in R 3 given by

where t ∈ [ T 0 , T 1 ] and h ∈ [0 , W ] . In our experiments we used T 0 = 3 2 π , T 1 = 9 2 π , and W = 5 . We use two sampling mechanisms for generating the data points : uniformly and clustered. In the first, we sample points uniformly at random in the [ T 0 , T 1 ] × [0 , W ] plane. In the second, we sample according to a mixture of isotropic multivariate Gaussian distributions in the same plane with equal weights, means [(7 , W/ 2) , (12 , W/ 2)] , and standard deviations [1 , 1] . In the clustered case, data samples are given a label y according to the Gaussian mixture component from which they were sampled.

We consider variations of the Swiss roll by projecting the data samples in higher dimension using a random rotation matrix sampled from the Haar distribution. We use three different ambient dimensions: 3, 10, and 50.

Finally, we add isotropic Gaussian noise to the data points in the ambient space with a standard deviation σ .

B.1.2 Tree

The tree dataset is created by generating K branches from a D -dimensional Brownian motion that are eventually glued together. Each branch is sampled from a multidimensional Brownian motion d X k = 2 d W ( t ) at times t = 0 , 1 , 2 , ..., L -1 for k ∈ [ K ] . The first branch is taken as the main branch and the remaining branches are glued to the main branch by setting X k = X k + X 0 [ i k ] where i k is a random index of the main branch vector. The total number of samples is thus L · K

In our experiments, we used L = 500 , K = 5 , and D = 5 , 10 ( i.e., two versions with different dimensions of the ambient space).

B.2 Evaluation Metrics

We compare the performance of the different methods according to several metrics. For synthetic datasets, where ground truth geodesic distance is available, we directly compare the estimated distance matrices and ground truth geodesic distance matrices. For real-world datasets, we use clustering quality and continuous interpolation as evaluation metrics.

B.2.1 Distance matrix evaluation

The following methods use an explicit distance matrix: diffusion maps, Heat Geodesic Embedding, Heat-Phate, Phate, Rand-Geo and Shortest Path. For these methods, we compare their ability their ability to recover the ground truth distance matrix several metrics. Letting D and ˆ D the ground truth and inferred distance matrices respectively, and N the number of points in the dataset, we use the following metrics.

Pearson ρ We compute the average Pearson correlation between the rows of the distance matrices, 1 N ∑ N i =1 r D i , ˆ D i , where r x,y is the Pearson correlation coefficient between vectors x and y . D i stands for the i -th row of D .

Spearman ρ We compute the average Spearman correlation between the rows of the distance matrices, 1 N ∑ N i =1 r D i , ˆ D i , where r x,y is the Spearman correlation coefficient between vectors x and y . D i stands for the i -th row of D .

Maximum Norm We use ∥ D -ˆ D ∥ ∞ , where ∥ A ∥ ∞ = max i,j | A i,j |

B.2.2 Embedding evaluation

Some methods produce low-dimensional embeddings without using an explicit distance matrix for the data points. This is the case for UMAP and t-SNE. To compare against these methods, we use the distance matrix obtained by considering Euclidean distance between the low-dimensional embeddings. We used 2-dimensional embeddings in our experiments. For diffusion maps, we obtain

these embeddings by using the first two eigenvectors of the diffusion operator only. For Heat Geodesic Embedding, Heat-PHATE, PHATE, Rand-GEO and Shortest Path, we use multidimensional scaling (MDS) on the originally inferred distance matrix.

Clustering We evaluate the ability of Heat Geodesic Embedding to create meaningful embeddings when clusters are present in the data. To this end, we run a k-means clustering on the two dimensional embeddings obtained with each method and compare them against the ground truth labels. For the Tree dataset, we use the branches as clusters. For the Swiss roll dataset, we sample data points on the manifold according to a mixture of Gaussians and use the mixture component as the ground truth cluster label.

Interpolation To quantitatively evaluate the quality of the continuous embeddings, we first embed the entire dataset and obfuscate all samples from a particular time point ( e.g., t = 2 ). We then estimate the distribution of the missing time point by using displacement interpolation [35] between the adjacent time points ( e.g., t = 1 and t = 3 ). We report the Earth Mover Distance (EMD) between the predicted distribution and true distribution. A low EMD suggests that the obfuscated embeddings are naturally located between the previous and later time points, and that the generated embedding captures the temporal evolution of the data adequately.

B.3 Hyperparameters

In Table 5, we report the values of hyperparameters used to compute the different embeddings.

B.4 Hardware

The experiments were performed on a compute node with 16 Intel Xeon Platinum 8358 Processors and 64GB RAM.

C Additional results

C.1 HeatGeo weighted

Following Sec. 5, we know that weighting the MDS loss by the heat kernel corresponds to a specific parametrization of SNE, and thus promote the identification of cluster. In Fig. 5, we show the embeddings of four Gaussian distributions in 10 dimensions (top), and the PBMC dataset (bottom). The reference embedding is using t-SNE, as it models as it also minimizes the KL between the ambient and embedded distributions. We see that HeatGeo weighted form cluster that are shaped like a Gaussian. This is expected as Prop. 5.5, indicates that this is equivalent to minimizing the D KL between the heat kernel and a Gaussian affinity kernel.

Ref

HeatGeo

HeatGeoWeighted

10D

Gaussians

PBMC

C.2 Truncated distance

In Fig.6, we discretize the interval [0 , 51] in 51 nodes, and we compute the heat-geodesic distance of the midpoint with respect to the other points, effectively approximating the Euclidean distance. Using Chebyshev polynomials of degree of 20, we see that the impact of the truncation is greater as the diffusion time increases. The backward Euler methods does not result in a truncated distance.

Squared Heat-Geodesic distance on R

t=1

t=10

t=50

150

1500

Exact

6000

Chebyshev

100

1000

4000

50

500

V(x,Vt)

2000

0

0

0

0

25

50

75

100

0

25

50

75

100

0

25

50

75

100

order = 1

order = 2

order =3

order = 4

order = 5

order =6

order = 7

order=8

6 = p

order =10

order = 11

order = 12

order = 13

order = 14

order = 15

order =16

order = 17

order = 18

order =19

order = 20

order = 21

order = 22

order = 23

order = 24

order = 25

C.3 Harnack inequality

For complete Riemannian manifolds that satisfy the parabolic Harnack inequality (PHI) we have h t ( x, y ) ≃ V -1 ( x, √ t ) e -d ( x,y ) 2 /t so that -t log h t ( x, y ) ≃ t log V ( x, √ t ) + d 2 ( x, y ) [25].

C.3.1 Case studies for specific manifolds

The circle S 1 We now show that our expression for the Heat Geodesic Embedding-distance is monotonically increasing with respect to the ground truth geodesic distance d ∈ R + for a fixed diffusion time t and for any Harnack regularization in S 1 .

We then have,

Our expression for the Heat Geodesic Embedding-distance is

As the square-root is monotonic, and 4 t log h t (0) is constant with respect to d , we need to show that f ( d ) = -log ( h t ( d )) is monotonically increasing.

For S 1 , we have

Recalling Harnack inequality,

With c 2 = c 4 = 4 , we have

As log is monotonically increasing, it suffices to show that ∑ m ∈ Z e -( d +2 πm ) 2 4 t is monotonically decreasing , which is the case as for any d ′ > d , ∀ m ∈ Z , we have

In general, one can see that (1) the heat kernel depending only on the geodesic distance and (2) the heat kernel being monotonically decreasing with respect to the geodesic distance are sufficient conditions for preserving ordering of pair-wise distances with Heat Geodesic Embedding.

The sphere S n The above result can be applied to the higher-dimensional sphere S n . It is known that the heat kernel on manifold of constant curvatures is a function of the the geodesic distance ( d ) and time only. For S n the heat kernel is given by

with I the regularized incomplete beta function and C the Gegenbauer polynomials.

Furthermore, Nowak et al. [24] showed that the heat kernel of the sphere is monotonically decreasing. The distance inferred from Heat Geodesic Embedding thus preserves ordering of the pair-wise distances.

Euclidean ( R 3 ) For the euclidean space, we have for the volume of √ t -geodesic ball and for the heat kernel:

In this case, the bound can be made tight, by setting

we recover the exact geodesic distance.

C.4 Quantitative results

C.4.1 Distance matrix evaluation

We report the performance of the different methods in terms of the ground truth geodesic matrix reconstruction in Table. 6 for the Swiss roll dataset and in Table. 7, for the Tree dataset.

C.4.2 Distance matrix evaluation via two-dimensional embeddings

We report the performance of the different methods in terms of the ground truth geodesic matrix reconstruction in Table 8 for the Swiss roll dataset and in Table 9, for the Tree dataset.

C.4.3 Clustering quality evaluation

On Tables 10, we report the performance on clustering quality for the synthetic datasets with different noise level.

C.5 Impact of the different hyperparameters

We investigate the impact of the different hyperparameters on the quality of the embeddings. In Figure 8, we show the embeddings of HeatGeo for different values of diffusion time, number of neighbours, order, and Harnack regularization.

In Figures 9, 10, 11, and 12, we show the impact of different hyperparameters on the Pearson correlation between the estimated distance matrix and ground truth distance matrix for different methods on the Swiss roll dataset.

C.6 Graph construction

We compare the embeddings of the heat-geodesic distance for different graph construction. Throughout the paper we used the graph construction from PHATE [22]. In the following we present additional results depending on the choice of kernel to construct the graph. Specifically, we use a simple nearest neighbor (kNN) graph implemented in [7], the graph from UMAP [19], and the implementation in the package Scanpy [36] for single-cell analysis. In figure, we present the embeddings 2500 points of a tree with five branches in 10 dimensions, where the observations are perturbed with a standard Gaussian noise. All methods used five nearest neighbors and a diffusion time of 20. In Figure 13, we show the evolution of the Pearson correlation between estimated and ground truth distance matrices for the 10-dimensional Swiss roll dataset for various graph constructions. We note that the results are stable across different graph construction strategies.

C.7 Time Complexity

In Table 12, we present the average computing time for creating embeddings and corresponding distance matrix for the different methods. All methods are applied on the Swiss roll dataset in three dimension with 2000 samples. We present empirical averages and standard deviations over ten repetitions. The experiments were run on a Apple M2 Pro chip with 16G RAM.

Embeddings of HeatGeo for different hyperparameters

0.02

0.02

0.04

0.075

0.02

0.01

10°0

0.02

0.050

00°0

0.00

0.00

0.00

0.025

10'0-

10'0—

0.02

0.02

000°0

0.02

0°0-

0.025

0.02

0.02

000100-

0.01

0.02

0.02

L

00°0100-

10'0

0.02

0.02

0.00

0.02

0.025 0.0000.0250.050

0.075

0.05

0.00

0.05

0.10

0.15

Harnack p = 0

Harnack p = 0.5

Harnack p = 1.0

Harnack p = 1.5

Harnack p = 2

Diffusion Time

0.02

0.02

0.02

0.02

0.02

0.01

0.01

10'0

0.01

0.01

0.00

0.00

0.00

0.00

0.00

100-

10'0

0.01

0.01

0.01

0.02

0.02

0.02

0.02

0.02

0.02

0.00

0.02

0.02 0.010.00

0.01

0.02

0.02 0.010.00

0.010.02

0.02

100-

00°0

10'0

0.02

-0.02-0.01

0.00

0.01

0.02

Diffusion Time t = 1

Diffusion Time t = 5

Diffusion Time t = 10

Diffusion Time t = 20

Diffusion Time t = auto

Number of neighbours

0.02

0.02

0.02

0.02

0.02

0.01

0.01

100

0.01

10°0

0.00

0.00

0.00

0.00

0.00

100-

0.01

0.01

10'0-

10°0

0.02

20'0-

0.02

0.02

0.02

0.02

100-

0.00

10'0

0.02

0.02

00-

0.00

0.01

0.02

0.02

10'0

0.00

100

0.02

0.02

0.01

0.02

0.02

0.01

0.00

0.01

0.02

K = 5

K =10

K =15

K = 20

Order of Heat Kernel Approximation (Euler)

0.02

0.02

0.02

0.02

0.02

0.01

0.01

100

0.01

0.01

00°0

0.00

0.00

00°0

0.00

10'0

0.01

0.01

I0'0

0.01

0.02

0.02

0.02

0.02

0.02

0.02

0.010.00

0.01

0.02

0.020.010.000.01

0.02

0.020.010.000.01

0.02

0.020.010.00

10°0

0.02

0.020.010.00

0.01

0.02

Order =10

Order =20

Order=30

Order=40

Order =50

Order of Heat Kernel Approximation (Chebyshev)

0.04

0.02

0.02

0.02

0.02

200

0.01

10'0

0.01

10°0

00°0

0.00

0.00

00°0

0.00

0.01

0.01

10'0-

0.01

0.02

0.02

0.02

0.02

0.02

0.02

0.00

0.02

0.02 0.010.000.01

0.02

0.02

0.010.000.01

0.02

0.02

0.010.00

10°0

0.02

0.02

0.010.00

10'0

0.02

Order = 10

Order = 20

Order =30

Order = 40

Order =50

Harnack Regularization

Pearson-p in function of T on Swiss Roll data

1.0

Configuration:

0.8

Dataset:Swiss Roll

Noise:0.5

Pearson-p

0.6

Heat Geo

0.4

Heat Geo Auto

Heat PHATE

Heat PHATE Auto

Rand Geo

0.2

Phate - auto

Phate

Diffusion Map

0

20

40

60

80

100

T

Pearson-p in function of Order of the heat kernel approximation on Swiss Roll data

1.0

Heat Geo

HeatPHATE

0.9

Configuration:

0.8

Dataset:SwissRoll

Noise:0.5

FilterMethod:ChebyshevApproximation

Pearson-p

0.7

T:auto

0.6

0.5

0.4

0.3

0.2

10

20

30

40

50

Order of theheatkernel approximation

Pearson-p in function of k on Swiss Roll data

1.0

Configuration:

0.9

Dataset:Swiss Roll

Noise:0.5

Pearson-p

0.8

0.7

Heat Geo

HeatPHATE

Rand Geo

0.6

Phate

Diffusion Map

ShortestPath

5

10

15

20

25

30

k

Pearson-p in function of Harnack Regularization p on Swiss Roll data

Heat Geo

0.99

Configuration:

Dataset:Swiss Roll

0.98

Noise:0.5

FilterMethod:EulerApproximation

Pearson-p

T:auto

0.97

0.96

0.95

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Harnack Regularization p

Pearson p in function of Graph type on Swiss Roll 10D data

HeatGeo

0.900

0.898

0.896

p

Pearson

0.894

0.892

0.890

0.888

alpha

knn

scanpy

umap

Graphtype

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper \"A heat diffusion perspective on geodesic preserving dimensionality reduction\", the authors develop a new embedding method (HeatGeo), conceptually close to PHATE (Moon et al 2019) in that it performs MDS on some kind of diffusion distances. The authors make some strong claims about HeatGeo: (1) it is SOTA in preserving manifold distances; (2) it is SOTA in preserving cluster structures; (3) it can approximate both PHATE and t-SNE depending on parameters.\n\nI did not fully understand/follow all the math, but I found the paper nevertheless interesting as it gives a systematic/principled treatment of heat diffusion MDS methods. That said, I was not entirely convinced by the evaluations and especially claims (2) and (3) seem to me too strong and IMHO not sufficiently supported by the shown evidence. Also, many places were not really clear to me. Overall I am somewhat on the fence about this paper: it seems it is not fully ready yet. I am giving it a borderline score.\n\nSTRENGTHS:\n* Systematic/principled mathematical treatment (which admittedly I did not fully grasp)\n* Benchmarks of several embedding methods on artificial datasets\n* Applications to real RNAseq data\n* Relationships between HeatGeo and existing methods (PHATE/t-SNE)\n\nWEAKNESSES:\n* Relationship to PHATE is not very clear\n* Very bold claims that are not fully supported by the evidence\n* Insufficient evaluation, in particular many embeddings are not shown\n* Runtime is not discussed\n\nNow in more detail:\n\n* In spirit, the method seems very similar to PHATE. First, kNN graph is used to construct NxN distance matrix between all points (using heat diffusion). Second, MDS is used to produce the embedding. Of course there are many small differences from PHATE in how exactly all of this is setup up, and the way this paper is written, it comes across as more mathematically principled than PHATE. But in the end, I was still confused about what is the MAIN conceptual/algorithmic difference. That's very important in order to judge the relative novelty of HeatGeo. Is it conceptually a minor tweak to / simplification of PHATE, or is it conceptually a very different thing?\n\n* One bold claim is that HeatGeo outperforms other methods in preserving clusters (supported by Table 2). This is conceptually surprising to me, because I expect some trade-off between preserving geodesic distances and preserving clusters (similar trade-off was discussed in Ref [2]: https://jmlr.org/papers/v23/21-0055.html), so it's odd if the same method (HeatGeo) wins in both competitions. Also, we know that methods like t-SNE and UMAP really excel in representing clusters in high-dim data (just try them on MNIST) but more geodesic/MDS methods like Isomap or PHATE perform worse (again, just try them on MNIST). \n\n  To be more convinced here, I would like to see the embeddings corresponding to Table 2. Also, I don't think \"Tree\" is a good dataset for clustering because we don't expect k-means to pick up the tree branches: just look at HeatGeo if Tree in Figure 1 -- How can k-means separate the branches? I don't really understand why HeatGeo performs well here in Table 2. Rather than the Tree, I would like to have some toy dataset with 100D Gaussians differently far away from each other. I would also love to see MNIST (a subset of MNIST).\n\n* Another bold claim is that HeatGeo approximates SNE/t-SNE. I was very surprised to read that, because I have never seen SNE/t-SNE formulated as an MDS problem, and assumed it is impossible. However, I don't really see much evidence that this is true. Can you do HeatGeo of MNIST that would look like t-SNE? In the EB dataset (Figure 3), can you change HeatGeo params to make it look like t-SNE? The same for the PBMC dataset (Figure 4)? \n\n* Figure 1: This is the main motivational figure showing that HeatGeo is very good at unrolling Swiss roll, whereas t-SNE creates \"artifical clusters\". Personally, I think it does not make much sense to apply t-SNE to an intrinsically 2D dataset because t-SNE is designed for reducing dimensionality (that's why it uses t-distribution kernel, unlike SNE). When working with intrinsically 2D data, it makes much more sense to use SNE instead of t-SNE. And indeed, SNE is much better at unrolling Swiss roll than t-SNE (example: https://jlmelville.github.io/smallvis/swisssne.html). As the authors discuss SNE in their paper anyway, I think it would be good to add SNE to the comparison tables and also to Figure 1 (can be done e.g. via openTSNE). By the way, Isomap can also unroll the Swiss roll perfectly.\n\n* As the HeatGeo relies on MDS, it constructs a dense NxN distance matrix and so has O(N^2) complexity. All the datasets used in the paper are very small. This is unlike t-SNE/UMAP which can be run on tens of millions of points easily. I think this has to be discussed, e.g. in Section 7.\n\n\nSMALLER POINTS AND QUESTIONS\n\n* line 124: classic MDS does not optimize objective (5).\n\n* line 148: Ref [2] shows that UMAP's effective loss function is very different from its stated loss function, so this formula is actually misleading. See also https://proceedings.neurips.cc/paper/2021/hash/2de5d16682c3c35007e4e92982f1a2ba-Abstract.html and https://openreview.net/forum?id=B8a1FcY0vi which develop this argument.\n\n* line 194: \"small t can lead to disconnected graphs\" -- I don't understand this sentence, I thought the graph does not depend on t, no? You first construct the graph and the laplacian L, and only later you use the diffusion time t.\n\n* line 211: Algorithm 1 -- I don't really understand how L is constructed. What is the underlying graph? Is it a kNN graph, what are the weights? What is the k value (number of neighbors)? I think this is not really spelled out in the text. PHATE used some complicated setup with alpha-kernels, are you also using this here?\n\n* Table 1: this table is missing t-SNE and UMAP rows. Also, I would like to see SNE in there, and Isomap.\n\n* Figure 3 caption: \"UMAP and t-SNE do not capture the continuous manifold\" -- actually UMAP looks pretty similar to HeatGeo to me. Also, Ref [2] argues that higher attraction in t-SNE leads to better manifold preservation, so UMAP is better at showing the manifold (which is what we see in your Fig 3) but one can do even better by using higher exaggeration in t-SNE. Maybe worth mentioning.\n\n* line 314: \"HeatGeo interpolated to PHATE for \\rho->1\" -- I was confused by that. What is rho? It wasn't really discussed before, I don't see it in section 5 that discusses relation to PHATE. I do see \\rho in Algorithm 1 but not in the text of Section 4. So it's mysterious.\n\n* Figure 4 does not show t-SNE, but Figure 5 in the Appendix does. I am surprised by how bad t-SNE looks. I have seen t-SNE of various PBMC RNAseq datasets many times, and t-SNE is usually very good at separating cell types. But in your appendix figure t-SNE looks very poor. Why is that?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-16"}

comments: SUMMARY:
The authors propose heat geodesic embeddings, a novel approach to perform graph embeddings that is motivated by Riemannian geometry (Varadhan's formula and the parabolic Harnack inequality), which allows for directly linking heat diffusion on graphs to manifold distances. They performed experiments to demonstrate the efficacy of their approach.

STRENGTHS:
- Originality: The proposed heat geodesic embedding and the approach is , to the best of my knowledge, original and novel. 

- Significance: It is certainly important to study spectral embeddings nowadays, and I find the authors' methodological contributions to be significant. 

- Quality: The quality of the theoretical derivations and experimental evaluations is sound, to the best of my knowledge. 

- Clarity: Despite the advanced mathematical material involved, the authors presented and motivated their approach in a clear and convincing manner.

WEAKNESSES:
- there are no obvious weaknesses in the paper. The method is presented well and is straightforward. The authors performed experiments which covered most of the bases.

QUESTIONS:
- see sections above.

LIMITATIONS:
- no negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors develop specific heat diffusion kernels for manifold embedding.\nThey define a heat kernel based on the Varadha formula.\nThe authors try to weave some connections with existing methods of DR such as PHATE, t-SNE, etc.\nThey run some experiments\n\nSTRENGTHS:\nThe paper is well written except for the organization, clarity, and self-containedness.\nThe presented developments about the geodesic distance and the heat diffusion kernel are of interest for the manifold learning community.\nThe authors quite honestly acknowledge some of the limitations of their work.\n\nWEAKNESSES:\nThe writing style is too dense, poorly structured, with a succession of (too) short sections, referring to each other too many times, and often missing key information.\nIsomap is not cited and not compared to (in Fig. 1 for instance; later it seems that Isomap is actually used, without saying the name and still without a bibliograpcal reference, or is 'shortest path' with stress-based MDS instead of classical MDS? This can be referenced too, most probably).\nThe mathematical notation is misleading and unusual for manifold learning (x and y in the same space! use x_i and x_j ?).\nThe experiments have unclear/contradictory objectives: to detect manifold structures and clusters? The COIL-20 or 100 data set should then be considered. The results in Fig. 1 are not convincing at all: Isomap does better. In experiments related to clustering, the reader might always question the purposefulness of assessing cluster separation in methods which are local and depending on graphs; a disconnected graph somehow automatically guarantees cluster separation. On the other hand, the arrangement of clusters with respect to each other might actually be more relevant. The clustering experiments on Swiss roll and tree are far-fetched (use COIL-20?), with completely artificial clusters. Over-expression of clusters or even spurious clusters have been reported with many 'local' methods of manifold learning. Assessing the preservation of distances or neighbourhoods might be less of a stretch between the focus on geodesic distances and the current interest for clustering with those methods in communities of practitioners.\n\nQUESTIONS:\nWhat is V(x,sqrt(t)) ? please define and/or give an intuitive interpretation. The section about Varadhan's formula seems to end prematurely. Probably Vradhan could gain to be detailed more while the subsequent paragraphs in Preliminaries could be shortened or dropped (except for the quick reminder about stress-based MDS).\nSection 3 about related work is going too fast and lacks details (reminder of P for PHATE? drop SNE that is quite well known).\nIn section 4, if distance order preserving matters, why not compare to non-metric MDS too?\nCan you clarify Section 4? Definition 4.1 is cryptic and unintuitive; what is the meaning of sigma?\nIn proposition 4.1, what is |x-y| ? absolute value? distance like ||.||_2 elsewhere?\nWhy come back on MDS? Merge the presentation of MDS and its specific weighting?\nSection 5 is also wanting to embrace everything within limited space: alonger version in appendix and more details about the core contribution of the paper\nRef 33: Cédric Villani has a twin brother?\n\nLIMITATIONS:\nYes. They have. No potential negative societal impact here.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a dimensionality reduction technique based on heat kernels and called HeatGeo. The main idea is to learn a mapping that projects points to a low-dimensional space (ideally 2 or 3-dimensional for visualization) so that relative distances between pairs of points satisfy the same order as distances on some (complete Riemannian) manifold that uses some distance based on the heat kernel.\n\nSTRENGTHS:\nOriginality: The idea of using heat kernels for dimensionality reduction is not new (e.g. Belkin & Niyogi [1] that proposed Laplacian eigenmaps). This paper proposes a generalization of previous techniques by exploiting Varadhan's formula (Eq. (1)) and by bounding the calculated distance on a low-dimensional complete Riemannian manifold. They show that the ordering of distances in the high-dimensional space tends to be preserved on the low-dimensional manifold.\n\nQuality and Clarity: The paper is well-written and mentions different approximation techniques that can be used to approximate the computation of the kernel matrix since the algorithmic complexity is cubic in the number of points. Overall, the paper shows that the proposed approach tends to preserve clusters and the ordering of distances in the low-dimensional space better than standard approaches such as t-SNE and UMAP.\n\nSignificance: The significance of the approach is difficult to quantify. As most dimensionality reduction techniques that are often used for qualitative evaluation in the machine learning literature, the proposed approach can improve the understanding of a data set with a visualization that preserves the original general structure.\n\nWEAKNESSES:\nI do not see a particular weakness in the formulation of the approach which has a nice theoretical justification. \n\nThe main weakness of the paper is that it mainly focuses on dimensionality reduction and visualization which already have widely used techniques (e.g. t-SNE and UMAP) that are used only for qualitative evaluation in practice.\n\nQUESTIONS:\nI have no questions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article deals with representation learning and dimensionality reduction of high dimensional noisy datasets. The proposed method relies on heat diffusion. More precisely, assuming their data can be viewed as Riemannian manifolds and based on Varadhan’s formula, the authors define the notion of heat-geodesic dissimilarity. According to the authors, using the heat equation guarantees the conservation of geodetic distances between the raw data and its integrated counterpart. This dissimilarity is then used through multidimensional scaling (MDS) to define their so-called heat geodesic embedding.\n\nAfterward, the authors theoretically compare their method to the start-of-the-are ones. In particular, in the appendix, they attempt to create a generalized framework for dimensionality reduction methods. \n\nLast, they conduct numerical experiments to test their algorithm on different data types. In particular, they want to ensure that their method effectively preserves geodetic distances. They also seek to ensure that the inherent data structures are conserved: Preserving clusters or data temporality, for instance.\n\nSTRENGTHS:\n- Interesting idea, based on the solid formalist of Riemannian manifolds.\n- Efforts to unify different embedding approaches (especially in the appendix) with comparisons with existing methods.\n- Numerical experiments demonstrating the benefits of heat geodesic embedding for understanding data.\n\nWEAKNESSES:\nThere are typos and undefined notations in several places (see below). This is a disservice to the paper.\n\nQUESTIONS:\nSome comments:\n- Line 66: $n$ not defined.\n- Line 67: I think it should be $\\frac{n}2$ instead of $2n$.\n- Eq (2): $V(x,\\sqrt{t})$ not defined.\n- Paragraph starting line 79 (Graph construction and diffusion): $n$ is not defined, although we understand it is the sample size in context.\n- Line 88: Notation $[n]$ not defined.\n- Line 94: Add references when referring to literature.\n- Line 160: Should the $n$ not be a $d$ here?\n- Line 181: they use the a distance... -> the _or_ a.\n- Line 304: Strange line break.\n\nLIMITATIONS:\nNot concerned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}