SpaceMAP: Visualizing Any Data in 2-dimension by Space Expansion

Dimensionality reduction ( DR ) and visualization of high-dimensional data is of theoretical and practical value in machine learning and related fields . In theory , there exists an intriguing , non-intuitive discrepancy between the geometry of highdimensional space and low-dimensional space . Based on this discrepancy , we propose a novel DR and visualization method called Space-based Manifold Approximation and Projection ( SpaceMAP ) . Our method establishes a quantitative space transformation to address the “ crowding problem ” in DR. With the proposed equivalent extended distance ( EED ) and function distortion ( FD ) theory , we are able to match the capacity of high-dimensional and low-dimensional space in a principled manner . To handle complex high-dimensional data with different manifold properties , SpaceMAP makes distinctions between the near field , middle field , and far field of data distribution in a data-specific , hierarchical manner . We evaluated SpaceMAP on a range of artificial and real datasets with different manifold properties , and demonstrated its excellent performance in comparison with classical and state-of-the-art DR methods . In addition , the concept of space expansion provides a generic framework for understanding nonlinear DR methods including t-distributed Stochastic Neighbor Embedding ( t-SNE ) and Uniform Manifold Approximation and Projection ( UMAP ) . 1 INTRODUCTION . Real-world data , including images , videos , genetic expressions , natural languages , or financial statistics , usually have a high dimensionality . The intrinsic dimensionality of these data , however , is typically much lower than its ambient dimension , which property is recognized as an important underlying reason for modern machine learning to work ( Levina & Bickel , 2004 ; Pope et al. , 2021 ; Wright & Ma , 2021 ) . To capture useful information from high-dimensional data , dimensionality reduction ( DR ) is of both theoretical and practical value . DR is essential for data visualization . A space with a dimensionality higher than 3 , however , is already beyond our accustomed way of observing data , and our intuition in 2-dimensional or 3-dimensional space may not apply . High-dimensional space is not a trivial extension of lowdimensional space ; theoretical research on high-dimensional geometry and statistics revealed a number of intriguing , non-intuitive phenomena in high-dimensional space ( Giraud , 2021 ) . Imagine a hyper-sphere with a radius r in a d-dimensional Euclidean space whose central point is at the origin . Consider a “ crust ” of the d-dimensional hyper-sphere , which is between the surfaces of this hypersphere and a slightly smaller concentric hyper-sphere with radius ( 1−ϵ ) r , where ϵ is small ( Figure 1 a ) . The ratio of the volume of the “ crust ” Cd ( r ) to the hyper-sphere is Vd ( r ) is Cd ( r ) Vd ( r ) = 1− ( 1−ϵ ) d. Take ϵ = 0.01 , it is easy to show that when d is small , the ratio is tiny ( as our intuition goes ) , however this ratio grows exponentially fast to near 100 % with the increase of dimensionality , as illustrated in figure 1 a-b . The volume of a high-dimensional hyper-sphere is therefore counter-intuitively concentrated on a crust ( 1 c ) . Such concentration explains the “ crowding problem ” of DR ( van der Maaten & Hinton , 2008 ) : a faithful preservation of distances in high-dimensional space would lead to crowded data points . t-distributed Stochastic Neighbor Embedding ( t-SNE ) and Uniform Manifold Approximation and Projection ( UMAP ) are among the most established DR methods today , both of which mitigate this crowding problem by modeling the low-dimensional similarity measure with a longer-tail distribution than high-dimensional similarity measure ( e.g . t distribution vs. Gaussian in the case of t-SNE ) . Such a disparity in similarity measure amounts to a distortion of distance between two spaces , but in a highly implicit manner . Both methods perform well empirically , but the underlying distortion of distance measure in two spaces can not be analytically expressed or validated . Furthermore , the implicity of distance transformation deters us from imposing priors in data if there is any ( e.g . a Swiss Roll has an intrinsic dimensionality of 2 ) . Another important property of real-world high-dimensional data is that they often exhibit a hierarchical structure ( sub-manifolds on large manifolds ) , governed by the underlying generative models . Such hierarchy demands different treatment of data points by DR at different relative position . The state-of-the-art DR methods , such as UMAP ( McInnes et al. , 2018 ) and Barnes-Hut t-SNE ( van der Maaten , 2014 ) , commonly consider on a selected neighborhood ( as a hyper-parameter ) while discarding the far field beyond it . Isomap ( Balasubramanian et al. , 2002 ) , on the other hand , taking both near and far field into account by calculating distances on a connected graph . t-SNE and UMAP works well on data of disjoint manifolds such as MNIST , while Isomap works well on data of continuous manifold such as Swiss roll . However , it is generally difficult for one method to succeed in both disjoint and continuous data manifolds . In this paper , we seek to develop a novel DR method that is based on the following two key ideas , and strive to address the two aforementioned issues : • Space Expansion : Matching the “ capacity ” in high-dimensional and low-dimensional space , by explicit , quantitative transformation of distance measure ; • Hierarchical Manifold Approximation : Data-specific , hierarchical modeling of similarities in high-dimensional data , to accommodate both disjoint and continuous data manifolds . 2 RELATED WORK . Over the past decades , there have been active development of DR methods , which can be roughly categorized into global and local methods . The most well-known global method is principal component analysis ( PCA ) ( Jolliffe , 2014 ) , which finds a low number of directions that accounts for most variations . While PCA is linear , Laplacian eigenmaps ( Belkin & Niyogi , 2001 ; 2003 ) and multidimensional scaling ( MDS ) ( Buja et al. , 2008 ) are nonlinear global methods . Local methods , including Isomap , Locally Linear Embedding ( LLE ) ( Roweis , 2000 ) , stochastic neighbor embedding ( SNE ) ( Hinton & Roweis , 2003 ) , and maximum variance unfolding ( MVU ) ( Weinberger & Saul , 2006 ) , focus on preserving local structure in data . While outperforming global methods on nonlinear datasets , local methods generally suffer from poor scalability , low speed , and stochasticity . In 2008 , t-distributed stochastic neighbor embedding ( t-SNE ) ( van der Maaten & Hinton , 2008 ) , a variant of SNE , was proposed and later became one of the most popular DR methods in research community . t-SNE alleviates the “ crowding problem ” by assuming t-distribution in low-dimensional space ( in contrast to Gaussian distribution in the high-dimensional space ) . t-SNE is followed up by a number of prominent work to improve its scalability and speed , including among others Barnes-Hut t-SNE ( BH t-SNE ) ( van der Maaten , 2014 ) , fast Fourier-transform-accelerated interpolation-based t-SNE ( FIt-SNE ) ( Linderman et al. , 2019 ) , and parametric t-SNE ( ptSNE ) ( van der Maaten , 2009 ) . Recently , uniform manifold approximation and projection McInnes et al . ( 2018 ) ( UMAP ) was proposed , which shows competitive performance to t-SNE . UMAP is grounded in the mathematical theories of Riemannian geometry and algebraic topology . It computes local neighbor graph as a fuzzy simplicial set , which glues together the high-dimensional sparse data and differentiates local and global topology . UMAP further incorporates a number efficient algorithms such as approximated nearest neighbor and stochastic gradient descent ( SGD ) to improve speed and scalability . TriMAP ( Amid & Warmuth , 2019 ) was recently proposed , which uses a triplet loss instead of pairwise . UMAP and TriMAP are reported to better preserve the global structure than t-SNE McInnes et al . ( 2018 ) ; Amid & Warmuth ( 2019 ) . We note that a different definition of “ distance ” in the low-dimensional space than in the highdimensional space is core to these successful DR algorithms . In t-SNE , with a heavier tail , tdistribution implicitly expands the low dimensional space distance metrics , to allow more space for data points to be arranged . Likewise , the definition of distances of UMAP in low-dimensional space is much more heavy-tailed ( inverse of polynomial ) than in high-dimensional space ( exponential ) , which also leads to implicit distortion of space . 3 METHOD . 3.1 SPACE EXPANSION . High-dimensional space , irrespective of the distances defined , possesses exponentially higher capacity to express data than low-dimensional space . Here we define “ capacity ” as a measure of space volume to accommodate data points . Our intuition is therefore to expand the low-dimensional space to make up for this space capacity lost due to DR. To transfer the high-dimensional geometry into a low-dimensional one , we introduce the concept of equivalent extended distance ( EED ) in Section 3.1.1 and function distortion ( FD ) in Section 3.1.2 . 3.1.1 EQUIVALENT EXTENDED DISTANCE ( EED ) . We first define space capacity to quantitatively calculate the amount of distortion needed to match low-dimensional space to high-dimensional space . Definition 3.1 ( Space Capacity ) . Let Rij = l ( xi , xj ) ∈ R be the distance between data point xi and xj in the D-dimensional space . The space capacity VD ( Rij ) from point i to point j is defined as the volume a D-dimensional ball with a radius Rij . In Euclidean space , the capacity VD ( Rij ) is simply the volume of a D-dimensional hyper-sphere SD ( Rij ) : VD ( Rij ) = SD ( Rij ) = πD/2 Γ ( D/2 + 1 ) RDij ( 1 ) As high-dimensional space has higher capacity than low-dimensional space , to preserve the capacity of the high-dimensional space in the low-dimensional space , the equivalent distance in the lowdimensional space naturally extends , hence Equivalent Extended Distance ( EED ) : Definition 3.2 ( Equivalent Extended Distance ( EED ) ) . Let Rij = l ( xi , xj ) ∈ R be the distance between data point xi and xj in the D-dimensional space . The equivalent extended distance ( EED ) R̃ij , D→d is defined as the equivalent distance between xi and xj in d-dimensional space which can reach the same Space Capacity : Vd ( R̃ij , D→d ) = VD ( Rij ) ( 2 ) Example 3.1 ( 2-dimensional EED in Euclidean space ) . Consider a D-dimensional Euclidean space , the distance between two points i and j is Rij = ∥xi − xj∥ . The 2-dimensional EED of Rij is : R̃ij , D→2 = √ SD ( Rij ) π = αR D/2 ij where α = π ( D−2 ) /4√ Γ ( D/2+1 ) and Γ is the gamma function . Example 3.1 shows that when embedding D-dimensional Euclidean space into d-dimensional space , EED can be expressed in a generic form of R̃ij , D→d = αRβ where β = Dd and α is a constant determined by D and d. In the following , we present prior theory of intrinsic dimension by the maximum likelihood estimation ( MLE ) method by Levina & Bickel ( 2004 ) , and show that our proposed EED fits in this classical framework . Theorem 3.1 ( MLE of the intrinsic dimension ( Levina & Bickel , 2004 ) ) . Let Rik ∈ R be the Euclidean distance between point i and its k-th nearest neighbor . Assume the distribution of the data points in the small sphere Si ( R ) is uniform . The maximum likelihood estimation ( MLE ) of the intrinsic dimension of the data around point i is : d̂i ( Rik ) = 1 k − 1 k−1∑ j=1 log Rik Rij −1 ( 3 ) The MLE method provides a way to estimate the intrinsic dimension at point i in its k neighborhood based on the distribution of distances . By applying EED transformation to a space with intrinsic dimension D , the following Lemma holds : Lemma 3.2 ( Transformation of the intrinsic dimension by applying EED ) . ∀Rij = l ( xi , xj ) ∈ R , if the MLE of the intrinsic dimension of the high-dimensional dataset around point i is d̂i ( Rij ) = D , then the MLE of the intrinsic dimension after applying EED is d̂i ( R̃ij , D→d ) = d. Proof . By replacing Rij with R̃ij , D→d = αRD/dij into Equation ( 3 ) , we have d̂i ( R̃ik , D→d ) = 1 k − 1 k−1∑ j=1 log αR D/d ik αR D/d ij −1 = 1 k − 1 k−1∑ j=1 D d log Rik Rij −1 = d D d̂i ( Rik ) = d ( 4 ) Specifically , when d = 2 for visualization purposes , the MLE of the intrinsic dimension of the expanded space will be exactly 2 . Essentially , the validity of this statement arises from the proposed EED transformation R̃ij , D→d = αR D d , where the exponential form of Dd transforms the intrinsic dimension from D to d , based on the MLE formula ( 3 ) where the distances are taken logarithmic . In the following sections , the distance Rij between point i and point j is described as l ( xi , xj ) , which is suitable for several different distance metrics .

Summary: This work studies dimensionality reduction and visualization of high-dimensional data. The idea is to construct two graphs in the original high-dimensional space and the low-dimensional space, respectively, based on space expansion and hierarchical manifold approximation. Space expansion is to map the distance in high-dimensional space to an equivalent extended distance (EED) in low-dimensional space such that the volumes of the two (hyper-)spheres in the two spaces are equal. Hierarchical manifold approximation divides the points into near field, middle field, and far field, and assigns EED based similarity for points in the near field. Then a problem minimizing the sum of a KL divergence and a repulsive force term is solved by SGD to determine the projected coordinates. In numerical experiments, the effectiveness of the proposed method is demonstrated by comparing with some baseline methods on synthetic and real-world datasets.