SpaceMAP: Visualizing Any Data in 2-dimension by Space Expansion

Dimensionality reduction ( DR ) and visualization of high-dimensional data is of theoretical and practical value in machine learning and related fields . In theory , there exists an intriguing , non-intuitive discrepancy between the geometry of highdimensional space and low-dimensional space . Based on this discrepancy , we propose a novel DR and visualization method called Space-based Manifold Approximation and Projection ( SpaceMAP ) . Our method establishes a quantitative space transformation to address the “ crowding problem ” in DR. With the proposed equivalent extended distance ( EED ) and function distortion ( FD ) theory , we are able to match the capacity of high-dimensional and low-dimensional space in a principled manner . To handle complex high-dimensional data with different manifold properties , SpaceMAP makes distinctions between the near field , middle field , and far field of data distribution in a data-specific , hierarchical manner . We evaluated SpaceMAP on a range of artificial and real datasets with different manifold properties , and demonstrated its excellent performance in comparison with classical and state-of-the-art DR methods . In addition , the concept of space expansion provides a generic framework for understanding nonlinear DR methods including t-distributed Stochastic Neighbor Embedding ( t-SNE ) and Uniform Manifold Approximation and Projection ( UMAP ) . 1 INTRODUCTION . Real-world data , including images , videos , genetic expressions , natural languages , or financial statistics , usually have a high dimensionality . The intrinsic dimensionality of these data , however , is typically much lower than its ambient dimension , which property is recognized as an important underlying reason for modern machine learning to work ( Levina & Bickel , 2004 ; Pope et al. , 2021 ; Wright & Ma , 2021 ) . To capture useful information from high-dimensional data , dimensionality reduction ( DR ) is of both theoretical and practical value . DR is essential for data visualization . A space with a dimensionality higher than 3 , however , is already beyond our accustomed way of observing data , and our intuition in 2-dimensional or 3-dimensional space may not apply . High-dimensional space is not a trivial extension of lowdimensional space ; theoretical research on high-dimensional geometry and statistics revealed a number of intriguing , non-intuitive phenomena in high-dimensional space ( Giraud , 2021 ) . Imagine a hyper-sphere with a radius r in a d-dimensional Euclidean space whose central point is at the origin . Consider a “ crust ” of the d-dimensional hyper-sphere , which is between the surfaces of this hypersphere and a slightly smaller concentric hyper-sphere with radius ( 1−ϵ ) r , where ϵ is small ( Figure 1 a ) . The ratio of the volume of the “ crust ” Cd ( r ) to the hyper-sphere is Vd ( r ) is Cd ( r ) Vd ( r ) = 1− ( 1−ϵ ) d. Take ϵ = 0.01 , it is easy to show that when d is small , the ratio is tiny ( as our intuition goes ) , however this ratio grows exponentially fast to near 100 % with the increase of dimensionality , as illustrated in figure 1 a-b . The volume of a high-dimensional hyper-sphere is therefore counter-intuitively concentrated on a crust ( 1 c ) . Such concentration explains the “ crowding problem ” of DR ( van der Maaten & Hinton , 2008 ) : a faithful preservation of distances in high-dimensional space would lead to crowded data points . t-distributed Stochastic Neighbor Embedding ( t-SNE ) and Uniform Manifold Approximation and Projection ( UMAP ) are among the most established DR methods today , both of which mitigate this crowding problem by modeling the low-dimensional similarity measure with a longer-tail distribution than high-dimensional similarity measure ( e.g . t distribution vs. Gaussian in the case of t-SNE ) . Such a disparity in similarity measure amounts to a distortion of distance between two spaces , but in a highly implicit manner . Both methods perform well empirically , but the underlying distortion of distance measure in two spaces can not be analytically expressed or validated . Furthermore , the implicity of distance transformation deters us from imposing priors in data if there is any ( e.g . a Swiss Roll has an intrinsic dimensionality of 2 ) . Another important property of real-world high-dimensional data is that they often exhibit a hierarchical structure ( sub-manifolds on large manifolds ) , governed by the underlying generative models . Such hierarchy demands different treatment of data points by DR at different relative position . The state-of-the-art DR methods , such as UMAP ( McInnes et al. , 2018 ) and Barnes-Hut t-SNE ( van der Maaten , 2014 ) , commonly consider on a selected neighborhood ( as a hyper-parameter ) while discarding the far field beyond it . Isomap ( Balasubramanian et al. , 2002 ) , on the other hand , taking both near and far field into account by calculating distances on a connected graph . t-SNE and UMAP works well on data of disjoint manifolds such as MNIST , while Isomap works well on data of continuous manifold such as Swiss roll . However , it is generally difficult for one method to succeed in both disjoint and continuous data manifolds . In this paper , we seek to develop a novel DR method that is based on the following two key ideas , and strive to address the two aforementioned issues : • Space Expansion : Matching the “ capacity ” in high-dimensional and low-dimensional space , by explicit , quantitative transformation of distance measure ; • Hierarchical Manifold Approximation : Data-specific , hierarchical modeling of similarities in high-dimensional data , to accommodate both disjoint and continuous data manifolds . 2 RELATED WORK . Over the past decades , there have been active development of DR methods , which can be roughly categorized into global and local methods . The most well-known global method is principal component analysis ( PCA ) ( Jolliffe , 2014 ) , which finds a low number of directions that accounts for most variations . While PCA is linear , Laplacian eigenmaps ( Belkin & Niyogi , 2001 ; 2003 ) and multidimensional scaling ( MDS ) ( Buja et al. , 2008 ) are nonlinear global methods . Local methods , including Isomap , Locally Linear Embedding ( LLE ) ( Roweis , 2000 ) , stochastic neighbor embedding ( SNE ) ( Hinton & Roweis , 2003 ) , and maximum variance unfolding ( MVU ) ( Weinberger & Saul , 2006 ) , focus on preserving local structure in data . While outperforming global methods on nonlinear datasets , local methods generally suffer from poor scalability , low speed , and stochasticity . In 2008 , t-distributed stochastic neighbor embedding ( t-SNE ) ( van der Maaten & Hinton , 2008 ) , a variant of SNE , was proposed and later became one of the most popular DR methods in research community . t-SNE alleviates the “ crowding problem ” by assuming t-distribution in low-dimensional space ( in contrast to Gaussian distribution in the high-dimensional space ) . t-SNE is followed up by a number of prominent work to improve its scalability and speed , including among others Barnes-Hut t-SNE ( BH t-SNE ) ( van der Maaten , 2014 ) , fast Fourier-transform-accelerated interpolation-based t-SNE ( FIt-SNE ) ( Linderman et al. , 2019 ) , and parametric t-SNE ( ptSNE ) ( van der Maaten , 2009 ) . Recently , uniform manifold approximation and projection McInnes et al . ( 2018 ) ( UMAP ) was proposed , which shows competitive performance to t-SNE . UMAP is grounded in the mathematical theories of Riemannian geometry and algebraic topology . It computes local neighbor graph as a fuzzy simplicial set , which glues together the high-dimensional sparse data and differentiates local and global topology . UMAP further incorporates a number efficient algorithms such as approximated nearest neighbor and stochastic gradient descent ( SGD ) to improve speed and scalability . TriMAP ( Amid & Warmuth , 2019 ) was recently proposed , which uses a triplet loss instead of pairwise . UMAP and TriMAP are reported to better preserve the global structure than t-SNE McInnes et al . ( 2018 ) ; Amid & Warmuth ( 2019 ) . We note that a different definition of “ distance ” in the low-dimensional space than in the highdimensional space is core to these successful DR algorithms . In t-SNE , with a heavier tail , tdistribution implicitly expands the low dimensional space distance metrics , to allow more space for data points to be arranged . Likewise , the definition of distances of UMAP in low-dimensional space is much more heavy-tailed ( inverse of polynomial ) than in high-dimensional space ( exponential ) , which also leads to implicit distortion of space . 3 METHOD . 3.1 SPACE EXPANSION . High-dimensional space , irrespective of the distances defined , possesses exponentially higher capacity to express data than low-dimensional space . Here we define “ capacity ” as a measure of space volume to accommodate data points . Our intuition is therefore to expand the low-dimensional space to make up for this space capacity lost due to DR. To transfer the high-dimensional geometry into a low-dimensional one , we introduce the concept of equivalent extended distance ( EED ) in Section 3.1.1 and function distortion ( FD ) in Section 3.1.2 . 3.1.1 EQUIVALENT EXTENDED DISTANCE ( EED ) . We first define space capacity to quantitatively calculate the amount of distortion needed to match low-dimensional space to high-dimensional space . Definition 3.1 ( Space Capacity ) . Let Rij = l ( xi , xj ) ∈ R be the distance between data point xi and xj in the D-dimensional space . The space capacity VD ( Rij ) from point i to point j is defined as the volume a D-dimensional ball with a radius Rij . In Euclidean space , the capacity VD ( Rij ) is simply the volume of a D-dimensional hyper-sphere SD ( Rij ) : VD ( Rij ) = SD ( Rij ) = πD/2 Γ ( D/2 + 1 ) RDij ( 1 ) As high-dimensional space has higher capacity than low-dimensional space , to preserve the capacity of the high-dimensional space in the low-dimensional space , the equivalent distance in the lowdimensional space naturally extends , hence Equivalent Extended Distance ( EED ) : Definition 3.2 ( Equivalent Extended Distance ( EED ) ) . Let Rij = l ( xi , xj ) ∈ R be the distance between data point xi and xj in the D-dimensional space . The equivalent extended distance ( EED ) R̃ij , D→d is defined as the equivalent distance between xi and xj in d-dimensional space which can reach the same Space Capacity : Vd ( R̃ij , D→d ) = VD ( Rij ) ( 2 ) Example 3.1 ( 2-dimensional EED in Euclidean space ) . Consider a D-dimensional Euclidean space , the distance between two points i and j is Rij = ∥xi − xj∥ . The 2-dimensional EED of Rij is : R̃ij , D→2 = √ SD ( Rij ) π = αR D/2 ij where α = π ( D−2 ) /4√ Γ ( D/2+1 ) and Γ is the gamma function . Example 3.1 shows that when embedding D-dimensional Euclidean space into d-dimensional space , EED can be expressed in a generic form of R̃ij , D→d = αRβ where β = Dd and α is a constant determined by D and d. In the following , we present prior theory of intrinsic dimension by the maximum likelihood estimation ( MLE ) method by Levina & Bickel ( 2004 ) , and show that our proposed EED fits in this classical framework . Theorem 3.1 ( MLE of the intrinsic dimension ( Levina & Bickel , 2004 ) ) . Let Rik ∈ R be the Euclidean distance between point i and its k-th nearest neighbor . Assume the distribution of the data points in the small sphere Si ( R ) is uniform . The maximum likelihood estimation ( MLE ) of the intrinsic dimension of the data around point i is : d̂i ( Rik ) = 1 k − 1 k−1∑ j=1 log Rik Rij −1 ( 3 ) The MLE method provides a way to estimate the intrinsic dimension at point i in its k neighborhood based on the distribution of distances . By applying EED transformation to a space with intrinsic dimension D , the following Lemma holds : Lemma 3.2 ( Transformation of the intrinsic dimension by applying EED ) . ∀Rij = l ( xi , xj ) ∈ R , if the MLE of the intrinsic dimension of the high-dimensional dataset around point i is d̂i ( Rij ) = D , then the MLE of the intrinsic dimension after applying EED is d̂i ( R̃ij , D→d ) = d. Proof . By replacing Rij with R̃ij , D→d = αRD/dij into Equation ( 3 ) , we have d̂i ( R̃ik , D→d ) = 1 k − 1 k−1∑ j=1 log αR D/d ik αR D/d ij −1 = 1 k − 1 k−1∑ j=1 D d log Rik Rij −1 = d D d̂i ( Rik ) = d ( 4 ) Specifically , when d = 2 for visualization purposes , the MLE of the intrinsic dimension of the expanded space will be exactly 2 . Essentially , the validity of this statement arises from the proposed EED transformation R̃ij , D→d = αR D d , where the exponential form of Dd transforms the intrinsic dimension from D to d , based on the MLE formula ( 3 ) where the distances are taken logarithmic . In the following sections , the distance Rij between point i and point j is described as l ( xi , xj ) , which is suitable for several different distance metrics .

Summary: This paper proposes Space-based Manifold Approximation and Projection (SpaceMAP) method to solve two existing issues in dimensionality reduction task: (1) capacity matching between high-dimensional data space and low-dimensional visualization space. (2) complexity in various manifold properties. To address these issues, SpaceMAP proposes two ideas: (1) space expansion (Section 3.2): inferring equivalent extended distance (EED) (Definition 3.2) for low-dimensional space based on volume matching and introducing function distortion (FD) (Section 3.2.2) to transform the similarity functions in high-dimensional and low-dimensional space; (2) hierarchical manifold approximation (Section 3.3): making distinctions between the near field, middle field, and far field of data distribution in a data-specific hierarchical manner. SpaceMAP achieves competitive performance on a diversity of datasets.