Uniform Manifold Approximation with Two-phase Optimization

1 INTRODUCTION . We present a novel dimensionality reduction method , Uniform Manifold Approximation with Twophase Optimization ( UMATO ) to obtain less biased and robust embedding over diverse initialization methods . One effective way of understanding high-dimensional data in various domains is to reduce its dimensionality and investigate the projection in a lower-dimensional space . The limitation of previous approaches such as t-Stochastic Neighbor Embedding ( t-SNE , Maaten & Hinton ( 2008 ) ) and Uniform Manifold Approximation and Projection ( UMAP , McInnes et al . ( 2018 ) ) is that they are susceptible to different initialization methods , generating considerably different embedding results ( Section 5.5 ) . t-SNE adopts Kullback-Leibler ( KL ) divergence as its loss function . The fundamental limitation of the KL divergence is that the penalty for the points that are distant in the original space being close in the projected space is too little ( Appendix B ) . This results in only the local manifolds being captured , while clusters that are far apart change their relative locations from run to run . Meanwhile , UMAP leverages the cross-entropy loss function , which is known to charge a penalty for points that are distant in the original space being close in the projection space and for points that are close in the original space being distant in the projection space ( Appendix B ) . UMAP considers all points in the optimization at once with diverse sampling techniques ( i.e. , negative sampling and edge sampling ) . Although the approximation technique in UMAP optimization makes the computation much faster , this raises another problem that the clusters in the embedding become dispersed as the number of epochs increases ( Appendix K ) , which can lead to misinterpretation . UMAP tried to alleviate this by using a fixed number ( e.g. , 200 ) , which is ad hoc , and by applying a learning rate decay . However , the optimal number of epochs and decay schedule for each initialization method needs to be found in practice . To solve the aforementioned problems , we avoid using approximation during the optimization process , which normally would result in greatly increased computational cost . Instead , we first run optimization only with a small number of points that represent the data ( i.e. , hub points ) . Finding the optimal projection for a small number of points using a cross-entropy function is relatively easy and robust , making the additional techniques employed in UMAP unnecessary . Furthermore , it is less sensitive to the initialization method used ( Section 5.5 ) . After capturing the overall skeleton of the high-dimensional structure , we gradually append the rest of the points in subsequent phases . Although the same approximation technique as UMAP is used for these points , as we have already embedded the hub points and use them as anchors , the projections become more robust and unbiased . The gradual addition of points can in fact be done in a single phase ; we found additional phases do not result in meaningful improvements in the performance but only in the increased computation time ( Section 4.5 ) . Therefore , we used only two phases in UMAP : global optimization to capture the global structures ( i.e. , the pairwise distances in a high-dimensional space ) and local optimization to retain the local structures ( i.e. , the relationships between neighboring points in a high-dimensional space ) of the data . We compared UMATO with popular dimensionality reduction techniques including PCA , Isomap ( Tenenbaum et al . ( 2000 ) ) , t-SNE , UMAP , topological autoencoders ( Moor et al . ( 2020 ) ) and At-SNE ( Fu et al . ( 2019 ) ) . We used one synthetic ( 101-dimensional spheres ) and three realworld ( MNIST , Fashion MNIST , and Kuzushiji MNIST ) datasets and analyzed the projection results with several quality metrics . In conclusion , UMATO demonstrated better performance than the baseline techniques in all datasets in terms of KLσ with different σ values , meaning that it reasonably preserved the density of data over diverse length scales . Finally , we presented the 2D projections of each dataset , including the replication of an experiment using the synthetic Spheres dataset introduced by Moor et al . ( 2020 ) where data points locally constitute multiple small balls globally contained in a larger sphere . Here , we demonstrate that UMATO can better preserve both structures compared to the baseline algorithms ( Figure 3 ) . 2 RELATED WORK . Dimensionality reduction . Most previous dimensionality reduction algorithms focused on preserving the data ’ s local structures . For example , Maaten & Hinton ( 2008 ) proposed t-SNE , focusing on the crowding problem with which the previous attempts ( Hinton & Roweis ( 2002 ) ; Cook et al . ( 2007 ) ) have struggled , to visualize high-dimensional data through projection produced by performing stochastic gradient descent on the KL divergence between two density functions in the original and projection spaces . Van Der Maaten ( 2014 ) accelerated t-SNE developing a variant of the Barnes-Hut algorithm ( Barnes & Hut ( 1986 ) ) and reduced the computational complexity from O ( N2 ) into O ( N logN ) . After that , grounded in Riemannian geometry and algebraic topology , McInnes et al . ( 2018 ) introduced UMAP as an alternative to t-SNE . Leveraging the cross-entropy function as its loss function , UMAP reduced the computation time by employing negative sampling from Word2Vec ( Mikolov et al . ( 2013 ) ) and edge sampling from LargeVis ( Tang et al . ( 2015 ; 2016 ) ) ( Table 1 ) . Moreover , they showed that UMAP can generate stable projection results compared to t-SNE over repetition . On the other hand , there also exist algorithms that aim to capture the global structures of data . Isomap ( Tenenbaum et al . ( 2000 ) ) was proposed to approximate the geodesic distance of highdimensional data and embed it onto the lower dimension . Global t-SNE ( Zhou & Sharpee ( 2018 ) ) converted the joint probability distribution , P , in the high-dimensional space from Gaussian to Student ’ s-t distribution , and proposed a variant of KL divergence . By adding it with the original loss function of t-SNE , Global t-SNE assigns a relatively large penalty for a pair of distant data points in high-dimensional space being close in the projection space . Another example is topological autoencoders ( Moor et al . ( 2020 ) ) , a deep-learning approach that uses a generative model to make the latent space resemble the high-dimensional space by appending a topological loss to the original reconstruction loss of autoencoders . However , they required a huge amount of time for hyperparameter exploration and training for a dataset , and only focused on the global aspect of data . Unlike other techniques that presented a variation of loss functions in a single pipeline , UMATO is novel as it preserves both structures by dividing the optimization into two phases ; this makes it outperform the baselines with respect to quality metrics in our experiments . Hubs , landmarks , and anchors . Many dimensionality reduction techniques have tried to draw sample points to better model the original space ; these points are usually called hubs , landmarks , or anchors . Silva & Tenenbaum ( 2003 ) proposed Landmark Isomap , a landmark version of classical multidimensional scaling ( MDS ) to alleviate its computation cost . Based on the Landmark Isomap , Yan et al . ( 2018 ) tried to retain the topological structures ( i.e. , homology ) of high-dimensional data by approximating the geodesic distances of all data points . However , both techniques have the limitation that landmarks were chosen randomly without considering their importance . UMATO uses a k-nearest neighbor graph to extract significant hubs that can represent the overall skeleton of high-dimensional data . The most similar work to ours is At-SNE ( Fu et al . ( 2019 ) ) , which optimized the anchor points and all other points with two different loss functions . However , since the anchors wander during the optimization and the KL divergence does not care about distant points , it hardly captures the global structure . UMATO separates the optimization process into two phases so that the hubs barely moves but guides other points so that the subareas manifest the shape of the highdimensional manifold in the projection . Applying different cross-entropy functions to each phase also helps preserve both structures . 3 UMAP . Since UMATO shares the overall pipeline of UMAP ( McInnes et al . ( 2018 ) ) , we briefly introduce UMAP in this section . Although UMAP is grounded in a sophisticated mathematical foundation , its computation can be simply divided into two steps , graph construction and layout optimization , a configuration similar to t-SNE . In this section , we succinctly explain the computation in an abstract manner . For more details about UMAP , please consult the original paper ( McInnes et al . ( 2018 ) ) . Graph Construction . UMAP starts by generating a weighted k-nearest neighbor graph that represents the distances between data points in the high-dimensional space . Given an input dataset X = { x1 , . . . , xn } , the number of neighbors to consider k and a distance metric d : X × X → [ 0 , ∞ ) , UMAP first computes Ni , the k-nearest neighbors of xi with respect to d. Then , UMAP computes two parameters , ρi and σi , for each data point xi to identify its local metric space . ρi is a nonzero distance from xi to its nearest neighbor : ρi = min j∈Ni { d ( xi , xj ) | d ( xi , xj ) > 0 } . ( 1 ) Using binary search , UMAP finds σi that satisfies : ∑ j∈Ni exp ( −max ( 0 , d ( xi , xj ) − ρi ) /σi ) = log2 ( k ) . ( 2 ) Next , UMAP computes : vj|i = exp ( −max ( 0 , d ( xi , xj ) − ρi ) /σi ) , ( 3 ) the weight of the edge from a point xi to another point xj . To make it symmetric , UMAP computes vij = vj|i + vi|j − vj|i · vi|j , a single edge with combined weight using vj|i and vi|j . Note that vij indicates the similarity between points xi and xj in the original space . Let yi be the projection of xi in a low-dimensional projection space . The similarity between two projected points yi and yj is wij = ( 1 + a||yi − yj ||2b2 ) −1 , where a and b are positive constants defined by the user . Setting both a and b to 1 is identical to using Student ’ s t-distribution to measure the similarity between two points in the projection space as in t-SNE ( Maaten & Hinton ( 2008 ) ) . Layout Optimization . The goal of layout optimization is to find the yi that minimizes the difference ( or loss ) between vij and wij . Unlike t-SNE , UMAP employs the cross entropy : CUMAP = ∑ i 6=j [ vij · log ( vij/wij ) − ( 1− vij ) · log ( ( 1− vij ) / ( 1− wij ) ) ] , ( 4 ) between vij andwij as the loss function . UMAP initializes yi through spectral embedding ( Belkin & Niyogi ( 2002 ) ) and iteratively optimize its position to minimize CUMAP . Given the output weight wij as 1/ ( 1 + ad2bij ) , the attractive gradient is : CUMAP yi + = −2abd2 ( b−1 ) ij 1 + ad2bij vij ( yi − yj ) , ( 5 ) and the repulsive gradient is : CUMAP yi − = 2b ( + d2ij ) ( 1 + ad 2b ij ) ( 1− vij ) ( yi − yj ) , ( 6 ) where is a small value added to prevent division by zero and dij is a Euclidean distance between yi and yj . For efficient optimization , UMAP leverages the negative sampling technique from Word2Vec ( Mikolov et al . ( 2013 ) ) . After choosing a target point and its negative samples , the position of the target is updated with the attractive gradient , while the positions of the latter do so with the repulsive gradient . Moreover , UMAP utilizes edge sampling ( Tang et al . ( 2015 ; 2016 ) ) to accelerate and simplify the optimization process ( Table 1 ) . In other words , UMAP randomly samples edges with a probability proportional to their weights , and subsequently treats the selected ones as binary edges . Considering the previous sampling techniques , the modified objective function is : O = ∑ ( i , j ) ∈E vij ( log ( wij ) + M∑ k=1 Ejk∼Pn ( j ) γ log ( 1− wijk ) ) . ( 7 ) Here , vij and wij are the similarities in the high and low-dimensional spaces respectively , M is the number of negative samples and Ejk∼Pn ( j ) indicates that jk is sampled according to a noisy distribution , Pn ( j ) , from Word2Vec ( Mikolov et al . ( 2013 ) ) .

Summary: This work proposed a dimensionality reduction algorithm called Uniform Manifold Approximation with Two-phase Optimization (UMATO), which is an improved version of UMAP (Ref. [3] see below). UMATO has a two-phase optimization approach: global optimization to obtain the overall skeleton of data & local optimization to identify the local structures.