A Unifying Perspective on Neighbor Embeddings along the Attraction-Repulsion Spectrum

1 Introduction . T-distributed stochastic neighbor embedding ( t-SNE ) ( van der Maaten & Hinton , 2008 ) is arguably among the most popular methods for low-dimensional visualizations of complex high-dimensional datasets . It defines pairwise similarities called affinities between points in the high-dimensional space and aims to arrange the points in a low-dimensional space to match these affinities ( Hinton & Roweis , 2003 ) . Affinities decay exponentially with high-dimensional distance , making them infinitesimal for most pairs of points and making the n×n affinity matrix effectively sparse . Efficient implementations of t-SNE suitable for large sample sizes n ( van der Maaten , 2014 ; Linderman et al. , 2019 ) explicitly truncate the affinities and use the k-nearest-neighbor ( kNN ) graph of the data with k n as the input . We use the term neighbor embedding ( NE ) to refer to all dimensionality reduction methods that operate on the kNN graph of the data and aim to preserve neighborhood relationships ( Yang et al. , 2013 ; 2014 ) . A prominent recent example of this class of algorithms is UMAP ( McInnes et al. , 2018 ) , which has become popular in applied fields such as single-cell transcriptomics ( Becht et al. , 2019 ) . It is based on stochastic optimization and typically produces more compact clusters than t-SNE . Another example of neighbor embeddings are force-directed graph layouts ( Noack , 2007 ; 2009 ) , originally developed for graph drawing . One specific algorithm called ForceAtlas2 ( Jacomy et al. , 2014 ) has recently gained popularity in the single-cell transcriptomic community to visualize datasets capturing cells at different stages of development ( Weinreb et al. , 2018 ; 2020 ; Wagner et al. , 2018a ; Tusi et al. , 2018 ; Kanton et al. , 2019 ; Sharma et al. , 2020 ) . Here we provide a unifying account of these algorithms . We studied the spectrum of t-SNE embeddings that are obtained when increasing/decreasing the attractive forces between kNN graph neighbors , thereby changing the balance between attraction and repulsion . This led to a trade-off between faithful representations of continuous and discrete structures ( Figure 1 ) . Remarkably , we found that ForceAtlas2 and UMAP could both be accurately positioned on this spectrum ( Figure 1 ) . For UMAP , we used mathematical analysis and Barnes-Hut re-implementation to show that increased attraction is due to the negative sampling optimisation strategy . 2 Related work . Various trade-offs in t-SNE generalizations have been studied previously ( Yang et al. , 2009 ; Kobak et al. , 2020 ; Venna et al. , 2010 ; Amid et al. , 2015 ; Amid & Warmuth , 2019 ; Narayan et al. , 2015 ; Im et al. , 2018 ) , but our work is the first to study the exaggeration-induced trade-off . Prior work used ‘ early exaggeration ’ only as an optimisation trick ( van der Maaten & Hinton , 2008 ) that allows to separate well-defined clusters ( Linderman & Steinerberger , 2019 ; Arora et al. , 2018 ) . Carreira-Perpinán ( 2010 ) introduced elastic embedding algorithm that has an explicit parameter λ controlling the attraction-repulsion balance . However , that paper suggests slowly increasing λ during optimization , as an optimisation trick similar to the early exaggeration , and does not discuss tradeoffs between high and low values of λ . Our results on UMAP go against the common wisdom on what makes UMAP perform as it does ( McInnes et al. , 2018 ; Becht et al. , 2019 ) . No previous work suggested that negative sampling may have a drastic effect on the resulting embedding . 3 Neighbor embeddings . We first cast t-SNE , UMAP , and ForceAtlas2 in a common mathematical framework , using consistent notation and highlighting the similarities between the algorithms , beforewe investigate the relationships between them empirically and analytically in more detail . We denote the original high-dimensional points as xi and their low-dimensional positions as yi . 3.1 t-SNE . T-SNE measures similarities between points by affinities vij and normalized affinities pij : pij = vij n , vij = pi|j + pj|i 2 , pj|i = vj|i∑ k 6=i vk|i , vj|i = exp ( − ‖xi − xj‖ 2 2σ2i ) . ( 1 ) For fixed i , pj|i is a probability distribution over all points j 6= i ( all pi|i are set to zero ) , and the variance of the Gaussian kernel σ2i is chosen to yield a pre-specified value of the perplexity of this probability distribution , P = 2H , where H = − ∑ j 6=i pj|i log2 pj|i . The affinities vij are normalized by n for pij to form a probability distribution on the set of all pairs of points ( i , j ) . Modern implementations ( van der Maaten , 2014 ; Linderman et al. , 2019 ) construct a kNN graph with k = 3P neighbors and only consider affinities between connected nodes as non-zero . The default perplexity value in most implementations is P = 30 . Similarities in the low-dimensional space are defined as qij = wij Z , wij = 1 1 + d2ij , dij = ‖yi − yj‖ , Z = ∑ k 6=l wkl , ( 2 ) with all qii set to 0 . The points yi are then rearranged in order to minimise the Kullback-Leibler ( KL ) divergence DKL ( { pij } ‖ { qij } ) = ∑ i , j pij log ( pij/qij ) between pij and qij : Lt-SNE = − ∑ i , j pij log wij Z = − ∑ i , j pij logwij + log ∑ i , j wij , ( 3 ) where we dropped constant terms and took into account that ∑ pij = 1 . The first term can be interpreted as contributing attractive forces to the gradient while the second term yields repulsive forces . Using ∂wij/∂yi = −2w2ij ( yi − yj ) , the gradient , up to a constant factor , can be written as : ∂Lt-SNE ∂yi ∼ ∑ j vijwij ( yi − yj ) − n Z ∑ j w2ij ( yi − yj ) . ( 4 ) 3.2 Exaggeration in t-SNE . A standard optimisation trick for t-SNE called early exaggeration ( van der Maaten & Hinton , 2008 ; van der Maaten , 2014 ) is to multiply the first sum in the gradient by a factor ρ = 12 during the initial iterations of gradient descent . This increases the attractive forces and allows similar points to gather into clusters more effectively . Carreira-Perpinán ( 2010 ) and Linderman & Steinerberger ( 2019 ) noticed that the attractive term in the t-SNE loss function is related to the loss function of Laplacian eigenmaps ( LE ) ( Belkin & Niyogi , 2002 ; Coifman & Lafon , 2006 ) . Indeed , if ρ→∞ , the relative repulsion strength goes to zero and the embedding shrinks to a point with all wij → 1 . This implies that , asymptotically , gradient descent becomes equivalent to Markov chain iterations with the transition matrix closely related to the graph Laplacian L = D−V of the affinity matrix V = [ vij ] ( here D is diagonal matrix with row sums ofV ; see Appendix ) . The entire embedding shrinks to a single point , but the leading eigenvectors of the Laplacian shrink the slowest . This makes t-SNE with ρ → ∞ produce embeddings very similar to LE , which computes the leading eigenvectors of the normalized Laplacian ( see Appendix and Figure 1 ) . This theoretical finding immediately suggests that it might be interesting to study t-SNE with exaggeration ρ > 1 not only as an optimisation trick , but in itself , as an intermediate method between LE and standard t-SNE . The gradient of t-SNE with exaggeration can be written as ∂Lt-SNE ( ρ ) ∂yi ∼ ∑ j vijwij ( yi − yj ) − n ρZ ∑ j w2ij ( yi − yj ) ( 5 ) and the corresponding loss function is Lt-SNE ( ρ ) = DKL ( { pij } ‖ { wij/Z 1 ρ } ) = ∑ i , j pij log pij wij/Z 1 ρ . ( 6 ) 3.3 UMAP . Using the same notation as above , UMAP optimizes the cross-entropy loss between vij and wij , without normalizing them into probabilities : LUMAP = ∑ i , j [ vij log vij wij + ( 1− vij ) log 1− vij 1− wij ] , ( 7 ) where the 1− vij term is approximated by 1 as most vij are 0 . Note that UMAP differs from t-SNE in how exactly it defines vij but this difference is negligible , at least for the data considered here.1 Dropping constant terms , we obtain LUMAP ∼ − ∑ i , j vij logwij − ∑ i , j log ( 1− wij ) , ( 8 ) which is the same loss function as the one introduced earlier by LargeVis ( Tang et al. , 2016 ) . The first term , corresponding to attractive forces , is the same as in t-SNE , but the second , repulsive , term is different . Taking wij = 1/ ( 1 + d2ij ) as in t-SNE,2 the UMAP gradient is given by ∂LUMAP ∂yi ∼ ∑ j vijwij ( yi − yj ) − ∑ j 1 d2ij + wij ( yi − yj ) , ( 9 ) where = 0.001 is added to the denominator to prevent numerical problems for dij ≈ 0 . If = 1 ( which does not strongly affect the result ; Figure S1 ) , the gradient becomes identical to the t-SNE gradient , up to the n/Z factor in front of the repulsive forces . Moreover , UMAP allows to use an arbitrary γ factor in front of the repulsive forces , which makes it easier to compare the loss functions3 : ∂LUMAP ( γ ) ∂yi ∼ ∑ j vijwij ( yi − yj ) − γ ∑ j 1 d2ij + wij ( yi − yj ) . ( 10 ) Whereas it is possible to approximate the full repulsive term with the same techniques as used in t-SNE ( van der Maaten , 2014 ; Linderman et al. , 2019 ) , UMAP took a different approach and followed LargeVis in using negative sampling ( Mikolov et al. , 2013 ) of repulsive forces : on each gradient descent iteration , only a small number ν of randomly picked repulsive forces are applied to each point for each of the ∼k attractive forces that it feels . Other repulsive terms are ignored . The default value is ν = 5 . The effect of this negative sampling on the resulting embedding has not been studied before . 3.4 ForceAtlas2 . Force-directed graph layouts are usually introduced directly via attractive and repulsive forces , even though it is easy to write down a suitable loss function ( Noack , 2007 ) . ForceAtlas2 ( FA2 ) has attractive forces proportional to dij and repulsive forces proportional to 1/dij ( Jacomy et al. , 2014 ) : ∂LFA2 ∂yi = ∑ j vij ( yi − yj ) − ∑ j ( hi + 1 ) ( hj + 1 ) d2ij ( yi − yj ) , ( 11 ) where hi denotes the degree of node i in the input graph . This is known as edge repulsion in the graph layout literature ( Noack , 2007 ; 2009 ) and is important for embedding graphs that have nodes of very different degrees . For symmetrized kNN graphs , hi ≈ k , so ( hi + 1 ) ( hj + 1 ) term contributes a roughly constant factor of ∼k2 to the repulsive forces .

Summary: the authors study a number of neighbor embedding methods in terms of attraction-repulsion forces. The authors show that t-SNE, UMAP, FA2, and LE can be (approximately) unified as a common approach that use different levels of tradeoff between these two terms. They also discuss the increased attraction in UMAP as a result of negative sampling.