A More Globally Accurate Dimensionality Reduction Method Using Triplets

1 INTRODUCTION . Information visualization using dimensionality reduction ( DR ) is a fundamental step for gaining insight about a dataset . Motivated by the fact that the humans essentially think in two or three dimensions , DR has been studied extensively ( Tenenbaum et al. , 2000 ; Kohonen , 1998 ; Roweis & Saul , 2000 ; Hinton & Roweis , 2003 ; Maaten & Hinton , 2008 ) . However , how to choose the best DR method for visualizing a given dataset is yet unresolved . Despite the plethora of quantitative measures such as trustworthiness-continuity ( Kaski et al. , 2003 ) , mean ( smoothed ) precision-recall ( Venna et al. , 2010 ) , and nearest-neighbor accuracy ( Van Der Maaten et al. , 2009 ) , there is no standard procedure to assess the quality of a low-dimensional embedding in general . Also these measures only focus on the local neighborhood of each point in the original space and low-dimensional embedding and fail to reflect global aspects of the data . We will argue that selecting a DR method based on these local measures is misleading . In this paper , we mainly focus on the global properties of the data . These properties include the relative placements of the clusters in the dataset as well as revelation of outlier points that are located far away from the rest of the points in the high-dimensional space . It appears that quantifying the global properties is significantly more complicated and certainly local quality measures of DR ( such as precision-recall1 ) do not capture these properties . Thus we resort to an evaluation based on visual clues that allows the practitioner to build confidence about the obtained DR results . To this end , we propose a number of transformations that happen naturally to real-world datasets . We argue that any DR method that claims to preserve the global structure of the data , should be able to handle these transformations . Ackerman et al . ( 2010 ) took a similar more theoretical approach for the task of clustering where certain natural properties are used to taxonomize different clustering algorithms . Next , we introduce a new DR method , called TriMap . The main idea behind TriMap is to capture higher orders of structure in the data by considering the relative similarities of a chosen set of triplets of points . The method minimizes a loss over triplets that measures how well a triplet is satisfied . This loss is made robust by capping it with a damping function . We provide compelling experimental 1We verified on small datasets that other common measures such as trustworthiness-continuity ( Kaski et al. , 2003 ) are also inadequate for evaluating the global structure . Also these measures are too expensive to calculate for moderately large datasets . results on large natural datasets . We show that t-SNE and LargeVis , which are both based on preserving the pairwise ( dis ) similarities of the points , tend to form spurious clusters with clean boundaries and fail to properly reflect the global aspects of the data . On the other hand , TriMap ’ s DR results preserve more of the global structure of the data as well as reveals outliers . 2 TESTING THE GLOBAL ACCURACY OF DR METHODS . Our proposed tests mimic the natural scenarios that happen in the real-world datasets . In each case we run the test on four DR methods : t-SNE , which is currently the most widely used DR method ; LargeVis , a more recent method that is also based on pairwise ( dis ) similarities ; TriMap , our new method based on triplet information ; and PCA a global method which projects the data onto the two directions of largest variance . We defer the description of t-SNE & LargeVis as well as our new TriMap method to later sections . As a running example , we perform the tests on the MNIST dataset2 which contains 70,000 images of handwritten digits ( represented by 784 dimensions ) 3 . We normalized the data to have pixel values between [ 0 , 1 ] and verified that the results are consistent over multiple runs of all methods . For each embedding , we calculated the area under the mean precision- 2 http : //yann.lecun.com/exdb/mnist/ 3For ease of comparison , we use the same initial solution for t-SNE and TriMap , whenever possible . The current implementation of LargeVis does not support initial solutions . recall curve ( AUC ) ( Venna et al. , 2010 ) .4 We also calculate the nearest-neighbor classification accuracy for each embedding . The ( AUC , NN-Accuracy ) values are shown on the bottom of each figure . The main reason for including these local measure in the results is to emphasize the fact that these measures fail to reflect the globally properties of the embedding , discussed in the following . Figure 5 ( ? ) shows the results on the full dataset by the four methods . Partial observation test A DR tool should be invariant to removing a subset of points from the dataset . The placement of the remaining points ( or clusters ) after running the DR method on the reduced data set should remain relatively unchanged compared to the embedding created from the full dataset . In a first test , the subset of removed points are selected at random . This test mimics the fact that in reality we only see a sample from the data distribution and the DR method should be stable w.r.t . the sample size . Figure 5 ( 2.1.a ) shows the results after removing % 90 of the dataset at random . t-SNE and TriMap both produce good results in that the clusters of the reduced datasets are located roughly in the same arrangement as in the full dataset . However , LargeVis moves the clusters around ( e.g . cluster ‘ 1 ’ ) . For labeled datasets , we might be interested in visualizing a particular subset of the classes , leading to our second test . Figure 5 ( 2.1.b ) gives the results on running the methods on just the subset of even digits of MNIST . As can be seen , only the TriMap method is able to preserve the relative distances between all the clusters after removing the odd digits , while t-SNE and LargeVis place the remaining clusters at arbitrary positions . 4More precisely , for each point we fix a neighborhood size of 20 in the high-dimension as the “ relevant points ” and vary the neighborhood size in the low-dimension between 1 and 100 . This gives us a precision and recall curve for each point . The AUC measure is the area under the mean curve for all points . Outlier test Natural datasets frequently contain outliers due to measurement or experimental errors . Detecting and removing the outliers is key step in statistical analysis that should be facilitated with the DR methods . We can check the ability of the DR methods to reveal outliers by adding artificial outlier points to the original dataset . Figure 5 ( 2.2 ) shows the DR results after shifting point X ( originally placed inside cluster ‘ 8 ’ by all methods ) far away in a random direction . The point X is clearly an outlier in the new dataset ( as verified by the PCA result ) . Surprisingly , t-SNE and LargeVis both place the outlier inside the cluster ‘ 3 ’ , which happens to be the nearest cluster to this point in higher dimension . More disturbingly , adding a single outlier this way consistently rearranges the relative location of the clusters for t-SNE and LargeVis . TriMap shows the outlier and preserve the structure . Multiple scales test A DR tool should be able to reflect the global structure of the data at different scales . For instance , a dataset might consist of multiple clusters where each cluster itself may contain multiple sub-clusters and so on . The practitioner can do a rudimentary test by duplicating and shifting the natural dataset at hand . That is , for each point xn of the original dataset , we add a point xn + c , where c is a fixed random shift which is chosen large enough such that the two copies are guaranteed to be far apart in the high-dimensional space5 . When applied on the duplicated dataset , the DR method should be able to show each copy separately . Figure 5 ( 2.3 ) illustrates the results on a duplicated MNIST dataset . We expect to see two identical copies of the same dataset in the low-dimensional embedding , which can be verified by the PCA result . Curiously enough , both t-SNE and LargeVis fail to provide any meaningful global structure6 . In fact , both methods tend to split some of the clusters into smaller sub-clusters . On the other hand , TriMap successfully recovers the two identical copies , which look similar to the original dataset in Figure 5 ( ? ) . 3 SKETCH OF THE T-SNE AND LARGEVIS . The t-SNE method ( Maaten & Hinton , 2008 ) is perhaps the most commonly used DR method in practice . The main idea of t-SNE is to reflect the pairwise similarities of the points by minimizing a KL divergence between the normalized pairwise similarities in the high-dimensional space and the low-dimensional embedding . The t-SNE method has O ( N2 ) time complexity for N points . However , the complexity can be reduced to O ( N logN ) by approximating the gradient using treebased algorithms ( Van Der Maaten , 2014 ) . t-SNE is easy to apply to general datasets and usually produces nicely separated clusters . Also it has been proven theoretically ( Arora et al. , 2018 ) that if the clusters are separated far enough in higher dimension , then t-SNE separates them in the embedding 7 . However , as we showed in the previous section , the DR results produced by t-SNE are sometimes extremely misleading : the whole dataset is simply collapsed into an “ orb ” and the outliers are shown as “ inliers ” . The latter can be explained by fact that for an outlier point , the pairwise similarities to the points that are closest to the outlier point dominate the rest , causing the method to pull back the outlier point to the closest cluster . LargeVis is a more recent method ( Tang et al. , 2016 ) that also aims to preserve the pairwise ( dis ) similarities of the points in the high-dimensional space in the low-dimensional embedding . To avoid the O ( N2 ) complexity of t-SNE , LargeVis uses a negative sampling approach ( Mikolov et al. , 2013 ) to randomly select a subset of the dissimilar pairs . However , the weights of all the dissimilar pairs are set to a positive constant . This causes the LargeVis to lose global information in the data , as we showed in the previous section . Overall , LargeVis forms well-separated clusters . However , the outlier points that are far away or in-between the clusters are pushed back into the closest clusters . This is a consequence of using a noisy distribution for negative sampling which tends to pick points from the denser regions with higher probability . For example , moving single points that lie between multiple large clusters inside the nearest cluster increases the likelihood of the model ( because a single point has a very small probability of being selected as a dissimilar example by any of points that lie inside the large clusters ) . 5Similar results obtained by shifting the datasets in 4 directions . 6Verified with a large range of the perplexity parameter for t-SNE ( from the default of 30 to 150 ) . 7The proof requires that ” early exaggeration ” is used in all iterations of training .

Summary: In this paper, the authors present a novel dimensionality reduction method named TriMap. TriMap attempts to improve upon the widely-adopted t-SNE algorithm by incorporating global distances through the use of triplets, rather than pairwise comparisons. The authors compare to t-SNE, as well as a newer method called LargeVis which also claims to impose a global distances metric. The authors show that their method is more robust to the addition or removal of clusters and outliers and provides a more meaningful global distance relative to the methods against which they compare.