Dimensionality Reduction for Data Visualization: PCA vs TSNE vs UMAP vs LDA | by Sivakar Sivarajah | Towards Data Science

In this story, we are gonna go through three Dimensionality reduction techniques specifically used for Data Visualization: PCA, t-SNE, LDA and UMAP. We are going to explore them in details using the Sign Language MNIST Dataset, without going in-depth with the maths behind the algorithms.
Many Machine Learning problems involve thousands of features, having such a large number of features bring along many problems, the most important ones are:
Makes the training extremely slow
Makes it difficult to find a good solution
This is known as the curse of dimensionality and the Dimensionality Reduction is the process of reducing the number of features to the most relevant ones in simple terms.
Reducing the dimensionality does lose some information, however as most compressing processes it comes with some drawbacks, even though we get the training faster, we make the system perform slightly worse, but this is ok! “sometimes reducing the dimensionality can filter out some of the noise present and some of the unnecessary details”.
Most Dimensionality Reduction applications are used for:
Data Compression
Noise Reduction
Data Classification
Data Visualization
One of the most important aspects of Dimensionality reduction, it is Data Visualization. Having to drop the dimensionality down to two or three, make it possible to visualize the data on a 2d or 3d plot, meaning important insights can be gained by analysing these patterns in terms of clusters and much more.
The two main approaches to reducing dimensionality: Projection and Manifold Learning.
Projection: This technique deals with projecting every data point which is in high dimension, onto a subspace suitable lower-dimensional space in a way which approximately preserves the distances between the points.
Manifold Learning: Many dimensionality reductions algorithm work by modelling the manifold on which the training instance lie; this is called Manifold learning. It relies on the manifold hypothesis or assumption, which holds that most real-world high-dimensional datasets lie close to a much lower-dimensional manifold, this assumption in most of the cases is based on observation or experience rather than theory or pure logic.[4]
Now let's briefly explain the three techniques before jumping into solving the use case.
One of the most known dimensionality reduction “unsupervised” algorithm is PCA(Principal Component Analysis).
This works by identifying the hyperplane which lies closest to the data and then projects the data on that hyperplane while retaining most of the variation in the data set.
The axis that explains the maximum amount of variance in the training set is called the Principal Components.
The axis orthogonal to this axis is called the second principal component. As we go for higher dimensions, PCA would find a third component orthogonal to the other two components and so on, for visualization purposes we always stick to 2 or maximum 3 principal components.
It is very important to choose the right hyperplane so that when the data is projected onto it, it the maximum amount of information about how the original data is distributed.
(t-SNE) or T-distributed stochastic neighbour embedding created in 2008 by (Laurens van der Maaten and Geoffrey Hinton) for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets.
(t-SNE) takes a high dimensional data set and reduces it to a low dimensional graph that retains a lot of the original information. It does so by giving each data point a location in a two or three-dimensional map. This technique finds clusters in data thereby making sure that an embedding preserves the meaning in the data. t-SNE reduces dimensionality while trying to keep similar instances close and dissimilar instances apart.[2]
For a quick a Visualization of this technique, refer to the animation below (it is taken from an amazing tutorial by Cyrille Rossant, I highly recommend to check out his amazing tutorial.
link: https://www.oreilly.com/content/an-illustrated-introduction-to-the-t-sne-algorithm/
Linear Discriminant Analysis (LDA) is most commonly used as a dimensionality reduction technique in the pre-processing step for pattern-classification.
The goal is to project a dataset onto a lower-dimensional space with good class-separability in order to avoid overfitting and also reduce computational costs.
The general approach is very similar to PCA, rather than finding the component axes that maximize the variance of our data, we are additionally interested in the axes that maximize the separation between multiple classes (LDA)[5].
LDA is “supervised” and computes the directions (“linear discriminants”) that will represent the axes that maximize the separation between multiple classes.
Uniform Manifold Approximation and Projection created in 2018 by (Leland McInnes, John Healy, James Melville) is a general-purpose manifold learning and dimension reduction algorithm.
UMAP is a nonlinear dimensionality reduction method, it is very effective for visualizing clusters or groups of data points and their relative proximities.
The significant difference with TSNE is scalability, it can be applied directly to sparse matrices thereby eliminating the need to applying any Dimensionality reduction such as PCA or Truncated SVD(Singular Value Decomposition) as a prior pre-processing step.[1]
Put simply, it is similar to t-SNE but with probably higher processing speed, therefore, faster and probably better visualization. (let’s find it out in the tutorial below)
Now we are going to go through the above-mentioned use case where all the three techniques will be applied: specifically, we will try to visualize a high dimensional dataset using these techniques: The Sign-Language-MNIST Dataset:https://www.kaggle.com/datamunge/sign-language-mnist
Note that: There are 25 unique labels representing the number of distinct sign-languages. Now for better visualization (“ it is very difficult to observe all 24 classes in a single visualization”) and for faster computation. I am only keeping the first 10 labels, omitting the rest.
After applying PCA, the new dimensionality of the data now has only 3 features compared to the 784 features of the x data.
The number of dimensions has been cut down drastically whilst trying to retain as much of the ‘variation’ in the information as possible.
plot_2d(principalComponents[:, 0],principalComponents[:, 1])
From the 2D plot, we can see the two components definitely hold some information, especially for specific digits, but clearly not enough to set all of them apart.
plot_3d(principalComponents[:, 0],principalComponents[:, 1],principalComponents[:, 2])
One thing to note down is that t-SNE is very computationally expensive, hence it is mentioned in its documentation that :
“It is highly recommended to use another dimensionality reduction method (e.g. PCA for dense data or TruncatedSVD for sparse data) to reduce the number of dimensions to a reasonable amount (e.g. 50) if the number of features is very high. This will suppress some noise and speed up the computation of pairwise distances between samples.”[2]
Thus, I have applied PCA choosing to retain 50 principal components from the original data to cut down the need for more processing power and it will require time to compute the dimensionality reduction if we had considered the original data.
The speed of the three techniques will be analysed and compared in the following sections further down in details.
plot_2d(tsne[:, 0],tsne[:, 1])
Compared to the PCA 2d result, we can clearly see the presence of different clusters and how they are positioned.
plot_3d(tsne[:, 0],tsne[:, 1],tsne[:, 2])
UMAP has different hyperparameters that can have an impact on the resulting embeddings:
n_neighbors
This parameter controls how UMAP balances local versus global structure in the data. This low values of n_neighbours forces UMAP to focus on very local structures while the higher values will make UMAP focus on the larger neighbourhoods.
min_dist
This parameter controls how tightly UMAP is allowed to pack points together. Lower values mean the points will be clustered closely and vice versa.
n_components
This parameter allows the user to determine the dimensionality of the reduced dimension space.
metric
This parameter controls how distance is computed in the ambient space of the input data.
For more detailed information, I suggest to check out the UMAP documentation://umap-learn.readthedocs.io/en/latest/
For this tutorial, I have chosen to keep the default setting apart for n_components which I set to 3 for the 3d space plot. It would be best to experiment with different hyper-parameter settings to obtain the best out of the algorithm.
plot_2d(reducer.embedding_[:, 0],reducer.embedding_[:, 1])
We can clearly see that the UMAP does a great job in separating the data points compared to t-SNE and PCA in terms of separation. However, there are no big clusters separating the sign sufficiently, There are similar data points agglomerated together in other parts too from a 2d prospective.
plot_3d(reducer.embedding_[:, 0],reducer.embedding_[:, 1],reducer.embedding_[:, 2])
plot_2d(X_LDA[:, 0],X_LDA[:, 1])
With LDA we can clearly identify the presence of these nine clusters with a significant separation. If with UMAP and t-SNE we could barely see the backbone of the clusters with LDA we can see the whole clusters of data points agglomerated in the same cluster zones.
plot_3d(X_LDA[:, 0],X_LDA[:, 1],X_LDA[:, 2])
Note that: the above table was constructed considering the computation time taken on a Kernel on Kaggle using their GPU.
PCA was not able to do such a good job in differentiating the signs. The main drawback of PCA is that it is highly influenced by outliers present in the data. PCA is a linear projection, which means it can’t capture non-linear dependencies, its goal is to find the directions (the so-called principal components) that maximize the variance in a dataset.
t-SNE does a better job(it tries to preserve topology neighbourhood structure) as compared to PCA when it comes to visualising the different patterns of the clusters. Similar labels are clustered together, even though there are big agglomerates of data points on top of each other, certainly not good enough to expect a clustering algorithm to perform well.
UMAP outperformed t-SNE and PCA, if we look at the 2d and 3d plot, we can see mini-clusters that are being separated well. It is very effective for visualizing clusters or groups of data points and their relative proximities. However, for this use case certainly not good enough to expect a clustering algorithm to distinguish the patterns.UMAP is much faster than t-SNE, another problem faced by the latter is the need for another dimensionality reduction method prior, otherwise, it would take a longer time to compute.
Finally, LDA outperformed all the previous techniques in all aspects. Excellent computation time (second fastest) as well as proving the well-separated clusters we were expecting.
Note: it is not surprising that LDA performed better than the other techiniques ,that was what we were expecting . PCA, TSNE and UMAP are performed without the knowledge of the true class label, unlike LDA.
We have explored four dimensionality reduction techniques for data visualization : (PCA, t-SNE, UMAP, LDA)and tried to use them to visualize a high-dimensional dataset in 2d and 3d plots.
Note: It is not difficult to fall into the trap of considering one technique better than the other, at the end of the day there is no way to map a high-dimensional data into low dimensions and preserving the whole structure at the same time, there is always a trade-off of qualities one technique will have compared to the other.
Thanks again for reaching until here, hope it has been an informative post! worth your time. There are many other variants and many other use cases, I highly encourage you to explore this amazing and well-developed area of Science.
[1] McInnes, L., & Healy, J. (2018). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. ArXiv e-prints.
[2] van der Maaten, L.J.P. t-Distributed Stochastic Neighbor Embedding
lvdmaaten.github.io
[3]Kaggle.com. 2020. Visualizing Kannada MNIST With T-SNE. Available at: https://www.kaggle.com/parulpandey/visualizing-kannada-mnist-with-t-sne
[4]Hands on Machine Learning with Scikit-Learn, Keras & Tensorflow by Aurelien Geron