Dimensionality Reduction for Data Visualization: PCA vs TSNE vs UMAP vs LDA
By: Sivakar Siva
Tags: Dimensionality Reduction, Artificial Intelligence, Machine Learning, Data Visualization, Data Science

In this story, we are gonna go through three Dimensionality reduction techniques specifically used for Data Visualization: PCA, t-SNE, LDA and UMAP. We are going to explore them in details using the Sign Language MNIST Dataset, without going in-depth with the maths behind the algorithms.

What is Dimensionality Reduction?

Many Machine Learning problems involve thousands of features, having such a large number of features bring along many problems, the most important ones are:

Makes the training extremely slow

Makes it difficult to find a good solution

This is known as the curse of dimensionality and the Dimensionality Reduction is the process of reducing the number of features to the most relevant ones in simple terms.

Reducing the dimensionality does lose some information, however as most compressing processes it comes with some drawbacks, even though we get the training faster, we make the system perform slightly worse, but this is ok! “sometimes reducing the dimensionality can filter out some of the noise present and some of the unnecessary details”.

Most Dimensionality Reduction applications are used for:

Data Compression

Noise Reduction

Data Classification

Data Visualization

One of the most important aspects of Dimensionality reduction, it is Data Visualization. Having to drop the dimensionality down to two or three, make it possible to visualize the data on a 2d or 3d plot, meaning important insights can be gained by analysing these patterns in terms of clusters and much more.

Main Approaches for Dimensionality Reduction

The two main approaches to reducing dimensionality: Projection and Manifold Learning.

Projection : This technique deals with projecting every data point which is in high dimension, onto a subspace suitable lower-dimensional space in a way which approximately preserves the distances between the points.

: This technique deals with projecting every data point which is in high dimension, onto a subspace suitable lower-dimensional space in a way which approximately preserves the distances between the points. Manifold Learning: Many dimensionality reductions algorithm work by modelling the manifold on which the training instance lie; this is called Manifold learning. It relies on the manifold hypothesis or assumption, which holds that most real-world high-dimensional datasets lie close to a much lower-dimensional manifold, this assumption in most of the cases is based on observation or experience rather than theory or pure logic.[4]

Now let's briefly explain the three techniques before jumping into solving the use case.

PCA (Principal Component Analysis)

One of the most known dimensionality reduction “unsupervised” algorithm is PCA(Principal Component Analysis).

This works by identifying the hyperplane which lies closest to the data and then projects the data on that hyperplane while retaining most of the variation in the data set.

Principal Components

The axis that explains the maximum amount of variance in the training set is called the Principal Components.

The axis orthogonal to this axis is called the second principal component. As we go for higher dimensions, PCA would find a third component orthogonal to the other two components and so on, for visualization purposes we always stick to 2 or maximum 3 principal components.

Source: Packt_Pub, via Hackernoon

It is very important to choose the right hyperplane so that when the data is projected onto it, it the maximum amount of information about how the original data is distributed.

t-SNE ( T-distributed stochastic neighbour embedding )

(t-SNE) or T-distributed stochastic neighbour embedding created in 2008 by (Laurens van der Maaten and Geoffrey Hinton) for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets.

(t-SNE) takes a high dimensional data set and reduces it to a low dimensional graph that retains a lot of the original information. It does so by giving each data point a location in a two or three-dimensional map. This technique finds clusters in data thereby making sure that an embedding preserves the meaning in the data. t-SNE reduces dimensionality while trying to keep similar instances close and dissimilar instances apart.[2]

For a quick a Visualization of this technique, refer to the animation below (it is taken from an amazing tutorial by Cyrille Rossant, I highly recommend to check out his amazing tutorial.

link: https://www.oreilly.com/content/an-illustrated-introduction-to-the-t-sne-algorithm/

Source: Cyrille Rossant ,via OReilly

LDA ( Linear Discriminant Analysis )

Linear Discriminant Analysis (LDA) is most commonly used as a dimensionality reduction technique in the pre-processing step for pattern-classification.

The goal is to project a dataset onto a lower-dimensional space with good class-separability in order to avoid overfitting and also reduce computational costs.

The general approach is very similar to PCA, rather than finding the component axes that maximize the variance of our data, we are additionally interested in the axes that maximize the separation between multiple classes (LDA)[5].

LDA is “supervised” and computes the directions (“linear discriminants”) that will represent the axes that maximize the separation between multiple classes.

UMAP ( Uniform Manifold Approximation and Projection )

Uniform Manifold Approximation and Projection created in 2018 by (Leland McInnes, John Healy, James Melville) is a general-purpose manifold learning and dimension reduction algorithm.

UMAP is a nonlinear dimensionality reduction method, it is very effective for visualizing clusters or groups of data points and their relative proximities.

The significant difference with TSNE is scalability, it can be applied directly to sparse matrices thereby eliminating the need to applying any Dimensionality reduction such as PCA or Truncated SVD(Singular Value Decomposition) as a prior pre-processing step.[1]

Put simply, it is similar to t-SNE but with probably higher processing speed, therefore, faster and probably better visualization. (let’s find it out in the tutorial below)

Use Case

Now we are going to go through the above-mentioned use case where all the three techniques will be applied: specifically, we will try to visualize a high dimensional dataset using these techniques: The Sign-Language-MNIST Dataset:https://www.kaggle.com/datamunge/sign-language-mnist

(Sign-Language-MNIST Dataset), screenshot from kaggle.com

The Data

Size of the train Data

The number of unique labels

Note that: There are 25 unique labels representing the number of distinct sign-languages. Now for better visualization (“ it is very difficult to observe all 24 classes in a single visualization”) and for faster computation. I am only keeping the first 10 labels, omitting the rest.

Implementing Plotting Functions

Standardizing the data

Implementing PCA

After applying PCA, the new dimensionality of the data now has only 3 features compared to the 784 features of the x data.

The number of dimensions has been cut down drastically whilst trying to retain as much of the ‘variation’ in the information as possible.

— PCA — 2D —