Dimensionality Reduction using t-Distributed Stochastic Neighbor Embedding (t-SNE) on the MNIST Dataset | by Dehao Zhang | Towards Data Science

It is easy for us to visualize two or three dimensional data, but once it goes beyond three dimensions, it becomes much harder to see what high dimensional data looks like.
Today we are often in a situation that we need to analyze and find patterns on datasets with thousands or even millions of dimensions, which makes visualization a bit of a challenge. However, a tool that can definitely help us better understand the data is dimensionality reduction.
In this post, I will discuss t-SNE, a popular non-linear dimensionality reduction technique and how to implement it in Python using sklearn. The dataset I have chosen here is the popular MNIST dataset.
What is t-SNE and how does it work?How is t-SNE different with PCA?How can we improve upon t-SNE?What are the limitations?What can we do next?
What is t-SNE and how does it work?
How is t-SNE different with PCA?
How can we improve upon t-SNE?
What are the limitations?
What can we do next?
T-Distributed Stochastic Neighbor Embedding, or t-SNE, is a machine learning algorithm and it is often used to embedding high dimensional data in a low dimensional space [1].
In simple terms, the approach of t-SNE can be broken down into two steps. The first step is to represent the high dimensional data by constructing a probability distribution P, where the probability of similar points being picked is high, whereas the probability of dissimilar points being picked is low. The second step is to create a low dimensional space with another probability distribution Q that preserves the property of P as close as possible.
In step 1, we compute the similarity between two data points using a conditional probability p. For example, the conditional probability of j given i represents that x_j would be picked by x_i as its neighbor assuming neighbors are picked in proportion to their probability density under a Gaussian distribution centered at x_i [1]. In step 2, we let y_i and y_j to be the low dimensional counterparts of x_i and x_j, respectively. Then we consider q to be a similar conditional probability for y_j being picked by y_i and we employ a student t-distribution in the low dimension map. The locations of the low dimensional data points are determined by minimizing the Kullback–Leibler divergence of probability distribution P from Q.
For more technical details of t-SNE, check out this paper.
I have chosen the MNIST dataset from Kaggle (link) as the example here because it is a simple computer vision dataset, with 28x28 pixel images of handwritten digits (0–9). We can think of each instance as a data point embedded in a 784-dimensional space.
To see the full Python code, check out my Kaggle kernel.
Without further ado, let’s get to the details!
Note that in the original Kaggle competition, the goal is to build a ML model using the training images with true labels that can accurately predict the labels on the test set. For our purposes here we will only use the training set.
As usual, we check its shape first:
train.shape--------------------------------------------------------------------(42000, 785)
There are 42K training instances. The 785 columns are the 784 pixel values, as well as the ‘label’ column.
We can check the label distribution as well:
label = train["label"]label.value_counts()--------------------------------------------------------------------1    46847    44013    43519    41882    41776    41370    41324    40728    40635    3795Name: label, dtype: int64
Principal Component Analysis (PCA)
Before we implement t-SNE, let’s try PCA, a popular linear method for dimensionality reduction.
After we standardize the data, we can transform our data using PCA (specify ‘n_components’ to be 2):
from sklearn.preprocessing import StandardScalerfrom sklearn.decomposition import PCAtrain = StandardScaler().fit_transform(train)pca = PCA(n_components=2)pca_res = pca.fit_transform(train)
Let’s make a scatter plot to visualize the result:
sns.scatterplot(x = pca_res[:,0], y = pca_res[:,1], hue = label, palette = sns.hls_palette(10), legend = 'full');
As shown in the scatter plot, PCA with two components does not sufficiently provide meaningful insights and patterns about the different labels. We know one drawback of PCA is that the linear projection can’t capture non-linear dependencies. Let’s try t-SNE now.
T-SNE with sklearn
We will implement t-SNE using sklearn.manifold (documentation):
from sklearn.manifold import TSNEtsne = TSNE(n_components = 2, random_state=0)tsne_res = tsne.fit_transform(train)sns.scatterplot(x = tsne_res[:,0], y = tsne_res[:,1], hue = label, palette = sns.hls_palette(10), legend = 'full');
Now we can see that the different clusters are more separable compared with the result from PCA. Here are a few observations on this plot:
The “5” data points seem to be more spread out compared with the other clusters such as “2” and “4”.There are a few “5” and “8” data points that are similar to “3”s.There are two clusters of “7” and “9” where they are next to each other.
The “5” data points seem to be more spread out compared with the other clusters such as “2” and “4”.
There are a few “5” and “8” data points that are similar to “3”s.
There are two clusters of “7” and “9” where they are next to each other.
An Approach that Combines Both
It is generally recommended to use PCA or TruncatedSVD to reduce the number of dimension to a reasonable amount (e.g. 50) before applying t-SNE [2].
Doing so can reduce the level of noise as well as speed up the computations.
Let’s try PCA (50 components) first and then apply t-SNE. Here is the scatter plot:
Compared with the previous scatter plot, wecan now separate out the 10 clusters better. here are a few observations:
Most of the “5” data points are not as spread out as before, despite a few that still look like “3”.There is one cluster of “7” and one cluster of “9” now.
Most of the “5” data points are not as spread out as before, despite a few that still look like “3”.
There is one cluster of “7” and one cluster of “9” now.
Besides, the runtime in this approach decreased by over 60%.
For more interactive 3D scatter plots, check out this post.
Here are a few limitations of t-SNE:
Unlike PCA, the cost function of t-SNE is non-convex, meaning there is a possibility that we would be stuck in a local minima.Similar to other dimensionality reduction techniques, the meaning of the compressed dimensions as well as the transformed features becomes less interpretable.
Unlike PCA, the cost function of t-SNE is non-convex, meaning there is a possibility that we would be stuck in a local minima.
Similar to other dimensionality reduction techniques, the meaning of the compressed dimensions as well as the transformed features becomes less interpretable.
Here are a few things that we can try as next steps:
Hyperparameter tuning — Try tune ‘perplexity’ and see its effect on the visualized output.Try some of the other non-linear techniques such as Uniform Manifold Approximation and Projection (UMAP), which is the generalization of t-SNE and it is based on Riemannian geometry.Train ML models on the transformed data and compare its performance with those from models without dimensionality reduction.
Hyperparameter tuning — Try tune ‘perplexity’ and see its effect on the visualized output.
Try some of the other non-linear techniques such as Uniform Manifold Approximation and Projection (UMAP), which is the generalization of t-SNE and it is based on Riemannian geometry.
Train ML models on the transformed data and compare its performance with those from models without dimensionality reduction.
Let’s quickly recap.
We implemented t-SNE using sklearn on the MNIST dataset. We compared the visualized output with that from using PCA, and lastly, we tried a mixed approach which applies PCA first and then t-SNE.
I hope you enjoyed this blog post and please share any thoughts that you may have :)
Check out my other post on Chi-square test for independence:
towardsdatascience.com
[1] https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding[2] https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html