What, Why and How of t-SNE. Dimensionality Reduction using t-SNE in... | by Ramya Vidiyala | Towards Data Science

Imagine the data we create in a single day; the news generated, posts, videos, images on social media platforms, messages on communication channels, websites which help business and many more... Huge! Right?
Now, imagine the task of analyzing this humongous data and obtaining useful insights to make data-driven decisions. Complex! Right?
What we can do about this problem is that we can remove redundant information and analyze only the high impact information.Dimensionality reduction comes into the picture at this very initial stage of any data analysis or data visualization.
Dimensionality Reduction means projecting data to a lower-dimensional space, which makes it easier for analyzing and visualizing data. However, the reduction of dimension requires a trade-off between accuracy (high dimensions) and interpretability (low dimensions).
But the key here is to retain maximum variance features and eliminate redundant features.
In this article, we will be focussing on the why, what, how, and also what not of t-SNE for Dimensionality Reduction.
Here is the flow of discussion:
Why not PCA? Why t-SNE?What’s t-SNE?How does t-SNE work?How to implement t-SNE?How to effectively use tSNE?
Why not PCA? Why t-SNE?
What’s t-SNE?
How does t-SNE work?
How to implement t-SNE?
How to effectively use tSNE?
When it comes to Dimensionality Reduction, PCA is famous because it is simple, fast, easy to use, and retains the overall variance of the dataset. For more about PCA at a glance, check this.
Though PCA is great, it does have some drawbacks. One of the major drawbacks of PCA is that it does not retain non-linear variance. This means PCA will not be able to get results for figures like this.
In simple terms, PCA works on retaining only global variance, and thus retaining local variance was the motivation behind t-SNE.
t-SNE is a nonlinear dimensionality reduction technique that is well suited for embedding high dimension data into lower dimensional data (2D or 3D) for data visualization.
t-SNE stands for t-distributed Stochastic Neighbor Embedding, which tells the following : Stochastic → not definite but random probability Neighbor →concerned only about retaining the variance of neighbor pointsEmbedding → plotting data into lower dimensions
In short, t-SNE is a machine learning algorithm that generates slightly different results each time on the same data set, focusing on retaining the structure of neighbor points.
2 parameters that can highly influence the results are a) n_iter: The number of iterations that the algorithm runs b) perplexity: This can be thought of as the number of neighboring points t-SNE must consider
Step 1: t-SNE constructs a probability distribution on pairs in higher dimensions such that similar objects are assigned a higher probability and dissimilar objects are assigned lower probability.
Step 2: Then, t-SNE replicates the same probability distribution on lower dimensions iteratively till the Kullback-Leibler divergence is minimized.
Kullback-Leibler divergence is a measure of the difference between the probability distributions from Step1 and Step2. KL divergence is mathematically given as the expected value of the logarithm of the difference of these probability distributions.
In this section, let us use Digit Recognizer data set from Kaggle for Dimensionality Reduction using t-SNE.
www.kaggle.com
Data Description:
The data file contains grey-scale images of hand-drawn digits, from zero through nine.
Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.
The training data set, (train.csv), has 785 columns. The first column, called “label”, is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.Each pixel column in the data set has a name like pixelx, where x is an integer between 0 and 783, inclusive. To locate this pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27, inclusive. Then pixelx is located on row i and column j of a 28 x 28 matrix, (indexing by zero).
Loading the data :
import pandas as pdmnist_data = pd.read_csv("mnist.csv")
The data file which is in CSV format (Comma Separated Value) is loaded into the data frame using Pandas.
mnist_data.shape
Output : (42000, 785)
Looking at the data, we observe that there are 42,000 data points and 785 features.
mnist_data.head()
Output:
The first column label is the target variable. And the remaining 784 columns are the features.
target_variable = mnist_data["label"]features_variable=mnist_data.drop("label",axis=1)
Assigning target_variable with label column and features_variable with the remaining columns.
print(target_variable.shape)print(features_variable.shape)
Output: (42000,)(42000, 784)
Now, let us see how this features_variable of dimensions 42000 data points and 784 features are reduced using t-SNE.
Implementing Dimensionality Reduction using t-SNE:
STEP 1: Standardization of data
from sklearn.preprocessing import StandardScalerstandarized_data = StandardScaler().fit_transform(features_variable)
Using StandardScaler( ).fit_transform( ), data can be Standarized in a single step.
STEP 2: Application of t-SNE on standardized data
from sklearn.manifold import TSNEmodel = TSNE(n_components=2, random_state=0,perplexity=50, n_iter=5000)tsne_data = model.fit_transform(standarized_data)
Here, we are creating an object of TSNE, and setting perplexity and n_iter values. We have used the fit_transform( ) method on the standardized data to get dimensionally reduced data using t-SNE.
To check that, let us just print the shape of tsne_data
print(standarized_data.shape)print(tsne_data.shape)
Output: (42000, 784)(42000,2)
We can now play along with various perplexity and n_iter to observe the results.
t-SNE plots are highly influenced by parameters. Thus it is necessary to perform t-SNE using different parameter values before analyzing results.Since t-SNE is stochastic, each run may lead to slightly different output. This can be solved by fixing the value of random_state parameter for all the runs.t-SNE doesn’t retain the distance between clusters from the raw data. Distance between clusters might vary post dimensionality reduction in t-SNE. It is recommended not to obtain any conclusions solely from the distance between the clusters.t-SNE shrinks widespread data and expands densely packed data. It is hence suggested not to decide the size and density/spread/variance of the clusters based on the output.Lower perplexity values might result in fewer clusters. It is hence recommended to try various perplexity values ranging from 2 to the number of data points to obtain better results.
t-SNE plots are highly influenced by parameters. Thus it is necessary to perform t-SNE using different parameter values before analyzing results.
Since t-SNE is stochastic, each run may lead to slightly different output. This can be solved by fixing the value of random_state parameter for all the runs.
t-SNE doesn’t retain the distance between clusters from the raw data. Distance between clusters might vary post dimensionality reduction in t-SNE. It is recommended not to obtain any conclusions solely from the distance between the clusters.
t-SNE shrinks widespread data and expands densely packed data. It is hence suggested not to decide the size and density/spread/variance of the clusters based on the output.
Lower perplexity values might result in fewer clusters. It is hence recommended to try various perplexity values ranging from 2 to the number of data points to obtain better results.
Resources :
This blog post by distill.pub made with the support of Google Brain. Do check this blog for more insights about how t-SNE effectively helps in reducing various data patterns for visualization.
Thanks for the read. I am going to write more beginner-friendly posts in the future. Follow me up on Medium to be informed about them. I welcome feedback and can be reached out on Twitter ramya_vidiyala and LinkedIn RamyaVidiyala. Happy learning!