Visualize multi-dimension datasets in a 2D graph using t-SNE (Airbnb bookings dataset as an example)
By: Paul Lo
Tags: Machine Learning, Dimensionality Reduction, Clustering, Data Visualization, Data Exploration

Visualize multi-dimension datasets in a 2D graph using t-SNE (Airbnb bookings dataset as an example) Paul Lo Follow Jan 15 · 6 min read

Using 31 numeric features in the user booking dataset which has 12 different travel destinations for prediction— and yeah, I know it’s really messy, at least we immediately know we have some feature engineering work to do :D

t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm

First of all, what is t-SNE and when and why are we using it? It is an unsupervised and non-linear dimension reduction algorithm, people usually use it during exploratory data analysis , an early stage in the whole machine learning pipeline. It helps us surface high-dimensional datasets (e.g. many features) through a 2D or 3D plot (or other relatively low numbers), and thus get a quick intuition about the data. It is NOT designed in a way to apply it directly to a classification task.

How about PCA? In the dimension reduction area, people often compare it with PCA , or Principal Component Analysis . Actually, t-SNE is a much newer approach that was developed by Laurens van der Maatens and Geoffrey Hinton in 2008 (see paper “Visualizing Data using t-SNE” here), while PCA was developed by Hotelling H. back in 1933 (Analysis of a complex of statistical variables into principal components), almost 3 generations ago!

As mentioned in the t-SNE paper, there is certainly some limitations for the linear type of models like PCA, “For high-dimensional data that lies on or near a low-dimensional, non-linear manifold it is usually more important to keep the low-dimensional representations of very similar datapoints close together, which is typically not possible with a linear mapping.”

To understand better about this, we can take a look at the underlying algorithm (and many great ‘PCA vs t-SNE’ articles online), aside from the Algorithm section in the original paper, I also highly recommend this An illustrated introduction to the t-SNE algorithm read, which provides a very intuitive but mathematical perspective to the model. In order to move on to the coding section, here, let’s just say t-SNE is more effective in handling certain types of complicated data, compared with PCA ‘s linear approach. As shown in the following pictures, a Kaggle script created by puyokw demonstrates t-SNE’s capabilities clearly.

Code

For R

Step 1: Install and load Rtsne package

install.packages("Rtsne") # Install Rtsne from CRAN library

library(Rtsne)

Step 2. Load a dataset for our example use case

> iris_unique <- unique(iris) # remove duplicate rows

> head(iris_unique) Sepal.Length Sepal.Width Petal.Length Petal.Width Species

1 5.1 3.5 1.4 0.2 setosa

2 4.9 3.0 1.4 0.2 setosa

3 4.7 3.2 1.3 0.2 setosa

4 4.6 3.1 1.5 0.2 setosa

5 5.0 3.6 1.4 0.2 setosa

Step 3. Fit with t-SNE and visualize

Yes — it’s really that simple

> iris_matrix = as.matrix(iris_unique[,1:4]) # note: we can only pass in numeric columns

> tsne_out <- Rtsne(iris_matrix)

> plot(tsne_out$Y,col=iris$Species) # graph is now generated

t-SNE visualization for a simple Iris dataset, the three types of flowers are clearly divided into different 3 clusters

Now, let’s try with another real-world dataset, but much more complicated — Airbnb’s user booking dataset in a Kaggle competition. With the following code, we can check out its visualization in a 2D space.

Step 1. Load the data: Airbnb dataset (there are 213,451 rows in training dataset)

> library(readr)

> df_train = read_csv(“train_users.csv”) # subset numerical features

> numeric_columns = sapply(df_train, is.numeric)

> countries = as.factor(df_train$country_destination)

> df_train = df_train[, numeric_columns]

> df_train$country_destination = countries # put destination column back

> df_train_unique <- unique(train) # de-duplication

> dim(df_train_unique) [1] 213451 31

Step 2. Fit t-SNE and generate the plot

> matrx = as.matrix(df_train_unique)

> tsne_out <- Rtsne(matrx)

> plot(tsne_out$Y, col=countries, main=’t-SNE of airbnb dataset on Kaggle (31 features)’)

31 numeric features and 12 different target variables

It took 20–30 minutes on Rtsne() and unique() , and the resulting graph was bad, which implied that I hadn’t come up with good features yet to separate 12 different country destinations for an accurate prediction.

(The speed can be improved with parameters like theta, max_iter, etc.)

> colnames(df_train) [1] “timestamp_first_active” “age” [3] “signup_flow” “-unknown-” [5] “Android App Unknown Phone/Tablet” “Android Phone” [7] “Blackberry” “Chromebook” [9] “Linux Desktop” “Mac Desktop” [11] “Opera Phone” “Tablet” [13] “Windows Desktop” “Windows Phone” [15] “iPad Tablet” “iPhone” [17] “iPodtouch” “total_elapsed_time” [19] “-unknown-_pct” “Android App Unknown Phone/Tablet_pct” [21] “Android Phone_pct” “Blackberry_pct” [23] “Chromebook_pct” “Linux Desktop_pct” [25] “Mac Desktop_pct” “Opera Phone_pct” [27] “Tablet_pct” “Windows Desktop_pct” [29] “Windows Phone_pct” “iPad Tablet_pct” [31] “iPhone_pct”

Python code

For Python folks, we’ll be using TSNE package under sklearn.manifold , a simple use case looks like the following, while there are optional parameters including learning_rate, n_components (dimension of the embedded space, default=2), n_iter (maximum number of iterations for the optimization) to play with

from sklearn.manifold import TSNE

import matplotlib.pyplot as plt X_tsne = TSNE().fit_transform(df_train_unique)

scatter(X_tsne[:, 0], X_tsne[:, 1],

c=countries,cmap=plt.cm.spectral,alpha=.4,

edgecolor='k')

The Tradeoff: High time and space complexity

When I passed in completed 150+ features (including categorical fields added by dummyVars() ) of 200K data points to build a 2D t-SNE visualization, it took forever and ate up 25GB of memory in my MBP, pretty scary….!

Other than that, t-SNE is computationally expensive. As shown in the sklearn documents, in the same Manifold Learning (non-linear dimensionality reduction) family, t-SNE can take 6–100 times more compared with other models such as Spectral Embedding (SE) and Multi-dimensional Scaling (MDS).

t-SNE is very computationally intensive, even compared with other non-linear dimensional reduction models

When we think about the underlying algorithm, t-SNE has to compute the distances between all the points and maintain a pairwise N by N distance matrix (N = # of examples). Therefore, its space and time complexity are at quadratic level, O(n²), and this problem naturally becomes another popular research domain people are trying to optimize, one example is Fast Fourier Transform-accelerated Interpolation-based t-SNE (FIt-SNE), the details can be found in the paper here.

To be continued……

Going back to the Airbnb dataset example, we should be able to generate a better visualization result with t-SNE — please stay tuned, let me share more next time!

Reference

End Notes: