How to Analyze 100-Dimensional Data with UMAP in Breathtakingly Beautiful Ways | Towards Data Science

No matter how powerful machine learning models are, they can't quite beat the feeling you get during those "Aha!" moments of exploring data through rich visuals. But, there are so many histograms, scatterplots, heatmaps you can create before you say, "this is really getting old."
In those moments, you need something that reminds you of how amazing and mesmerizing data can be. You need to get inspiration from masterpiece visuals like at FlowingData or relevant subreddits, but you don't have to go that far. Recently, I have been fortunate to come across UMAP — a Python package to visualize and cluster high-dimensional data in breathtakingly beautiful ways. It was just what I needed to remember why I got into learning data science two years ago.
Today, we will learn how to analyze multi-dimensional datasets by projecting them to 2D using Uniform Manifold Approximation & Projection (UMAP) package through visuals like below:
UMAP is a dimensionality reduction algorithm and a powerful data analysis tool.
It is similar to PCA (Principal Component Analysis) in terms of speed and resembles tSNE to reduce dimensionality while preserving as much information of the dataset as possible. Before the UMAP algorithm was introduced in 2018, PCA and tSNE had two most significant flaws:
PCA was very fast at the cost of losing finer details of the data after the reductiontSNE was extremely slow even though it preserves the underlying structure of the data.
PCA was very fast at the cost of losing finer details of the data after the reduction
tSNE was extremely slow even though it preserves the underlying structure of the data.
We will talk more about these differences later. Now, it is time to get the first taste of UMAP, and we will immediately start with a challenging dataset:
>>> tps.shape(957919, 119)
The Kaggle TPS September dataset contains ~1M rows and ~120 features with a binary target. They are all numerical, and we are pretty helpless in performing proper EDA on this dataset. Our options are limited to only printing summary statistics and plotting histograms of each feature.
Let's see what UMAP can do for us. Before using it, we will sample the dataset to avoid overplotting and fill in the missing values:
Here, the target represents whether or not a client claims their insurance.
After installing and importing UMAP, we initialize the manifold algorithm and fit it to X, y in the familiar Sklearn fit/transform pattern:
>>> X_reduced.shape(100000, 2)
By default, UMAP projects the data into two components (2D). Let's create a scatterplot colored by the target class:
>>> plt.scatter(X_reduced[:, 0], X_reduced[:, 1], c=y, s=0.5);
Even though it is fun to look at, the plot does not show any clear patterns. It is because we didn't scale the features before fitting UMAP. The algorithm uses distance metrics to group similar data points, and features with higher scales bias such calculations.
So, we will choose Quantile Transformer to scale the features based on their quantiles and median. This scaling method suits the dataset better since it contains many skewed and bimodal features:
# Plot the resultsplt.scatter(X_reduced_2[:, 0], X_reduced_2[:, 1], c=y, s=0.5);
Now, we are talking! UMAP managed to capture the hidden distinction between the target classes perfectly. We can also see some outliers (the dot around the yellow blob). The dataset was not so challenging after all.
But, this plot is in no way close to what I have shown you. It is still overplotted to see structural patterns within each cluster. To take this to the next level, we will use the default UMAP visualization package with many more features. And we will need a better dataset.
This section will analyze the Kaggle TPS May competition data that categorizes ~200k eCommerce listings based on ~75 numeric qualities. Let's import it and take a quick look:
The target contains nine classes.
As before, we will scale all features, but this time with a straightforward log transform. Then, we fit the UMAP manifold:
After the fit is done, we will import the umap.plot package (installed separately) and plot a point cloud:
Doesn't it resemble a nebula from space? We can distinctly see that Class 8 dominates the space and is clustered around the center. Class 6 is also clearly distinguished from the rest. We see half a circle of mixed data points around class 8. Regarding the singleton data points, they may be classified as outliers.
A note on visualization above — we are simply passing the fitted manifold (not transformed data!) to the points function and specifying the labels for color encoding. I have also chosen the fire as a dark theme.
You can also create connectivity plots with umap.plot.connectivity for diagnostics purposes and to better understand the manifold structure. Do note that creating these plots is time-consuming and computation/memory-heavy.
You can learn more about UMAP visualizations from this section of the documentation.
The underlying reduction algorithm has many parameters that can significantly impact the manifold and hence, the visuals. The four most important ones are:
n_components
n_neighbors
min_dist
metric
As you might have guessed, n_components controls the number of dimensions after the projection. The default is 2 for its ease of visualization. However, for datasets with more than 100 features, 2D may not be enough to preserve the underlying topological structure of the data fully. I recommend trying values between 2-20 at steps of 5 and evaluate different baseline models to see the change in accuracy.
Next, we have n_neighbors. It controls the area of the local neighborhood UMAP looks at for each sample when building the manifold. Smaller values narrow the focus to local structure, taking into account peculiarities and small patterns, potentially losing the big picture.
Higher values to n_neighbors give more flexibility and allows UMAP to focus on a broader "view" of the data in the corresponding dimension. This, of course, comes at the cost of losing the finer details of the structure. The default value for this parameter is 15.
Another critical parameter is min_dist which controls the literal distance between data points. You can tweak the default value of 0.1 to control the tightness of distinct point clouds. Lower values will result in clumpier embeddings, allowing you to see individual clusters more easily. This may be useful during clustering. In contrast, values close to 1 give points more breathing room and enable you to see the broader topological structure.
metric represents the formula to calculate the distance between points. The default is euclidean but you can choose among many others, including manhattan, minkowski and chebyshev.
First consideration while using UMAP is the RAM consumption. Under the hood, UMAP consumes a lot of memory, especially during fitting and creating diagrams like connectivity plots. I suggest running UMAP on a machine with at least 16GB of RAM.
For example, even the 200k-row dataset in the plotting section consumed ~18GB of RAM when creating the connectivity plot. The documentation suggests setting low_memory to True as a possible fix. Additionally, I recommend reducing the memory usage of the dataset by casting each column to the smallest subtype possible using NumPy. I have discussed dealing with out-of-memory issues at length in a previous article:
towardsdatascience.com
Also, don't forget to transform/scale numerical features as they tend to have different scales by default. I suggest QuantileTransformer for crazy distributions like bimodals, trimodals, etc. PowerTransformer works best for skewed features. Whichever transformer you choose, the goal is always to make features as normally distributed as possible.
Today, we have covered the basics of UMAP and learned only a thin slice of what it can do. For further reading, I suggest checking out the documentation of the package. There, you will see different use cases of UMAP and how it is used by tech giants like Google in various projects.
There are also dedicated sections for comparing UMAP to other dimensionality reduction algorithms. For a math enthusiast, you can also read how UMAP works and proofs of its formulas. Thank you for reading!