Clustering analyses with highly imbalanced datasets
By: Cole Brendel
Tags: Imbalanced Data, Hierarchical Clustering, K Means, Clustering, Data Science

Clustering analyses with highly imbalanced datasets

Introduction

Clustering analyses allow one to identify and interpret structure within a data set. As data scientists, we are often asked questions which can be answered only by understanding the latent structure within our data. For example, the marketing team might want to understand the “categories” of users that are using a product, so that advertisements or other promotional material can be targeted. Or perhaps your leadership knows that there are power-users of your product, but their characteristics are not well understood — are all power users interacting with your product in the same way? (However you choose to define “power user”.) Beyond understanding the inherent structure in our datasets, clustering analyses can be leveraged to improve the performance of predictive models (q.v., cluster-then-predict for classifiers).

When I say clustering, the algorithm that most probably comes to mind is k-Means; however, this algorithm is not a silver bullet. Two areas where the shortcomings of k-Means become apparent are (1) highly imbalanced datasets and (2) data in high-dimensional space — both spaces in which we data scientists often find ourselves. To make matters worse, we are often confronted with datasets which are both high dimensional and imbalanced!

In this article, I will show how to leverage hierarchical clustering analysis to identify structure in imbalanced data while not falling prey to the shortcomings of k-Means.

When and why k-Means Falls Apart

To understand why k-Means fails in high-dimensional data, it is informative to review how this algorithm works. Formally, the objective of k-means is to take a dataset of n observations and partition it into k < n clusters, such that the sum of squared errors from each datapoint to the cluster mean is minimized. (See this article for a more thorough introduction).

The complexity of the k-means algorithm is NP-Hard and takes exponential time to solve. Given n observations with d dimensions (features) binned into k clusters, the time complexity is: O( n^dk). So, a dataset of 1k observations with 10 features clustered into 3 groups would take 1x10³⁰ calculations to solve. This problem is well known as the curse of dimensionality.

In practice, approximate algorithms are used; however, complexity alone is not the only problem with this k-means. When dealing with datasets with high-levels of imbalance, the algorithm will often fair poorly at identifying structure in the small non-majority class. Let us consider a concrete example of this.

K-Means with Imbalanced Data

First, let’s generate a dataset of 3 known groups: a, b, c consisting of 2 features x and y, each with ever decreasing sample size, randomly decreasing mean (x,y) values, and consistent standard deviations.

import pandas as pd

import numpy as np mean = 5

std = 2

targets = [0, 1, 2]

sizes = [1500,750,150] dfs = []

for target, size in zip(targets, sizes):

df = pd.DataFrame()

df['x'] = np.random.normal(mean - np.random.rand(), std, size)

df['y'] = np.random.normal(mean + np.random.rand(), std, size)

df['target'] = target

dfs.append(df)

mean -= np.random.randint(low=2,high=5)



data = pd.concat(dfs, ignore_index=True)

We have a simple 2 dimensional dataset with high imbalance. Here are the means and size of each target group in the dataset. We can see target=0 is the largest group at n=1500, followed by target=1 at n=750, and finally we have target=2 at n=150. Inspecting the means of these groups, we would expect that all 3 groups should be easily separable.

data.groupby('target').agg(['mean', 'count']) '''

target x y

mean count mean count

0 4.972052 1500 5.737989 1500

1 0.689288 750 1.424916 750

2 -3.471737 150 -1.623322 150 '''

Now, let’s see what happens when we fit k-means to this dataset and plot the cluster centroids. I have added code to plot the means of each group in red and the location of the centroids in blue. Recall that one must specify the number of clusters you would like to identify in the dataset. We know there are 3, so let’s use that!

from sklearn.cluster import KMeans # set 3 clusters, random_state to make it repeatable

kmeans = KMeans(n_clusters=3, random_state=8675309).fit(data[['x','y']]) # fit and leave clusters as col in data

data['k-cluster'] = kmeans.fit_predict(data[['x','y']]) def k_means_plotter(model, data, x='x', y='y', target='target'):

'''plots means of targets wrt clusters learned from model'''

import seaborn as sns

import matplotlib.pyplot as plt # private generator to yield new markers

def _get_markers():

from matplotlib.lines import Line2D

yield from list(Line2D.filled_markers) # set the scatter plot

g = sns.scatterplot(x=x, y=y, hue=target, data=data, alpha=0.2) #vis_buffer = 0.25 # adds separation for text for annotations

marker = _get_markers() # generator for getting next marker

for i, c in enumerate(model.cluster_centers_):

next_marker = next(marker)



# add annots for centriods

plt.text(c[0], c[1], f'centriod {i}') # plot the annotation

plt.scatter(x=c[0], y=c[1], color='b', marker=next_marker) # plot the centriod



# add annots for mean of each target

plt.text(data.loc[data[f'{target}']==i][f'{x}'].mean(),

data.loc[data[f'{target}']==i][f'{y}'].mean(),

f'target {i}')

plt.scatter(data.loc[data[f'{target}']==i][f'{x}'].mean(),

data.loc[data[f'{target}']==i][f'{y}'].mean(),

color='r', marker=next_marker)





return g k_means_plotter(kmeans, data)

Target 0 pulls all centroids towards it… like a black hole

Upon visual inspection, we can see that the large cluster target=0 is like a blackhole; it is pulling all the centroids nearer to it. Notice how centroid 2 is located right at the border between target=0 and target=1 . When we inspect the cluster assignments learned by k-Means, we see a lot of “confusion” between target=0 and target=1 , poor tiny target=2 is simply lumped in with the target=1 class.

data.groupby(['target', 'k-cluster'])['target'].count() '''

target k-cluster

0 1 1109 *** target 0 is majority in k-cluster 1

2 391

1 0 354 *** target 1 tied between k-cluster 0, 2

1 6

2 390 ***

2 0 148 ** target 2 is lumped into k-cluster 0

2 2

Name: target, dtype: int64

'''

Inspecting the assignments reveals that target=1 is torn between 2 clusters, that of the majority class target=0 and the minority class target=2 . It is clear that k-Means has difficulty separating the targets in this dataset.

Hierarchical Clustering to the Rescue

So, how can we overcome this problem? One potential solution is to use Hierarchical Clustering. Hierarchical Clustering directly optimizes for distance-based loss functions and is more robust in the face of highly imbalanced datasets.

The agglomerative clustering algorithm proceeds with each observation in its own cluster, and then iteratively, the nearest observations (according to some distance metric) are merged until all of the data hierarchically belong to a single massive cluster.

This highlights one very nice property of this algorithm: you do not need to set the n_clusters a priori. After the clustering model has been fit, you can select the number of clusters you would like to retain using as many clusters as you have observations to as few as one single cluster. Although, if you are clustering, why would anyone ever do that!?

Wait, what about high dimensional datasets?

In the case of a high-dimensional dataset, it may make sense to run the clustering on a correlation matrix of features. Stay tuned for a detailed tutorial.

Hierarchical Clustering

Now, let’s see how Hierarchical Clustering fairs on this dataset. One should experiment with different linkage parameters. This stack exchange provides a good summary. For our case, I have used complete as this linkage tends to produce very compact clusters. As a convenience, sklearn allows you to set the n_clusters you would like to extract for predictions; however, do not forget that this is not necessary for this model.

from sklearn.cluster import AgglomerativeClustering # set to complete linkage for

hclust = AgglomerativeClustering(linkage='complete', n_clusters=3)

data['h-cluster'] = hclust.fit_predict(data[['x','y']]) data.groupby(['target', 'h-cluster'])['target'].count() '''

target h-cluster

0 0 574 ** some confusion into h-clust 0

1 926 *** target 0 clusters into h-clust 1

1 0 721 *** target 1 maps heavily into h-clust 0

1 3

2 26

2 0 33

2 117 ** target 2 maps heavily to h-clust 2

Name: target, dtype: int64

'''

From the numbers alone, it is clear that hierarchical clustering has done a better job at detecting the structure in this dataset. Each target in the dataset is mapped to its own cluster.

The minority target=2 class is almost entirely mapped to h-cluster=2 . When we consider target=1 , we see a strong mapping to h-cluster=0 and lastly, target=0 is majority mapped to h-cluster=1 .

No algorithm will be perfect in real life. As we see here, there is still some confusion between targets 0 and 1: around 1/3 of the target=0 observations have slipped into this same cluster as target=1 . No clustering model is perfect, but it is clear that hierarchical clustering outperforms on this dataset.

As the last step, let’s visualize the clusters using a dendrogram — the traditional method to examine h-clust models.

## codes modified from: https://scikit-learn.org/stable/auto_examples/cluster/plot_agglomerative_dendrogram.html def plot_dendrogram(model, **kwargs):

import matplotlib.pyplot as plt

from scipy.cluster.hierarchy import dendrogram

# Create linkage matrix and then plot the dendrogram # create the counts of samples under each node

counts = np.zeros(model.children_.shape[0])

n_samples = len(model.labels_)

for i, merge in enumerate(model.children_):

current_count = 0

for child_idx in merge:

if child_idx < n_samples:

current_count += 1 # leaf node

else:

current_count += counts[child_idx - n_samples]

counts[i] = current_count linkage_matrix = np.column_stack([model.children_, model.distances_,

counts]).astype(float) plt.title('Hierarchical Clustering Dendrogram')

dendrogram(linkage_matrix, **kwargs)

plt.xticks(rotation = 45)

plt.xlabel("Number of points in node (or index of point if no parenthesis).") # set n_clusters to none for plotting

hclust_plotting = AgglomerativeClustering(linkage='complete', distance_threshold=0, n_clusters=None)

hclust_plotting.fit(data[['x','y']])

plot_dendrogram(hclust_plotting, truncate_mode='level', p=3)

Conclusion

In this article, I’ve shown you the how and why of failures when using k-means in imbalanced datasets and have demonstrated one potential recourse: hierarchical clustering. In short, k-means performs poorly when data are imbalanced and tends to pull all centroids to the mean of the majority class. Hierarchical clustering can help you to identify latent structure in such datasets.