Less Known Applications of k-Means Clustering — Dimensionality Reduction, Anomaly Detection and Data Representation
By: Nitish Kumar Thakur
Tags: K Means, Dimensionality Reduction, Clustering, Artificial Intelligence, Machine Learning

k-Means is a data partitioning technique which is widely used for clustering. It has variants(like the mini-batch k-Means) which are incredibly fast for large amounts of data. Its clustering results are also easy to interpret. However, there are a lot of applications of k-Means which are not talked about a lot. They are:

Non-Linear Dimensionality Reduction — It can be used to represent thousands of features using only a few “transformed” features — which can be used as engineered features in a ML Pipeline or can be used for Data Visualization. Multivariate Outlier/Anomaly Detection Data Representation(For Input to other Algorithms) —Generally, k-means is only able to detect spherical clusters. However, it is possible to feed the output of k-means to a Hierarchical Clustering algorithm to detect non-convex clusters of complicated shapes.

We will discuss the above applications in details with examples and Python code. We will use scikit-learn. Let us load the data First.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

from sklearn import linear_model, preprocessing, model_selection, pipeline, ensemble, tree, datasets, cluster

sns.set(style = 'white', font_scale = 1.4) ######## Load the Data

data = pd.DataFrame(datasets.load_boston().data, columns = datasets.load_boston().feature_names)

y = datasets.load_boston().target features = ['CRIM', 'LSTAT', 'RM', 'AGE', 'INDUS', 'NOX', 'DIS']

data = data[features]

data.head()

Non Linear Dimensionality Reduction using K-Means

The idea is to use k-Means to calculate the cluster centers, setting the number of clusters to the number of dimensions we want in our transformed data. For our example, since we want to reduce our data to 2 dimensions, we set the number of clusters to 2. The new features in the transformed space are the distances of each point from each cluster center.

Here is how we perform Non-Linear Dimensionality using K-Means:

Fit k-Means to data. Set number of clusters to the number of dimensions we want to transform our data to — which is 2 in this case. Calculate the distance of each data point from each cluster center. This way, we get 2 features for each data point namely — (distance of point from cluster 1, distance of point from cluster 2). If we had k cluster centers, we would get k features per data point. These represent our transformed features. Luckily, the transform() method of the fitted K-means object does this for us.

Let us consider 7 features from the Boston Housing Dataset. We will attempt to reduce them to 2 dimensions using K-Means for Visualization.

Here is what the data looks like:

7 Features of the Boston Housing Data

We will Use Scikit-Learn Pipelines to perform scaling and fit k-Means.

#### Scale and Fit KMeans to data as part of the pipeline ####

kmeans = pipeline.make_pipeline(preprocessing.StandardScaler(), cluster.KMeans(n_clusters = 2)).fit(X_train) # Store transformed dimensions

lower_dim = pd.DataFrame(kmeans.transform(X_train), columns = ['Comp 1', 'Comp 2']) lower_dim.plot.scatter( x='Comp 1',y= 'Comp 2', grid = True, figsize = (10, 7))

2 Reduced Dimensions — using K-Means on the Boston Housing Data

In the above diagram, Component 1 measures the distance of each point from Cluster Center #1. Component 2 measures the distance of each point from Cluster Center #2. As we can see, we have some isolated points in the top right corner. As we will discuss later, points like this may be considered as outliers.

Following are some ways of Using the transformed features:

Data Visualization: For data with hundreds of features, this allows us to visualize and code data on a 2D Screen. The data can be color-coded according to various quantities of interest. For Predictive Modeling: Here, we use the transformed features as inputs to a model. This is specially useful for Linear models because the features created by K-Means are non-linear transformations — and can help account for non-linear relations in data. Anomaly detection: The larger the distance of a point from the cluster centers, the more the more chances it has of being an anomaly.

Let us take a look at how to use it for Predictive Modeling. We use the sklearn pipelines. Let us first use Linear Regression on the original data WITHOUT performing the transformation:

### Train Test Split

X_train, X_test, y_train, y_test = model_selection.train_test_split(data, y, test_size = .2, random_state = 10) ### Fit a Linear Model

model = linear_model.LinearRegression() score = model_selection.cross_val_score(model, X_train, y_train, cv = 10, scoring = 'r2')

print(f'Average r2: {np.mean(score)}') Average r2: 0.6534847470004668

Let us now perform the transformation using K-Means and train a linear model on the transformed features. We use sklearn pipelines and transform data to have 5 features:

km = pipeline.make_pipeline(preprocessing.StandardScaler(), cluster.KMeans(n_clusters = 5),

linear_model.LinearRegression()) score = model_selection.cross_val_score(km, X_train, y_train, cv = 10, scoring = 'r2')

print(f'Average r2: {np.mean(score)}') Average r2: 0.6751640572278654

In this case, the model performance improved on using 5 transformed features. However, the number of transformed features must be found via cross-validation or decided based on domain/business expertise in the general case.