Handling Outliers in Clusters using Silhouette Analysis
By: Satyam Kumar
Tags: Artificial Intelligence, Machine Learning, Data Science, Education, Clustering

Handling Outliers in Clusters using Silhouette Analysis

Identify and remove outliers in each cluster from K-Means clustering

Image by Gerd Altmann from Pixabay

The real-world data often has a lot of outlier values. The cause of outliers can be data corruption or failure to record data. The handling of outliers is very important during the data preprocessing pipeline as the presence of outliers can prevent the model to perform best.

There are various strategies to handle outliers in the dataset. This article will cover how to handle outliers after clustering data into several clusters using Silhouette Analysis.

Silhouette Analysis:

The silhouette method is a method to find the optimal number of clusters and interpretation and validation of consistency within clusters of data. The silhouette method computes silhouette coefficients of each point that measure how much a point is similar to its own cluster compared to other clusters. by providing a succinct graphical representation of how well each object has been classified. The analysis of these graphical representations is called Silhouette Analysis.

The silhouette value is a measure of how similar an object is to its own cluster (cohesion) compared to other clusters (separation). The value of the silhouette ranges between [1, -1].

Important Points:

The Silhouette coefficient of +1 indicates that the sample is far away from the neighboring clusters.

The Silhouette coefficient of 0 indicates that the sample is on or very close to the decision boundary between two neighboring clusters.

Silhouette coefficient <0 indicates that those samples might have been assigned to the wrong cluster or are outliers.

Computing Silhouette Coefficient:

Steps to find the silhouette coefficient of an i’th point:

Compute an (i): The average distance of that point with all other points in the same clusters. Compute b(i): The average distance of that point with all the points in the closest cluster to its cluster. Compute s(i) — silhouette coefficient or i’th point using below mentioned formula.

(Image by Author), Diagramatic representation of a(i) and b(i) from the above-mentioned formula to compute silhouette coefficient — s(i)

Find the optimal value of ‘k’ using Silhouette Analysis:

Use the Silhouette Method to find the optimal number of clusters. It can also be found using the Elbow method, but the Silhouette Method is considered a better approach than the Elbow method. Read the below article to know more:

(Image by Author), Left: Avg distance vs the number of clusters, Right: Silhouette score vs the number of clusters

The silhouette plot displays a measure of how close each point in one cluster is to points in the neighboring clusters and thus provides a way to assess parameters like the number of clusters visually.

Key Takeaways from Silhouette Analysis:

From the silhouette line plot and silhouette analysis for different values of n_clusters, it is observed that n_cluster=3 is the best value of the number of clusters (k). In the above image for “silhouette analysis for KMeans clustering on sample data with n_clusters=3”, it is observed that all the clusters [0,1,2] have most of their points have silhouette coefficients more than average silhouette score. In the above image for “silhouette analysis for KMeans clustering on sample data with n_clusters=3”, it is observed that for cluster_label 2, there are few points that have negative silhouette coefficients, that can be considered as outliers. Also for cluster_label 1, some points have silhouette coefficients less than the average silhouette score, which are points on cluster boundaries away from its cluster center.

To find the outliers, find the points that have a negative silhouette coefficient, and remove it. Points lying on the cluster boundaries away from its cluster center can also be removed to create a robust model, but it depends on the case study.