Handling Outliers in Clusters using Silhouette Analysis | by Satyam Kumar | Towards Data Science

The real-world data often has a lot of outlier values. The cause of outliers can be data corruption or failure to record data. The handling of outliers is very important during the data preprocessing pipeline as the presence of outliers can prevent the model to perform best.
There are various strategies to handle outliers in the dataset. This article will cover how to handle outliers after clustering data into several clusters using Silhouette Analysis.
The silhouette method is a method to find the optimal number of clusters and interpretation and validation of consistency within clusters of data. The silhouette method computes silhouette coefficients of each point that measure how much a point is similar to its own cluster compared to other clusters. by providing a succinct graphical representation of how well each object has been classified. The analysis of these graphical representations is called Silhouette Analysis.
The silhouette value is a measure of how similar an object is to its own cluster (cohesion) compared to other clusters (separation). The value of the silhouette ranges between [1, -1].
Important Points:The Silhouette coefficient of +1 indicates that the sample is far away from the neighboring clusters.The Silhouette coefficient of 0 indicates that the sample is on or very close to the decision boundary between two neighboring clusters.Silhouette coefficient <0 indicates that those samples might have been assigned to the wrong cluster or are outliers.
Steps to find the silhouette coefficient of an i’th point:
Compute an (i): The average distance of that point with all other points in the same clusters.Compute b(i): The average distance of that point with all the points in the closest cluster to its cluster.Compute s(i) — silhouette coefficient or i’th point using below mentioned formula.
Compute an (i): The average distance of that point with all other points in the same clusters.
Compute b(i): The average distance of that point with all the points in the closest cluster to its cluster.
Compute s(i) — silhouette coefficient or i’th point using below mentioned formula.
Use the Silhouette Method to find the optimal number of clusters. It can also be found using the Elbow method, but the Silhouette Method is considered a better approach than the Elbow method. Read the below article to know more:
towardsdatascience.com
The silhouette plot displays a measure of how close each point in one cluster is to points in the neighboring clusters and thus provides a way to assess parameters like the number of clusters visually.
From the silhouette line plot and silhouette analysis for different values of n_clusters, it is observed that n_cluster=3 is the best value of the number of clusters (k).In the above image for “silhouette analysis for KMeans clustering on sample data with n_clusters=3”, it is observed that all the clusters [0,1,2] have most of their points have silhouette coefficients more than average silhouette score.In the above image for “silhouette analysis for KMeans clustering on sample data with n_clusters=3”, it is observed that for cluster_label 2, there are few points that have negative silhouette coefficients, that can be considered as outliers. Also for cluster_label 1, some points have silhouette coefficients less than the average silhouette score, which are points on cluster boundaries away from its cluster center.
From the silhouette line plot and silhouette analysis for different values of n_clusters, it is observed that n_cluster=3 is the best value of the number of clusters (k).
In the above image for “silhouette analysis for KMeans clustering on sample data with n_clusters=3”, it is observed that all the clusters [0,1,2] have most of their points have silhouette coefficients more than average silhouette score.
In the above image for “silhouette analysis for KMeans clustering on sample data with n_clusters=3”, it is observed that for cluster_label 2, there are few points that have negative silhouette coefficients, that can be considered as outliers. Also for cluster_label 1, some points have silhouette coefficients less than the average silhouette score, which are points on cluster boundaries away from its cluster center.
To find the outliers, find the points that have a negative silhouette coefficient, and remove it. Points lying on the cluster boundaries away from its cluster center can also be removed to create a robust model, but it depends on the case study.
Read this article to get deep dive explanation of Silhouette Analysis, and refer gist to get python code implementation.
[1] Wikipedia: Silhouette (clustering), (14 Sep 2020)
[2] Scikit Learn documentation: Selecting the number of clusters with silhouette analysis on KMeans clustering
Thank You for Reading