How to Determine the Right Number of Clusters (with Code) | by Adam Brownell | Towards Data Science

Clustering is a fundamental skill in your Data Science toolkit. It can solve a huge array of problems — from user segmentation to anomaly detection — and can help your team derive very interesting insights.
Determining the right number of clusters for your project is a little more art than science. In this article, I will go over a few common ways to determine the right number of clusters.
All of the following examples will involve the following imports:
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltfrom sklearn.cluster import KMeans
and the following variables:
min_clusters = 2max_clusters = 15
The objective of this metric is to find the “Elbow” of the WSS curve in order to determine the smallest number of clusters that captures the most amount of signal in your data. If this definition sounds vague and subjective, it’s because it is — finding the “elbow” of the chart is certainly a subjective decision.
Within-Cluster Sum of Squares (WSS) is a measure of how far away each centroid is from their respective class instances. The larger the WSS, the more dispersed the cluster values are from the centroid. We can define WSS as:
def within_sum_of_squares(data, centroids, labels):      SSW = 0    for l in np.unique(labels):        data_l = data[labels == l]        resid = data_l - centroids[l]        SSW += (resid**2).sum()    return SSW
And by simply iterating through our data for every cluster, we can generate the elbow curve:
wss_list = []for i in range(min_clusters, max_clusters+1):  print('Training {} cluster algo...'.format(i))  km = KMeans(n_clusters=i)  km.fit(data)  wss = within_sum_of_squares(np.array(data),km.cluster_centers_, km.predict(data))      wss_list.append(wss)plt.plot(wss_list)plt.title('WSS Plot')plt.xlabel('# of Clusters')plt.ylabel('WSS')plt.show()
From the above example, we may decide that 4–5 cluster is the appropriate number of clusters, since adding additional clusters.
Another way to cut up this data is by the relative improvement of each cluster. By setting a threshold, say 5%, we can see at what number of clusters would start to see diminished returns on additional clusters of less than 5%
perc_improve_list = [0]rel_improvement = []base_wss = wss_list[0]for i in range(len(wss_list)):  improvement = (wss_list[0] - wss_list[i])/wss_list[0]  rel_improvement.append(improvement - perc_improve_list[-1])  perc_improve_list.append(improvement)  threshold = 0.05plt.plot([i for i in range(min_clusters+1,max_clusters+1)], rel_improvement[1:])plt.axhline(threshold, linestyle='--', color='grey')plt.title('WSS Improvement Plot')plt.xlabel('# of Clusters')plt.ylabel('% improvement in WSS')plt.ylim([0,1])plt.show()
This chart points to the same outcome as the first, 4–5 clusters, albeit a little clearer. I prefer this output since it provides an objective threshold to the chart that we can modify at a later date.
“The optimal number of clusters k is the one that maximize the average silhouette over a range of possible values for k” — (Kaufman and Rousseeuw 1990)
A Silhouette Score is (other than being a word I can never spell correctly on first attempt) how close a specific example is to examples from other clusters. The larger the distance from other clusters’ examples, the cleaner the division between clusters and therefore the better clustering.
As the above quote states, by the number of clusters that maximizes the average silhouette score is preferred over other options.
To calculate silhouette scores, we can use SKLearns built in function:
from sklearn.metrics import silhouette_scoresil_score_list = []for i in range(min_clusters, max_clusters+1):  print('Training {} cluster algo...'.format(i))  km = KMeans(n_clusters=i)  km.fit(data)  sil = silhouette_score(np.array(data),km.predict(data))  sil_score_list.append(sil)  plt.bar([i for i in range(2,len(sil_score_list)+2)],sil_score_list)plt.title('Sil Plot')plt.xlabel('# of Clusters')plt.ylabel('Sil Score')plt.show()
For this metric, it seems to point to using 3 clusters to maximize the average Silhouette score. WSS and Sil scores won’t always agree, as they are measuring different things. Using both in conjunction will help you make the most informed decision, along with method #3 outlined below.
Please Note: While I don’t know the Big O Notation for the SKLearn Silhouette algo (someone please comment if you know!) but this took significantly longer than the WSS search outlined above. Whereas the WSS code finish in about a half hour, the Silhouette code took well over 4 hours. Be prepared!
The real answer to how many clusters are right is:
Whatever makes the most sense for your team & project
It may seem like a cop-out answer (and it certainly isn’t a data science answer) but what matters more than WSS or Silhouette scores is that the clustering you generate helps you accomplish the task at hand.
There are many more ways to validate clustering strength, from Dunn Statics to Gap Metrics, but the real way to validate your clustering approach is to ensure that it is able to answer the questions you are seeking. Oftentimes Data Scientists can’t see the forest for the trees, and stressing over multiple data-driven success criteria rather than on producing outcomes for your team is a common mistake.
If you are looking to these clusters to generate human-interpretable archetypes for your business, perhaps to inform marketing, then you don’t want to use more than 10 clusters; otherwise, your marketing team will get bogged down with too many personas to generate copy for.
On the other end of the spectrum, if you are using clustering for personalization, you want as many meaningful clusters as possible, since the more clusters you create the more unique experiences you can provide your customers. Micro-segmentation is excellent for recommendations but at the cost of interpretability.
So when determining clusters — ensure you are on the same page with the stakeholder on the project and feel confident that you can justify the number of clusters, whether it be 2 or 200, from a non-technical perspective.
Hope you enjoyed!