K-Means Clustering: How It Works & Finding The Optimum Number Of Clusters In The Data
By: Serafeim Loukas
Tags: Data Science, Unsupervised Learning, Clustering, K Means, Machine Learning

Python working example

For this example we will create artificial data i.e. artificial clusters. This way we will know in advance the ground through i.e. the exact number of clusters in our dataset.

Let’s start with importing the required python libraries:



from sklearn.cluster import

from sklearn.metrics import silhouette_score



import matplotlib.pyplot as plt

import matplotlib.cm as cm

import numpy as np from sklearn.datasets import make_blobs from sklearn.cluster import KMeans from sklearn.metrics import silhouette_samples import matplotlib.pyplot as pltimport matplotlib.cm as cmimport numpy as np

Next, let’s create some artificial data containing 500 samples, 2 features/variables, and K=4 clusters.



# This particular setting has one distinct cluster and 3 clusters placed close together.

X, y =

n_features=2,

centers=4,

cluster_std=1,

center_box=(-10.0, 10.0),

shuffle=True,

random_state=1) # Generating the data# This particular setting has one distinct cluster and 3 clusters placed close together.X, y = make_blobs (n_samples=500,n_features=2,centers=4,cluster_std=1,center_box=(-10.0, 10.0),shuffle=True,random_state=1)

We know that we have K=4 clusters in the data however, in order to understand how the Silhouette Score works we will fit the model using a range of different number of clusters.

Each time, we will estimate the Silhouette Score and also plot the data with the final (converged) centroids. All these are done by the following code:

range_n_clusters = [3, 4, 5]

# Create a subplot with 1 row and 2 columns

fig, (ax1, ax2) =

fig.set_size_inches(18, 7)



# The 1st subplot is the silhouette plot

# The silhouette coefficient can range from -1, 1 but in this example all

# lie within [-0.1, 1]

ax1.set_xlim([-0.1, 1])

# The (n_clusters+1)*10 is for inserting blank space between silhouette

# plots of individual clusters, to demarcate them clearly.

ax1.set_ylim([0, len(X) + (n_clusters + 1) * 10])



# Initialize the clusterer with n_clusters value and a random generator

# seed of 10 for reproducibility.

clusterer =

cluster_labels = clusterer.fit_predict(X)



# The silhouette_score gives the average value for all the samples.

# This gives a perspective into the density and separation of the formed

# clusters

silhouette_avg =

print("For n_clusters =", n_clusters,

"The average silhouette_score is :", silhouette_avg)



# Compute the silhouette scores for each sample

sample_silhouette_values =



y_lower = 10

for i in range(n_clusters):

# Aggregate the silhouette scores for samples belonging to

# cluster i, and sort them

ith_cluster_silhouette_values = \

sample_silhouette_values[cluster_labels == i]



ith_cluster_silhouette_values.sort()



size_cluster_i = ith_cluster_silhouette_values.shape[0]

y_upper = y_lower + size_cluster_i



color = cm.nipy_spectral(float(i) / n_clusters)

ax1.fill_betweenx(

0, ith_cluster_silhouette_values,

facecolor=color, edgecolor=color, alpha=0.7)



# Label the silhouette plots with their cluster numbers at the middle

ax1.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i))



# Compute the new y_lower for next plot

y_lower = y_upper + 10 # 10 for the 0 samples



ax1.set_title("The silhouette plot for the various clusters.")

ax1.set_xlabel("The silhouette coefficient values")

ax1.set_ylabel("Cluster label")



# The vertical line for average silhouette score of all the values

ax1.axvline(x=silhouette_avg, color="red", linestyle="--")



ax1.set_yticks([]) # Clear the yaxis labels / ticks

ax1.set_xticks([-0.1, 0, 0.2, 0.4, 0.6, 0.8, 1])



# 2nd Plot showing the actual clusters formed

colors = cm.nipy_spectral(cluster_labels.astype(float) / n_clusters)

ax2.scatter(X[:, 0], X[:, 1], marker='.', s=30, lw=0, alpha=0.7,

c=colors, edgecolor='k')



# Labeling the clusters

centers = clusterer.cluster_centers_

# Draw white circles at cluster centers

ax2.scatter(centers[:, 0], centers[:, 1], marker='o',

c="white", alpha=1, s=200, edgecolor='k')



for i, c in enumerate(centers):

ax2.scatter(c[0], c[1], marker='$%d$' % i, alpha=1,

s=50, edgecolor='k')



ax2.set_title("The visualization of the clustered data.")

ax2.set_xlabel("Feature space for the 1st feature")

ax2.set_ylabel("Feature space for the 2nd feature")





"with n_clusters = %d" % n_clusters),

fontsize=14, fontweight='bold')



() for n_clusters in range_n_clusters:# Create a subplot with 1 row and 2 columnsfig, (ax1, ax2) = plt.subplots (1, 2)fig.set_size_inches(18, 7)# The 1st subplot is the silhouette plot# The silhouette coefficient can range from -1, 1 but in this example all# lie within [-0.1, 1]ax1.set_xlim([-0.1, 1])# The (n_clusters+1)*10 is for inserting blank space between silhouette# plots of individual clusters, to demarcate them clearly.ax1.set_ylim([0, len(X) + (n_clusters + 1) * 10])# Initialize the clusterer with n_clusters value and a random generator# seed of 10 for reproducibility.clusterer = KMeans (n_clusters=n_clusters, random_state=10)cluster_labels = clusterer.fit_predict(X)# The silhouette_score gives the average value for all the samples.# This gives a perspective into the density and separation of the formed# clusterssilhouette_avg = silhouette_score (X, cluster_labels)print("For n_clusters =", n_clusters,"The average silhouette_score is :", silhouette_avg)# Compute the silhouette scores for each samplesample_silhouette_values = silhouette_samples (X, cluster_labels)y_lower = 10for i in range(n_clusters):# Aggregate the silhouette scores for samples belonging to# cluster i, and sort themith_cluster_silhouette_values = \sample_silhouette_values[cluster_labels == i]ith_cluster_silhouette_values.sort()size_cluster_i = ith_cluster_silhouette_values.shape[0]y_upper = y_lower + size_cluster_icolor = cm.nipy_spectral(float(i) / n_clusters)ax1.fill_betweenx( np.arange (y_lower, y_upper),0, ith_cluster_silhouette_values,facecolor=color, edgecolor=color, alpha=0.7)# Label the silhouette plots with their cluster numbers at the middleax1.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i))# Compute the new y_lower for next ploty_lower = y_upper + 10 # 10 for the 0 samplesax1.set_title("The silhouette plot for the various clusters.")ax1.set_xlabel("The silhouette coefficient values")ax1.set_ylabel("Cluster label")# The vertical line for average silhouette score of all the valuesax1.axvline(x=silhouette_avg, color="red", linestyle="--")ax1.set_yticks([]) # Clear the yaxis labels / ticksax1.set_xticks([-0.1, 0, 0.2, 0.4, 0.6, 0.8, 1])# 2nd Plot showing the actual clusters formedcolors = cm.nipy_spectral(cluster_labels.astype(float) / n_clusters)ax2.scatter(X[:, 0], X[:, 1], marker='.', s=30, lw=0, alpha=0.7,c=colors, edgecolor='k')# Labeling the clusterscenters = clusterer.cluster_centers_# Draw white circles at cluster centersax2.scatter(centers[:, 0], centers[:, 1], marker='o',c="white", alpha=1, s=200, edgecolor='k')for i, c in enumerate(centers):ax2.scatter(c[0], c[1], marker='$%d$' % i, alpha=1,s=50, edgecolor='k')ax2.set_title("The visualization of the clustered data.")ax2.set_xlabel("Feature space for the 1st feature")ax2.set_ylabel("Feature space for the 2nd feature") plt.suptitle (("Silhouette analysis for KMeans clustering on sample data ""with n_clusters = %d" % n_clusters),fontsize=14, fontweight='bold') plt.show ()

Code output in the console (image created by the author)

We observe that the average/mean Silhouette Score is the highest in the case of K=4 clusters.

This verifies that the Silhouette Score is a good measure of K-means fitting goodness.

We also produced these Figures:

Silhouette score & the data with the final (converged) centroids (image created by the author)

Silhouette score & the data with the final (converged) centroids (image created by the author)

Silhouette score & the data with the final (converged) centroids (image created by the author)

The vertical line is the average silhouette score of all the values.

Again, we can also visually verify that the Silhouette Score is a good measure of K-means fitting goodness for the specific example.