K-Means Clustering: How It Works & Finding The Optimum Number Of Clusters In The Data | by Serafeim Loukas | Towards Data Science

K-means is one of the most widely used unsupervised clustering methods.
The K-means algorithm clusters the data at hand by trying to separate samples into K groups of equal variance, minimizing a criterion known as the inertia or within-cluster sum-of-squares. This algorithm requires the number of clusters to be specified. It scales well to large number of samples and has been used across a large range of application areas in many different fields.
The k-means algorithm divides a set of N samples (stored in a data matrix X) into K disjoint clusters C, each described by the mean μj of the samples in the cluster. The means are commonly called the cluster “centroids”.
K-means algorithm falls into the family of unsupervised machine learning algorithms/methods. For this family of models, the research needs to have at hand a dataset with some observations without the need of having also the labels/classes of the observations. Unsupervised learning studies how systems can infer a function to describe a hidden structure from unlabeled data.
Now let’s discover the mathematical foundations of the algorithm.
Given a set of observations (x1, x2, ..., xn), where each observation is a d-dimensional real vector, k-means clustering aims to partition the n observations into k (≤ n) sets S = {S1, S2, ..., Sk} so as to minimize the within-cluster sum of squares (WCSS) (i.e. variance). Formally, the objective is defined as follows:
where μi is the mean of points in group/cluster Si.
The algorithm can also be understood through the concept of Voronoi diagrams.
First, the Voronoi diagram of the points is calculated using the current centroids. Initially, the centroids are usually chosen randomly but this depends on the underlying package / library / software used.
Each segment in the Voronoi diagram becomes a separate cluster.
Secondly, the centroids are updated to the mean of each segment. The algorithm then repeats this until a stopping criterion is fulfilled.
The algorithm stops when the relative decrease in the objective function between iterations is less than the given tolerance (ε) value or when centroids move (in space) less than the tolerance.
As stated in the introduction, the K-Means algorithm requires the number of clusters to be specified beforehand. In case that there is no valid assumption about the underlying number of clusters in the data, this task can be difficult to tackle.
Fortunately, there are some methods for estimating the optimum number of clusters in our data such as the Silhouette Coefficient or the Elbow method. If the ground truth labels are not known, evaluation must be performed using the model itself. In this article we will only use the Silhouette Coefficient and not the Elbow method which is much simpler. Briefly, the elbow method looks at the percentage of variance explained as a function of the number of clusters: One should choose a number of clusters so that adding another cluster doesn’t give much better modeling of the data.
A higher Silhouette Coefficient score relates to a model with better defined clusters. The Silhouette Coefficient is defined for each sample and is composed of two scores:
a: The mean distance between a sample and all other points in the same class.
b: The mean distance between a sample and all other points in the next nearest cluster.
The Silhouette Coefficient s for a single sample is then defined as:
Finally, the Total Silhouette Coefficient for a set of samples is given as the mean of the Silhouette Coefficient for each sample.
The best value is 1 and the worst value is -1. Values near 0 indicate overlapping clusters. Negative values generally indicate that a sample has been assigned to the wrong cluster, as a different cluster is more similar.
For this example we will create artificial data i.e. artificial clusters. This way we will know in advance the ground through i.e. the exact number of clusters in our dataset.
Let’s start with importing the required python libraries:
from sklearn.datasets import make_blobsfrom sklearn.cluster import KMeansfrom sklearn.metrics import silhouette_samples, silhouette_scoreimport matplotlib.pyplot as pltimport matplotlib.cm as cmimport numpy as np
Next, let’s create some artificial data containing 500 samples, 2 features/variables, and K=4 clusters.
# Generating the data# This particular setting has one distinct cluster and 3 clusters placed close together.X, y = make_blobs(n_samples=500,                  n_features=2,                  centers=4,                  cluster_std=1,                  center_box=(-10.0, 10.0),                  shuffle=True,                  random_state=1)
We know that we have K=4 clusters in the data however, in order to understand how the Silhouette Score works we will fit the model using a range of different number of clusters.
Each time, we will estimate the Silhouette Score and also plot the data with the final (converged) centroids. All these are done by the following code:
range_n_clusters = [3, 4, 5]for n_clusters in range_n_clusters:    # Create a subplot with 1 row and 2 columns    fig, (ax1, ax2) = plt.subplots(1, 2)    fig.set_size_inches(18, 7)    # The 1st subplot is the silhouette plot    # The silhouette coefficient can range from -1, 1 but in this example all    # lie within [-0.1, 1]    ax1.set_xlim([-0.1, 1])    # The (n_clusters+1)*10 is for inserting blank space between silhouette    # plots of individual clusters, to demarcate them clearly.    ax1.set_ylim([0, len(X) + (n_clusters + 1) * 10])    # Initialize the clusterer with n_clusters value and a random generator    # seed of 10 for reproducibility.    clusterer = KMeans(n_clusters=n_clusters, random_state=10)    cluster_labels = clusterer.fit_predict(X)    # The silhouette_score gives the average value for all the samples.    # This gives a perspective into the density and separation of the formed    # clusters    silhouette_avg = silhouette_score(X, cluster_labels)    print("For n_clusters =", n_clusters,          "The average silhouette_score is :", silhouette_avg)    # Compute the silhouette scores for each sample    sample_silhouette_values = silhouette_samples(X, cluster_labels)    y_lower = 10    for i in range(n_clusters):        # Aggregate the silhouette scores for samples belonging to        # cluster i, and sort them        ith_cluster_silhouette_values = \            sample_silhouette_values[cluster_labels == i]        ith_cluster_silhouette_values.sort()        size_cluster_i = ith_cluster_silhouette_values.shape[0]        y_upper = y_lower + size_cluster_i        color = cm.nipy_spectral(float(i) / n_clusters)        ax1.fill_betweenx(np.arange(y_lower, y_upper),                          0, ith_cluster_silhouette_values,                          facecolor=color, edgecolor=color, alpha=0.7)        # Label the silhouette plots with their cluster numbers at the middle        ax1.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i))        # Compute the new y_lower for next plot        y_lower = y_upper + 10  # 10 for the 0 samples    ax1.set_title("The silhouette plot for the various clusters.")    ax1.set_xlabel("The silhouette coefficient values")    ax1.set_ylabel("Cluster label")    # The vertical line for average silhouette score of all the values    ax1.axvline(x=silhouette_avg, color="red", linestyle="--")    ax1.set_yticks([])  # Clear the yaxis labels / ticks    ax1.set_xticks([-0.1, 0, 0.2, 0.4, 0.6, 0.8, 1])    # 2nd Plot showing the actual clusters formed    colors = cm.nipy_spectral(cluster_labels.astype(float) / n_clusters)    ax2.scatter(X[:, 0], X[:, 1], marker='.', s=30, lw=0, alpha=0.7,                c=colors, edgecolor='k')    # Labeling the clusters    centers = clusterer.cluster_centers_    # Draw white circles at cluster centers    ax2.scatter(centers[:, 0], centers[:, 1], marker='o',                c="white", alpha=1, s=200, edgecolor='k')    for i, c in enumerate(centers):        ax2.scatter(c[0], c[1], marker='$%d$' % i, alpha=1,                    s=50, edgecolor='k')    ax2.set_title("The visualization of the clustered data.")    ax2.set_xlabel("Feature space for the 1st feature")    ax2.set_ylabel("Feature space for the 2nd feature")    plt.suptitle(("Silhouette analysis for KMeans clustering on sample data "                  "with n_clusters = %d" % n_clusters),                 fontsize=14, fontweight='bold')plt.show()
We observe that the average/mean Silhouette Score is the highest in the case of K=4 clusters.
This verifies that the Silhouette Score is a good measure of K-means fitting goodness.
We also produced these Figures:
The vertical line is the average silhouette score of all the values.
Again, we can also visually verify that the Silhouette Score is a good measure of K-means fitting goodness for the specific example.
K-means is one of the most widely used unsupervised clustering methods. The algorithm clusters the data at hand by trying to separate samples into K groups of equal variance, minimizing a criterion known as the inertia or within-cluster sum-of-squares. This algorithm requires the number of clusters to be specified.
However, if the ground truth labels are not known, the evaluation must be performed using the model itself.
In this article we used the Silhouette Coefficient. From the working example, we can verify that the Silhouette Score is a good measure of K-means fitting goodness.
That’s all! Hope you liked this new article — more to come soon!
[1] Peter J. Rousseeuw (1987). “Silhouettes: a Graphical Aid to the Interpretation and Validation of Cluster Analysis”. Computational and Applied Mathematics 20: 53–65. doi:10.1016/0377–0427(87)90125–7.
[2] https://en.wikipedia.org/wiki/Silhouette_(clustering)
towardsdatascience.com
towardsdatascience.com
If you liked and found this article useful, follow me to be able to see all my new posts.
Questions? Post them as a comment and I will reply as soon as possible.