K-Means Clustering with Scikit-learn
By: Rukshan Pramoditha
Tags: Machine Learning, K Means Clustering, Data Science, Unsupervised Learning, Scikit Learn

You’re all welcome to another exciting ML topic — K-Means Clustering. To implement the algorithm to a real-world data set, I’ll use the Scikit-learn machine learning library in Python.

What is K-Means Clustering?

Clustering is the task of partitioning a dataset into groups, called Clusters. The objective of clustering is to identify distinct groups in the dataset such that the observations within a group are similar to each other but different from observations in other groups. Clustering is often used to find patterns in unlabeled data which has no label.

K-Means Algorithm is one of the simplest and most commonly used clustering algorithms. In k-means clustering, the algorithm attempts to group observations into k groups, with each group having roughly equal variance. The number of groups, k, is specified by the user as a hyperparameter.

K-means clustering is a type of unsupervised machine learning algorithm. To learn more about supervised vs unsupervised learning, you can read my Getting Started with Machine Learning (ML) article. There, you will find the difference between those two learning methods.

In scikit-learn, k-means clustering is implemented using the KMeans() class. When using this class, the user must specify the value of the hyperparameter k by setting the n_clusters parameter to an integer greater than 1 since the algorithm does not learn k by itself.

The Goals of K-Means Clustering

We implement k-means clustering to an unlabelled dataset to achieve the following goals.

K number of centroids representing the centre of each cluster

number of centroids representing the centre of each cluster A number to each data point, indicating which cluster a particular data point belongs to

Assumptions in K-Means Clustering

It is important to note the following three assumptions in k-means clustering.

First, k-means clustering assumes the clusters are convex shaped (e.g., a circle, a sphere).

Second, all features are equally scaled.

Third, the groups are balanced (i.e., have roughly the same number of observations).

Note: When we are applying k-means to real-world data, we want to make sure that the features are measured on the same scale. If it is not, you should standardize the features to meet the second assumption. So, you can apply z-score standardization or min-max scaling by using Scikit-learn StandardScaler() class or MinMaxScaler() class respectively.

How the K-Means Algorithm Works

The k-means algorithm can be summarized in the following five steps:

Randomly pick K (predefined) number of centroids (cluster centres) from the data points as initial cluster centres. For each data point in the dataset, calculate the Euclidean distance between itself and each of the K centroids. Assign each data point to the closest cluster centre according to the distance measured in step 2. [After doing step 1, 2 and 3, the first iteration is completed and the points would be clustered.] Now take the average of all of the points in each cluster and reposition the centroids using the newly calculated average. Measure the distance between each of the old centroids and the new centroids.

[If the distance is 0, that means that the centroids did not change their positions and hence the centroids are found. Otherwise, you repeat the steps 2, 3, 4 and 5 with the new centroids until all the centroids do not change position anymore (i.e. distance is 0).]

The Euclidean distance between two data points (x and y) in an n-dimensional space is defined by the following formula. We use that distance to measure the similarity between data points in the K-Means Algorithm.

Image by author

Based on this Euclidean distance metric, we can describe the k-means algorithm as a simple optimization problem, an iterative approach for minimizing the within-cluster Sum of Squared Errors (SSE), which is sometimes also called cluster inertia defined by the following formula.

Formula by saedsayad

Using Scikit-learn for K-Means Clustering

Now let’s work on an example to see how k-means clustering works and how to implement it using Scikit-learn. For this example, we have a dataset named iris which is a built-in dataset in Scikit-learn. The dataset contains four columns — sepal length, sepal width, petal length, and petal width measured in cm.

This dataset is often used for multi-class classification, but it can also be used for k-means clustering when we ignore the target array (label) column of the dataset. The following steps describe the process of implementing k-means clustering to the dataset with Scikit-learn.

Step 1: Import libraries and set plot styles

As the first step, we import various Python libraries which are useful for our data analysis, data visualization, calculation and model building tasks. When importing these libraries, we use community-accepted standard conventions.

Note that we use ggplot as the plot style for our plots. It is a good-looking plot style adapted from R ggplot2 data visualization library based on the Grammar of Graphics. matplotlib inline with % notation is a magic command which is used to display the plot directly below the code cell in the Jupyter notebook. Also, note that we haven’t imported any Scikit-learn class or function yet. We import them one by one when we need to use them. This is because Scikit-learn consists of a large set of functions and classes and here, we do not need all of them. We need only a few of them.

Step 2: Get and prepare data

The dataset that we use here is available in Scikit-learn. But it is not in the correct format that we want. So, we have to do some manipulations to get the dataset ready for our task. Let’s load the dataset using Scikit-learn load_iris() function.

The load_iris() function returns a dictionary-like object called a bunch which contains two arrays of features and target values. This is stored in the variable iris. It also contains some metadata such as feature names, target names and other important information. If you want to see all of them, just type iris and execute!

Now, we want to convert this data into a pandas DataFrame which is the format we are familiar with. The data attribute of iris returns the feature matrix. The target attribute of iris returns the target vector. The feature_names attribute of iris returns the feature names. We can use all of them with pandas DataFrame() class constructor to convert the iris object into a pandas DataFrame. Note that we do not need the target vector (label column) and we do not include it as a column of our dataset. This is because k-means clustering is an unsupervised machine learning algorithm and we do not consider the target vector (label column).

Now, df contains a pandas DataFrame. We can see its first 5 rows by calling the head() method.

If you want to see the shape of the dates, you can use the shape attribute.

You can see that there are 150 observations (rows) and 4 features (columns) in the dataset.

The next step is to check the data type of each variable. For this, we can use the DataFrame info() method.

The dataset contains all numerical values.

Next, we need to check if there are any missing values. For this, we can use the DataFrame isnull() method combined with sum().

The dataset is good, as it does not have any missing values.

Step 3: Visualize data

Now we create a scatterplot between sepal length and petal length using seaborn scatterplot() function. For the purpose of visualization, we use only two features in the dataset for our k-means clustering although k-means clustering can be applied to data in higher dimensions (more than two features).

The output is:

Now, we want to group these data points based on their feature similarities. This can be achieved using the k-means algorithm.

Step 4: Build the model

4.a: Obtain the feature matrix

The feature matrix contains the values of sepal length and petal length. It is a two-dimensional Numpy array. It is stored in the X variable.

4.b: Standardize the features if necessary

One of the assumptions in k-means clustering is that all features are equally scaled. You can see that the two features that we are interested in are equally scaled and we do not need to apply z-score standardization or min-max scaling anymore.

4.c: Choose the right estimator

In Scikit-learn, an estimator refers to a machine learning model. Every estimator is represented by a Python class. Since our interest is to do k-means clustering, we can import the KMeans() class which is in the cluster sub-module in Scikit-learn.

If you want to get help on the KMeans() class, you can execute help(KMeans).

4.d: Create and train (fit) the model

Now, you have chosen the right estimator. To do k-means clustering, all you need to do is to create an instance of KMeans() class use X to train the model using the fit() method of that class.

It is important to note that the general KMeans() class is not the same as an instance of KMeans() class. This is because there are still some options open to us for changing the default values of KMeans() class parameters.

When using the KMeans() class, the most important hyperparameter that we need to consider is the value of k (number of clusters) which is represented by the n_clusters parameter in the KMeans() class. The algorithm does not learn k by itself. If we set n_clusters=10, the algorithm happily returns 10 clusters and if n_clusters=25, the algorithm happily returns 25 clusters and so on. So, the data scientist’s responsibility is to choose the optimal number of clusters (k).

There are some standard methods to determine the optimal number of clusters k. Later in this tutorial, I will discuss those methods. For now, I set k=3 for our case. After discussing some standard methods to determine the optimal number of clusters k, I’ll verify that k=3 is the correct choice for our example.

Now, I create the model using the KMeans() class.

Now, the variable model refers to an instance of the KMeans() class with the above-specified parameters. Since k=3, n_clusters=3. The random_state parameter is also important. It determines the random number generation for centroid initialization. We can use an integer to make the randomness deterministic for each execution.

Now it is time to apply our model to data. This can be done with the fit() method of the model.

4.e: Get cluster centres (centroids)

We can use the cluster_centers_ attribute of the KMeans() class to get the coordinates of cluster centres.

This returns a two-dimensional Numpy array which consists of coordinates of centres for the three clusters. For example, 5.00784314 and 1.49215686 are the coordinates for the centre of the first cluster.

4.f: Assign each data point to cluster index (label)

For this, we can use the labels_ attribute of the KMeans() class or the predict() method of the KMeans() class.

From the output, we can see that the 1st data point belongs to cluster 0, the 150th data point belongs to cluster 2 and so on.

4.g: Visulaize the clusters and centroids

This is the fun part 😊😊😊. The following scatterplot shows the three clusters and their centroids marked by red stars.

The output is:

The clusters are well balanced having roughly the same number of observations. The clusters are convex shaped. All these things verify the assumptions of k-means clustering that we made earlier.

Finding the Optimal Number of Clusters

One of the main challenges in unsupervised learning is to evaluate model performance. An inappropriate choice for k in k-means clustering can result in poor clustering performance. There are some standard methods to determine the optimal number of clusters k. These methods include both graphical and numeric techniques. Let’s discuss them now.

The Elbow Method

Before discussing the elbow method, it is very important to define the term cluster inertia and how it measures the performance of each cluster.

Cluster inertia (distortion): This is defined as the within-cluster Sum of Squared Error (SSE) for each cluster. Therefore the smaller the inertia, the denser the cluster (closer together all the points are). When we are using Scikit-learn, this can be accessed via the inertia_ attribute after fitting a KMeans model. The inertia of our model when k=3 is:

Based on the cluster inertia, we can create a graphical tool, called the elbow plot, to estimate the optimal number of clusters k for a given task.

The idea behind the elbow method is to identify the value of k where the inertia (distortion) begins to decrease most slowly as we keep increasing k. This is the point where the elbow occurs (hence the name, elbow plot). The following lines of code create an elbow plot for the values of k from 2 to 10.

The output is:

As you can see, the inertia drops very quickly as we increase k up to 4, but then it decreases much more slowly as we keep increasing k. This curve has roughly the shape of an arm, and there is an “elbow” at k = 4. From this plot, we can say that the optimal number of clusters is 4. The elbow method is not a good method to find the optimal number of clusters because the distortion always keeps getting lower as we increase k. There are other precise methods to find the optimal K — silhouette score, silhouette plot.

Note 1: When we create the above elbow plot, we ran K-Means Algorithm 9 times by automating with a for loop. With modern computational techniques, running a machine learning algorithm several times is just a matter of seconds. We can automate the task using a for loop.

Note 2: Another way to create the elbow plot is to use the Yellowbrick machine learning visualization library. Using the KElbowVisualizer (an object that learns from data to produce a visualization), we can create an even more informative elbow plot with a just few lines of codes.

The output is:

The KElbowVisualizer automatically does things for us. It locates the elbow at k=4 and gives the corresponding distortion score of 34.317. That’s great!👍👍👍

The Silhouette Score

A more precise approach to find the optimal number of clusters is to use the Silhouette Score which is the mean of the Silhouette Coefficients over all the instances.

The silhouette coefficient is a measure of the quality of clustering that you have achieved. It measures cluster cohesion which is the space between clusters and cluster separation. An instance’s silhouette coefficient is equal to:

Image by author

Where,

S(i): The silhouette coefficient of i th instance.

The silhouette coefficient of th instance. a(i): The mean distance of a point i to all other points in the same cluster; if a(i) is small, cluster cohesion is good, as all of the points are close together.

The mean distance of a point to all other points in the same cluster; if is small, cluster cohesion is good, as all of the points are close together. b(i): The mean distance of a point i to the instances of the closest cluster; if b(i) is large, cluster separation is good, as the nearest cluster is far apart.

The silhouette coefficient can vary between –1 and +1. A coefficient close to +1 means that the instance is well inside its own cluster and far from other clusters, while a coefficient close to 0 means that it is close to a cluster boundary, and finally a coefficient close to –1 means that the instance may have been assigned to the wrong cluster.

If a(i) is small and b(i) is large, the Silhouette Coefficient is high. The Silhouette Score is calculated by getting the mean of the Silhouette Coefficients over all the instances. The value of k that yields the highest Silhouette Score is known as the optimal k.

Calculating the silhouette coefficient of an instance

Let’s walk through an example of how to calculate the Silhouette Coefficient of a point. Consider the seven points and the clusters (K=3) to which they belong, as shown in the following figure.

Image by author

Let’s calculate the Silhouette Coefficient of a particular point. Consider the point (10,10) in cluster 0:

Step 1: Calculate a(i)

Now we calculate its average distance to all other points in the same cluster:

Distance from (10, 10) to (12, 12) = 2.828

Distance from (10, 10) to (10, 13) = 3.0

Average: (2.828 + 3.0) / 2 = 2.914

So, a(i) = 2.914

Step 2: Calculate b(i)

Now we calculate its average distance to all other points in cluster 1.

Distance from (10, 10) to (25, 20) = 18.028

Distance from (10, 10) to (26, 21) = 19.416

Average: (18.028 + 19.416) / 2 = 18.722

Now we calculate its average distance to all other points in cluster 2.

Distance from (10, 10) to (25, 5) = 15.811

Distance from (10, 10) to (25, 4) = 16.155

Average: (15.811 + 16.156) / 2 = 15.983

So, b(i) = min(18.722,15.983); b(i) = 15.983

Step 3: Calculate S(i)

Therefore, the Silhouette Coefficient of point (10,10) = (15.983–2.914) / max(2.914, 15.983); S(i) = 0.81768.

This is just for one point in the dataset. To get the Silhouette Score, you need to calculate the Silhouette Coefficients of the other six points in the dataset and take the mean of all 7 Silhouette Coefficients.

Now, we compute silhouette scores for different k (number of clusters) for our data. To compute the silhouette score, we can use Scikit-Learn’s silhouette_score() function, giving it all the instances in the dataset and the labels they were assigned.

By automating the process with a for loop, I have calculated silhouette scores from k=2 to k=10. Let’s plot these values against k.

The output is:

The value of k that yields the highest Silhouette Score is considered as the optimal k. So, the optimal k is 2. [Remember that, from the elbow method, we got 3 as the optimal k.]

The Silhouette Diagram

An even more informative visualization is obtained when you plot every instance’s silhouette coefficient, sorted by the cluster they are assigned to and by the value of the coefficient. This is called a silhouette diagram. Each diagram contains one knife shape per cluster. A knife shape is created by bars which represent all the instances (data points) in the cluster. Therefore, the width of a knife shape represents the number of instances in the cluster. The bar length represents the Silhouette Coefficient for each instance. Bars in a knife shape are sorted in the descending order of the Silhouette Coefficient. The dashed line indicates the silhouette score. Let’s create silhouette diagrams for k=2, k=3 and k=4 using Yellowbrick SilhouetteVisualizer (an object that learns from data to produce a visualization).

When k=2,

The silhouette diagram for k=2 shows that the data is split into two clusters of unequal size. This is because the widths of the two shapes are significantly different. But, most of the points in the two clusters have large silhouette values (0.75 or greater) and extend beyond the dashed line, to the right, indicating that the clusters are well separated. I do not choose k=2 as the optimal number of clusters because of unequal clusters. [Remember the assumption: the clusters are balanced (i.e., have roughly the same number of observations).]

When k=3,

When k=4,

When k=3 and k=4, both diagrams show that the clusters are well separated since most instances extend beyond the dashed line and have a silhouette score closer to 0.6. When k=4, widths of the four shapes are significantly different and clusters are unbalanced.

It seems like a good idea to use k=3 to get clusters of similar sizes, even though the overall silhouette score for k=2 is slightly greater than for k=3. Cluster sizes for different values of k can be verified from the following figure.