K-Means Clustering for Beginners. An in-depth explanation and... | by Aden Haussmann | Towards Data Science

K-Means clustering was one of the first algorithms I learned when I was getting into Machine Learning, right after Linear and Polynomial Regression.
But K-Means diverges fundamentally from the the latter two. Regression analysis is a supervised ML algorithm, whereas K-Means is unsupervised.
What does this mean?
Supervised and unsupervised learning are two broad categories for Machine Learning algorithms:
You feed the program labelled data. In other words, you train the algorithm on data where the correct answers are provided, then apply the learned rules to new data to predict their answers.
This is useful for regression and classification.
You don’t supply labels for the data, and it is up to the program to discover them.
This is useful for clustering and discovering hidden patterns in data.
If you are a beginner, I do recommend you read these articles on Linear and Polynomial Regression first, which I’ve linked below. In them, I cover some foundational Machine Learning knowledge and terminology, upon which I will be building throughout this text.
towardsdatascience.com
towardsdatascience.com
Imagine a dataset with a large number of data points. Our goal is to assign each point to a cluster, or group. To do this, we need to find out where the clusters are, and which points should belong to each one. In our example, we will have two variables: the birth rate and life expectancy of different countries. We can create a scatter plot of this data to visualise our groups.
Why would we want to perform K-Means clustering on this data?
Here’s a practical example: let’s say that the UN wants to group countries into three categories based on these two metrics, so that they can deliver proportionate aid packages depending on respective need.
Instead of eyeballing it, we can use K-Means to automate this process (where K represents the number of clusters we want to create, and Mean represents the average).
There are two key assumptions behind K-means:
The centre of each cluster is the mean of all the data points that belong to the cluster.Each data point belongs to the cluster with the nearest centre point.
The centre of each cluster is the mean of all the data points that belong to the cluster.
Each data point belongs to the cluster with the nearest centre point.
These two simple assumptions describe the entire algorithm. All our program will do is iterate over a few steps, each trying to satisfy the above conditions.
Before we continue, we must discuss the concept of distance. We can use a number of distance metrics in ML, such as Manhattan and Chebyshev, but today we will stick to the more familiar Euclidian, which you will likely remember from high school maths.
In two dimensions, the Euclidian distance between two points is
u0016√((xj — xi)2 + (yj — yi)2)
This is an outline of the algorithm:
Initialise a mean for each cluster by randomly picking points from the dataset and using these as starting values for the means.Assign each point to the nearest cluster.Compute the means for each cluster as the mean for all the points that belong to it.Repeat 2 and 3 either a pre-specified number of times, or until convergence.
Initialise a mean for each cluster by randomly picking points from the dataset and using these as starting values for the means.
Assign each point to the nearest cluster.
Compute the means for each cluster as the mean for all the points that belong to it.
Repeat 2 and 3 either a pre-specified number of times, or until convergence.
As usual, we begin with the imports:
Matplotlib (pyplot & rcParams) — To create our data visualisationScikit-Learn (pairwise_distances_argmin) — To perform Machine LearningNumPy — To do scientific computingcsv — To read csv filescollections (Counter and defaultdict) — For counting
Matplotlib (pyplot & rcParams) — To create our data visualisation
Scikit-Learn (pairwise_distances_argmin) — To perform Machine Learning
NumPy — To do scientific computing
csv — To read csv files
collections (Counter and defaultdict) — For counting
import matplotlib.pyplot as pltimport numpy as npimport csvfrom sklearn.metrics import pairwise_distances_argminfrom collections import Counter, defaultdict
I’ve got a csv file containing the data that looks like this:
Countries,BirthRate(Per1000 - 2008),LifeExpectancy(2008)Afghanistan,46.613,47.532Albania,14.69,76.492Algeria,20.804,72.44...
To wrangle this data, we’re going to need some variables to hold our x values, y values, labels and country names, in the case of this csv file. We can extract all of this information from the file in a single function:
x, y, x_label, y_label, countries = read_csv()
My definition for the read_csv() function is below. Of course, you should adjust it to suit your csv file.
def read_csv():    x = []    y = []    countries = []    x_label = ""    y_label = ""    with open('dataBoth.csv') as csvfile:        reader = csv.reader(csvfile, delimiter=',')        lines = 0        for row in reader:            if lines >= 1:                print(', '.join(row))                x.append(float(row[1]))                y.append(float(row[2]))                countries.append(row[0])                lines += 1            else:                x_label = row[1]                y_label = row[2]                print(', '.join(row))                lines += 1    return x, y, x_label, y_label, countries
So now that we have our data processed, we need to combine x and y into a 2D list of (x, y) pairs, which we can do like this:
X = np.vstack((x, y)).T
Now we’ve got a 2D list (actually a numpy array) that looks like this:
[[46.613, 47.532] [14.69, 76.492] [20.804, 72.44] ...]
Let’s draw a scatter plot of these data points to see what we’re dealing with here:
plt.xlabel(x_label)plt.ylabel(y_label)plt.scatter(x, y, color='#76c2b4')plt.show()
As you can see, a negative correlation between birth rate and life expectancy is evident. However, 3 distinct groups do not immediately appear. This is where K-Means comes in.
We’ll write the function that will iteratively create our clusters next. A line-by-line explanation of the function is included in the code snippet below as comments, but I will also provide some clarification here.
We can choose random centres using np.random.RandomState(rseed), calling .permutation on that to find an i, and then choosing the ith element of X, which is our 2D list of (x, y) pairs, remember. We have a loop, in which we use pairwise_distance_argmin to calculate the distances between points and centres, then find new centres from the means of the points, then check if the clusters have converged (if no new centre can be chosen such that the means become smaller). This loop terminates when the clusters have converged:
def find_clusters(X, n_clusters, rseed=2):    # 1. Randomly choose clusters    rng = np.random.RandomState(rseed)    i = rng.permutation(X.shape[0])[:n_clusters]    centers = X[i]    # The main loop    # This loop continues until convergence.    # You could make it run a set number of times by changing    # it to say while x > 5, for example, and removing the break    print("\nConverging centres:")    while True:        # 2a. Assign labels based on closest center        # I am using the pairwise_distances_argmin method to        # calculate distances between points to centres        labels = pairwise_distances_argmin(X, centers)        # 2b. Find new centers from means of points        new_centers = np.array([X[labels == i].mean(0) for i in        range(n_clusters)])        # 2c. Check for convergence        if np.all(centers == new_centers):            break        centers = new_centers        # Print converging centres        print(centers)        print()    return centers, labels
Let’s set our number of clusters as 3:
clust_num = 3
All that’s left to do is visualise the result of our K-Means algorithm applied to the data:
centers, labels = find_clusters(X, clust_num)plt.scatter(X[:, 0], X[:, 1], c=labels, s=50, cmap='viridis')plt.title('K-Means clustering of countries by birth rate vs life expectancy')plt.xlabel(x_label)plt.ylabel(y_label)plt.show()
That’s the algorithm successfully implemented!
But what if we want more information?
After all, the UN would want to see the names of the countries in each cluster.
We can extract all this information, and print it to the terminal:
print("\nNumber of countries in each cluster:")print(Counter(labels))# Get cluster indicesclusters_indices = defaultdict(list)for index, c in enumerate(labels):    clusters_indices[c].append(index)# Print countries in each cluster and meansx = 0while x < clust_num:    print("\nCluster " + str(x + 1))    print("----------")    for i in clusters_indices[x]:        print(countries[i])    print("----------")    print("Mean birth rate:")    print(centers[x][0])    print("Mean life expectancy:")    print(centers[x][1])    x+=1
This will print the countries in each cluster as well as that cluster’s mean birth rate and life expectancy.
It should at least be clear that K-Means clustering is a very useful algorithm with many real-world applications. Hopefully, you’ve learnt enough to perform your own implementation on some interesting data and discover some hidden clusters.
A summary of the content covered:
A brief comparison of supervised vs. unsupervised Machine Learning.An example of the technique’s application.An outline of the algorithm.An example of implementation.
A brief comparison of supervised vs. unsupervised Machine Learning.
An example of the technique’s application.
An outline of the algorithm.
An example of implementation.
If you found the article helpful, I’d love to engage with you! Follow me on Instagram for more Machine Learning and Software Engineering content.
Happy coding!
Skikit Learn sklearn.cluster.KMeans https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html
Python Documentation collections https://docs.python.org/2/library/collections.html
SciPy numpy.random.RandomState https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.RandomState.html
Scikit Learn sklearn.metrics.pairwise_distance_argmin https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise_distances_argmin.html
Numpy.org numpy.vstack https://numpy.org/doc/stable/reference/generated/numpy.vstack.html