Build Your Own K-Means Clustering Algorithm
By: Saurav Suresh
Tags: Machine Learning, K Means, Algorithms, Unsupervised Learning, Artificial Intelligence

Build Your Own K-Means Clustering Algorithm

Humans have an innate desire to create; to mold their surroundings as they will. This desire can manifest in a child turning a pile of multicolored legos into a spaceship or a woodworker turning a pile of wood into a canoe. Such acts of creating from scratch can lead to a sense of mastery and a deep understanding of the final creation. This experience is not confined to only woodworkers and children, rather the computer scientist is situated perfectly for the task of creating from scratch. Such is my justification for momentarily ignoring the plethora of machine learning libraries, which can accomplish K-Means clustering in a few lines of code, in favor of creating a K-Means clustering algorithm from the ground-up.

At its core, a K-Means clustering algorithm must be able to determine clusters in a dataset. While locating clusters is trivial for the human brain, it becomes a more involved process when we must code it. To enable a computer to locate cluster we must use something called a centroid.

The Centroid

The centroid is a point that is located at the center of a cluster of points. The above image shows three clusters, each with a centroid point that is situated more or less at the center of its cluster. Once each centroid is appropriately located, then each point can be assigned to the centroid it is closest to. As a result, each centroid will be assigned all the points in the cluster it is at the center of. So now our larger problem of finding clusters can be turned into finding where each cluster’s centroid is located.

Tweaking The Centroids

Suppose we start our centroid points at random positions as shown in the image below(the three black points not at the center of a cluster).

If we pretend these three points are our centroids, we get the following clusters(this is an estimate).

These are obviously not accurate clusters so we need to change our centroids to become closer to its points. But the question remains: how do we change the centroids’ positions? Say we get the average of all the distances between a centroid and its assigned data points. If we change the centroid to minimize this average, then we will ultimately get accurate centroids for our data. Now we have reached a common problem in data science: minimizing a function. This can be accomplished with gradient-descent as I discuss below for the case of one centroid.

Let us restrict ourselves to one centroid and its collection of points. We will also stay in two dimensions for now, but this notation can be expanded to n-dimensions.

Let the centroid be a point C

Let the data points be defined as a set of P

Where any point n can be denoted as follows:

Let some function S, be the distance between a point n and the centroid

Let some function L, be the average of all the distances between the points and the centroid

Our objective is to find in what way can we change the centroid in order to decrease the function, L. To accomplish this we can use gradient-descent.

The gradient of the function, L, will be a vector that points in the direction of the steepest ascent of the function. So we need only take the antiparallel vector and change the coordinates of the centroid accordingly.

Simply change each centroid coordinate as dictated above. You may have noticed an extra symbol in the above equations, eta. The eta represents some step size which will dictate how much we move the centroid in the direction given by the gradient.

Now we have a way to position centroids at the center of clusters. Once the centroids are randomly positioned, we will use the above equation to change the centroids’ positions. After a certain amount of tweaking, all the centroids will be correctly located at their respective clusters.

Wrapping Up

Finally, we have the conceptual framework of a gradient-descent K-Means clustering algorithm. All that is left to do is coding the algorithm. This may seem like a daunting task but we have already covered all the difficult parts. The code only has to initialize random centroids, assign each data point to the centroid it is closest to, change each centroid to make it closer to its points, then repeat this process (except randomly initializing the centroids) until the centroids stop changing significantly.

Why Go Through All This Work?

It may seem that all this work for just one algorithm is not worth it when there are libraries that accomplish the same thing and are easier to implement. But the struggles and obstacles one must overcome when building from the ground up, develop a deeper understanding and a greater mastery over the craft. It is like deriving a formula by yourself and using it to solve a problem instead of just looking up the formula or memorizing it. Obviously building these algorithms from the ground up is not realistic in all situations, but doing so as a personal pursuit can greatly bolster one’s understanding of machine learning.

Help for Implementation

If you need some insight into how to implement the algorithm, you can use my implementation for guidance.