Unsupervised Learning: Clustering Algorithms
By: Max Miller
Tags: Clustering, Machine Learning, K Means, Data Science Ground Up, Unsupervised Learning

Data Science from the Ground Up

Unsupervised Learning: Clustering Algorithms

Predictive models generally require what is called ‘labeled’ data to train on- that is that the data has some target variable that you have filled out already. Of course, the goal is to use the model on unseen data where you don’t know the value of the target variable, but without properly labeled training data, you have no way of validating your model. For this reason, data production is often the hardest part of a data science project. Say, for instance, you want to teach a computer to read handwriting. It’s not enough to collect hundreds of pages of written words and scan them in. You also need to label this data, to have an accompanying ‘correct reading’ for each word. With something as complicated as handwriting, you might need hundreds of thousands of training examples and hand labeling that many entries will be laborious.

Perhaps if you encounter a dataset without a labeled target variable there’s a way to glean some insight out of it without going through the hassle of labeling it. Maybe you simply don’t have the time or resources to label the data or maybe there isn’t a clear target variable to predict, is there still a way to use the data? This is the world of ‘unsupervised learning’. One of the more common goals of unsupervised learning is to cluster the data, to find reasonable groupings where the points in each group seem more similar to each other than to those in the other groups. Two of the most common clustering methods are K-means clustering and agglomerative clustering.

K-means clustering

So, you have some data, you don’t have any category labels for it, but you’re pretty sure you can segment the data into sensible groups, you just don’t know what those groups are. Let’s generate some data to experiment with:

Some hypothetical data, generated as three blobs

This data was generated in three clumps, as you can see. For this example, that guarantees that there is a sensible grouping into three clusters, but remember that the point here is to see if you can find that grouping on your own, so let’s remove the color coding:

Our points without the benefit of color coding

Visually you can probably still see the clusters, at least a little bit, but how do we get the computer to discover the clusters on its own? The first strategy we’ll discuss is k-means. In the k-means strategy, you first specify how many clusters you’re looking for, in this case we’ll say 3, and the computer starts by placing that many new points randomly on the graph:

Our data with three new points — our centroids — in red

These new points are called ‘centroids’ — they will be the centers of our clusters. Why place them randomly? Well, there isn’t really a way for the computer to tell ahead of time where would be a sensible place to put the cluster centers, so the strategy is to place them randomly and then step by step move them to improve the clustering. To decide how to move the centroids, you first assign each of your points to the nearest centroid to it:

The points have been assigned to the nearest centroids which are now star shaped

I’ve made the centroids star shaped and then color coded the points according to which centroid is closest. You can see that the red and green centroids are near each other and towards the edge of the data, this doesn’t seem to make a lot of sense — for one thing the red cluster only has two points in it. So we’ll try moving these centroids someplace more central. We move each centroid to the center of the cluster, that is the average location within the cluster. Once we’ve moved the centroids, we re-assign the points to the nearest centroid. Here’s what the new assignments look like one step into the process:

After moving each centroid to the center of its cluster and reassigning the points

The red centroid did not move far, it still only has two points in its cluster. The green centroid, however, took a big step towards the center of the cluster. When it did that, it captured a bunch of points from the blue cluster. As a result, when we repeat this process, the blue centroid will move further into the top corner, leaving the bottom to the green and the red:

One more step into the process

Repeating the process again yields:

We continue this process, moving the centroids step by step and reassigning the points to the associated clusters, until we find a stable set of centers which have ceased to move. In this case, that equilibrium looks like this:

Which isn’t a bad way to segment this data set at all! It may also help to visualize the movement of the centroids themselves:

How each of the centroids moved step by step

In this process, the first fews steps were pretty big as the centroids get pulled towards the center of their clusters, but afterwards the steps they take become smaller and smaller as only a handful of points get reassigned in each step.

In this instance, you’ll notice that the cluster algorithm ends up essentially discovering the groups that were made when I generated the data. Crucially, though, remember that this being an unsupervised learning problem, there’s no way for the algorithm to validate these groups in any way. The groupings were made without any reference to what ‘class’ the points belonged, merely their position, and anyways, the computer wouldn’t be able to tell what these groupings represented. If you wanted an interpretation of the clusters, a person would need to look at the groups and think about them, and there’s no guarantee that a person would be able to immediately see what features typified the clusters that had been created — especially if there were more than two feature variables making the data hard to visualize.

Another challenge is that the groupings you create depend on where your initial centroids land and also how many there are, and, again, you may not have a way of validating whether the grouping you find is the best possible one or if you even have an appropriate number of clusters! Consider this example, which also has three clusters, but the centroids start in different places:

The red and green clusters end up both migrating towards the blob in the upper right, splitting it up between themselves, leaving the blue centroid all alone with two thirds of the data. The k-means algorithm has no way of knowing whether this set of clusters makes more or less sense than the first one we found. Similarly, we have to specify how many clusters we want to group the data into, we didn’t need to choose 3, we could have chosen any number. Here’s another grouping with 5 clusters instead of 3:

In some cases you might have a specific reason to choose one particular number of groups — perhaps you are dividing customer support cases up among some number of support staff so you just want as many groupings as staff members — but generally one of the challenges of unsupervised clustering is deciding on the number of clusters.

One other thing to bear in mind when setting up a k-means algorithm or an agglomerative one which we’ll discuss shortly is that there are actually a few different ways to calculate the ‘distance’ between two points in abstract space. Maybe the most common is ‘Euclidean distance’ which you’ll remember from the Pythagorean Theorem. There’s also ‘Manhattan Distance’ so-called because it adds up the distance along each axis individually as if you were walking from one intersection in Manhattan to another along the grid of streets at right angles. More generally, Euclidean and Manhattan distances are actually both specific cases of the general metric called Minkowski Distance. If you’re dealing with many-dimensional but sparse data, you might conceivably also use Cosine Similarity as a measure of similarity — while not really a measure of distance per se, Cosine Similarity gets at something similar in a way that can be computationally quicker if you have a great many variables and each row tends to be ‘sparse’ (perhaps you’re looking at customers who each may purchased a handful of products from an online store that has many thousands of different products). Different measures of distance may also yield slightly different results in the final groupings.

Hierarchical Agglomerative Clustering

K-means is sort of a ‘top-down’ approach; it groups everything into clusters at once and then tweaks the clusters over a number of steps. Agglomerative clustering is sort of bottom-up; it starts with no clusters, just all of the individual points, and then slowly groups points together one at a time. The process is simple enough. First you calculate the distance between each point and every other point. Choose the points that are closest to each other and group them together. You continue to repeat the process, calculating the distance between any individual point or cluster to all the other points and clusters, slowly joining points together into clusters and clusters into other clusters until you have only as many clusters left as you set out to find. As an illustration, here is an example of the algorithm at work on a smaller dataset with only 20 points:

Here is how this agglomerative algorithm ends up dividing the first data set (you’ll see a few points floating out by themselves that appear to have been grouped by themselves, but the algorithm has actually settled on three clusters, and those points are actually being grouped with the bigger nearby groups. Their seeming to float by themselves is a quirk of the kernel density estimator I used to automatically draw the enclosures around the clusters):

The original dataset grouped with agglomerative clustering

With an algorithm like this, in addition to deciding on a measure of distance between two points, you actually also need to decide how you measure the distance between two clusters. You can measure the shortest distance between a point in one cluster to a point in the other, sort of like the distance between the inside edges of the two clusters — this is called ‘single linkage’. You could also measure the furthest distance between a point in one to a point in the other — this is called ‘complete linkage’. Or you could take the average distance between any point in one cluster to any point in the other — ‘average linkage’. Choosing a different measure of distance or type of linkage may result in different grouping decisions.

One benefit of this style of clustering is that it is stable. It doesn’t depend on a random seed like the k-means method. Once you decide on a measure of distance and a number of clusters, it will always end up with the same clusters. One downside is that it can be computationally expensive. You need to measure the distance between every two set of points (for n points the formula for the number of such links is n(n-1)/2) and that’s just in order to decide on the first linkage you make!

Deciding on a number of clusters

I mentioned earlier that deciding on a number of clusters was one of the challenges of these clustering algorithms. So, how would you go about actually choosing the number of clusters? There is no hard and fast rule, but there are a couple of measures you can use to guide your choices. Metrics to judge clusters tend to focus on how ‘tight’ the clusters are, that is, how similar the points within them are. One simple metric might be the Within cluster Sum of Square (WSS) which will be larger if the points within the group are more spread out and smaller if the points are closer together. The challenge with using this method on its own is that as you add more clusters the WSS will always go down. You could minimize WSS by having as many clusters as points, then the WSS would be 0!

How do you know when to stop adding clusters? The goal is to find a number of clusters where adding one more cluster no longer has a really significant benefit to the WSS measure. If the clusters are well defined, it will generally be pretty obvious when this is the case. Consider this dataset, generated to have 4 reasonably distinct blobs:

Now I’ll use k-means to group these in to clusters numerous times, each time increasing the number of clusters. I’ll record the WSS value for each of these and then I plot it against the number of clusters to get this graph:

You’ll notice that my WSS measure drops a lot when I increase the number of clusters from 2 to 3, and again from 3 to 4, but not very much for each subsequent increase. Once I hit 4 clusters, the algorithm generally finds the reasonable groupings. When I add more clusters above 4, the algorithm starts to split up the natural clusters and the marginal benefit for my compactness measure isn’t as large.

This sort of method to determine the appropriate number of clusters is sometimes called an ‘elbow method’; you’re looking for the crook or elbow in the graph where increasing k no longer yields big steps in terms of compactness. This is a sensible place to start, but no way of choosing a number of clusters is infallible. This method may be tricked by clusters that overlap, are a little fuzzy or are very different sizes.

Choosing a clustering method

While these different algorithms are generally getting at the same sort of thing — namely, tightly grouped clusters — they go about the task in different ways and so respond to different shapes and distributions in different ways. As a maybe extreme example, consider these points:

Perhaps I’ve given the game away a little bit by color coding them, but I think many people would agree that there do seem to be two distinct classes here: an inner ring and an outer one. How would the k-means algorithm handle these points? Well, the results are not great:

k-means can’t properly distinguish nested rings

This example may strike you as a little contrived, perhaps even unfair. Can a simple algorithm possibly pick out the nested shapes? The answer, though, is actually yes; in this case the agglomerative method using single linkage to find the distance between clusters actually does identify the nested rings:

Single linkage agglomerative clustering succeeds!

There is, unfortunately, no master clustering algorithm that works equally well for all shapes, but after working and playing with clustering methods for a while, hopefully you’ll develop an intuition for which situations are better served by one algorithm or one measure of distance.