Hierarchical clustering with a work-out example
By: Hussain Safwan
Tags: Machine Learning, Clustering, Hierarchical Clustering, Artificial Intelligence

Say you’ve got a bunch of points (or objects, suppose, with reasonable similarity among each) and now you want to arrange them into nice little buckets, but the problem is you don’t know how many different sets to partition them into. So you can’t use k means or any related ones (obviously because the value of k is not known or cant feasibly be guessed). What can you do now? We go HIERARCHICAL then!

Okay, funs aside😉. The idea of hierarchical clustering is to build clusters that have predominant ordering from top to bottom (head on to this site, quite awesome explanation). In this episode, we have a look at the ground on which hierarchical clustering is built on, it’s classification, algorithm and as always a worked-out example to build firm the concepts.

Please note that this is not introductory writing(if that’s what you’re looking for, then here’s a master article), so I’ll assume you’ve got a rough understanding of hierarchical clustering. The purpose of this blogpost is to show the internal mechanisms of the algorithm which I aim to achieve by solving an example.

Why hierarchical clustering

The fact of not having known the value of k isn’t intuitive enough. So I came up with this example. Say, you’re researching the effect of geography on culture and suppose you start with Asia. In Asia, there are countries under the cultural influence of India, which might form a cluster (say the name of the cluster is India), then there are ones under that of China forming another cluster (cluster China), these are level 1 clusters. These two clusters would merge to form a bigger cluster (say Indochina) since there might be similarities between Indian and Chinese cultures, this is a level 2 cluster. Say this level 2 cluster merges with another level 2 cluster (say Persian cluster) to form a level 3 cluster which is Asia itself (oversimplified for mere illustrative purpose). Here at the beginning, each individual country was a cluster by itself, of level 0. What I’m trying to explain is that this technique of clustering is used when along with clustering the relations between the clusters and their components and aggregate are also needed to be known. Hope this insane example makes a point to the readers.

Classification.

Broadly of two types.

Agglomerative: In this class, the individual clusters, in each iterative step, merge into a larger one, until there one big cluster. ONE CLUSTER TO INCLUDE THEM ALL!

Divisive: As the name suggests, it works just the opposite of agglomerative and breaks down the larger cluster until each cluster comprises of a single item. We’re mainly gonna focus on agglomerative clustering, since we believe in building and not breaking, just kiddin’😜.

The algorithm

Described in words, we’re given a distance matrix, to begin with (we’ll see how that looks in a minute) containing the pairwise distance between the points. What we have to do is to pack up the independent points into clusters. We begin with the two points with the minimum distance and include them in a cluster. Then measure reconsider the matrix and pack up the points with current minimum distance, note that each newly formed clusters are treated as ordinary points while determining the distance. Now the big question that arises is how is the distance between a cluster and a point calculated? To put that into perspective, say we’ve got a cluster [A, B] and an independent point C, how’s the distance to be calculated? Well, as it turns out there are quite a few ways to do so.

Min approach: Distance here is the minimum of the pairwise distances, like in above example,

d([A, B], C) = min(d(A, C), d(B, C)).

This is also called single linkage.

Max approach: Max of the pairwise distances taken. Average: Average of the pairwise distances taken. Like

d([A, B], C) = (d(A, C) + d(B, C)) / 2

Now, these are by no means the only metrics used, there are quite a few out there like weighted/unweighted average, Ward’s distance, and others. Okay, I can hear you asking which one to use. That depends upon the dataset you’re working with, how scattered the data is, what you wan to make out of this and some other factors. According to data scientists, it might be a wise choice to try them all out and then decide where the best of results come from. As of now, I’d be using the single linkage metric for the rest of the article. Semi-formal pseudocode might look something like this,