Hierarchical Clustering Explained
By: Mazen Ahmed
Tags: Machine Learning, Hierarchical Clustering, Clustering, Data Science, Statistics

Hierarchical Clustering Explained

Unsupervised Algorithms | Data Series | Episode 8.3

In the previous episode we have taken a look at the popular clustering technique called K-means clustering. In this episode we will take a look at another widely used clustering technique called Hierarchical clustering.

Please consider watching this video if any section of this article is unclear:

Video Link

What is Hierarchical clustering?

Hierarchical clustering is an unsupervised machine learning algorithm where its job is to find clusters within data. We can then use these clusters identified by the algorithm to make predictions for which group or cluster a new observation belongs to.

Overview

Similar to K-means clustering, Hierarchical clustering takes data and finds clusters:

What differs, however, is the algorithm to identify clusters. We will discuss at the end the relative advantages and disadvantages of Hierarchical clustering compared to K-means clustering.

The Algorithm

Step 1:

Treat each data point as a cluster. Calculate the Euclidian distance each cluster is away from each other:

Step 2:

Using the distance matrix identify the clusters closest to each other:

Step 3:

Link these clusters together to form a new cluster:

Step 4:

Calculate the distance each cluster’s mean point is away from each other:

Step 5:

Repeat steps 2 to 4 until a single cluster is formed.

One cluster has been formed so we stop.

Step 6:

Cut our dendrogram at a chosen point to give the clusters identified by our algorithm at that point. The point we choose to cut is usually done visually.

and we are done!

Linkage Methods

Note that in step 4 we calculated the distance each cluster is away from each other (known as dissimilarity) based on the centroid or mean point of each cluster. We then link the clusters with the smallest of such dissimilarity. This is known as Centroid Linkage:

There are however other methods to link clusters:

Ward’s method shares the same objective function as K-means clustering discussed in the previous episode.

Considerations of Hierarchical clustering

Advantages

Do not have to manually select number of clusters K.

Easy to implement.

Dendrogram can give useful information.

No need for many random centroid initializations as with K-means clustering.

Disadvantages

With large datasets it is difficult to determine the number of suitable clusters from the dendrogram.

Computationally expensive, slower than K-mean clustering.

Sensitive to outliers.

In the next episode we will be implementing Hierarchical clustering on a real-life dataset using Python.

Summary

Hierarchical clustering is an unsupervised machine learning algorithm that is used to cluster data into groups

machine learning algorithm that is used to cluster data into groups The algorithm works by linking clusters, using a certain linkage method (mean, complete, single, ward’s method etc.) to form new clusters.

The above process produces a dendrogram where we can see the linkages of each cluster.

We can cut our dendrogram at a certain point to obtain suitable clusters from our data.

Prev Episode _______ Next Episode