Everything on Hierarchical Clustering
By: Renu Khandelwal
Tags: Clustering, Machine Learning, Hierarchical Clustering, Unsupervised Clustering, Unsupervised Learning

Everything on Hierarchical Clustering

An unsupervised clustering algorithm to hierarchically cluster data sharing common characteristics into distinct groups Renu Khandelwal Follow Jun 26 · 6 min read

In this article, you will learn.

What is Hierarchical clustering, and where is it used?

Two different types of Hierarchical clustering -Agglomerative and Divisive Clustering

How does the Hierarchical Clustering algorithm work with an understanding of different linkages and metrics?

What is a dendrogram?

Finding the optimal number of clusters from a dendrogram

Implement Hierarchical clustering using python

Clustering is the most common form of unsupervised learning on unlabeled data to clusters objects with common characteristics into discrete clusters based on a distance measure.

Common Clustering Algorithms are

Centroid based clustering like KMeans which is efficient but sensitive to initial conditions and outliers

like KMeans which is efficient but sensitive to initial conditions and outliers Density-based clustering like DBSCAN clusters data into a high-density area separated by a low-density area.

like DBSCAN clusters data into a high-density area separated by a low-density area. Distribution based clustering like Gaussian Mixture model using Expectation-Maximization(EM), which is a generative probabilistic model that attempts to find Gaussian probability distributions that best model the dataset

like Gaussian Mixture model using Expectation-Maximization(EM), which is a generative probabilistic model that attempts to find Gaussian probability distributions that best model the dataset Hierarchical clustering

Hierarchical clustering builds a hierarchy of clusters without prespecifying the number of clusters based on similarity score

Hierarchical clustering is useful for

Customer Segmentation where customers can be segmented using demographic or income or purchasing patterns

where customers can be segmented using demographic or income or purchasing patterns Social Network Analysis to understand the dynamics of individuals and groups based on their interests and access to information available.

to understand the dynamics of individuals and groups based on their interests and access to information available. Arranging Genomic Data into meaningful biological structures based on a common characteristic

into meaningful biological structures based on a common characteristic City Planning Clustering ensures a commercial zone is not within an industrial zone, or a residential area is not within an industrial zone.

Hierarchical Clustering is either bottom-up, referred to as Agglomerative clustering, or Divisive, which uses a top-down approach.

Agglomerative clustering

A bottom-up approach where each data point is considered a singleton cluster at the start, clusters are iteratively merged based on similarity until all data points have merged into one cluster.

Agglomerative clustering agglomerates pairs of clusters based on maximum similarity calculated using distance metrics to obtain a new cluster, thus reducing the number of clusters with every iteration. They consider only local patterns and do not account for global distribution.

Divisive clustering

A top-down approach, opposite to Agglomerative cluster's bottom-up approach. The divisive cluster begins with one cluster encompassing all data points from the dataset. It then iteratively splits the cluster using a flat clustering algorithm like KMeans until each data point belongs to a singleton cluster. Divisive cluster produces more accurate hierarchies than bottom-up Agglomerative clusters. Divisive clustering accounts for the global pattern by looking at the complete information present in the dataset.

Deep Dive into Working of Agglomerative Clustering

Step 1: All the data points are assigned as a single point cluster. If there are m observations in the dataset, each point is assigned to a cluster, and we will have m clusters.

Step 2: Find the closets or the most similar pair of clusters to merge them into one cluster. Hierarchical clustering use similarity measure to combine the most similar cluster pair. The similarity between the clusters is measured using either Euclidean distance, Manhattan distance(City Block distance), Minkowski distance, or cosine similarity.

Step 3: After identifying the two closest clusters, use the linkage method to determine how to merge the two clusters. Several linkage methods are single linkage, complete linkage, average linkage, centroid method, or ward’s method.

Different linkage to merge two closest or similar clusters

Repeat step 2 and step 3 till all observations are clustered. into one single cluster

Dendrogram

Hierarchical clustering is typically visualized using a dendrogram. Dendrograms are a tree-like representation of points based on similarity or dissimilarity metrics. A dendrogram has data items of one axis and the distances along the other axis where the distance between two merged points or clusters monotonically increases.

The distance score is calculated between every pair of points

Dendrograms based on different Linkage methods

from scipy.cluster.hierarchy import dendrogram, linkage

import matplotlib.pyplot as plt

%matplotlib inline

plt.figure(figsize=(15,5))

data= [[i] for i in [9, 3, 6, 4, 11]]

linked_s = linkage(data, 'single', metric='euclidean')

plt.subplot(2, 5, 1) dendrogram(linked , labels=data)

plt.ylabel('Distance')

plt.title("single linkage") linked_s = linkage(data, 'complete', metric='euclidean')

plt.subplot(2, 5,2)

dendrogram(linked_s , labels=data)

plt.title("complete linkage") linked_s = linkage(data, 'average', metric='euclidean')

plt.subplot(2, 5,3)

dendrogram(linked_s , labels=data)

plt.title("average linkage") linked_s = linkage(data, 'ward', metric='euclidean')

plt.subplot(2, 5,4)

dendrogram(linked_s , labels=data)

plt.title("ward method") linked_s = linkage(data, 'centroid', metric='euclidean')

plt.subplot(2, 5,5)

dendrogram(linked_s , labels=data)

plt.title("centroid method")

plt.show()

Agglomerative Clustering Implementation

Dataset-Mall Customer Dataset

import pandas as pd

# Read the dataset into a dataframe

dataset = pd.read_csv('Mall_Customers.csv',index_col='CustomerID')

# Drop duplicates

dataset.drop_duplicates(inplace=True)

plt.figure(figsize=(10,5))

# Creating the input variable

X= dataset.iloc[:, [1,2,3]].values

linked_s = linkage(X, 'complete', metric='CityBlock')

dendrogram(linked , labels=X)

plt.ylabel('Distance')

plt.axhline(y=1.5, color='orange')

plt.title("Mall Customer HCA")

plt.tight_layout(pad=3.0)

plt.show()

Finding the optimal number of clusters from a Dendrogram

Find the largest difference of heights in the vertical lines shown by the blue arrow above. The threshold is selected in such a way that it cuts the tallest vertical line.

The number of clusters will be the number of vertical lines intersected by the line drawn using the threshold.

In the above case, the optimal number of clusters is 4

Running Agglomerative Clustering

from sklearn.cluster import AgglomerativeClustering

import seaborn as sns

agg_cluster = AgglomerativeClustering(n_clusters=4).fit_predict(X) #Visualising the clusters

plt.figure(figsize=(15,7))

sns.scatterplot(X[agg_cluster == 0, 2], X[agg_cluster == 0, 1], color = 'yellow', label = 'Cluster 1',s=50)

sns.scatterplot(X[agg_cluster == 1, 2], X[agg_cluster == 1, 1], color = 'blue', label = 'Cluster 2',s=50)

sns.scatterplot(X[agg_cluster == 2, 2], X[agg_cluster == 2, 1], color = 'green', label = 'Cluster 3',s=50)

sns.scatterplot(X[agg_cluster == 3, 2], X[agg_cluster == 3, 1], color = 'grey', label = 'Cluster 4',s=50)

plt.grid(False)

plt.title('Clusters of customers')

plt.xlabel('Annual Income')

plt.ylabel('Spending Score')

plt.legend()

plt.show()

Additional things to know about Hierarchical Clustering

Hierarchical clustering is sensitive to outliers and does not work with missing data

Hierarchical clustering works especially well with smaller data sets. When the clustering is complete, it becomes more computationally expensive as more data points are considered.

Conclusion:

Hierarchical clustering is one of the unsupervised clustering methodologies to clusters objects with common characteristics into discrete clusters based on a distance measure. The hierarchical algorithm builds clusters by merging or splitting them successively and without prespecifying the number of clusters. The similarity score is calculated using different methods like Euclidean distance or a city block distance. When we successively merge the clusters, it is Agglomerative clustering, and when we successively split, the clustering is referred to as Divisive clustering. The hierarchical clustering is visualized using Dendrograms.

References:

Hierarchical agglomerative clustering (stanford.edu)

hierarchical-clustering.pdf (princeton.edu)

https://developers.google.com/machine-learning/clustering