Introduction to Unsupervised Learning Using Clustering
By: Ipsita Shee
Tags: Python, Unsupervised Learning, Data Visualization, Clustering, Data Science

Understanding the types of clustering

Photo by Pierre Bamin on Unsplash

What is Clustering?

Clustering literally means grouping. In Machine Learning, clustering involves dividing the data points in a dataset into groups or clusters, based on their similarities and dissimilarities. A cluster consists of similar data points.

Clustering is of two types:

1) Hard Clustering

In hard clustering, the grouping is done in such a manner that a data point can belong to one cluster only. In hard clustering, a data point either completely belongs to a cluster or not at all.

Hard clustering example. (Image source: By Author)

2) Soft Clustering

In soft clustering the grouping style allows a data point to be present in more than one cluster. Here instead of putting a data point in a separate cluster, it is assigned the probability of it being in a certain cluster.

Soft clustering example. (Image source: By Author)

Types of Clustering Algorithm:

1) Connectivity models

These models are based on the conception that the data points which lie closer to each other are similar whiles the ones that are further from each other are not.

There are 2 ways in which such models perform clustering. One method involves forming as many clusters as there are datapoints then grouping these data points into clusters based on how close they lie to each other.

In the second method, the exact opposite happens. All the data points are put in one cluster and then segregated based on their proximity to each other.

Example:- Hierarchical clustering and its variants.

The first type of connectivity model. Each data point is assigned to a cluster and these clusters are grouped according to proximity. (Image Source: By Author)

The second type of connectivity model. The data points are put in one cluster and then separated into different clusters according to their proximity. (Image Source: By Author)

2) Centroid models

These models on based on the conception that data points lying close to the centroid are similar and hence belong in that cluster. K-means clustering, a widely used clustering algorithm is a centroid type model.

3) Distribution model

In this model, data points are clustered based on the distribution they follow. The data points following similar distribution (example:- Gaussian, Normal) are considered similar and for a cluster. These models often suffer from overfitting.

4) Density model

In these models, the data space is searched for a varied density of data points. These dense areas are isolated and the data points around it are assigned to it to form a cluster.

Example:- DBSCAN and OPTICS

Hierarchical Clustering

HIerarchial clustering is a connectivity type model. Just as the name suggests, this algorithm works by building a hierarchy of clusters by grouping them on the basis of similarity of clusters. It is of two types.

Agglomerative hierarchical cluster (bottom-up)

Agglomerate means to collect or form into a mass or group. An agglomerative hierarchical cluster is a bottom-up approach. The steps are as follows:

Here we assign each data point to an individual cluster, which means that initially, the number of clusters formed is equal to the number of data points. Then 2 closest clusters are joined to form one cluster. Now again from the clusters formed in step 2, we find the clusters closest to each other and join them. The third step keeps repeating until we get one single cluster.

For hierarchical clustering, we often use a dendrogram. A dendrogram is a type of tree diagram showing hierarchical clustering. Let us understand this better with the help of the diagrams shown below.

Let us plot some sample data points on a graph.

Sample data points plot (Image Source: By Author)

We can easily see from the diagram above that the sample data points that we have plotted form two distinct clusters. However, in real life, we may come across a data set with a lot of data points spread across multiple dimensions, where spotting the clustering by mere observation will not be possible. For such scenarios, we make use of the dendrogram.

Dendrogram of our sample data points. (Image source: By Author)

From the plot for our sample data points, we can see that the points 10 and 11, 7 and 8, 3 and 4, and 2 and 12 are closest to each other. Hence they form a cluster. We can see from the diagram above that dendrogram has been created by joining points 10 with 11, 7 with 8, 3 with 4, and 2 with 12.

The next cluster is formed by joining the cluster formed by joining two points to the next nearest cluster or point. We can see from the plot of random data points that the clusters with points 10, 11, and 7,8 are close, hence they join to form a single cluster. Similarly, the cluster formed by 3,4 is close to 5, hence they join to form a single cluster. This process continues until all the points are joined together to form one big cluster.

Once this process terminates we have to determine the largest vertical distance that doesn’t intersect any of the other clusters and draw a horizontal line through it. The number of vertical lines this newly drawn horizontal line crosses is the number of clusters.

Determining the number of clusters (Image source: By Author)

For the dendrogram of our sample data points, the horizontal line passes through two vertical lines, hence the number of clusters will be 2, just like we had observed earlier.

Divisive hierarchical clustering (top-down)

This method of clustering works in a totally opposite manner of Agglomerative hierarchical clustering.

K means clustering

This is an iterative algorithm. Each iteration aims to find the local maxima. First, we choose a K value to decide the number of centroids. So if K = 3, it means that we have 3 centroids and hence 3 clusters. After the number of centroids is decided, the data points are randomly assigned to the clusters. Then the centroid for these clusters is computed by the algorithm. Data points are assigned to these centroids based on their distance from the centroids. Once the initial clusters are formed, the centroid for these clusters are re-computed by the algorithm, and the data points are reassigned accordingly. The re-computing of centroid and re-assignment of data point continues until a global optimum is obtained.

Once there is no further re-assignment of data points between the clusters for two successive repeats, it will mark the termination of the algorithm if not specifically mentioned.

Before You Go

Thanks for reading! If you want to get in touch with me, feel free to reach me on ipsitashee4@gmail.com or my LinkedIn Profile.