Understanding Clustering
By: Bala Priya C
Tags: Machine Learning, Unsupervised Learning, Women Who Code, Clustering, Python

Understanding Clustering

Introduction to unsupervised learning

Photo by Ian Baldwin on Unsplash

Outline - Introduction to Clustering

- Motivating Example

- Common Clustering Algorithms

- K-Means Clustering

- Hierarchical Clustering

- Density Based Clustering (DBSCAN)

- Understanding K-Means Clustering

- Choosing the optimal value for K

- Limitations of K-Means Clustering

Introduction to Clustering

Clustering is a commonly used unsupervised learning algorithm that aims at dividing data points into groups/clusters based on some similarity measure. Data points in a cluster are more similar to those within the cluster than to data points in other clusters.

Classification is a supervised learning problem where we know the class labels associated with the data points and given a new data point, we check how well the model performs in assigning the data point to the correct class label; Clustering on the other hand is an unsupervised learning problem where the data points are not labelled but we’d like to group together the data points that are similar under a certain similarity measure.

Given a dataset that we do not know anything about, clustering essentially helps in understanding intrinsic grouping among the data points.

Motivating Example

A cake company wants to open a chain of stores across a city and wants to find the optimal store locations to maximize revenue.

What would be the possible challenges?

✅Analyze the areas from where the cake is being ordered frequently.

✅Understand how many stores to open to covers the buyers in the area.

✅Figure out the locations for the stores within all these areas so as to keep the distance between the store and buyers small.

Well, here’s a simple way to solve the above problem; It may not seem very practical but helps understand the intuition behind the approach.

Initialize locations of cake stores to certain number of random locations on the map Assign each buyer to the nearest cake store Update the store location depending on the average number of buyers

Do you already see that this is just one of the many possible ways we could do this? That’s absolutely correct. This is just one of the valid hypotheses.

In clustering, every possible hypothesis results in different but valid clusters.

Customer Segmentation, Social network analysis, Biological data analysis are some applications where clustering is commonly used.

Common Clustering Algorithms

In this section we’ll look at some of the commonly used clustering algorithms.

K-Means Clustering

K-Means clustering aims at partitioning the unlabeled data into a distinct number of groups (K) where the grouping is done based on the similarity measure(distance). The primary idea is that the closer the data points are, the more likely they are to be similar and are therefore more likely to belong to the same cluster. We shall look at K-Means Clustering in greater detail in a subsequent section.

Hierarchical Clustering

Hierarchical clustering is a method of clustering that uses top-down(divisive) or bottom-up(agglomerative) approaches to cluster similar data points together.

Agglomerative Clustering (Bottom-up Approach)

Start by treating each observation as a separate cluster in itself.

Identify two clusters that are the closest to each other Merge the two most similar clusters

Repeat steps (1) and (2) until all the clusters are merged together. Here’s a simple illustration.

Illustrating Agglomerative Clustering (Image Source)

Divisive clustering (Top-down Approach)

In divisive clustering (not very frequently used in practice), we do the following:

Start by englobing all points in a single cluster Split the cluster iteratively into smaller clusters until each cluster has only one data point in it.

A simple illustration is shown below.

Illustrating Divisive Clustering (Image Source)

Density Based Clustering (DBSCAN)

DBSCAN stands for Density Based Spatial Clustering of Applications with Noise. It’s a clustering method that takes two parameters epsilon ( eps ) and MinPoints for cluster analysis.

eps — this essentially quantifies how close two points should be to each other to be considered a part of the cluster.

MinPoints — the minimum number of points to form a dense region. For example, if we set the minPoints parameter as 3, then we need at least 3 points to form a dense region.

These parameters should be chosen to be in their optimal value range.

Understanding K-Means Clustering

K-Means Clustering is a simple clustering algorithm that groups data points in an unlabeled dataset into a certain number (K) of coherent subsets called clusters. It finds K centers called cluster centroids, one for each cluster and hence the name K-Means Clustering. The algorithm finds the closest cluster to which a data point belongs based on distance metric such as Euclidean distance.

The most commonly used distance metric is the square of the Euclidean distance between two points x and y in an m-dimensional space.

Euclidean Distance (Image Source)

Here, j is the jth dimension (or feature column) of the sample points x and y

We define cluster inertia, a quantity that the clustering algorithm aims to minimize in the Sum of Squared Errors (SSE) sense as follows

Cluster Inertia (Image Source)

Here, μ(j) is the centroid for cluster j, and w(i,j) = 1 if the sample x(i) is in cluster j and 0 otherwise

The working of K-Means algorithm essentially involves the following steps

Random initialization of K cluster centroids Cluster Assignment Updating Centroids

The pseudocode for the algorithm is shown below

Image by Author

The following is a good illustration of K-Means Clustering;

We initialize 4 random points as cluster centroids; And follow the steps in the pseudocode above and we can see how the distance between all other data points to the cluster centroids are computed and each data point gets assigned to a cluster and the centroids get updated and the process repeats until convergence.

Choosing the optimal value for K

As we see, K is a hyperparameter and we have to choose the optimal number of clusters K , But how do we do it? Well, here are a few methods to find the optimal value of K .

Method 1: Run k-means multiple times, each time with different randomized initial centroids, and choose a set of values that leads to the minimum value of

distortion cost function given by

Distortion Cost Function (Image Source)

where u_c is the cluster centroid.

Smaller values of cost function correspond to better initialization

Well, this can be very time-inefficient and computationally expensive.

Method 2 — Elbow Method: Plot the values of K against the distortion cost function or inertia.

Choose the value of K at which the graph elbows out or has a sharp turn. To determine the optimal number of clusters, we have to select the value of K at the “elbow” , that is, the point after which the distortion/inertia starts decreasing linearly.

The figure below shows a case where for the dataset used, the optimal number of clusters K turns out to be 3.

Optimal K Value by Elbow Method (Image Source)

Limitations of K-Means Clustering

While K-Means is a simple and useful clustering algorithm, there are a few limitations which are stated below;

We need to specify the number of clusters ( K is a hyperparameter)

is a hyperparameter) Results can change depending on the location of the initial centroids (random initialization)

Not recommended if there are lot of categorical variables (depends on the numeric values to compute the distances)

Assumes that clusters are spherical, distinct and approximately equal in size

Not suitable for clustering data where clusters are of varying sizes and density

References