Finding Optimal Number Of Clusters for Clustering Algorithm — With python code
By: Mohamed Asarudheen
Tags: Machine Learning, Pyhton, Statistics, Unsupervised Learning, Artificial Intelligence

WHAT IS CLUSTERING?

It is basically a type of unsupervised learning method. An unsupervised learning method is a method in which we draw references from datasets consisting of input data without labeled responses. Generally, it is used as a process to find meaningful structure, explanatory underlying processes, generative features, and groupings inherent in a set of examples.

Clustering is the task of dividing the population or data points into a number of groups such that data points in the same groups are more similar to other data points in the same group and dissimilar to the data points in other groups. It is basically a collection of objects on the basis of similarity and dissimilarity between them.

There are multiple methods you can use in order to determine what is the optimal number of clusters for your data and that is what will see in detail below.

DETERMINING THE OPTIMAL NUMBER OF CLUSTERS

1. ELBOW METHOD

The Elbow method is a heuristic method of interpretation and validation of consistency within-cluster analysis designed to help to find the appropriate number of clusters in a dataset.

Elbow method example

The example code below creates finds the optimal value for k.

# clustering dataset

# determine k using elbow method

from sklearn.cluster import KMeans

from sklearn import metrics

from scipy.spatial.distance import cdist

import numpy as np

import matplotlib.pyplot as plt

x1 = np.array([3, 1, 1, 2, 1, 6, 6, 6, 5, 6, 7, 8, 9, 8, 9, 9, 8])

x2 = np.array([5, 4, 5, 6, 5, 8, 6, 7, 6, 7, 1, 2, 1, 2, 3, 2, 3])

plt.plot()

plt.xlim([0, 10])

plt.ylim([0, 10])

plt.title(‘Dataset’)

plt.scatter(x1, x2)

plt.show()

# create new plot and data

plt.plot()

X = np.array(list(zip(x1, x2))).reshape(len(x1), 2)

colors = [‘b’, ‘g’, ‘r’]

markers = [‘o’, ‘v’, ‘s’]

# k means determine k

distortions = []

K = range(1,10)

for k in K:

kmeanModel = KMeans(n_clusters=k).fit(X)

kmeanModel.fit(X)

distortions.append(sum(np.min(cdist(X, kmeanModel.cluster_centers_, ‘euclidean’), axis=1)) / X.shape[0])

# Plot the elbow

plt.plot(K, distortions, ‘bx-’)

plt.xlabel(‘k’)

plt.ylabel(‘Distortion’)

plt.title(‘The Elbow Method showing the optimal k’)

plt.show()

2. SILHOUETTE ANALYSIS

Silhouette analysis can be used to study the separation distance between the resulting clusters. The silhouette plot displays a measure of how close each point in one cluster is to points in the neighboring clusters and thus provides a way to assess parameters like number of clusters visually. This measure has a range of [-1, 1].

Where 1 means that points are very close to their own cluster and far from other clusters, whereas -1 indicates that points are close to the neighboring clusters.

Scikit learn link — Python code can be found in this sci-kit learn.

3. GAP STATISTICS

Gap statistic is a goodness of clustering measure, where for each hypothetical number of clusters k, it compares two functions: log of within-cluster sum of squares (wss) with its expectation under the null reference distribution of the data.

# compute gap statistic

set.seed(123)

gap_stat <- clusGap(df, FUN = kmeans, nstart = 25,

K.max = 10, B = 50)

# Print the result

print(gap_stat, method = "firstmax")

## Clustering Gap statistic ["clusGap"] from call:

## clusGap(x = df, FUNcluster = kmeans, K.max = 10, B = 50, nstart = 25)

## B=50 simulated reference sets, k = 1..10; spaceH0="scaledPCA"

## --> Number of clusters (method 'firstmax'): 4

## logW E.logW gap SE.sim

## [1,] 3.458369 3.638250 0.1798804 0.03653200

## [2,] 3.135112 3.371452 0.2363409 0.03394132

## [3,] 2.977727 3.235385 0.2576588 0.03635372

## [4,] 2.826221 3.120441 0.2942199 0.03615597

## [5,] 2.738868 3.020288 0.2814197 0.03950085

## [6,] 2.669860 2.933533 0.2636730 0.03957994

## [7,] 2.598748 2.855759 0.2570109 0.03809451

## [8,] 2.531626 2.784000 0.2523744 0.03869283

## [9,] 2.468162 2.716498 0.2483355 0.03971815

## [10,] 2.394884 2.652241 0.2573567 0.04104674

We can visualize the results with fviz_gap_stat which suggests four clusters as the optimal number of clusters.

fviz_gap_stat(gap_stat)

4. HIERARCHICAL CLUSTERING- Dendrogram

A dendrogram is a tree-like diagram that records the sequences of merges or splits.

Hierarchical clustering is usually used to better understand the structure and relationships in your data and based on them you decide what number of clusters seems appropriate for your purpose

How to choose the optimal number of clusters based on the output of this analysis, the dendogram? As a rule of thumb, look for the clusters with the longest ‘branches’, the shorter they are, the more similar they are to following ‘twigs’ and ‘leaves’. But keep in mind that, as always, the optimal number will also depend on the context, expert knowledge, application, etc.

>>> from scipy.cluster import hierarchy

>>> import matplotlib.pyplot as plt

>>> ytdist = np.array([662., 877., 255., 412., 996., 295., 468., 268.,

... 400., 754., 564., 138., 219., 869., 669.])

>>> Z = hierarchy.linkage(ytdist, 'single')

>>> plt.figure()

>>> dn = hierarchy.dendrogram(Z)

>>> hierarchy.set_link_color_palette(['m', 'c', 'y', 'k'])

>>> fig, axes = plt.subplots(1, 2, figsize=(8, 3))

>>> dn1 = hierarchy.dendrogram(Z, ax=axes[0], above_threshold_color='y',

... orientation='top')

>>> dn2 = hierarchy.dendrogram(Z, ax=axes[1],

... above_threshold_color='#bcbddc',

... orientation='right')

>>> hierarchy.set_link_color_palette(None) # reset to default after use

>>> plt.show()

Dendrogram

How to choose the optimal number of clusters based on the output of this analysis, the dendrogram? As a rule of thumb, look for the clusters with the longest ‘branches’, the shorter they are, the more similar they are to following ‘twigs’ and ‘leaves’. But keep in mind that, as always, the optimal number will also depend on the context, expert knowledge, application, etc.

Thanks for reading my first post in Medium.