10 Tips for Choosing the Optimal Number of Clusters
Tags: Data Science, Data Visualization, Clustering, Rstats, Machine Learning

This shows that we can an indication of the correct clustering resolution by examining the edges and we can overly information to assess the quality of the clustering.

Choosing the appropriate algorithm

What about choice of appropriate clustering algorithm? The cValid package can be used to simultaneously compare multiple clustering algorithms, to identify the best clustering approach and the optimal number of clusters. We will compare k-means, hierarchical and PAM clustering.

intern <- clValid(mammals_scaled, nClust = 2:24,

clMethods = c("hierarchical","kmeans","pam"), validation = "internal") # Summary

summary(intern) %>% kable() %>% kable_styling() Clustering Methods:

hierarchical kmeans pam



Cluster sizes:

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24



Validation Measures:

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24



hierarchical Connectivity 4.1829 10.5746 13.2579 20.1579 22.8508 25.8258 32.6270 35.3032 38.2905 39.2405 41.2405 45.7742 47.2742 50.6075 52.6075 55.8575 58.7242 60.7242 63.2242 65.2242 67.2242 69.2242 71.2242

Dunn 0.3595 0.3086 0.3282 0.2978 0.3430 0.3430 0.4390 0.4390 0.5804 0.5938 0.5938 0.8497 0.8497 0.5848 0.5848 0.4926 0.9138 0.9138 0.8892 0.9049 0.9335 1.0558 2.1253

Silhouette 0.5098 0.5091 0.4592 0.4077 0.4077 0.3664 0.3484 0.4060 0.3801 0.3749 0.3322 0.3646 0.3418 0.2650 0.2317 0.2166 0.2469 0.2213 0.1659 0.1207 0.1050 0.0832 0.0691

kmeans Connectivity 7.2385 10.5746 15.8159 20.1579 22.8508 25.8258 33.5198 35.3032 38.2905 39.2405 41.2405 45.7742 47.2742 51.8909 53.8909 57.1409 58.7242 60.7242 63.2242 65.2242 67.2242 69.2242 71.2242

Dunn 0.2070 0.3086 0.2884 0.2978 0.3430 0.3430 0.3861 0.4390 0.5804 0.5938 0.5938 0.8497 0.8497 0.5866 0.5866 0.5725 0.9138 0.9138 0.8892 0.9049 0.9335 1.0558 2.1253

Silhouette 0.5122 0.5091 0.4260 0.4077 0.4077 0.3664 0.3676 0.4060 0.3801 0.3749 0.3322 0.3646 0.3418 0.2811 0.2478 0.2402 0.2469 0.2213 0.1659 0.1207 0.1050 0.0832 0.0691

pam Connectivity 7.2385 14.1385 17.4746 24.0024 26.6857 32.0413 33.8913 36.0579 38.6607 40.6607 42.7869 45.7742 47.2742 51.7242 53.7242 56.9742 58.7242 60.7242 62.7242 64.7242 66.7242 69.2242 71.2242

Dunn 0.2070 0.1462 0.2180 0.2180 0.2978 0.2980 0.4390 0.4390 0.4390 0.4390 0.4390 0.8497 0.8497 0.5314 0.5314 0.4782 0.9138 0.9138 0.8333 0.8189 0.7937 1.0558 2.1253

Silhouette 0.5122 0.3716 0.4250 0.3581 0.3587 0.3318 0.3606 0.3592 0.3664 0.3237 0.3665 0.3646 0.3418 0.2830 0.2497 0.2389 0.2469 0.2213 0.1758 0.1598 0.1380 0.0832 0.0691



Optimal Scores:



Score Method Clusters

Connectivity 4.1829 hierarchical 2

Dunn 2.1253 hierarchical 24

Silhouette 0.5122 kmeans 2

Connectivity and Silhouette are both measurements of connectedness while the Dunn Index is the ratio of the smallest distance between observations not in the same cluster to the largest intra-cluster distance.

Extracting Features of Clusters

As mentioned earlier it’s difficult to assess the quality of results from clustering. We don’t have true labels so so it’s unclear how one would measure “how good it actually works” in term of interal validation. However, clustering is an excellent EDA starting point for exploring the differences between clusters in greater detail. Think of clustering like manufacturing shirt sizes. We could choose to only make three sizes: small, medium and large. we’re sure to cut down cost but not everyone is going to have a great fit. Think about pant sizes now (or shirt brands with many sizes (XS, XL, XXL, etc.)) where you have many more categories (or clusters). For some fields the choice of optimal cluster may depend on some external knowledge like cost of producing k clusters to satisfy customers with the best possible fit. In other domains like biology where you are trying to determine the exact number of cells a more in-depth approach would be required. For example, many of the above heuristics contradicted each other for what the optimal number of clusters was. Keep in mind these were all evaluating the k-means algorithm at different numbers of k. This could potentially mean that the k-means algorithm fails and no k is good. The k-means algorithm is not a very robust algorithm that is sensitive to outliers and this data set is quit small. The best thing to do would be to explore the above methods on the output of other algorithms (for example hierarchical clustering which clValid suggested), collect more data, or spend some time labelling samples for other ML methods if possible.

Ultimately, we would like to answer questions like “what is it that makes this cluster unique from others?” and “what are the clusters that are similar to one another”. Let’s select five clusters and interrogate the features of these clusters.

# Compute dissimilarity matrix with euclidean distances

d <- dist(mammals_scaled, method = "euclidean") # Hierarchical clustering using Ward's method

res.hc <- hclust(d, method = "ward.D2" ) # Cut tree into 5 groups

grp <- cutree(res.hc, k = 5) # Visualize

plot(res.hc, cex = 0.6) # plot tree

rect.hclust(res.hc, k = 5, border = 2:5) # add rectangle