Get the Optimal K in K-Means Clustering
By: Satsawat Natakarnkitkul
Tags: Artificial Intelligence, Machine Learning, K Means Clustering, Tutorial, Unsupervised Learning

Get the Optimal K in K-Means Clustering

Provide the quick starting guide to find the optimal number of clusters in K-means clustering

Introduction

Clustering (or Cluster Analysis) helps to identify and group the data points, which is closely related using some measure of distance, to a blob of data or segment. Clustering is classified as unsupervised learning within machine learning space, this means there is no labeled.

It is one of the important techniques in machine learning for business or corporate utilizing data science. Several use cases are:

In the financial industry, suspicious activities and individuals can be identified using anomaly detection

In biology, clustering is used to find groups of genes with similar property or expression patterns

In marketing science, clustering is used to identify the segments of similar customers by various characteristics. With these segments, a business can develop personalized marketing strategies and programs to tackle each segments accordingly

k-means clustering (not to be confused with machine learning technique: k-nearest neighbor classifier) is popular for cluster analysis in data mining. The method is aiming to partition the given observations into k clusters, in which each observation is in the cluster with the nearest mean.

K-means clustering is categorized as prototype-based clustering because each cluster is represented by a prototype, which can either be the centroid (average) of similar points with continuous features, or the medoid (the most representative or most frequently occurring point) in the case of categorical features.

Choosing the right number of k clusters can be a bit tricky, and this article will show the fundamental of k-means clustering and how to choose the right number of clusters, which should help you to kick start your clustering project.

Sample of clustering

Fundamentals

There are multiple clustering algorithms, which are based on alternative strategies to solve the problem. There are two main clustering techniques;

Hard clustering techniques, where each observation must only belong to a single cluster Soft (or fuzzy) clustering, which is an alternative approach and is based on a membership score that defines how much the elements are compatible with each cluster.

K-means clustering algorithm will

Randomly assign the simple points as the starting cluster centers (or centroids). The algorithm then will iteratively move the centroids to the centers of the samples. This iterative approach is minimizing the within cluster sum of squared errors (SSE), which sometimes called cluster inertia. Continue the step (2) until it reached the defined tolerance or the maximum number of iterations is reached.

When the centroids move, it will compute the squared Euclidean distance to measure the similarity between the samples and centroids. Hence, the k-means is very good at identifying clusters with a spherical shape (as it uses the squared Euclidean distance). That is also a drawback as in reality, the data points won’t be in such shape, k-means does not account for variance and has the effect from the curse of dimensionality at a high dimension.

Another drawback of k-means is the users need to specify the number of clusters, k. This can cause analysis and clustering performance to be poor. In this article, I will show several methods to effectively select the number of clusters for k-means clustering.