K-Means Algorithm: Dealing with Unlabeled Data
By: Srijarko Roy
Tags: K Means, Unsupervised Learning, Data Science, Machine Learning

What is Unsupervised Learning?

You may have heard about Market Segmentation where there is a database of customers and we group them into different market segments so as to serve them better, or maybe about Social Network Analysis to find out who are the coherent groups of friends in a social network. However can you imagine any labels in the data that is used in the training examples for any of the mentioned sector? If no, then how do we develop Machine Leaning Algorithms to make predictions on unlabeled data? The answer is Unsupervised Learning. The Unsupervised Learning Algorithms learn from unlabeled data by finding some structure in the given training set. The following figure will illustrate the concept:

Unlabeled data grouped in 2 clusters

Clustering and working of K-Means Algorithm:

The above figure gives us an intuition about how an unsupervised learning algorithm can automatically group unlabeled data into coherent clusters. The K-Means algorithm is the most widely used clustering algorithm. So how does the K-Means Algorithm work?

K-Means is an iterative algorithm. Let us say we have an unlabeled dataset as shown:

Initial Unlabeled Dataset

Our objective is to group the data into 2 clusters. So, first we are going to randomly initialize two points (since I want to group the data into 2 clusters), known as cluster centroids.

Cluster centroids randomly initialized

Now my algorithm goes through each of the examples and depending on whether they are closer to the Red or the Blue centroid, it assigns each of the data points to one of the cluster centroids. This step is called the “Cluster Assignment Step”.

Cluster Assignment Step

The second part of the iteration is called the “Move Centroid Step”. The two cluster centroids are taken and moved to the average of their corresponding data points. Specifically, we are going to compute the mean of the points’ location for each of the two clusters already formed and move the centroids to their respective cluster means.

Cluster centroids moved to their respective means

As I have stated earlier, K-Means is an iterative algorithm. So now we go back to the Cluster Assignment Step, go through the training examples and assign each data points to the nearest cluster centroid.

Cluster Assignment for next iteration

The algorithm continues with yet another Move Centroid Step and computes the average of the data points for each of the 2 clusters and move the centroids to their new position.

Move Centroid Step for next iteration

As we can see 2 clusters have been formed quite distinctly. The algorithm will continue to run for as many iterations that has been specified but our intuition says that there will not be any change further while either of the 2 steps are performed since K-Means has converged.

Generalized Explanation:

To write more formally, the K-Means Algorithm takes two inputs. The first is a parameter (K) for the number of clusters we want to find, and the second is our unlabeled training set having ‘m’ examples.

Input for K-Means Algorithm

Each of my training examples has ‘n’ features, hence each of them is taken as an R(n) dimensional vector. Now that the inputs have been taken, we run the K-Means Algorithm.

First we randomly initialize K cluster centroids u(1) through u(K). Then our iterative process of Cluster Assignment and Move Centroid begins.

K-Means Algorithm representation

The first inner loop (Cluster Assignment Step) computes a variable ‘c(i)’ to store the index of the cluster centroid closest to training example ‘x(i)’. So, ‘c(i)’ is a number between 1 and K which denotes that the ‘i’th training example is closest to which cluster centroid ‘k’ (1 through K). Now the question arises that how the operation is done.

The loop runs for each of the training examples from 1 through m. In order to compute c(i), the ‘i’th example x(i) is taken and its distance is measured from each of the cluster centroids u(k), and the value of ‘k’ for which we get the minimum distance between x(i) and u(k), is what gets set in c(i). Hence, c(i) is set to be that value of ‘k’ that minimizes the square of the norm between (x(i)-u(k)) thereby selecting the cluster centroid having the shortest distance from my training example ‘x(i)’.

Cluster Assignment (computing c(i))

The other inner loop for Move Centroid Step computes the ‘u(k)’ for each cluster as the mean of the training example points assigned them. Suppose the examples x(1), x(7), x(9) and x(15) has been assigned to cluster centroid 1. This implies that c(1) = c(7) = c(9) = c(15) = 1. The Move Centroid step would thus compute u(1) as:

Move Centroid (computing u(k) for k=1)

As a result the cluster centroids will move to the mean of the points assigned to them. The process continues until each of the clusters centroids have moved to their desired positions and the iterative K-Means algorithm continues to perform the steps and will eventually converge.

Cost-Function:

The K-Means algorithm has a cost function that it tries to optimize, just like supervised algorithms, which helps in debugging and avoiding local minima. We have used variables c(i) and u(k) for the Cluster Assignment and Move Centroid steps, respectively. We have to find a proper value of the cluster centroid of the cluster u(c(i)) to which a particular training example x(i) has been assigned.

Suppose x(i) has been assigned to cluster 7. So by the Cluster Assignment step, c(i) = 7. So the cluster centroid of the cluster to which x(i) has been assigned will be:

Cluster centroid of cluster no. 7

The Cost-function that K-Means optimizes is a function J of c(1) through c(m) and u(1) through u(m). These parameters are varied to an ambient value such that the Cost Function is minimized.

Cost Function

The Cost function is also known as the Distortion Cost function, and is implemented in the iterative process of Cluster Assignment step and Move Centroid step as:

Implementation of the cost function

So, the 2 sets of variables are taken and sort of partitioned. J is first minimized with respect to c, then with respect to u, and the process repeats itself the specified number of times. At the end of the iterative process K-Means converges and a structure in the data is eventually found.

Conclusion:

The K-Means Algorithm scales really well to large datasets, generalized to clusters of various shapes and guarantees convergence after a specific number of iterations. However, it fails to yield accurate results if outliers are present or density of the spread of data is not even. Nevertheless, equipped with the quality of adapting easily to new examples, K-Means is still the most widely used Unsupervised Learning Algorithm.