FREE DIVING INTO K-MEANS
By: Honghui Wang
Tags: Machine Learning, Big Data, K Means Clustering, Anomaly Detection, Unsupervised Learning

Content Contributors: Shray Khanna, Honghui Wang, and Chhavi Verma

Warming Up

K-means clustering is a type of unsupervised learning, which is used when you have unlabeled data (i.e., data without defined categories or groups). Wait, what?

So, basically, k-means helps you to form groups in the unlabeled data. These groups are in technical terms “k clusters” which helps you to do classification needed to be done by you.

We know, it’s interesting. In this article, you will learn in depth about k-means and it’s applications in a fun way such that you’ll answer anything on k-means (even in your sleep!). That’s right, so, let the fun begin!

Why k-means is important?

K-means has a high significance in the field of Machine Learning, we can do anomaly detection, document classification, image compression, image segmentation, fraud detection, trend analysis for many social networking sites and many more. See how one algorithm is applicable to so many real-world problems. This is the power of clustering and specifically k-means.

From scikit-learn’s algorithm map, we can see k-means deals with categorical and unlabeled data.

Spark’s creation Inspired by K-means

There is an interesting story about k-means. When Matei Zaharia was studying at UC Berkeley AMPlab as a Ph.D. student, he implemented a parallel k-means algorithm based on MapReduce and ran it on the cluster. It turned out to be slower than running it in one computer. After discussion with his colleagues, Matei Zaharia figured out that the problem resided on the repetition of loading and storing the data from and to the disk. Then Matei Zaharia created Spark.

Matei Zaharia- creator of Apache Spark and co-founder of Databricks

Get your feet wet

What is the K-means algorithm?

K-means is a kind of clustering algorithm. It can divide our dataset into K different groups which makes the data points in every group share some common features.

For example, given K=3 and the data, k-means will divide the dataset into three groups as shown below.

K-means algorithm is pretty simple, just four steps as shown below:

Given a dataset of m points, pick up K points randomly as cluster centroids. Assign each data point to the closest centroid. Update cluster centroids, each equals to the mean of all its assigned data points. Repeat Steps 2 and 3 until some termination conditions (i.e. the centroid does not change anymore) are met.

Working of K-means

Start Diving

The convergence of k-means

One of the most asked question in the interview of data scientist is: why k-means is guaranteed to converge?

In order to answer this question, we resort to mathematics.

We can see that the object of k-means is to minimize such a cost function:

So k-means is, in fact, coordinate descent on J. Specifically, the second step of k-means repeatedly minimizes J with respect to c while holding µ fixed, and the third step minimizes J with respect to µ while holding c fixed. Obviously, J decreases monotonically. So the value of J must converge. (As pointed out by Andrew Ng in his brilliant Coursera machine learning course, this implies that c and µ will converge too. In theory, it is possible for k-means to oscillate between a few different clusterings — i.e., a few different values for c and/or µ — that have exactly the same value of J, but this almost never happens in practice.)

Is it guaranteed to find the optimal solution?

Unfortunately, it is not. Because the object function is not convex, we may be stuck on the local optima.

For example, if we are lucky we may get the initial points as shown in (a), but sometimes, we may come up with initialization as shown in (b) and (c), which will obviously give us a local minimum solution.

We can see that k-means is vulnerable to the selection of centroids. To solve this problem, we can run k-means for many times (50–10000) and pick the one which has the minimal distortion cost.

The other way is to use K-means ++.

K-means ++

Looking at (a), (b) and (c) we can get this intuition: each of the k initial cluster centroids should stay away from each other as far as possible. In other words, spreading out the k initial cluster centroids is a good thing [4]. K-means ++ follows this intuition. It chooses the centroids one by one. Each time it computes the distance of each data point to its nearest centroid. The large the distance is, the more likely it will be chosen as the next centroid.

Here is the algorithm:

1. Select one data point uniformly at random as the first centroid.

2. Calculate the square distance of each point to its nearest centroid, and denoted it as d(x).

3. Select one data point as the new centroid with probability proportional to d(x).

4. Repeat Steps 2 and 3 until we find k centroids.

5. From here, we can use the standard k-means to cluster the dataset

David Arthur and Sergei Vassilvitskii concluded in their paper that k-means++ consistently outperformed k-means [6]. It not only achieves much better results it also made the algorithm converge faster. When K is small compared to the number of data points, k-means++ improve the performance by several orders of magnitude.

How to determine the number of clusters?

In practice, data scientists pick up this number by hand, by parameter tuning or by the demand of the problem.

Also, we can use Elbow Method to pick up one by visualizing the relation between the number of clusters and value of cost function J, as shown below. We select the one after which the figure becomes much smoother, just like an elbow in our arm.

But do not make your expectation too high, because the figure may look like the one shown on the right which has likely smoothness everywhere.

A Quick Peek into Abyss

ISODATA

ISODATA algorithm is proposed to generate the number of clusters automatically. It has two operations: divide and merge. It chooses to divide a cluster or merge two clusters based on the situation of the data points in the clusters.

It requires four more parameters:

k₀: the initial number of clusters, and it determines the range of the final number of clusters which will be [ k₀/2, 2* k₀] Nmin: the minimal size of every cluster. σ: when the variance of a cluster is bigger than it, the cluster will be divided. Dmin: If the distance between two cluster centroids is smaller than Dmin, then merge these two clusters.

As we can see, ISODATA requires much more parameters than standard k-means, and is hard to give a reasonable value for them. So in practice, people seldom use ISODATA.

The parallel version of k-means.

K-means works well with MapReduce process. Use map(point)=(point, c) to map each point to its nearest centroid, then aggregate them to get the new centroid.

K-means for Very Large scale dataset

For finding the optimal solution of a cost function with a large scale dataset, we usually resort to sequential methods. Botton and Bengio proposed an online, stochastic gradient (SGD) variant of k-means [1]. The algorithm computes a gradient descent step on one randomly chosen data point at a time. Due to stochastic noise, this algorithm finds lower quality solutions than the standard k-means, although it converges much more quickly.

Sculley David proposed mini batch k-means which select a small batch of the data points to compute the gradient [2]. As shown by Sculley in his paper, mini-batch k-means reduces computation cost by orders of magnitude while achieving a comparable result with standard k-means.

Resurfacing

Can not hold your breath anymore, right? Let’s resurface and go to see some interesting application that k-means can do.

We used k-means in three different situations based on scikit-learn, and we mostly used scikit-learn’s preset dataset such that anyone can implement without the problem of getting data from third party websites.

In the first example, we used k-means to detect anomaly on synthesized data points which have two-dimensional features.

We further used k-means for handwritten digitals recognition. Just by applying the k-means we see an accuracy of 73% which is not that bad. After some pre-processing, we can make it to 93%.

In the third example, we see color compression using k-means, by reducing the extra colors and values that are not needed to represent an image and not necessary for human vision.

Go and check the code in the link below, you’ll get to know the method in 9 easy steps.

Jake VanderPlas has an amazing book on data science: The Python Data Science Handbook by Jake VanderPlas (O’Reilly Media, 2016). We have implemented the K-means using his books and notes on the algorithm. Link to his website: https://jakevdp.github.io/pages/about.html

Reference

[1] Bottou, Leon, and Yoshua Bengio. “Convergence properties of the k-means algorithms.” Advances in neural information processing systems. 1995.

[2] Sculley, David. “Web-scale k-means clustering.” Proceedings of the 19th international conference on World wide web. ACM, 2010.

[3] Andrew NG;Machine Learning coursera online course;https://www.coursera.org/learn/machine-learning

[4] K-means ++; wikipedia;https://en.wikipedia.org/wiki/K-means%2B%2B

[5] SCI-KIT IMPLEMENTATION: https://scikit-learn.org/stable/index.html

[6] Arthur, David, and Sergei Vassilvitskii. “k-means++: The advantages of careful seeding.” Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms. Society for Industrial and Applied Mathematics, 2007.

[7] Liu, Hongfu, Jun Li, Yue Wu, and Yun Fu. “Clustering with Outlier Removal.” arXiv preprint arXiv:1801.01899 (2018).

[8] Christy, A., G. Meera Gandhi, and S. Vaithyasubramanian. “Cluster based outlier detection algorithm for healthcare data.” Procedia Computer Science 50 (2015): 209–215.