K-Means Clustering — Explained. Detailed theorotical explanation and... | by Soner Yıldırım | Towards Data Science

Let’s assume we start a business to sell some services to people. Business goes well and we have about ten thousand customers in our database within a few months. We want to keep our customers and also increase our revenue per customer. So we plan to offer a deal to the customers in our database. We have two options:
Offer same deal to all customers
Prepare customer-specific deals
First option is straightforward. “10% off on all services” deal would do the job. However, it is less efficient and less profitable compared to customer-specific deals. Furthermore, customer-specific deals are likely to be more appealing to customers. Some customers prefer a discount on a single item while others prefer a buy-one-get-one-free kind of deal. Some customers buy service A on the weekends while some others buy it on monday mornings. We can list many more different options depending on business size and number of customers.
We decide to prepare customer-specific deals. Next step is to decide what kind of deals to offer. We cannot just create a different deal for each customer. That is simply not manageable. A wise option could be detecting customers with similar interests or purchasing behavior and group them. The criteria for grouping can be customer preferences, tastes, interests, customer-service combinations and so on. Let’s say customer database has the following information for each customer:
Customer age, address
Average purchase amount
Average purchase quantity
Purchasing frequency
Time and type of purchase
This list can easily be extended. It is extremely difficult to group customers manually. Then we ask machine learning for help. The task of grouping similar customers is called clustering. A more formal definition on wikipedia:
Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense) to each other than to those in other groups (clusters).
Objects in our case are customers.
Clustering is unsupervised which means there is no label for samples (or data points). Clustering is used in many industries. Following are some examples of clustering:
Customer segmentation
Image segmentation
Image/color compression
Detecting anomalies or outliers
There are many different clustering algorithms. In this post, I will cover one of most common clustering algorithms: K-Means Clustering.
Before starting our discussion on k-means clustering, I would like point out the difference between clustering and classification.
Samples in a classification task have labels. Each data point is classified according to some measurements. Classification algorithms try to model the relationship between measurements (features) on samples and their assigned class. Then the model predicts the class of new samples.
Samples in clustering do not have labels. We expect the model to find structures in the data set so that similar samples can be grouped into clusters. We basically ask the model to label samples.
K-means clustering aims to partition data into k clusters in a way that data points in the same cluster are similar and data points in the different clusters are farther apart.
Similarity of two points is determined by the distance between them.
There are many methods to measure the distance. Euclidean distance (minkowski distance with p=2) is one of most commonly used distance measurements. The figure below shows how to calculate euclidean distance between two points in a 2-dimensional space. It is calculated using the square of the difference between x and y coordinates of the points.
In the case above, euclidean distance is the square root of (16 + 9) which is 5. Euclidean distance in two dimensions remind us the famous pythagorean theorem.
There are other ways to measure distance such as cosine similarity, average distance and so on. The similarity measure is at the core of k-means clustering. Optimal method depends on the type of problem. So it is important to have a good domain knowledge in order to choose the best measurement type.
K-means clustering tries to minimize distances within a cluster and maximize the distance between different clusters.
Let’s start with a simple example to understand the concept. As usual, we import the dependencies first:
# Importing necessary librariesimport numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom sklearn.datasets import make_blobsfrom sklearn.cluster import KMeans
Scikit-learn provides many useful functions to create synthetic datasets which are very helpful for practicing machine learning algorithms. I will use make_blobs function.
X, y = make_blobs(n_samples = 200, centers=4, cluster_std = 0.5, random_state = 0)plt.scatter(X[:, 0], X[:, 1], s=50)
Then we create a KMeans object and fit the data:
kmeans = KMeans(n_clusters = 4)kmeans.fit(X)
We can now partition the dataset into clusters:
y_pred = kmeans.predict(X)plt.scatter(X[:, 0], X[:, 1], c = y_pred, s=50)
Real life datasets are much more complex in which clusters are not clearly separated. However, the algorithm works in the same way.
K-means algorithm is not capable of determining the number of clusters. We need to define it when creating the KMeans object which may be a challenging task.
K-means is an iterative process. It is built on expectation-maximization algorithm. After number of clusters are determined, it works by executing the following steps:
Randomly select centroids (center of cluster) for each cluster.Calculate the distance of all data points to the centroids.Assign data points to the closest cluster.Find the new centroids of each cluster by taking the mean of all data points in the cluster.Repeat steps 2,3 and 4 until all points converge and cluster centers stop moving.
Randomly select centroids (center of cluster) for each cluster.
Calculate the distance of all data points to the centroids.
Assign data points to the closest cluster.
Find the new centroids of each cluster by taking the mean of all data points in the cluster.
Repeat steps 2,3 and 4 until all points converge and cluster centers stop moving.
Note: Initial centroids are chosen randomly which may cause final clusters to be somewhat different. To overcome this issue, scikit learn provides n_init parameter. The k-means algorithm run “n_init” times with different initial centroids and final results will be determined according to n_init consecutive runs.
You can visualize each step using this tool prepared by Naftali Harris on his blog. I took screenshot from this interactive tool so that the steps above become more comprehensible. I strongly suggest to spend some time with this tool.
Pros:
Easy to interpret
Relatively fast
Scalable for large data sets
Able to choose the positions of initial centroids in a smart way that speeds up the convergence
Guarantees convergence
Cons:
Number of clusters must be pre-determined. K-means algorithm is not able to guess how many clusters exist in the data. Determining number of clusters may well be a challenging task.
Can only draw linear boundaries. If there is a non-linear structure separating groups in the data, k-means will not be a good choice.
Slows down as the number of samples increases because at each step, k-means algorithm accesses all data points and calculates distances. An alternative way is to use a subset of data points to update the location of centroids (i.e. sklearn.cluster.MiniBatchKMeans)
Sensitive to outliers
Thank you for reading. Please let me know if you have any feedback.
Naive Bayes Classifier — Explained
Logistic Regression — Explained
K-Nearest Neighbors (kNN) — Explained
Support Vector Machine — Explained
Decision Trees and Random Forests — Explained
Gradient Boosted Decision Trees — Explained
Principal Component Analysis — Explained
Predicting Used Car Prices with Machine Learning