K-Means Clustering with Math. Common Unsupervised learning technique…
By: Sampath Kumar Gajawada
Tags: Clustering, Machine Learning, Mathematics, Unsupervised Learning, Data Science

When we are working with huge volumes of data, it makes sense to partition the data into logical groups and doing the analysis. We can use Clustering to make the data into groups with the help of several algorithms like K-Means.

In this article, I will try to address

a. Clustering

b. K-Means and working of the algorithm.

c. Choosing the right K Value

Clustering

A process of organizing objects into groups such that data points in the same groups are similar to the data points in the same group. A cluster is a collection of objects where these objects are similar and dissimilar to the other cluster.

K-Means

K-Means clustering is a type of unsupervised learning. The main goal of this algorithm to find groups in data and the number of groups is represented by K. It is an iterative procedure where each data point is assigned to one of the K groups based on feature similarity.

Algorithm

K-Means algorithm starts with initial estimates of K centroids, which are randomly selected from the dataset. The algorithm iterates between two steps assigning data points and updating Centroids.

Data Assignment

In this step, the data point is assigned to its nearest centroid based on the squared Euclidean distance. Let us assume a Cluster with c as centroid and a data point x is assigned to this cluster, based on the distance between c,x. There are some other distance measures like Manhattan, Jaccard, and Cosine which are used based on the appropriate type of data.

Centroid Update

Centroids are recomputed by taking the mean of all data points assigned to a particular cluster.

image by George Seif

Let us go through the above steps using the example below.

Consider 4 data points A,B,C,D as below

Observations

2. Choose two centroids AB and CD, calculated as

AB = Average of A, B CD = Average of C,D

Two centroids AB, CD

3. Calculate squared euclidean distance between all data points to the centroids AB, CD. For example distance between A(2,3) and AB (4,2) can be given by s = (2–4)² + (3–2)².

A is very near to CD than AB

4. If we observe in the fig, the highlighted distance between (A, CD) is 4 and is less compared to (AB, A) which is 5. Since point A is close to the CD we can move A to CD cluster.

5. There are two clusters formed so far, let recompute the centroids i.e, B, ACD similar to step 2.

ACD = Average of A, C, D B = B

New centroids B, ACD

6. As we know K-Means is iterative procedure now we have to calculate the distance of all points (A, B, C, D) to new centroids (B, ACD ) similar to step 3.

Clusters B, ACD

7. In the above picture, we can see respective cluster values are minimum that A is too far from cluster B and near to cluster ACD. All data points are assigned to clusters (B, ACD ) based on their minimum distance. The iterative procedure ends here.

8. To conclude, we have started with two centroids and end up with two clusters, K=2.

Choosing K

One method of choosing value K is the elbow method. In this method we will run K-Means clustering for a range of K values lets say ( K= 1 to 10 ) and calculate the Sum of Squared Error (SSE). SSE is calculated as the mean distance between data points and their cluster centroid.

Then plot a line chart for SSE values for each K, if the line chart looks like an arm then the elbow on the arm is the value of K that is the best.

Choose the Best K

Hope you enjoyed it!! Please do comment on any queries or suggestions.