Clustering Analysis in R using K-means | by Luiz Fonseca | Towards Data Science

The purpose of clustering analysis is to identify patterns in your data and create groups according to those patterns. Therefore, if two points have similar characteristics, that means they have the same pattern and consequently, they belong to the same group. By doing clustering analysis we should be able to check what features usually appear together and see what characterizes a group.
In this post, we are going to perform a clustering analysis with multiple variables using the algorithm K-means. The intention is to find groups of mammals based on the composition of the species’ milk. The main points covered here are:
A brief description of the data used in the clustering;An explanation of how the K-means algorithm works;How to decide the number of groups we want as input;Clustering validation using silhouette width.How to interpret the results and show them in a visualization.
A brief description of the data used in the clustering;
An explanation of how the K-means algorithm works;
How to decide the number of groups we want as input;
Clustering validation using silhouette width.
How to interpret the results and show them in a visualization.
The dataset used is part of the package cluster.datasets and contains 25 observations on the following 6 variables:
name — a character vector for the name of the animals water — a numeric vector for the water content in the milk sample protein — a numeric vector for the amount of protein in the milk samplefat — a numeric vector for the fat content in the milk sample lactose — a numeric vector for the amount of lactose in the milk sample ash — a numeric vector for the amount of mineral in the milk sample
Let’s take a look at a sample of the data
library(cluster.datasets)data(all.mammals.milk.1956)head(all.mammals.milk.1956)##        name water protein fat lactose  ash## 1     Horse  90.1     2.6 1.0     6.9 0.35## 2 Orangutan  88.5     1.4 3.5     6.0 0.24## 3    Monkey  88.4     2.2 2.7     6.4 0.18## 4    Donkey  90.3     1.7 1.4     6.2 0.40## 5     Hippo  90.4     0.6 4.5     4.4 0.10## 6     Camel  87.7     3.5 3.4     4.8 0.71
The charts below show us the distribution for each variable. Each point represents a mammal species (25 in total).
Each variable has different behavior and we could identify groups of mammals on each one individually, but that’s not the purpose here.
All the variables will be used in the clustering on a linear scale. Sometimes, when the values (for each feature) are in a big range, for example from 0 up to 1 million, it’s interesting to use a logarithmic scale because on a log scale we would highlight bigger differences between the values and smaller differences would be considered less important. Since the values in our dataset vary between 0 and 100, we are going to use a linear scale, which considers differences between values equally important.
The clustering algorithm that we are going to use is the K-means algorithm, which we can find in the package stats. The K-means algorithm accepts two parameters as input:
The data;
A K value, which is the number of groups that we want to create.
Conceptually, the K-means behaves as follows:
It chooses K centroids randomly;Matches each point in the data (in our case, each mammal) with the closest centroid in an n-dimensional space where n is the number of features used in the clustering (in our example, 5 features — water, protein, fat, lactose, ash). After this step, each point belongs to a group.Now, it recalculates the centroids as being the mean point (vector) of all other points in the group.It keeps repeating the steps 2 and 3 until either when the groups are stabilized, that is, when no points are reallocated to another centroid or when it reaches the maximum number of iterations (the stats library uses 10 as default).
It chooses K centroids randomly;
Matches each point in the data (in our case, each mammal) with the closest centroid in an n-dimensional space where n is the number of features used in the clustering (in our example, 5 features — water, protein, fat, lactose, ash). After this step, each point belongs to a group.
Now, it recalculates the centroids as being the mean point (vector) of all other points in the group.
It keeps repeating the steps 2 and 3 until either when the groups are stabilized, that is, when no points are reallocated to another centroid or when it reaches the maximum number of iterations (the stats library uses 10 as default).
The bigger is the K you choose, the lower will be the variance within the groups in the clustering. If K is equal to the number of observations, then each point will be a group and the variance will be 0. It’s interesting to find a balance between the number of groups and their variance. A variance of a group means how different the members of the group are. The bigger is the variance, the bigger is the dissimilarity in a group.
How do we choose the best value of K in order to find that balance?
To answer that question, we are going to run K-means for an arbitrary K. Let’s pick 3.
## K-means clustering with 3 clusters of sizes 7, 2, 16## ## Cluster means:##      water   protein      fat  lactose      ash## 1 69.47143  9.514286 16.28571 2.928571 1.311429## 2 45.65000 10.150000 38.45000 0.450000 0.690000## 3 86.06250  4.275000  4.17500 5.118750 0.635625## ## Clustering vector:##  [1] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 2 2## ## Within cluster sum of squares by cluster:## [1] 300.1562  27.1912 377.2215##  (between_SS / total_SS =  89.9 %)## ## Available components:## ## [1] "cluster"      "centers"      "totss"        "withinss"    ## [5] "tot.withinss" "betweenss"    "size"         "iter"        ## [9] "ifault"
The kmeans() function outputs the results of the clustering. We can see the centroid vectors (cluster means), the group in which each observation was allocated (clustering vector) and a percentage (89.9%) that represents the compactness of the clustering, that is, how similar are the members within the same group. If all the observations within a group were in the same exact point in the n-dimensional space, then we would achieve 100% of compactness.
Since we know that, we will use that percentage to help us decide our K value, that is, a number of groups that will have satisfactory variance and compactness.
The function below plots a chart showing the “within sum of squares” (withinss) by the number of groups (K value) chosen for several executions of the algorithm. The within sum of squares is a metric that shows how dissimilar are the members of a group., the greater is the sum, the greater is the dissimilarity within a group.
By Analysing the chart from right to left, we can see that when the number of groups (K) reduces from 4 to 3 there is a big increase in the sum of squares, bigger than any other previous increase. That means that when it passes from 4 to 3 groups there is a reduction in the clustering compactness (by compactness, I mean the similarity within a group). Our goal, however, is not to achieve compactness of 100% — for that, we would just take each observation as a group. The main purpose is to find a fair number of groups that could explain satisfactorily a considerable part of the data.
So, let’s choose K = 4 and run the K-means again.
## K-means clustering with 4 clusters of sizes 10, 2, 7, 6## ## Cluster means:##      water   protein      fat  lactose       ash## 1 88.50000  2.570000  2.80000 5.680000 0.4850000## 2 45.65000 10.150000 38.45000 0.450000 0.6900000## 3 81.18571  7.428571  6.90000 4.014286 0.9314286## 4 68.33333  9.550000 17.41667 2.916667 1.3300000## ## Clustering vector:##  [1] 1 1 1 1 1 1 1 3 3 3 3 1 1 3 1 3 3 4 4 4 4 4 4 2 2## ## Within cluster sum of squares by cluster:## [1]  59.41225  27.19120  63.53491 191.96100##  (between_SS / total_SS =  95.1 %)## ## Available components:## ## [1] "cluster"      "centers"      "totss"        "withinss"    ## [5] "tot.withinss" "betweenss"    "size"         "iter"        ## [9] "ifault"
Using 3 groups (K = 3) we had 89.9% of well-grouped data. Using 4 groups (K = 4) that value raised to 95.1%, which is a good value for us.
We may use the silhouette coefficient (silhouette width) to evaluate the goodness of our clustering.
The silhouette coefficient is calculated as follows:
For each observation i, it calculates the average dissimilarity between i and all the other points within the same cluster which i belongs. Let’s call this average dissimilarity “Di”.Now we do the same dissimilarity calculation between i and all the other clusters and get the lowest value among them. That is, we find the dissimilarity between i and the cluster that is closest to i right after its own cluster. Let’s call that value “Ci”The silhouette (Si) width is the difference between Ci and Di (Ci — Di) divided by the greatest of those two values (max(Di, Ci)).Si = (Ci — Di) / max(Di, Ci)
For each observation i, it calculates the average dissimilarity between i and all the other points within the same cluster which i belongs. Let’s call this average dissimilarity “Di”.
Now we do the same dissimilarity calculation between i and all the other clusters and get the lowest value among them. That is, we find the dissimilarity between i and the cluster that is closest to i right after its own cluster. Let’s call that value “Ci”
The silhouette (Si) width is the difference between Ci and Di (Ci — Di) divided by the greatest of those two values (max(Di, Ci)).Si = (Ci — Di) / max(Di, Ci)
So, the interpretation of the silhouette width is the following:
Si > 0 means that the observation is well clustered. The closest it is to 1, the best it is clustered.
Si < 0 means that the observation was placed in the wrong cluster.
Si = 0 means that the observation is between two clusters.
The silhouette plot below gives us evidence that our clustering using four groups is good because there’s no negative silhouette width and most of the values are bigger than 0.5.
##   cluster size ave.sil.width## 1       1   10          0.65## 2       2    2          0.76## 3       3    7          0.58## 4       4    6          0.49
The following plot shows the final result of our clustering. The actual plot is interactive, but the image below is not. You can reproduce the plot using the code below. In the interactive plot, you may isolate the groups to better understand each one individually.
The purpose of clustering analysis is to identify patterns in the data. As we can see in the plot above, observations within the same group tend to have similar characteristics.
Let’s take the green group as an instance to evaluate. The two mammal species that belong to that group, namely seal and dolphin, they have the lowest percentage of water (44.9% and 46.4%); they both have around 10% of protein in their milk; they have the highest percentage of fat in the milk among all other species as well as the lowest percentage of lactose. This is the pattern found that puts seals and dolphins together in the same group. We can identify such patterns in the other groups as well.
Thank you for reading. I hope it was a pleasurable and useful reading.