Perform K-Means Clustering in R
By: Jinhang Jiang
Tags: R, K Means, Unsupervised Learning, Data Science, Machine Learning

Plot by author

Introduction

K-means clustering is one of the most popular unsupervised learning methods in machine learning. This algorithm helps identify “k” possible groups (clusters) from “n” elements based on the distance between the elements.

A more detailed explanation would be: the algorithm finds out the distance among each element in your data, then find the number of centroids, allocate the element to the nearest centroids to form clusters, and the ultimate goal is to keep the size of each cluster as small as possible.

K-means can be used in multiple ways. For instance, customer segmentation, insurance fraud detection, document classification, etc.

One of the common questions regarding the K-means algorithm is if it can handle non-numeric data. The short answer is NO since the algorithm is using the distance between the observations. However, there are many algorithms out there that can help convert non-numeric features to numeric features, which will allow you to apply the K-means algorithm to your data. For example, the “node2vec” model, an algorithmic framework for representational learning on graphs, can convert the data to embeddings, which can later be used in K-means.

Another interesting fact is unlike most unsupervised learning algorithms that you have to make empirical decisions on tuning parameters (or keep trying different parameters until you reach a state where you are satisfied), you can somehow perform hyperparameters tuning on K-means. I will demonstrate how to do that in this blog.

Case Study

Loda Data

I used the “iris” dataset in R in this case study.

data("iris")

?iris

Edgar Anderson's Iris Data Description

This famous (Fisher's or Anderson's) iris data set gives the measurements in centimeters of the variables sepal length and width and petal length and width, respectively, for 50 flowers from each of 3 species of iris. The species are Iris setosa, versicolor, and virginica. Source

Fisher, R. A. (1936) The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7, Part II, 179–188.

The data were collected by Anderson, Edgar (1935). The irises of the Gaspe Peninsula, Bulletin of the American Iris Society, 59, 2–5. References

Becker, R. A., Chambers, J. M. and Wilks, A. R. (1988) The New S Language. Wadsworth & Brooks/Cole. (has iris3 as iris.)

Preview of the data

Before I performed the K-means algorithm, I first checked the labels to see how many clusters were presented in this dataset.

> levels(iris$Species)

[1] "setosa" "versicolor" "virginica"

Then I separated the labels from the original data. In this way, we can treat the data as new, unlabeled data for unsupervised learning purposes.

iris1 <- iris[,-5]

iris_label <- iris[,5]

Since all the variables in the iris data are numeric, I did not have to do any data preprocessing. I pretended that I never saw this iris data before, so I wondered what the best K for the model (hyperparameters tuning I mentioned at the beginning).

Find K

In this step, we would need to know two terms, “wss” and “elbow rule,” to help us find the best number of centroids.

WSS: The sum distance within the centroids. Since the K-means algorithm's goal is to keep the size of each cluster as small as possible, the small wss indicates that every data point is close to its nearest centroids, or say the model has returned good results.

Elbow rule/method: a heuristic used in determining the number of clusters in a dataset. You first plot out the wss score against the number of K. Because with the number of K increasing, the wss will always decrease; however, the magnitude of decrease between each k will be diminishing, and the plot will be a curve which looks like an arm that curled up. In this way, you secondly need to find out which point falls on the elbow.

# to reproduce the results

# you have to use set.seed()

set.seed(1)

wss<- NULL

for (i in 1:10){

fit = kmeans(iris1,centers = i)

wss = c(wss, fit$tot.withinss)

} plot(1:10, wss, type = "o")

WSS vs. K

Read the Plot and Apply the Elbow Method

Based on the graph above, we can tell that k = 3 probably will be the answer. And if you recall, the original data indeed has three different species. So far, we are doing a good job! However, you have to keep in mind that the data tends to be neither perfect nor labeled in the real world. And the relationship among the groups is way more complicated than what we have here. So you might discover multiple K that could meet your needs. Then, you will have to try them out one by one, study the groups, and justify your answer by doing more in-depth research.

Fit & Plot

Then, I fitted a K-means model with k = 3 and plotted the clusters with the “fpc” package.

fit <- kmeans(iris[,-5], 3,) library(fpc)

plotcluster(iris[,-5],fit$cluster,pointsbyclvecd=FALSE)

Clusters Plot with k = 3

Note: When pointsbyclvecd = TRUE, the graph's data points will be represented by more reasonable symbols like numbers of letters. When the parameter is set to FALSE, the symbols will become less “reasonable,” like what the featured image of this blog displays.

Evaluation

When you apply the K-means algorithm in R, the function will help you generate multiple statistics of the model simultaneously, including TSS, BSS, and what we have discussed above— WSS.

TSS: It stands for Total-Sum-of-Squares, the total distance of data points from the global mean of the data.

BSS: It is the sum of each cluster's distance to the global mean of the data.

And, TSS = BSS + WSS.

Here, we can use another method to evaluate the model you fit. Since I have repeatedly mentioned that we want to keep the WSS as small as possible, therefore, theoretically, a high ratio of BSS to TSS is what we are looking for.

> fit$betweenss/fit$totss

[1] 0.8842753

If you want to test the accuracy of your model, here is how I did:

# First, relabel the data with the cluster number iris$cluster = fit$cluster for (i in 1:length(iris$Species)){

if (iris$cluster[i] == 1){

iris$label[i] = "setosa"

} else if (iris$cluster[i] == 3){

iris$label[i] = "versicolor"

} else {

iris$label[i] = "virginica"

}

} # Second, calculate the accuracy score

> mean(iris$label ==iris$Species)

[1] 0.8933333

Sometimes, if you have multiple groups, it would be hard to relabel the data properly with the for loop. Or you just want to explore the data from the same cluster since your original data is not labeled. You may subsample them based on the cluster numbers. I think this is also a universal way to handle your data. You may modify the code according to your needs.

subsample <- list()

for(i in 1:3){

subsample[[i]]<- iris[fit$cluster==i,]

} > table(subsample[[1]]$Species)

setosa versicolor virginica

50 0 0 > table(subsample[[2]]$Species)

setosa versicolor virginica

0 2 36 > table(subsample[[3]]$Species)

setosa versicolor virginica

0 48 14 > 134/150

[1] 0.8933333

Conclusion

As one of the most popular unsupervised learning algorithms, K-means can help us study and discover the complicated relationship, which will rather likely be ignored if we observe by eyes only, among the unlabeled data.

In this blog, I’ve discussed fitting a K-means model in R, finding the best K, and evaluating the model. And I’ve talked about calculating the accuracy score for the labeled data as well.

Related Blogs

Use R to Calculate Boilerplate for Accounting Analysis

Linear Regression Analysis in R

Analyzing Disease Co-occurrence Using NetworkX, Gephi, and Node2Vec