Cluster analysis: theory and implementation of unsupervised algorithms | by Mahbubul Alam | Towards Data Science

In simple terms, clustering is nothing but separating observations based on certain properties. In a more technical term, clustering is an unsupervised machine learning algorithm, a process, by which observations (data) are grouped in a way that similar observations are put closer to each other. It is an “unsupervised” algorithm because unlike supervised algorithms (e.g. random forest) you do not have to train it with labeled data, and instead, you put your data into a “clustering machine” along with some instructions (e.g. # of clusters you want), and the machine will figure out the rest and cluster the data based on the underlying patterns and properties.
Lee RCT (1981):
Clustering analysis is a newly developed computer-oriented data analysis technique. It is a product of many research fields: statistics, computer science, operations research, and pattern recognition.
The purpose of this article is to highlight some industry applications as well as discuss pros and cons of most frequently used clustering algorithms. In the second part I will demonstrate an implementation of K-means clustering as an example in Python environment. In the end I’ll leave some additional technical notes for practicing data scientists.
Why is clustering so popular in statistics and machine learning fields? This is because cluster analysis is a powerful data mining tool in a wide range of business application cases. Here are just a few of many applications:
Exploratory data analysis (EDA): Clustering is part of the most basic data analysis techniques employed in understanding and interpreting data and developing initial intuition about the features and patterns in data.
Statistics analysis: Frequently used in identifying the (dis)similarities of the same variable in different samples (e.g. math scores among children in city A vs city B).
Urban planning: Clustering helps identify households and communities of similar characteristics to implement appropriate community development policies.
Anomaly detection: Insurance industries use clustering to identify anomalous and potentially fraudulent transactions.
Customer segmentation: Clustering is widely used in developing marketing strategies, for example, in targeting different categories of customers for different kinds of promotions.
Computer vision: In computer vision/image segmentation clustering is used to partition data into disjoint groups based on pattern recognition process.
In Biology: Clustering is an essential tool in genetic and, taxonomic classification and understanding the evolution of living and extinct organisms.
And many others: Clustering has a wide range of other applications such as building recommendation systems, social media network analysis, spatial analysis in land use classification etc.
There are several variants of clustering algorithms family: K-means, hierarchical, DBSCAN, spectral, gaussian, birch, mean shift and affinity propagation are some of them. Below I am highlighting some key points on the first three algorithms— the most commonly applied ones.
K-means: First, “K” refers to the number of clusters you want. That is, K = n means n number of clusters to be identified. Then there’s something called “centroid”, which is an imaginary/artificial data point (an average of data points) around which each cluster of data is partitioned. So K = 2 means that the algorithm will partition the observations (data) into 2 clusters such that the distances between the centroids and observations are minimized.
Advantages: simple to understand, easy to implement
Disadvantages: sometimes difficult to choose the K; outliers can drag the centroid in their direction; scaling data can change the clusters
Hierarchical clustering: Hierarchical clustering works in two different ways: the first one is called a “bottom-up” or agglomerative clustering, where each observation gets its own cluster, then each pair of clusters are merged together to form another cluster, and so on. The other one (a.k.a. “top-down” or divisive clustering) works in the opposite direction, i.e., all observations start with one cluster, then repeatedly divided into smaller cluster sizes.
Advantages: easy to implement; number of clusters is easy to identify by looking at the dendrogram; more informative than K-means clustering
Disadvantages: highly sensitive to outliers; can be time consuming for large datasets
DBSCAN: Proposed in 1996, it is a density-based algorithm, where observations are clustered based on how close they are to each other given a minimum number of points. It takes two parameters: (i) ε (epsilon) — determining the radius within which the points should be in one cluster; and (ii) minPts — specifying a minimum number of points to form a dense space/cluster. Interesting enough, the 1996 paper that proposed this algorithm won the “ Test of Time Award” in the 2014 KDD conference.
Advantages: unlike K-means and hierarchical clustering, DBSCAN is robust in the presence of outliers; thus can be used in anomaly (i.e. outliers) detection.
Disadvantages: it is sensitive to parameter values (ε and minPts); fails to identify any clusters appropriately in varying data density.
The overall process for implementing a clustering algorithm is as simple as it gets since there is lesser human decision-making in the process and parameter tuning compared to other supervised machine learning algorithms. In this section, I am demonstrating a K-means clustering implementation in Python environment using sklearn library.
Step 1. Install dependencies
Basically you need three libraries: pandas to work with data, seaborn for visualization, sklearn for input pre-processing and modeling.
# to import and work with dataimport pandas as pd # to visualize data and resultsimport seaborn as snsimport matplotlib.pyplot as plt # to pre-process model inputsfrom sklearn import preprocessing # clustering algorithmfrom sklearn.cluster import KMeans
Step 2. Data
The data that I am using for this demo is the famous iris dataset. I have chosen this dataset because the clusters can be easily/visually separated in a scatter plot.
In many instances, after importing data you may need to do some processing such as encoding categorical variables etc. Also need to make sure there’s no NaN values.
# import datadf = pd.read_csv("iris.csv")df.head()
# select datadf = df[["petal_length", "petal_width"]]# see if there's NA valuesdf.isna().sum()
Step 3. Prepare model inputs
After you have chosen the dataset, next up pre-processing/formatting input in a way that the model can use it. Two things happen in this phase: normalization of selected features (to reduce variability in the data) and converting the data frame into numpy arrays.
# data normalizationdf = preprocessing.scale(df)df = pd.DataFrame(df)# inputimport numpy as npX = df.iloc[:, [0,1]].values
Step 4. Determine number of clusters
In K-means algorithm you need to define the number of clusters you want. The so-called “elbow method” can help determine that by minimizing the sum of squared errors.
########################### The "elbow" method ###########################k_range = range(1,10)sse = []for k in k_range:    km = KMeans(n_clusters = k)    km.fit(X)    sse.append(km.inertia_)plt.xlabel("K")plt.ylabel("Sum of squared errors")plt.plot(k_rng, sse, marker='o')
Step 5. Model implementation
Once you have made a determination on the only required parameter in the previous step, you are good to fit the model, visualize the number of clusters in a two-dimensional plot and do further analysis to answer the research question you are looking for.
# modelkm = KMeans(n_clusters = 3)y_km=km.fit_predict(X)# plot the 3 clustersplt.scatter(    X[y_km == 0, 0], X[y_km == 0, 1],    s=50, c='lightgreen',    marker='s', edgecolor='black',    label='cluster 1')plt.scatter(    X[y_km == 1, 0], X[y_km == 1, 1],    s=50, c='orange',    marker='o', edgecolor='black',    label='cluster 2')plt.scatter(    X[y_km == 2, 0], X[y_km == 2, 1],    s=50, c='lightblue',    marker='v', edgecolor='black',    label='cluster 3')
# value counts in different clustersframe = pd.DataFrame(X)frame['cluster'] = y_kmframe['cluster'].value_counts()
Just like other machine learning algorithms if there are categorical features, they need to be encoded in numeric features.It needs to be reiterated that the clustering algorithms are sensitive to scale since the algorithm uses Euclidean distance. So make sure that input data is pre-processed appropriately.Notice that unlike supervised classification implementations, where a model is fitted first and then used to predict, in clustering model fitting and predicting happens altogether (hence the use of fit_predict())Check out this StackOverflow page if you are interested in what works and what doesn’t in K-means implementation.If you are into rstat you can check out this UC Business Analytics page that has codes as well as some useful discussion.
Just like other machine learning algorithms if there are categorical features, they need to be encoded in numeric features.
It needs to be reiterated that the clustering algorithms are sensitive to scale since the algorithm uses Euclidean distance. So make sure that input data is pre-processed appropriately.
Notice that unlike supervised classification implementations, where a model is fitted first and then used to predict, in clustering model fitting and predicting happens altogether (hence the use of fit_predict())
Check out this StackOverflow page if you are interested in what works and what doesn’t in K-means implementation.
If you are into rstat you can check out this UC Business Analytics page that has codes as well as some useful discussion.