Clustering Techniques. Clustering falls under the unsupervised... | by M Bharathwaj | Towards Data Science

Clustering falls under the unsupervised learning technique. In this technique, the data is not labelled and there is no defined dependant variable. This type of learning is usually done to identify patterns in the data and/or to group similar data.
In this post, a detailed explanation on the type of clustering techniques and a code walk-through is provided.
Clustering is a method of grouping of similar objects. The objective of clustering is to create homogeneous groups out of heterogeneous observations. The assumption is that the data comes from multiple population, for example, there could be people from different walks of life requesting loan from a bank for different purposes. If the person is a student, he/she could ask for an education loan, someone who is looking to buy a house can ask for home loan and so on. Clustering helps to identify similar group and cater the needs better.
Clustering is a distance-based algorithm. The purpose of clustering is to minimize the intra-cluster distance and maximize the inter-cluster distance.
Clustering as a tool can be used to gain insight into the data. Huge amount of information can be obtained by visualizing the data. The output of the clustering can also be used as a pre-processing step for other algorithms. There are several use cases of this technique that is used widely — some of the important ones are market segmentation, customer segmentation, image processing.
Before proceeding further, let us understand the core of clustering.
Clustering is all about distance between two points and distance between two clusters. Distance cannot be negative. There are a few common measures of distance that the algorithm uses for the clustering problem.
EUCLIDEAN DISTANCE
It is a default distance used by the algorithm. It is best explained as the distance between two points. If the distance between two points p and q are to be measured, then the Euclidean distance is
MANHATTAN DISTANCE
It is the distance between two points calculated on a perpendicular angle along the axes. It is also called taxicab distance as this represents how vehicles in the city of Manhattan drive where the streets intersect at right angles.
MINKOWSKI DISTANCE
In an n-dimensional space, the distance between two points is called Minkowski distance.
It is a generalization of the Euclidean and Manhattan distance that if the value of p is 2, it becomes Euclidean distance and if the value of p is 1, it becomes Manhattan distance.
There are two major types of clustering techniques
Hierarchical or Agglomerativek-means
Hierarchical or Agglomerative
k-means
Let us look at each type along with code walk-through
It is a bottom-up approach. Records in the data set are grouped sequentially to form clusters based on distance between the records and also the distance between the clusters. Here is a step-wise approach to this method -
Start with n clusters where each row is considered as a clusterUsing distance based approach, two records that are closest to each other are merged into a cluster. In Fig 3, for the given five records, assuming A and C are closest in distance, they form a cluster and likewise B and E form another cluster and so on
Start with n clusters where each row is considered as a cluster
Using distance based approach, two records that are closest to each other are merged into a cluster. In Fig 3, for the given five records, assuming A and C are closest in distance, they form a cluster and likewise B and E form another cluster and so on
3. At every step, two closest clusters will be merged. Either a single record(singleton) is added to the existing cluster or two clusters are combined. After at least one multiple-element cluster is formed, a scenario where the distance needs to be computed for a singleton and a set of observations and that is where the concept of linkages comes into picture. There are five major types of linkages. By using one of the below concepts, the clustering happens-
Single linkage: It is the shortest distance between any two points in both the clusters
Complete linkage: It is the opposite of single linkage. It is the longest distance between any two points in both the clusters
Average linkage: It is the average distance between each point in one cluster to every point in the other cluster
Centroid linkage: The distance between the center point in one cluster to the center point in the other cluster
Ward’s linkage: A combination of average and centroid methods. The within cluster variance is calculated by determining the center point of the cluster and the distance of the observations from the center. While trying to merge two clusters, the variance is found between the clusters and the clusters are merged whose variance is less compared to the other combination.
A point to note is that each linkage method produces an unique result. When each of these methods are applied on the same data set, it may be clustered differently.
4. Repeat the steps until there is a single cluster with all the records
To visualize the clustering, there is a concept called dendrogram. The dendrogram is a tree diagram summarizing the clustering process. The records are on the x-axis. Similar records are joined by lines whose vertical length reflects the distance between the records. The greater the difference in height, more the dissimilarity. A sample dendrogram is shown -
The code for hierarchical clustering is written in Python 3x using jupyter notebook. Let’s begin by importing the necessary libraries.
#Import the necessary librariesimport numpy as npimport pandas as pdimport seaborn as snsimport matplotlib.pyplot as plt%matplotlib inlinefrom scipy.cluster.hierarchy import linkage, dendrogram, fcluster
Next, load the data set. Here, a data set on the food menu of Starbucks is used.
#Read the datasetdf = pd.read_csv('starbucks_menu.csv')#Look at the top 5 rows df.head()
Do the necessary Exploratory Data Analysis like looking at the descriptive statistics, checking for null values, duplicate values. Perform uni-variate and bi-variate analysis, do outlier treatment(if any). Since this is a distance based algorithm, it is necessary to perform normalization wherever applicable so that all the variables are free of any units of measurement. This enables the model to perform at its best.
from scipy.stats import zscoredf.iloc[:,1:6] = df.iloc[:,1:6].apply(zscore)#Check the head after scalingdf.head()
Once the data is ready, let us work towards building the model. A label list needs to be assigned which is a list of unique value of categorical variable. Here, label list is created from the Food variable.
#Before clustering, setup label list from the food variablelabelList = list(df.Food.unique())labelList
Next step is to form a linkage to cluster a singleton and another cluster. In this case, ward’s method is preferred.
#Create linkage method using Ward's methodlink_method = linkage(df.iloc[:,1:6], method = 'ward')
Visualize the clustering with the help of a dendrogram. In this case, a truncated dendrogram by specifying the p value which displays the ante-penultimate and penultimate clusters.
#Generate the dendrogramdend = dendrogram(link_method,                  labels = labelList,                  truncate_mode='lastp',                   p=10)
Once the dendrogram is created, it is necessary to cut the tree to determine the optimum number of clusters. It can be done in one of two ways (explained in the diagram). In this case, 3 clusters are chosen. The clusters can be attached to the data frame as a new column for gaining insights.
#Method 1: criterion = 'maxclust' where a cut is defined based on the number of clustersclusters = fcluster(link_method, 3, criterion='maxclust') clusters#Method 2: criterion='distance' where a cut is defined based on distance in the y-axis#clusters = fcluster(link_method, 800, criterion='distance')#Apply the clusters back to the datasetdf['HCluster'] = clustersdf.head()
The last step is to do cluster profiling to extract information and insights from the algorithm to help with effective decision making. The cluster profiling is done by grouping the mean of the cluster and sorting based on the frequency.
aggdata=df.iloc[:,1:8].groupby('HCluster').mean()aggdata['Frequency']=df.HCluster.value_counts().sort_index()aggdata
A quick insight is that the first cluster has the food items that are usually low in calories and hence the macro nutrients are on the lower side. The second cluster has the food items with the most amount of calories and hence more in macro nutrients and there is a mid range in between cluster 1 and 2 which is the third cluster that has good amount of calories and macro nutrients. Overall, on a brief, this model has clustered well.
Let’s move on to the next method
K-Means is a non-hierarchical approach. The idea is to specify the number of clusters before hand. Based on the number of clusters, each record is assigned to the cluster based on the distance from each cluster. This approach is preferred when the data set is large. The word means in k-means refers to averaging of the data which is also referred to as finding the centroid. Here is a step-wise approach-
Specify the k value before handAssign each record to the cluster whose distance is smallest to the centroid. The K-Means by default uses Euclidean distanceRe-calculate the centroid for the newly formed clusters. There is a chance that some of the data points may move around based on the distance.Re-assignment may happen on an iterative basis and new centroid is formed. This process will stop until there is no jumping of observations from cluster to cluster.If there is any re-assignment, go back to step 3 and continue the steps. If not, the clusters are finalized.
Specify the k value before hand
Assign each record to the cluster whose distance is smallest to the centroid. The K-Means by default uses Euclidean distance
Re-calculate the centroid for the newly formed clusters. There is a chance that some of the data points may move around based on the distance.
Re-assignment may happen on an iterative basis and new centroid is formed. This process will stop until there is no jumping of observations from cluster to cluster.
If there is any re-assignment, go back to step 3 and continue the steps. If not, the clusters are finalized.
Although we have determined the number of clusters before hand, it may not be always right and it is necessary to determine the optimum number of clusters. There is no solid solution to determine the number of clusters, however, there is a common method in place. For each value of k, a Within Sum of Squares(WSS) value can be identified. A single cluster cannot be chosen so it is important to find that k value after which there is no significant difference in the WSS value. To make this more efficient, an elbow plot can be drawn with WSS scores in the y-axis and number of clusters in the x-axis to visualize the optimum number of clusters.
There is a way to understand how well the model has performed. It is done by checking two metrics, namely Silhouette Width and Silhouette Score. This helps us to analyse whether each and every observation mapped to the clusters is correct or not based on distance criteria. Silhoutte Width is calculated as
where b is the distance between the observation and the neighboring cluster’s centroid and a is the distance between the observation and the very own cluster’s centroid.
The Silhouette Width can have a value in the range of -1 to 1. If the value of Silhoutte Width is positive, then the mapping of the observation to the current cluster is correct. When a > b, Silhouette Width will return a negative value. The average of all the Silhouette Width is called the Silhouette Score. If the final score is a positive value and it is close to +1, the clusters are well separated on an average. If it is close to 0, it is not separated well enough. If it is negative value, the model has done a blunder in clustering.
Let’s begin by importing the necessary libraries
#Import the necessary librariesimport numpy as npimport pandas as pdimport seaborn as snsimport matplotlib.pyplot as plt%matplotlib inlinefrom sklearn.cluster import KMeansfrom sklearn.metrics import silhouette_score
Next, load the data set. The same data set used for Hierarchical clustering is used here.
Do the necessary Exploratory Data Analysis like looking at the descriptive statistics, checking for null values, duplicate values. Perform uni-variate and bi-variate analysis, do outlier treatment(if any). K-means clustering demands scaling. It is done so that all the variables are free of any units of measurement. This enables the model to perform at its best. In this case, StandardScaler method is put to use.
#Importing the standard scaler module and applying it on continuous variablesfrom sklearn.preprocessing import StandardScaler X = StandardScaler()scaled_df = X.fit_transform(df.iloc[:,1:6])scaled_df
Next is to invoke the KMeans method with defining the number of clusters before hand. Then fit the scaled data set to the model.
# Create K Means cluster and store the result in the object k_meansk_means = KMeans(n_clusters=2)# Fit K means on the scaled_dfk_means.fit(scaled_df)# Get the labelsk_means.labels_
Now is the time to find the optimum number of clusters by analyzing the values of Within Sum of Squares(WSS) for a given range of k.
#To determine the optimum number of clusters, check the wss score for a given range of kwss =[] for i in range(1,11):    KM = KMeans(n_clusters=i)    KM.fit(scaled_df)    wss.append(KM.inertia_)    wss
It is seen that there is a dip in WSS score after k=2 hence let’s keep an eye on k=3. The same can be visualized using an elbow plot
#Draw the elbow plotplt.plot(range(1,11), wss, marker = '*')
Another helping hand in deciding the number of clusters could be the value of Silhouette score. As discussed before, better the score, better the clustering. Let’s check the score.
#Checking for n-clusters=3k_means_three = KMeans(n_clusters = 3)k_means_three.fit(scaled_df)print('WSS for K=3:', k_means_three.inertia_)labels_three = k_means_three.labels_print(labels_three)#Calculating silhouette_score for k=3print(silhouette_score(scaled_df, labels_three))
The WSS for k=3 is 261.67 and the Silhouette score for those labels is 0.3054. Since the score is positive, it is a sign that good clustering has happened.
The final step is to do cluster profiling to understand how the cluster has happened and gain more insights.
clust_profile=df.iloc[:,1:8].groupby('KMCluster').mean()clust_profile['KMFrequency']=df.KMCluster.value_counts().sort_index()clust_profile
Just like Hierarchical clustering, these three clusters indicate three levels of foods with different calorie and macro nutrients range.
Data set source: https://www.kaggle.com/starbucks/starbucks-menu?select=starbucks-menu-nutrition-food.csv
To view the jupyter notebook files, click here