Unsupervised Learning Techniques using Python — K Means ++ and Silhouette Score for Clustering | by Angel Das | Towards Data Science

For those of you who are wondering what an unsupervised learning technique is, imagine you run a sports shop in Clayton (Go Melbourne Victory FC). You plan to introduce a promotional offer for your high valued customers as a token of appreciation. You have hundreds of customers visiting you every day. Who should you send this offer to? Tough choice! Hmm, maybe high valued ones? But what about those who purchased frequently but didn’t spend much. This is a big dilemma. Luckily your loyalty card program captures your customer’s information and your oracle database includes all transaction details. You can use this data, and form a group of customers with similar attributes without having a pre-defined notion of who belongs where. In short, a learning technique that doesn’t have a pre-defined outcome variable (in this case your customer segments) but forms one after studying the customer attributes is defined as an unsupervised learning technique.
Now theoretically, the more information you have about your customer, the better you can group them. Isn’t that true? However, in real life, it’s a little more complicated than you think. In practice, too many information (features) leads to a poor grouping of your customers. But why?
Features can be highly correlated — Higher the age, lower is the spend
Too many features can introduce a lot of noise making it difficult for algorithms to perform well on the data — Sam, your neighborhood customer dropped by almost every day to say “Hi” and purchase a protein bar worth a dollar. He is a highly valued customer?
The number of data points you need to train any unsupervised model increases exponentially with dimensionality (features). Too many information means you require more data to train your model
To overcome such issues different dimensionality reduction techniques like Principal Component Analysis, Factor Analysis, etc. can be used. We introduced the concept of Curse of Dimensionality to pave a pathway for assumptions surrounding unsupervised learning techniques.
Some of the common algorithms of unsupervised learning technique include:
K Means ClusteringHierarchical ClusteringK Mode ClusteringDBScan Clustering — Density-Based Spatial ClusteringPrincipal Component AnalysisFactor Analysis
K Means Clustering
Hierarchical Clustering
K Mode Clustering
DBScan Clustering — Density-Based Spatial Clustering
Principal Component Analysis
Factor Analysis
It is common for people to think of reality as a series of black and white events. This is a challenging issue as those suffering from this way of thinking may never feel that reality is good enough. Today, we try and address the problem of simplicity and depression while determining the relationship between higher levels of black and white thinking and higher levels of self-reported depression in psychiatric patients hospitalized for depression.
The data used for this analysis is from the Ginzberg data frame which is based on psychiatric patients hospitalized for depression. Data is collected from the book “Applied Regression Analysis and Generalized Linear Models”, Second Edition by Fox, J. (2008). The dataset includes three variables — simplicity (black and white thinking), fatalism, and depression ad their adjusted scores.
To break it down, K signifies the number of groups, and Means signifies average. Essentially we have K groups based on an average distance calculation. Not clear I guess! For any K Means clustering algorithm to work, we need to have the following parameters.
K — How many groups we want. Note — The algorithm doesn’t have the predicting capability to decide or come up with a value for K. It’s a user’s call guided by few techniques like Elbow Plot and Silhouette ScoreData points — In this case Simplicity, Fatalism, and Depression scores, i.e. your input dataK Centroids — Think of it like anchor/center values for each of the K groups. Selected randomly by the algorithm mostly. The selection of centroid impacts the way the final clusters/groups are formed. Also if you are familiar with the time complexity of any algorithm represented using O(n), it has been observed that suboptimal allocation of centroids makes the time complexity polynomial, i.e. larger time for the algorithm to run and converge. Hence a smarter way was proposed by David Arthur and Sergei Vassilvitskii (also known as the K Means ++) that does the following:- Choose one center C1 randomly from the input data points ( say X)- For each data point x ∈ X not chosen yet, compute D(x), i.e distance between x and the nearest centroid that has already been selected- For all x ∈ X calculate D(x)2 and sum them up; Select a new center Ci, selecting x ∈ X such that it has a probability of D(x)2 / Sum(For all x ∈ X D(x)2)- Repeat Steps 2 and 3 until k centers have been chosenDistance Measure— Mostly Euclidean distance as a measure of similarity between multiple patients. E.g. when we talk about soccer or popularly known as football, we often use total goals scored to group quality players. Say Ronaldo, Messi, Werner, and Puki scored 35, 32, 10, and 8 goals respectively. How will you measure similar players in terms of quality? Probably Ronaldo and Messi (35, 32 goals each) and Werner and Puki (10, 8 goals each). Notice we look into the difference of goals scored. If I add say “Assists” as another metric, we need to use a different measure. E.g. Assists are 10, 8, 2, and 1 respectively. So the new distance between Ronaldo and Messi becomes square root((35–32)2 + (10–8)2). Note — Other distance metrics like Manhattan, City block, Jacobi’s distance, etc. are also adopted by few of the other algorithms.
K — How many groups we want. Note — The algorithm doesn’t have the predicting capability to decide or come up with a value for K. It’s a user’s call guided by few techniques like Elbow Plot and Silhouette Score
Data points — In this case Simplicity, Fatalism, and Depression scores, i.e. your input data
K Centroids — Think of it like anchor/center values for each of the K groups. Selected randomly by the algorithm mostly. The selection of centroid impacts the way the final clusters/groups are formed. Also if you are familiar with the time complexity of any algorithm represented using O(n), it has been observed that suboptimal allocation of centroids makes the time complexity polynomial, i.e. larger time for the algorithm to run and converge. Hence a smarter way was proposed by David Arthur and Sergei Vassilvitskii (also known as the K Means ++) that does the following:- Choose one center C1 randomly from the input data points ( say X)- For each data point x ∈ X not chosen yet, compute D(x), i.e distance between x and the nearest centroid that has already been selected- For all x ∈ X calculate D(x)2 and sum them up; Select a new center Ci, selecting x ∈ X such that it has a probability of D(x)2 / Sum(For all x ∈ X D(x)2)- Repeat Steps 2 and 3 until k centers have been chosen
Distance Measure— Mostly Euclidean distance as a measure of similarity between multiple patients. E.g. when we talk about soccer or popularly known as football, we often use total goals scored to group quality players. Say Ronaldo, Messi, Werner, and Puki scored 35, 32, 10, and 8 goals respectively. How will you measure similar players in terms of quality? Probably Ronaldo and Messi (35, 32 goals each) and Werner and Puki (10, 8 goals each). Notice we look into the difference of goals scored. If I add say “Assists” as another metric, we need to use a different measure. E.g. Assists are 10, 8, 2, and 1 respectively. So the new distance between Ronaldo and Messi becomes square root((35–32)2 + (10–8)2). Note — Other distance metrics like Manhattan, City block, Jacobi’s distance, etc. are also adopted by few of the other algorithms.
Note: All images are reproduced using latex by the author.
Centroids
1. Ci - ith Centroid where i ∈ K (or K Groups of Cluster)2. Si - Group of Data points that belong to Cluster K with Ci as its centroid; j represents attributes of a point Xi3. ||Si|| - Number of data points from Si
Euclidean Distance
1. Dx is the distance of of a point Xi from it's centroid Ck
Convergence CriterionK Means is an iterative process. Once all data points are assigned to its centroid (based on the shortest Euclidean distance), the centroids are recalculated and this process continues till the centroids stop re-shifting, i.e. The new and previous centroid numbers didn’t change significantly
Intra and Inter-Cluster DistanceIntra Cluster distance is the total distance between points from the same cluster and Inter-Cluster is the distance between two points in the different clusters
K Means algorithm performs poorly if data contains outliersK Means algorithm can’t handle missing valuesClustering variables shouldn’t be highly correlated to each otherAll clustering variables should be standardized as the magnitude of variables impact distance calculation
K Means algorithm performs poorly if data contains outliers
K Means algorithm can’t handle missing values
Clustering variables shouldn’t be highly correlated to each other
All clustering variables should be standardized as the magnitude of variables impact distance calculation
# Numerical librariesimport numpy as npfrom sklearn.model_selection import train_test_splitfrom sklearn.cluster import KMeans# to handle data in form of rows and columns import pandas as pd# importing ploting librariesfrom matplotlib import pyplot as plt%matplotlib inline#importing seaborn for statistical plotsimport seaborn as snsfrom sklearn import metricsimport pandas as pd# reading the CSV file into pandas dataframemydata = pd.read_csv(“Depression.csv”)mydata.drop('id', axis=1, inplace=True)
# — — — — — — — — — — — — — — — -Heat map to identify highly correlated variables — — — — — — — — — — — — -#-------------------------------Heat map to identify highly correlated variables-------------------------plt.figure(figsize=(10,8))sns.heatmap(mydata.corr(),            annot=True,            linewidths=.5,            center=0,            cbar=False,            cmap="YlGnBu")plt.show()
mydata.drop(columns = {‘simplicity’, ‘fatalism’,’depression’}, inplace=True)#--Checking Outliersplt.figure(figsize=(15,10))pos = 1for i in mydata.columns:    plt.subplot(3, 3, pos)    sns.boxplot(mydata[i])    pos += 1
col_names=list(mydata.columns)display(col_names)for i in col_names: q1, q2, q3 = mydata[i].quantile([0.25,0.5,0.75]) IQR = q3 — q1 lower_cap=q1–1.5*IQR upper_cap=q3+1.5*IQR mydata[i]=mydata[i].apply(lambda x: upper_cap if x>(upper_cap) else (lower_cap if x<(lower_cap) else x))
##Scale the datafrom scipy.stats import zscoremydata_z = mydata.apply(zscore)mydata_z.head()
# List to store cluster and intra cluster distanceclusters = []inertia_vals = []# Since creating one cluster is similar to observing the data as a whole, multiple values of K are utilized to come up with the optimum cluster value#Note: Cluster number and intra cluster distance is appended for plotting the elbow curvefor k in range(1, 10, 1):        # train clustering with the specified K    model = KMeans(n_clusters=k, random_state=7, n_jobs=10)    model.fit(mydata_z)# append model to cluster list    clusters.append(model)    inertia_vals.append(model.inertia_)
n_clusters — The number of clusters to form; Also equivalent to the number of centroids to be generatedinit — Method specifying the initial centroid initialization (default K-means++)n_init — Number of times the k-means algorithm will be run with different centroid seedsmax_iter — Maximum number of iterations of the k-means algorithm for a single run. Stops if a convergence criterion is met, defined using the tol (tolerance option)random_state — random number generation assisting centroid initialization. Import to ensure a reproducible result
n_clusters — The number of clusters to form; Also equivalent to the number of centroids to be generated
init — Method specifying the initial centroid initialization (default K-means++)
n_init — Number of times the k-means algorithm will be run with different centroid seeds
max_iter — Maximum number of iterations of the k-means algorithm for a single run. Stops if a convergence criterion is met, defined using the tol (tolerance option)
random_state — random number generation assisting centroid initialization. Import to ensure a reproducible result
Inertia is defined as the sum of squared distances of data points to their closest cluster center (centroid). Lower the distance better the compactness of the clusters. However, inertia decreases as we keep increasing the value of K. An algorithm with two clusters will always have a higher inertia score than the one with four clusters and so on. This is where we combine elbow plot and silhouette score to decide the optimum value of K.
The idea here is to plot the total inertia score, for different values of K and identify K beyond which the inertia doesn’t change much. The formula for inertia is given below.
1. I = Inertial = Sum of the squared distance of data points to their closest cluster2. n = Total number of records in the dataset3. D()= Sum of squared distance4. Xi = Record i in the data (i.e. in this case record of a given patient)5. Ck = Cluster to which record Xi is assigned
# plot the inertia vs K valuesplt.plot(range(1,10,1),inertia_vals,marker='*')plt.show()
Studying the graph above reveals 4, 5, or 6 as the optimum value of K. Notice from 6 to 7 the curve tends to flatten out whereas the slope doesn’t observe a significant change post 4 clusters.
The Silhouette score is used to measure the degree of separation between clusters. In the formula above bi represents the shortest mean distance between a point to all points in any other cluster of which i is not a part whereas ai is the mean distance of i and all data points from the same cluster. Logically if bi > ai, then a point is well separated from its neighboring cluster whereas it is closer to all points from the cluster it belongs to.
from sklearn.metrics import silhouette_scorefor i in range(1,9,1):        print("---------------------------------------")    print(clusters[i])    print("Silhouette score:",silhouette_score(mydata_z, clusters[i].predict(mydata_z)))
The Output
KMeans(n_clusters=2, n_jobs=10, random_state=7)Silhouette score: 0.40099183297222984---------------------------------------KMeans(n_clusters=3, n_jobs=10, random_state=7)Silhouette score: 0.3191854112351335---------------------------------------KMeans(n_clusters=4, n_jobs=10, random_state=7)Silhouette score: 0.281779515317617---------------------------------------KMeans(n_clusters=5, n_jobs=10, random_state=7)Silhouette score: 0.2742499085300452---------------------------------------KMeans(n_clusters=6, n_jobs=10, random_state=7)Silhouette score: 0.2790023172083168---------------------------------------KMeans(n_clusters=7, n_jobs=10, random_state=7)Silhouette score: 0.2761554255999077---------------------------------------KMeans(n_jobs=10, random_state=7)Silhouette score: 0.2666698045792603---------------------------------------KMeans(n_clusters=9, n_jobs=10, random_state=7)Silhouette score: 0.27378613906697535
Silhouette score observing the local maxima is used to determine the optimum number of clusters along with the Elbow plot. Hence in this example we can go ahead with 6 as the optimal number of clusters.
About the Author: Advanced analytics professional and management consultant helping companies find solutions for diverse problems through a mix of business, technology, and math on organizational data. A Data Science enthusiast, here to share, learn and contribute; You can connect with me on Linked and Twitter;