Explaining K-Means Clustering. Comparing PCA and t-SNE dimensionality... | by Kamil Mysiak | Towards Data Science

Today’s data comes in all shapes and sizes. NLP data encompasses the written word, time-series data tracks sequential data movement over time (ie. stocks), structured data which allows computers to learn by example, and unclassified data allows the computer to apply structure. Whichever dataset you possess, you can be sure there is an algorithm ready to decipher its secrets. In this article, we want to cover a clustering algorithm named KMeans which attempts to uncover hidden subgroups hiding in your dataset. Furthermore, we will examine what effects dimension reduction has on the quality of the clusters obtained from KMeans.
In our example, we will be examining a human resources dataset consisting of 15,000 individual employees. The dataset contains employee job characteristics such as job satisfaction, performance score, workload, years of tenure, accidents, number of promotions. We will apply KMeans in order to uncover similar groups of employees.
Classification problems will have target labels which we are trying to predict. Famous datasets such as Titanic and Iris are both prime for classification as they both have targets we are trying to predict (ie. survived and species). Furthermore, classification tasks require us to split our data into training and test, where the classifier is trained on the training data and then its performance is measured via the test dataset.
When dealing with a clustering problem, we want to use an algorithm to uncover meaningful groups in our data. Perhaps we are trying to uncover customer segments or identify anomalies in our data. Either way, the algorithm uncovers the groups with little human intervention as we don’t have target labels to predict.
That said, we can partner unsupervised clustering algorithms with supervised algorithms by first identifying groups/clusters and then building supervised models to predict the cluster membership.
At its core, KMeans attempts to organize the data into a specified number of clusters. Unfortunately, it is up to us to determine the number of clusters we wish to find but fear not we have tools at our disposal that assist with this process. The goal of KMeans is to identify similar data points and cluster them together while trying to distance each cluster as far as possible. Its “similarity” calculation is determined via Euclidean distance or an ordinary straight line between two points (Wiki). The shorter the Euclidean distance the more similar the points are.
First, the user (ie. you or I) determines the number of clusters KMeans needs to find. The number of clusters cannot exceed the number of features in the dataset. Next, KMeans will select a random point for each centroid. The number of centroids is equal to the number of clusters selected. The centroid is the point around which each cluster is built around.
Second, the Euclidean distance is calculated between each point and each centroid. Each point will be initially assigned to the closest centroid/cluster based on the Euclidean distance. Each data point can belong to one cluster or centroid. The algorithm then averages Euclidean distance (between each point and centroid) for each cluster and this point becomes the new centroid. This process of averaging the Euclidean distances within clusters and assigning new centroids repeats until cluster centroids no longer move. The animation below shows the process, refresh the page if needed.
We need to be aware of how KMeans selects the initial centroid/s and what problems might this produce. Without our intervention, KMeans will randomly select the initial centroid/s which ultimately can cause different resulting clusters on the same data. From the plots below we can see how running the KMeans algorithm on two different occasions resulted in different initial centroids. The same data points were assigned to different clusters between the first and second time running KMeans.
Have no fear Sklearn has our backs! The Sklearn KMeans library has certain parameters such as “n_int” and “max_iter” to mitigate this issue. The “n_int” parameter determines the number of times KMeans will randomly select different centroids. “Max_iter” determines how many iterations will run. An iteration is the process of finding the distance, taking the average distance, and moving the centroid.
If we set our parameters at n_int=25 and max_iter=200 KMeans will randomly select 25 initial centroids and run each centroid up to 200 iterations. The best out of those 25 centroids will be the final cluster.
The Sklearn also has an “int” parameter which will select the first centroid at random and locate all the data points which are furthest away from the first centroid. Then, the second centroid is assigned nearby those far points as they are less likely to belong to the first centroid. Sklearn selects “int=kmeans++” by default which applies the above logic.
“You mentioned something about needing to select the number of clusters....? Just how do we do that?”
Domain Knowledge: Very often we have a certain level of knowledge and experience in the domain from which our dataset was gathered. This expertise can allow us to set the number of clusters we believe exists in the general population.
Hypothesis Testing: Setting a specific number of clusters can also act as a test of a certain hypothesis we might have. For example, when analyzing marketing data we have a hunch there are 3 subgroups of customers who very likely, likely, and not likely to purchase our product.
Data Comes Pre-Labeled: There are times when the data we are analyzing comes with pre-labeled targets. These datasets are typically used for supervised ML problems but that doesn’t mean we cannot cluster the data. Pre-labeled data is unique as you need to remove the targets from your initial analysis and then use them to validate how well the model clustered the data.
Elbow Method: This is a very popular iterative statistical technique for determining the optimal number of clusters by actually running the K-Means algorithm for a range of cluster values. The elbow method calculates the sum of squared distances from each point to its assigned centroid for each iteration of KMeans. Each iteration runs through a different number of clusters. The result is a line chart that displays the sum of squared distances at each cluster. We want to select the number of clusters at the elbow of the line chart or the lowest sum of squared distances (ie. Inertia) at the lowest number of clusters. The lower the sum of squares distances means the data inside each cluster are more tightly grouped.
KMeans is very sensitive to scale and requires all features to be on the same scale. KMeans will put more weight or emphasis on features with larger variances and those features will impose more influence on the final cluster shape. For example, let’s consider a dataset of car information such as weight (lbs) and horsepower (hp). If there is a larger variation in weight between all the cars the average Euclidean distance will be more affected by weight. Ultimately, the membership of each cluster will be more affected by weight than horsepower.
Clustering algorithms such as KMeans have a difficult time accurately clustering data of high dimensionality (ie. too many features). Our dataset is not necessarily highly dimensional as it contains 7 features but even this amount will create issues for KMeans. I would suggest you explore the Curse of Dimensionality for more details. As we saw earlier, many clustering algorithms use a distance formula (ie. Euclidean distance) to determine cluster membership. When our clustering algorithm has too many dimensions, pairs of points will begin to have very similar distances and we wouldn’t be able to obtain meaningful clusters.
In this example, we are going to compare PCA and t-SNE data reduction techniques prior to running our K-Means clustering algorithm. Let’s take a few mins to explain PCA and t-SNE.
Principal component analysis or (PCA) is a classic method we can use to reduce high-dimensional data to a low-dimensional space. In other words, we simply cannot accurately visualize high-dimensional datasets because we cannot visualize anything above 3 features (1 feature=1D, 2 features = 2D, 3 features=3D plots). The main purpose behind PCA is to transform datasets with more than 3 features (high-dimensional) into typically a 2D or 3D plots for us feeble humans. That’s what is meant by low-dimensional space. The beauty behind PCA is the fact that despite the reduction into a lower-dimensional space we still retain most (+90%) of the variance or information from our original high-dimensional dataset. The information or variance from our original features is “squeezed” into what PCA calls principal components (PC). The first PC will contain the majority of the information from the original features. The second PC will contain the next largest amount of information, the 3rd PC the third largest amount of info and so on and so on. The PC are not correlated (ie. orthogonal) which means they all contain unique pieces of information. We can typically “squeeze” most (ie. 80–90%) of the information or variance contained in the original features into a few principal components. We use these principal components in our analyses instead of using the original features. This way we can perform an analysis with only 2 or 3 principal components instead of 50 features while still maintaining 80–90% of the information from our original features.
Let’s take a look at some of the details behind how PCA does its magic. To make the explanation a bit easier let’s see how we can reduce a dataset with 2 features (ie. 2D) in one principal component (1D). That said, reducing 50 features into 3 or 4 principal components utilizes the same methodology.
We have a 2D plot of weight and height. In other words, we have a dataset of 7 people and have plotted their height in relation to their weight.
First, PCA needs to center the data by measuring the distances from each point to the y-axis (height) and x-axis (weight). It then calculates the average distance for both axes (ie. height and weight) and centers the data using these averages.
Then PCA will plot the first principal component (PC1) or the best fitting line which maximizes the variance or the amount of information between weight and height. It determines the best fitting line by maximizing the distance from the point projected onto the best fitting line and the origin (ie. blue light below). It does it for each green point and then it squares each distance, to remove negative values, and sums everything up. The best-fitting line or PC1 will have the largest sum of squared distances from the origin to the projected points for all points.
Now, we simply rotate the axes to where PC1 is now our x-axis and we are finished.
If we wanted to reduce 3 features down to two principal components we would simply place a perpendicular (y-axis) to our best-fitting line in a 2D space. Why a perpendicular line? Because each principal component is orthogonal or uncorrelated with all other features. If we wanted to find a third principal component we would simply find another orthogonal line to PC1 and PC2 but this time in a 3D space.
As you can see from the bar plot below, we initially began with a dataset of 5 features. Remember the number of principal components will always equal the number of features. However, we can see that the first 2 principal components account for 90% of the variance or information contained in the original 5 features. This is how we determine the optimal number of principal components. It is important to remember PCA is very often used for visualizing very high dimensional data (ie. thousands of features), therefore, you will most often see PCA of 2 or 3 principal components.
Just like PCA, t-SNE takes high-dimensional data and reduces it to a low-dimensional graph (2-D typically). It is also a great dimensionality reduction technique. Unlike PCA, t-SNE can reduce dimensions with non-linear relationships. In other words, if our data had this “Swiss Roll” non-linear distribution where the change in X or Y does not correspond with a constant change in the other variable. PCA would not be able to accurately distill this data into principal components. This is because PCA would attempt to draw the best fitting line through the distribution. T-SNE would be a better solution in this case because it calculates a similarity measure based on the distance between points instead of trying to maximize variance.
Let’s examine how t-SNE converts high dimensional data space into lower dimensions. It looks at the similarity between local or nearby points by observing the distance (think Euclidean distance). Points that are nearby each other are considered similar. t-SNE then converts this similarity distance for each pair of points into a probability for each pair of points. If two points are close to each other in the high-dimensional space they will have a high probability value and vice versa. This way the probability of picking a set of points is proportional to their similarity.
Then each point gets randomly projected into a low dimensional space. For this example, we are plotting in a 1-D space but we can plot this in a 2-D or 3-D space as well. Why 2-D or 3-D? Because those are the only dimensions we (humans) can visualize. Remember t-SNE is a visualization tool first and a dimensionality reduction tool second.
Finally, t-SNE calculates the similarity probability score in a low dimensional space in order to cluster the points together. The result is a 1-D plot we see below.
One last thing we need to discuss about t-SNE is “Perplexity”, which is a required parameter when running the algorithm. “Perplexity” determines how broad or how tight of a space t-SNE captures similarities between points. If your perplexity is low (perhaps 2), t-SNE will only use two similar points and produce a plot with many scattered clusters. However, when we increase the perplexity to 10, t-SNE will consider 10 neighbor points as similar and cluster them together resulting in larger clusters of points. There is a point of diminishing returns where perplexity can become too large and we achieve a plot with one or two scattered clusters. At this point, t-SNE incorrectly considers points not necessarily related as belonging to a cluster. We typically set perplexity anywhere between 5 and 50, according to the original published paper (link).
One of the main limitations of t-SNE is its high computational costs. If you have a very large feature set it might be good to first use PCA to reduce the number of features to a few principal components and then use t-SNE to further reduce the data to 2 or 3 clusters.
Let’s get back to KMeans........
When using unsupervised ML algorithms we often cannot compare our results against a known true label. In other words, we do not have a test set to gauge the performance of our model. That said, we still need to understand how well K-Means managed to cluster our data. We already know how tightly the data is contained within our clusters by looking at the Elbow graph and the number of clusters we selected.
Silhouette Method: This technique measures the separability between clusters. First, an average distance is found between each point and all other points in a cluster. Then it measures the distance between each point and each point in other clusters. We subtract the two average measures and divide by whichever average is larger.
We ultimately want a high (ie. closest to 1) score which would indicate that there is a small intra-cluster average distance (tight clusters) and a large inter-cluster average distance (clusters well separated).
Visual Cluster Interpretation: Once you have obtained your clusters it is very important to interpret each cluster. This is typically done by merging the original dataset with the clusters and visualizing each cluster. The more clear and distinct each cluster is the better. We will review this process below.
Here’s the plan for the analysis below:
Standardize the dataApplying KMeans on the original datasetFeature Reduction via PCAApplying KMeans to PCA principal componentsFeature Reduction via t-SNEApplying KMeans to t-SNE clustersComparing PCA and t-SNE KMeans derived clusters
Standardize the data
Applying KMeans on the original dataset
Feature Reduction via PCA
Applying KMeans to PCA principal components
Feature Reduction via t-SNE
Applying KMeans to t-SNE clusters
Comparing PCA and t-SNE KMeans derived clusters
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltfrom mpl_toolkits.mplot3d import Axes3Dimport seaborn as snsimport plotly.offline as pyopyo.init_notebook_mode()import plotly.graph_objs as gofrom plotly import toolsfrom plotly.subplots import make_subplotsimport plotly.offline as pyimport plotly.express as pxfrom sklearn.cluster import KMeansfrom sklearn.preprocessing import StandardScalerfrom sklearn.decomposition import PCAfrom sklearn.manifold import TSNEfrom sklearn.metrics import silhouette_score%matplotlib inlinefrom warnings import filterwarningsfilterwarnings('ignore')with open('HR_data.csv') as f:    df =  pd.read_csv(f, usecols=['satisfaction_level', 'last_evaluation', 'number_project',       'average_montly_hours', 'time_spend_company', 'Work_accident','promotion_last_5years'])f.close()df.head()
This is a relatively clean dataset without any missing values or outliers. We do not see any mixed-type features, odd values or rare labels which need encoding. Features have low multicollinearity as well. Let’s move on to scaling our dataset.
As aforementioned, the standardization of data will ultimately bring all features to the same scale and bringing the mean to zero and the standard deviation to 1.
scaler = StandardScaler()scaler.fit(df)X_scale = scaler.transform(df)df_scale = pd.DataFrame(X_scale, columns=df.columns)df_scale.head()
Let’s utilize the Elbow Method to determine the optimal number of clusters KMeans should obtain. It seem 4 or 5 clusters would be best and for the sake of simplicity we’ll select 4.
sse = []k_list = range(1, 15)for k in k_list:    km = KMeans(n_clusters=k)    km.fit(df_scale)    sse.append([k, km.inertia_])    oca_results_scale = pd.DataFrame({'Cluster': range(1,15), 'SSE': sse})plt.figure(figsize=(12,6))plt.plot(pd.DataFrame(sse)[0], pd.DataFrame(sse)[1], marker='o')plt.title('Optimal Number of Clusters using Elbow Method (Scaled Data)')plt.xlabel('Number of clusters')plt.ylabel('Inertia')
Let’s apply KMeans on the original dataset requesting 4 clusters. We achieved a silhouette score of 0.25 which is on the low end.
df_scale2 = df_scale.copy()kmeans_scale = KMeans(n_clusters=4, n_init=100, max_iter=400, init='k-means++', random_state=42).fit(df_scale2)print('KMeans Scaled Silhouette Score: {}'.format(silhouette_score(df_scale2, kmeans_scale.labels_, metric='euclidean')))labels_scale = kmeans_scale.labels_clusters_scale = pd.concat([df_scale2, pd.DataFrame({'cluster_scaled':labels_scale})], axis=1)
Using PCA to reduce the dataset into 3 principal components we can plot the KMeans derived clusters into 2D and 3D visuals. PCA visualizations tend to aggregate clusters around a central point which makes interpretation difficult but we can see clusters 1 and 3 to have some distinct structure compared to clusters 0 and 2. However, when we plot the clusters into a 3D space we can clearly distinct all 4 clusters.
pca2 = PCA(n_components=3).fit(df_scale2)pca2d = pca2.transform(df_scale2)plt.figure(figsize = (10,10))sns.scatterplot(pca2d[:,0], pca2d[:,1],                 hue=labels_scale,                 palette='Set1',                s=100, alpha=0.2).set_title('KMeans Clusters (4) Derived from Original Dataset', fontsize=15)plt.legend()plt.ylabel('PC2')plt.xlabel('PC1')plt.show()
Scene = dict(xaxis = dict(title  = 'PC1'),yaxis = dict(title  = 'PC2'),zaxis = dict(title  = 'PC3'))labels = labels_scaletrace = go.Scatter3d(x=pca2d[:,0], y=pca2d[:,1], z=pca2d[:,2], mode='markers',marker=dict(color = labels, colorscale='Viridis', size = 10, line = dict(color = 'gray',width = 5)))layout = go.Layout(margin=dict(l=0,r=0),scene = Scene, height = 1000,width = 1000)data = [trace]fig = go.Figure(data = data, layout = layout)fig.show()
First, let’s determine what is the optimal number of principal components we need. By examining the amount of variance each principal component encompasses we can see that the first 3 principal components explain roughly 70% of the variance. Finally, we apply PCA again and reduce our dataset to 3 principal components.
#n_components=7 because we have 7 features in the datasetpca = PCA(n_components=7)pca.fit(df_scale)variance = pca.explained_variance_ratio_var = np.cumsum(np.round(variance, 3)*100)plt.figure(figsize=(12,6))plt.ylabel('% Variance Explained')plt.xlabel('# of Features')plt.title('PCA Analysis')plt.ylim(0,100.5)plt.plot(var)
pca = PCA(n_components=3)pca_scale = pca.fit_transform(df_scale)pca_df_scale = pd.DataFrame(pca_scale, columns=['pc1','pc2','pc3'])print(pca.explained_variance_ratio_)
Now that we have reduced the original dataset of 7 features to just 3 principal components let’s apply the KMeans algorithm. We once again needed to determine what is the optimal number of clusters and again it seems 4 is the right choice. It is important to remember we are now using the 3 principal components instead of the original 7 features to determine the optimal number of clusters.
sse = []k_list = range(1, 15)for k in k_list:    km = KMeans(n_clusters=k)    km.fit(pca_df_scale)    sse.append([k, km.inertia_])    pca_results_scale = pd.DataFrame({'Cluster': range(1,15), 'SSE': sse})plt.figure(figsize=(12,6))plt.plot(pd.DataFrame(sse)[0], pd.DataFrame(sse)[1], marker='o')plt.title('Optimal Number of Clusters using Elbow Method (PCA_Scaled Data)')plt.xlabel('Number of clusters')plt.ylabel('Inertia')
Now we are ready to apply KMeans on the PCA principal components. We can see that we were able to increase our silhouette score from 0.25 to 0.36 by passing KMeans a lower dimensional dataset. Looking at the 2D and 3D scatter plots we can see a significant improvement in the distinction between clusters.
kmeans_pca_scale = KMeans(n_clusters=4, n_init=100, max_iter=400, init='k-means++', random_state=42).fit(pca_df_scale)print('KMeans PCA Scaled Silhouette Score: {}'.format(silhouette_score(pca_df_scale, kmeans_pca_scale.labels_, metric='euclidean')))labels_pca_scale = kmeans_pca_scale.labels_clusters_pca_scale = pd.concat([pca_df_scale, pd.DataFrame({'pca_clusters':labels_pca_scale})], axis=1)
plt.figure(figsize = (10,10))sns.scatterplot(clusters_pca_scale.iloc[:,0],clusters_pca_scale.iloc[:,1], hue=labels_pca_scale, palette='Set1', s=100, alpha=0.2).set_title('KMeans Clusters (4) Derived from PCA', fontsize=15)plt.legend()plt.show()
We can see a definite improvement in KMeans ability to cluster our data when we reduce the number of dimensions to 3 principal components. In this section we will reduce our data once again using t-SNE and compare KMeans results to that of PCA KMeans. We will reduce down to 3 t-SNE components. Please keep in mind t-SNE is a computationally heavy algorithm. Computational time can be reduced using the ‘n_iter’ parameter. Furthermore, the code you see below is a result of dozens iterations of the ‘Perplexity’ parameter. Anything above a perplexity of 80 tended to aggregate our data into one large disperse cluster.
tsne = TSNE(n_components=3, verbose=1, perplexity=80, n_iter=5000, learning_rate=200)tsne_scale_results = tsne.fit_transform(df_scale)tsne_df_scale = pd.DataFrame(tsne_scale_results, columns=['tsne1', 'tsne2', 'tsne3'])plt.figure(figsize = (10,10))plt.scatter(tsne_df_scale.iloc[:,0],tsne_df_scale.iloc[:,1],alpha=0.25, facecolor='lightslategray')plt.xlabel('tsne1')plt.ylabel('tsne2')plt.show()
Below is the result of reducing our original dataset into 3 t-SNE components plotted into a 2D space.
Once again it seems 4 is the magic number of clusters for our KMeans analysis.
sse = []k_list = range(1, 15)for k in k_list:    km = KMeans(n_clusters=k)    km.fit(tsne_df_scale)    sse.append([k, km.inertia_])    tsne_results_scale = pd.DataFrame({'Cluster': range(1,15), 'SSE': sse})plt.figure(figsize=(12,6))plt.plot(pd.DataFrame(sse)[0], pd.DataFrame(sse)[1], marker='o')plt.title('Optimal Number of Clusters using Elbow Method (tSNE_Scaled Data)')plt.xlabel('Number of clusters')plt.ylabel('Inertia')
Applying KMeans to our 3 t-SNE derived components we were able to obtain a Silhouette score of 0.39. If you recall the Silhouette score obtained from KMeans on PCA’s 3 principal components was 0.36. A relatively small improvement but an improvement nonetheless.
kmeans_tsne_scale = KMeans(n_clusters=4, n_init=100, max_iter=400, init='k-means++', random_state=42).fit(tsne_df_scale)print('KMeans tSNE Scaled Silhouette Score: {}'.format(silhouette_score(tsne_df_scale, kmeans_tsne_scale.labels_, metric='euclidean')))labels_tsne_scale = kmeans_tsne_scale.labels_clusters_tsne_scale = pd.concat([tsne_df_scale, pd.DataFrame({'tsne_clusters':labels_tsne_scale})], axis=1)
The interpretation of t-SNE can be slightly counterintuitive as the density of t-SNE clusters (ie. low dimensional space) is not proportionally related to the data relationships in the original (high dimensional space) dataset. In other words, we can have quality dense clusters as derived from KMeans but t-SNE might display them as very broad or even as multiple clusters, especially when perplexity is too low. Density, cluster size, the number of clusters (under the same KMeans cluster) and shape really have little meaning when it comes to reading t-SNE plots. We can have very broad, dense or even multiple clusters (especially when perplexity is too low) for the same KMeans cluster but that does not relate to the quality of the cluster. Rather, t-SNE’s major advantage is the distance and location of each KMeans cluster. Clusters which are close to each other will be more related to each other. However, that does not mean clusters which are far away from each are dissimilar proportionally. Finally, we want to see a certain degree of separation between KMeans clusters as displayed using t-SNE.
plt.figure(figsize = (15,15))sns.scatterplot(clusters_tsne_scale.iloc[:,0],clusters_tsne_scale.iloc[:,1],hue=labels_tsne_scale, palette='Set1', s=100, alpha=0.6).set_title('Cluster Vis tSNE Scaled Data', fontsize=15)plt.legend()plt.show()
Scene = dict(xaxis = dict(title  = 'tsne1'),yaxis = dict(title  = 'tsne2'),zaxis = dict(title  = 'tsne3'))labels = labels_tsne_scaletrace = go.Scatter3d(x=clusters_tsne_scale.iloc[:,0], y=clusters_tsne_scale.iloc[:,1], z=clusters_tsne_scale.iloc[:,2], mode='markers',marker=dict(color = labels, colorscale='Viridis', size = 10, line = dict(color = 'yellow',width = 5)))layout = go.Layout(margin=dict(l=0,r=0),scene = Scene, height = 1000,width = 1000)data = [trace]fig = go.Figure(data = data, layout = layout)fig.show()
Staring at PCA/t-SNE plots and comparing evaluation metrics such as the Silhouette score will give us a technical perspective on the clusters derived from KMeans. If we want to understand the clusters from a business perspective we need to examine the clusters against the original features. In other words, what types of employees make up the clusters we obtained from KMeans. Not only will this analysis help guide the business and potentially continued analysis but also give us insights into the quality of the clusters.
Let’s first merge the KMeans clusters with the original unscaled features. We’ll create two separate data frames. One for the PCA derives KMeans clusters and one for the t-SNE KMeans clusters.
cluster_tsne_profile = pd.merge(df, clusters_tsne_scale['tsne_clusters'], left_index=True, right_index=True )cluster_pca_profile = pd.merge(df, clusters_pca_scale['pca_clusters'], left_index=True, right_index=True )
Let’s begin with a univariate review of the clusters by comparing the clusters based on each individual feature.
for c in cluster_pca_profile:    grid = sns.FacetGrid(cluster_pca_profile, col='pca_clusters')    grid.map(plt.hist, c)
for c in cluster_tsne_profile:    grid = sns.FacetGrid(cluster_tsne_profile, col='tsne_clusters')    grid.map(plt.hist, c)
Where the real fun begins when we examine the clusters from a bivariate perspective. When we examine employee satisfaction and last evaluation scores, we can see that KMeans clusters derived from our principal components are much more defined. We will focus our examination of this bivariate relationship between satisfaction and performance on the clusters derived from PCA.
Cluster 0 represents the largest amount of employees (7488) who on average are relatively satisfied at work with a very wide range of performance scores. Since this cluster represents almost 50% of the population it is not surprising we are seeing this wide range of satisfaction and performance scores. There is also a smaller but visible cluster of very satisfied and high performing employees.
Cluster 1 represents a critical part of the workforce as these are very high performing employees but unfortunately, have extremely low employee satisfaction. The fact cluster 1 represents 20% of the workforce only adds to the severity of the situation. Unsatisfied employees are at a much greater risk of voluntary turnover. Their high-performance scores indicate not only their proficiency but potential institutional knowledge. Losing these employees would create a significant reduction in organizational knowledge and performance. It would be interesting to see the organizational level (ie. mgr, director, executive) of these employees. Losing a large portion of senior management is a significant problem.
Cluster 2 seems to represent low performing employees who are on the lower end of the satisfaction continuum. Once again dissatisfied employees are at a higher risk of voluntary turnover and due to their low-performance evaluation we might consider these employees as ‘constructive turnover’. In other words, the voluntary turnover of these employees might actually be a good thing for the organization. We have to look at these results from two different perspectives. First, these employees make up over 25% of the population. If a significant portion of these employees quit the organization might be hard-pressed to have enough employees to successfully perform its function. Secondly, such a large portion of the population being dissatisfied and poorly performing speaks volumes about the recruiting, management, and training function of the organization. These employees might be the result of poorly constructed organizational initiatives. The company would be doing itself an enormous service if they delved deeper into what makes these employees dissatisfied and poorly performing.
Finally, cluster 3 contains only 2% of the population and has no discernible distribution. We need a larger dataset to fully explore these employees.
plt.figure(figsize=(15,10))fig , (ax1, ax2) = plt.subplots(1,2, figsize=(20,15))sns.scatterplot(data=cluster_pca_profile, x='last_evaluation', y='satisfaction_level',                 hue='pca_clusters', s=85, alpha=0.4, palette='bright', ax=ax1).set_title(    '(PCA) Clusters by satisfaction level and last_evaluation',fontsize=18)sns.scatterplot(data=cluster_tsne_profile, x='last_evaluation', y='satisfaction_level',                 hue='tsne_clusters', s=85, alpha=0.4, palette='bright', ax=ax2).set_title('(tSNE) Clusters by satisfaction level and last evaluation', fontsize=18)
Very interesting, the bivariate relationship between satisfaction and performance scores is almost identical to satisfaction and average monthly hours. Once again the KMeans clusters derived from PCA are significantly more distinct, let’s focus our attention on the PCA features.
It seems the employees in PCA cluster 0 not only have a wide range of performance but also average monthly hours. Overall, these results are promising as the majority of the workforce is overall happy, performing admirably, and working a significant amount of hours.
Cluster 1 employees are once again dissatisfied and working the largest average monthly hours. If you recall these were also the very high performing employees. These long hours could very well have an impact on their overall job satisfaction. Keep in mind there is a smaller yet emerging group of cluster 1 employees who are overall satisfied and very high performing. There could be more than just 4 clusters in our dataset.
The cluster 2 employees are not only low performing but also working the lowest average monthly hours. Keep in mind this dataset might be a little skewed as working 160 hours in a month is considered full-time.
Finally, cluster 3 again does not present itself in the plots.
plt.figure(figsize=(15,10))fig , (ax1, ax2) = plt.subplots(1,2, figsize=(20,15))sns.scatterplot(data=cluster_pca_profile, x='average_montly_hours', y='satisfaction_level',                 hue='pca_clusters',s=85, alpha=0.4, palette='bright', ax=ax1).set_title(    '(PCA) Clusters by Satisfaction Level and Average Monthly Hours',fontsize=18)sns.scatterplot(data=cluster_tsne_profile, x='average_montly_hours', y='satisfaction_level',                 hue='tsne_clusters', s=85, alpha=0.4, palette='bright', ax=ax2).set_title(    '(tSNE) Clusters by Satisfaction Level and Average Monthly Hours', fontsize=18)
As we continue with our bivariate plots we compare satisfaction and time spent in company or tenure. To no surprise, KMeans clusters from PCA principal components are much more distinct. The t-SNE plot has a similar shape to the PCA plot but its clusters are much more scattered. Looking at the PCA plots we have made an important discovery regarding cluster 0 or the vast majority (50%) of the employees. The employees in cluster 0 have primarily been with the company between 2 and 4 years. This is a fairly common statistic as the number of employees with extended tenure (ie 5 years plus) will decrease as tenure increases. Furthermore, their overall high satisfaction points to a potential increase in average tenure as satisfied employees are less likely to quit.
Cluster 1 is a little hard to describe as its data points are scattered around two distinct clusters. Overall, their tenure doesn’t vary much between 4 and 6 years of tenure, however, their satisfaction is the opposite of each other. There is a cluster is high job satisfaction and a larger cluster with very low job satisfaction. It would seem cluster 1 might encompass two distinct groups of employees. Those who are very happy and very dissatisfied with their jobs. That said, despite their differences in satisfaction, their job performance, average monthly hours and tenure are on the higher end. It seems cluster 1 are either happy or unhappy very hardworking and committed employees.
Cluster 2 follows the same pattern as the plots above. Not only are they dissatisfied at work, but they have one of the lowest average tenures as well. However, it is important to recall these employees account for 26% of the organization. Losing a significant portion of these employees would create issues. Lastly, dissatisfied employees are less likely to engage in organizational citizenship behaviors such as altruism, conscientiousness, helpfulness, and more likely to engage in counterproductive work behavior which can create toxic cultures. Dissecting who are these employees in terms of tenure, company level, turnover rates, location, and job types would be very helpful in diagnosing the issue and developing an action plan.
Finally, cluster 3 is yet again too scattered to provide us with any useful information. There simply aren’t enough employees which encompass this cluster to discern any useful patterns.
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='time_spend_company', y='satisfaction_level',                 hue='pca_clusters', jitter=0.47, s=7, alpha=0.5, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='time_spend_company', y='satisfaction_level',                 hue='tsne_clusters', jitter=0.47, s=7, alpha=0.5, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by satisfaction level and time_spend_company', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by satisfaction level and time_spend_company', fontsize=16)plt.close(1)plt.close(2)
PCA derived clusters once again outperformed t-SNE, however, only marginally. Cluster 0 has continued to follow a similar trend, overall happy employees with an average number of projects. Nothing to be overly excited or worried about. This might point to the ideal number of projects for the organization’s employees.
Cluster 1 employees are also following their trend we have seen from previous plots. Very dissatisfied employees who are being worked extremely hard when looking at the number of projects. If you recall, this group was also working very long hours during the month. We are once again seeing a smaller but visible group of satisfied employees working on an average number of projects. This was the same trend we saw with satisfaction X tenure, average monthly hours, and last evaluation score. There is definitely yet another group of very satisfied and hard-working employees. Gaining a more in-depth understanding of these employees in terms of recruiting (ie. sources, recruiters), management practices, compensation, might lead to insights into successful recruiting and hiring practices. These insights might also help the organization understand how to convert lower-performing employees.
Cluster 2 is following its trend as well. Less satisfied employees working on a very small number of projects. This trend has been seen across all bivariate relationships with satisfaction.
Cluster 3 is simply too small to make out any discernible trends.
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='number_project', y='satisfaction_level',                 hue='pca_clusters', jitter=0.47, s=7, alpha=0.5, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='number_project', y='satisfaction_level',                 hue='tsne_clusters', jitter=0.47, s=7,alpha=0.5, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by Satisfaction Level and Number of Projects', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by Satisfaction Level and Number of Projects', fontsize=16)plt.close(1)plt.close(2)
We can certainly continue examining the bivariate relationship between each cluster and our features but we are seeing a definitive trend. Below are the remaining relationship plots we urge you to examine.
The purpose of this article was to examine the effects of different dimension reduction techniques (ie. PCA and t-SNE) will have on the quality of clusters produced by KMeans. We examined a relatively clean human resources dataset consisting of over 15,000 employees along with their job characteristics. From the original 7 features, we managed to reduce the dataset into 3 principal components and 3 t-SNE components. Our silhouette scores ranged from 0.25 (unreduced dataset) to 0.36 (PCA) and 0.39 (t-SNE). Upon visual inspection, both PCA and t-SNE derived KMeans clusters did a much better job clustering the data compared to the unreduced original 7 features. It wasn’t until we began to compare the PCA and t-SNE KMeans derived clusters in terms of employee job characteristics that we saw significant differences. The PCA KMeans extracted clusters seemed to produce much clearer and defined clusters.
From a business perspective we would be doing the reader a disservice if the PCA KMeans clusters were not summarized as well.
Cluster 0: This cluster encompassed the majority of the employees (50%) in the dataset. This cluster contains mostly the average employees. They maintain an average level of job satisfaction, their number of monthly worked hours has a very wide range but not extending to the extreme. The same is true regarding their performance as they mostly maintain satisfactory performance scores. The number of projects this cluster of employees is involved in also average between 3 and 5. The only outlier we saw for these employees is their relatively young tenure (2–4 yrs) with the company. These results are not uncommon as even organizations with enough employees will follow a Gaussian distribution on many factors. As we increase the sample size we will statistically have a higher probability of selecting employees that fall towards the middle of the distribution and the outliers will have less and less pull on the distribution (ie. regression to the mean). It is easy to see why KMeans created this cluster of employees.
Cluster 1: There was a certain duality or juxtaposition to this group of employees at times. On one hand, we saw a very dissatisfied group of employees who achieved absolutely stunning performance scores, worked very long hours, managed an extreme number of projects, had an above-average tenure, and committing zero accidents. In other words, very hard working and valuable employees who were dissatisfied with the organization. The company would suffer a significant loss of productivity and knowledge if these employees began to turnover.
On the other hand, we were seeing a smaller but visible cluster of very satisfied employees with amazing performance scores, above-average number of projects, monthly hours, and tenure. By all means, the model employee the organization was lucky to have. Satisfaction was definitely the splitting factor as the two unique groups tended to merge when we compare the group’s performance with tenure or even number of projects.
It would seem there are additional clusters of employees KMeans did not identify. Perhaps 5 or even 6 clusters might have been better criteria for the KMeans algorithm.
Cluster 2: Although this cluster was not as well defined as cluster 0, we were able to see a definite trend among the data. On average these employees were not overly satisfied with the organization. Their performance typically ranked towards the lower end of the scale. They worked the lowest number of monthly hours on the lowest number of projects. Their tenure hovered around 3 years which was average.
Cluster 3: Finally, cluster 3 employees made up roughly 2% of the dataset which made it virtually impossible to identify any discernible trend in the data.
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='Work_accident', y='satisfaction_level',                 hue='pca_clusters', jitter=0.47, s=7, alpha=0.3, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='Work_accident', y='satisfaction_level',                 hue='tsne_clusters', jitter=0.47, s=7, alpha=0.3, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by Satisfaction Level and Work Accident', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by Satisfaction Level and Work Accident', fontsize=16)plt.close(1)plt.close(2)
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='number_project', y='last_evaluation',                 hue='pca_clusters', jitter=0.47, s=7, alpha=0.4, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='number_project', y='last_evaluation',                 hue='tsne_clusters', jitter=0.47, s=7, alpha=0.4, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by Last Evaluation and Number Projects', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by Last Evaluation and Number Projects', fontsize=16)plt.close(1)plt.close(2)
plt.figure(figsize=(15,10))fig , (ax1, ax2) = plt.subplots(1,2, figsize=(20,15))sns.scatterplot(data=cluster_pca_profile, x='average_montly_hours', y='last_evaluation',                 hue='pca_clusters', s=85, alpha=0.3, palette='bright', ax=ax1).set_title(    '(PCA) Clusters by Last Evaluation and Average Montly Hours',fontsize=18)sns.scatterplot(data=cluster_tsne_profile, x='average_montly_hours', y='last_evaluation',                 hue='tsne_clusters', s=85, alpha=0.3, palette='bright', ax=ax2).set_title(    '(tSNE) Clusters by Last Evaluation and Average Montly Hours', fontsize=18)
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='time_spend_company', y='last_evaluation',                 hue='pca_clusters', jitter=0.47, s=7, alpha=0.5, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='time_spend_company', y='last_evaluation',                 hue='tsne_clusters', jitter=0.47, s=7, alpha=0.5, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by Last Evaluation and Time Spend Company', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by  Last Evaluation and Time Spend Company', fontsize=16)plt.close(1)plt.close(2)
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='Work_accident', y='last_evaluation',                 hue='pca_clusters', jitter=0.47, s=7, alpha=0.4, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='Work_accident', y='last_evaluation',                 hue='tsne_clusters', jitter=0.47, s=7, alpha=0.4, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by Last Evaluation and Work Accident', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by Last Evaluation and Work Accident', fontsize=16)plt.close(1)plt.close(2)
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='number_project', y='average_montly_hours',                 hue='pca_clusters', jitter=0.47, s=7, alpha=0.5, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='number_project', y='average_montly_hours',                 hue='tsne_clusters', jitter=0.47, s=7,  alpha=0.5, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by Number Project and Average Montly Hours', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by Number Project and Average Montly Hours', fontsize=16)plt.close(1)plt.close(2)
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='number_project', y='time_spend_company',                hue='pca_clusters', jitter=0.4, s=30, alpha=0.2, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='number_project', y='time_spend_company',                 hue='tsne_clusters', jitter=0.4, s=30, alpha=0.2, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by Number Project and Time Spend Company', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by Number Project and Time Spend Company', fontsize=16)plt.close(1)plt.close(2)
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='Work_accident', y='number_project',                hue='pca_clusters', jitter=0.45, s=30, alpha=0.2, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='Work_accident', y='number_project',                 hue='tsne_clusters', jitter=0.45, s=30, alpha=0.2, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by Number Project and Work Accident', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by Number Project and Work Accident', fontsize=16)plt.close(1)plt.close(2)
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='time_spend_company', y='average_montly_hours',                 hue='pca_clusters', jitter=0.47, s=7, alpha=0.4, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='time_spend_company', y='average_montly_hours',                 hue='tsne_clusters', jitter=0.47, s=7, alpha=0.4, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by Time Spend Company and Average Monthly Hours', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by  Time Spend Company and Average Monthly Hours', fontsize=16)plt.close(1)plt.close(2)
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='Work_accident', y='average_montly_hours',                 hue='pca_clusters', jitter=0.47, s=7, alpha=0.4, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='Work_accident', y='average_montly_hours',                 hue='tsne_clusters', jitter=0.47, s=7, alpha=0.4, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by Work Accident and Average Monthly Hours', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by Work Accident and Average Monthly Hours', fontsize=16)plt.close(1)plt.close(2)
plt.figure(figsize=(15,10))fig = plt.subplots(1,2, figsize=(20,15))ax1 = sns.catplot(data=cluster_pca_profile, x='Work_accident', y='time_spend_company',                 hue='pca_clusters', jitter=0.45, s=40, alpha=0.2, height=8, aspect=1.5, palette='bright')ax2 = sns.catplot(data=cluster_tsne_profile, x='Work_accident', y='time_spend_company',                 hue='tsne_clusters', jitter=0.45, s=40, alpha=0.2, height=8, aspect=1.5, palette='bright')ax1.fig.suptitle('(PCA) Clusters by Work Accident and Time Spend Company', fontsize=16)ax2.fig.suptitle('(t-SNE) Clusters by Work Accident and Time Spend Company', fontsize=16)plt.close(1)plt.close(2)