Implementation of Principal Component Analysis(PCA) in K Means Clustering
By: Wamika Jha
Tags: Machine Learning, Data Visualization, Clustering Algorithm, Unsupervised Learning, Python

Let us code!

About the dataset : It contains 217 columns of hobbies, where 1 means yes.

So, first step will be to import all the necessary libraries.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

from sklearn.cluster import KMeans

from sklearn.decomposition import PCA

Next step will be to load our dataset. I have downloaded a dataset from Kaggle.

df = pd.read_csv("kaggle_Interests_group.csv")

Next step is data preprocessing. The data has a lot of NaN values, because of which we cannot train the model. So we simply replace those with 0 using this code.

df.fillna(0, inplace = True)

As we can see in the dataset that we don’t need the first two columns, so we will assign rest 217 columns to variable X.

x = df.iloc[:,2:]

Let us now move on to building and training the model. Even though it is specified in the dataset that it contains 4 groups, but still we will implement the “elbow method” to determine the number of clusters. This can be done by using WCSS (sum of squares of distances of datapoints)

wcss = []

for i in range(1,11):

model = KMeans(n_clusters = i, init = "k-means++")

model.fit(x)

wcss.append(model.inertia_)

plt.figure(figsize=(10,10))

plt.plot(range(1,11), wcss)

plt.xlabel('Number of clusters')

plt.ylabel('WCSS')

plt.show()

This should give the following graph as output

Graph to find out appropriate number of cluster

As it is evident that there is no particular elbow for this dataset, so in this article I will do it using 6 clusters. You can implement it using 4 clusters as a practice.

Next step is to convert our dataset from multidimensions to 2 dimensions.

pca = PCA(2)

data = pca.fit_transform(x)

You can specify any number of dimensions/features to reduce to while initialising PCA() functions, but for simplicity sake I have used 2. If you are using more than 2 components, you can use first two components with highest variance value to train and visualise the dataset.

Here we have used fit_transform() function to to both fit the dataset x and apply dimensionality reduction to it. This returns an ndarray of 2x2 dimensions.

Next we plot and check the variance of the components.

plt.figure(figsize=(10,10))

var = np.round(pca.explained_variance_ratio_*100, decimals = 1)

lbls = [str(x) for x in range(1,len(var)+1)]

plt.bar(x=range(1,len(var)+1), height = var, tick_label = lbls)

plt.show()

This returns the following screenplot

Variance levels of different components

Now we will train our model based on the new features generated by PCA(). Since we have only 2 Principal Components (PC1 and PC2), we will get a 2D figure with 6 clusters.

centers = np.array(model2.cluster_centers_)

model = KMeans(n_clusters = 6, init = "k-means++")

label = model.fit_predict(data)

plt.figure(figsize=(10,10))

uniq = np.unique(label)

for i in uniq:

plt.scatter(data[label == i , 0] , data[label == i , 1] , label = i) plt.scatter(centers[:,0], centers[:,1], marker="x", color='k')

#This is done to find the centroid for each clusters.

plt.legend()

plt.show()

This is the scatterplot we get.

PC1 vs PC2 graph, with centroids marked

Hurray! We have successfully coded and implemented K Means clustering with PCA. Give yourself a pat on the back ;)

You can checkout the python notebook as well as the dataset on this GitHub link.

Here is the Kaggle link for the dataset used in the above article.

Thank you for reading!