K-means and PCA for Image Clustering: a Visual Analysis | by Sunny K. Tuladhar | Towards Data Science

What we will be doing here is train a K-means clustering model on the f-MNIST data so that it is able to cluster the images of the data-set with relative accuracy and the clusters have some logic to them which we can understand and interpret. We will then visually analyze the results of the clustering using matplotlib and plotly with reference to the actual labels (y) and draw a rough conclusion on how k-means clustering performs on an image data-set. The final code is available on the link at the end.
The words features and components have been used interchangeably in this article.
Get straight to the code on Github.
Clustering is an unsupervised machine learning algorithm and it recognizes patterns without specific labels and clusters the data according to the features. In our case, we will see if a clustering algorithm (k-means) can find a pattern between different images of the apparel in f-MNIST without the labels (y).
K-means clustering works by assigning a number of centroids based on the number of clusters given. Each data point is assigned to the cluster whose centroid is nearest to it. The algorithm aims to minimize the squared Euclidean distances between the observation and the centroid of cluster to which it belongs.
Principal Component Analysis or PCA is a method of reducing the dimensions of the given dataset while still retaining most of its variance. Wikipedia defines it as, “PCA is defined as an orthogonal linear transformation that transforms the data to a new coordinate system such that the greatest variance by some scalar projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate, and so on.”
Basically PCA reduces the dimensions of the dataset while conserving most of the information. For e.g. if a data-set has 500 features, it gets reduced to 200 features depending on the specified amount of variance retained. Higher the variance retained,more information is conserved, but more the resulting dimensions will be.
Less dimensions means less time to train and test the model. In some cases models which use data-set with PCA perform better than the original dataset. The concept of PCA and the changes it causes on images by changing the retained variance is shown brilliantly here.
So the plan is to perform k-means on the data-set but only after applying PCA on it.
Load the data-set from kerasPre-process the data, flatten the data (from 60000 x 28 x 28 array to 60000 x 784 array)Apply PCA on it to reduce the dimensions (784 to 420 using 0.98 variance)Apply K-means clustering on the PC data-set (10 clusters)Observe and Analyze the results using matplotlib and plotly
Load the data-set from keras
Pre-process the data, flatten the data (from 60000 x 28 x 28 array to 60000 x 784 array)
Apply PCA on it to reduce the dimensions (784 to 420 using 0.98 variance)
Apply K-means clustering on the PC data-set (10 clusters)
Observe and Analyze the results using matplotlib and plotly
The f-MNIST dataset is a slightly advanced dataset compared to the MNIST numbers dataset and it consists of 28 x 28 pixel images of different variety of clothing with the numbers representing the following type of clothing.
#Loading required librariesimport kerasfrom keras.datasets import fashion_mnist import numpy as npimport matplotlib.pyplot as pltfrom sklearn.preprocessing import StandardScalerfrom sklearn.cluster import KMeans#Loading the dataset(X_train,y_train), (X_test,y_test) = fashion_mnist.load_data()
We reshape the data into 2D array from a 3D array by flattening the features into 1 dimension. Each image of 28 x 28 array is now a single array of 784 features. So our dataset becomes 60000, 784 array from the previous 60000,28,28.
#Reshaping X to a 2D array for PCA and then k-meansX = X_train.reshape(-1,X_train.shape[1]*X_train.shape[2]) #We will only be using X for clustering. No need of y.print ("The shape of X is " + str(X.shape))print ("The shape of y is " + str(y.shape)) #We will be using y only to check our clusteringOutput:The shape of X is (60000, 784)The shape of y is (60000,)
We will now use PCA on our dataset to reduce our dimensions. We will pick our retained variance of 0.98 (value selected through trial and error) and use it on our dataset.
from sklearn.decomposition import PCA# Make an instance of the Modelvariance = 0.98 #The higher the explained variance the more accurate the model will remain, but more dimensions will be presentpca = PCA(variance)pca.fit(Clus_dataSet) #fit the data according to our PCA instanceprint("Number of components before PCA  = " + str(X.shape[1]))print("Number of components after PCA 0.98 = " + str(pca.n_components_)) #dimension reduced from 784Output:Number of components before PCA  = 784Number of components after PCA 0.98 = 420
We now transform the data according to the PCA instance
Clus_dataSet = pca.transform(Clus_dataSet)print(“Dimension of our data after PCA = “ + str(Clus_dataSet.shape))Output:Dimension of our data after PCA  = (60000, 420)
We can also inverse transform the data to view how the data changes because of the PCA.
approximation = pca.inverse_transform(Clus_dataSet)#image reconstruction using the less dimensioned dataplt.figure(figsize=(8,4));n = 500 #index value, change to view different data# Original Imageplt.subplot(1, 2, 1);plt.imshow(X[n].reshape(X_train.shape[1], X_train.shape[2]), cmap = plt.cm.gray,);plt.xlabel(str(X.shape[1])+’ components’, fontsize = 14)plt.title(‘Original Image’, fontsize = 20);# 196 principal componentsplt.subplot(1, 2, 2);plt.imshow(approximation[n].reshape(X_train.shape[1], X_train.shape[2]), cmap = plt.cm.gray,);plt.xlabel(str(Clus_dataSet.shape[1]) +’ components’, fontsize = 14)plt.title(str(variance * 100) + ‘% of Variance Retained’, fontsize = 20);
Building the k-means models we need values for the following parameters.
init: Initialization method of the centroids. Value will be: “k-means++”. k-means++: Selects initial cluster centers for k-mean clustering in a smart way to speed up convergence.
n_clusters: The number of clusters to form as well as the number of centroids to generate. Value will be: 10 ( we have 10 classes according to INDEX, might not be best but good enough for our context)
n_init: Number of time the k-means algorithm will be run with different centroid seeds. The final results will be the best output of n_init consecutive runs in terms of inertia. Value will be: 35 based on our inertia results (might not be the best but good enough for our context)
So now we have our model and we fit it to our dataset. Try using the mentioned parameters first to check the results with this article. You can change them later and see different results.
k_means = KMeans(init = “k-means++”, n_clusters = 10, n_init = 35)k_means.fit(Clus_dataSet)
Now the model has been fit on our Image dataset on which we had performed PCA . The data is now divided into 10 clusters. Now we have to check what type of image ended up in each of these clusters and see if there is any pattern to it. Remember the distribution will be similar but the number(label) of the clusters might be different when you execute this.
After the fit, we use the following code to visualize our clusters.
G = len(np.unique(k_means_labels)) #Number of labels#2D matrix  for an array of indexes of the given labelcluster_index= [[] for i in range(G)]for i, label in enumerate(k_means_labels,0):    for n in range(G):        if label == n:            cluster_index[n].append(i)        else:            continue#Visualisation for clusters = clustplt.figure(figsize=(20,20));clust = 3 #enter label number to visualisenum = 100 #num of data to visualize from the clusterfor i in range(1,num):     plt.subplot(10, 10, i); #(Number of rows, Number of column per row, item number)    plt.imshow(X[cluster_index[clust][i+500]].reshape(X_train.shape[1], X_train.shape[2]), cmap = plt.cm.binary);    plt.show()
As you can see below this particular cluster ( 3 in my case) does a great job of clustering Ankle Boots.
The code for the following bar-graph visualization is available on the link at the end. This uses the category labels from the y_train of the dataset and checks the quantity of a certain category in a given cluster. The graph will be same (if you used the same parameters) while the number(label) of the clusters might be different when you execute it. E.g. in mine a cluster labelled 0 seems to be a cluster of Sandals, in yours this cluster might be labelled 4.
Note: Cluster Names are ABOVE the bargraph.
Cluster 0 seems to have mostly Sandals.
Cluster 1 seems random but mostly has only upper body clothes. (T-shirt, Pullover, Dress, Coat and Shirt)
Cluster 2 also has upper body clothes but less varied.(Pullover,Shirt and Coat)
Cluster 3 seems to have mostly Ankle Boots few Sneakers and sandals. Basically all are shoes.
Similarly we can observe clusters 4 ~7.In both the clusters 6 and 7 (in my case) there seem to be majority of Bags . But upon using the visualization of the above clusters (shown below) we see a sensible pattern. One cluster has the bag handles lifted (among other things), the other does not.
The clusters 8 and 9 shown below seem to be shoes. Cluster 8 has a few sandals and mostly Sneakers, Cluster 9 seems to have a few sandals and mostly Ankle Boots.
We will be visualizing the clusters in 3D using plotly. Plotly is an advanced visualization library for python. Use the following code to obtain a 3D scatter plot of the clustered data. We will using only be 3 features from the 420 features in our dataset. This visualization helps to understand how well the clusters have formed and how far out a single cluster is spread into other clusters.
#install these if you haven’t!pip install chart_studio !pip install plotlyimport plotly as pyimport plotly.graph_objs as goimport plotly.express as px#3D Plotly Visualization of Clusters using golayout = go.Layout(    title='<b>Cluster Visualisation</b>',    yaxis=dict(        title='<i>Y</i>'    ),    xaxis=dict(        title='<i>X</i>'    ))colors = ['red','green' ,'blue','purple','magenta','yellow','cyan','maroon','teal','black']trace = [ go.Scatter3d() for _ in range(11)]for i in range(0,10):    my_members = (k_means_labels == i)    index = [h for h, g in enumerate(my_members) if g]    trace[i] = go.Scatter3d(            x=Clus_dataSet[my_members, 0],# 0 is a component among the 420 components. Feel free to change it            y=Clus_dataSet[my_members, 1],# 1 is a component among the 420 components. Feel free to change it            z=Clus_dataSet[my_members, 2],# 2 is a component among the 420 components. Feel free to change it            mode='markers',            marker = dict(size = 2,color = colors[i]),            hovertext=index,            name='Cluster'+str(i),               )fig = go.Figure(data=[trace[0],trace[1],trace[2],trace[3],trace[4],trace[5],trace[6],trace[7],trace[8],trace[9]], layout=layout)    py.offline.iplot(fig)
Output:
Since the data-set (with PCA) actually has 420 dimensions, this visualization only shows 3 of those features in the scatter plot. The component to be plotted can be changed in the code to get an idea of this in term of different components.
(A further PCA can be done to reduce all the 784 components to 3 components to fully represent it in a 3D plot but a lot of the information will be lost while doing so)
The clustering does seem to group similar items together. A cluster either contains upper-body clothes(T-shirt/top, pullover, Dress, Coat, Shirt) or shoes (Sandals/Sneakers/Ankle Boots) or Bags. The clustering however performs poorly on trousers and seems to group it together with dresses.
But it distinguishes these two types of bags quite well. The pattern can be interpreted by us. One cluster has bags with raised handles, the other has the bags with unraised handles.
Similarly in the two clusters shown below both have a majority of Ankle boots. But the ankle boots seems to be of different sizes and the smaller ones do look similar to the sneakers which are in the same cluster.
Looking at the above results we can see the k -means algorithm distinguishes its clusters based on a reasonable pattern. Thus we can draw a rough conclusion that K-means clustering method after performing a Principal Component Analysis can get a decent result on classifying images without labels.
Click here for the link to the code
References
[1] A. Ng, Machine Learning by Stanford University. (n.d.). Retrieved from https://www.coursera.org/learn/machine-learning
[2] S Joel Franklin , K-Means Clustering for Image Classification(2020), Medium
[3] Michael Galarnyk, PCA + Logistic Regression (MNIST)(2018), github