K-means and PCA for Image Clustering: a Visual Analysis
By: Sunny K. Tuladhar
Tags: Principal Component, Image, Plotly, K Means, Clustering

K-means and PCA for Image Clustering: a Visual Analysis

The Fashion MNIST data set. Image from researchgate.net

What we will be doing here is train a K-means clustering model on the f-MNIST data so that it is able to cluster the images of the data-set with relative accuracy and the clusters have some logic to them which we can understand and interpret. We will then visually analyze the results of the clustering using matplotlib and plotly with reference to the actual labels (y) and draw a rough conclusion on how k-means clustering performs on an image data-set. The final code is available on the link at the end.

The words features and components have been used interchangeably in this article.

Get straight to the code on Github.

Introduction

Clustering is an unsupervised machine learning algorithm and it recognizes patterns without specific labels and clusters the data according to the features. In our case, we will see if a clustering algorithm (k-means) can find a pattern between different images of the apparel in f-MNIST without the labels (y).

A gif illustrating how K-means works. Each red dot is a centroid and each different color represents a different cluster. Every frame is an iteration where the centroid is relocated. (Source: gyfcat)

K-means clustering works by assigning a number of centroids based on the number of clusters given. Each data point is assigned to the cluster whose centroid is nearest to it. The algorithm aims to minimize the squared Euclidean distances between the observation and the centroid of cluster to which it belongs.

Principal Component Analysis or PCA is a method of reducing the dimensions of the given dataset while still retaining most of its variance. Wikipedia defines it as, “PCA is defined as an orthogonal linear transformation that transforms the data to a new coordinate system such that the greatest variance by some scalar projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate, and so on.”

PCA visualisation. The best PC (black moving line) is when the total length of those red lines are minimum. It will be used instead of the horizontal and vertical components (Source: giphy )

Basically PCA reduces the dimensions of the dataset while conserving most of the information. For e.g. if a data-set has 500 features, it gets reduced to 200 features depending on the specified amount of variance retained. Higher the variance retained,more information is conserved, but more the resulting dimensions will be.

Less dimensions means less time to train and test the model. In some cases models which use data-set with PCA perform better than the original dataset. The concept of PCA and the changes it causes on images by changing the retained variance is shown brilliantly here.

The plan

So the plan is to perform k-means on the data-set but only after applying PCA on it.

Load the data-set from keras Pre-process the data, flatten the data (from 60000 x 28 x 28 array to 60000 x 784 array) Apply PCA on it to reduce the dimensions (784 to 420 using 0.98 variance) Apply K-means clustering on the PC data-set (10 clusters) Observe and Analyze the results using matplotlib and plotly

The data-set

The f-MNIST dataset is a slightly advanced dataset compared to the MNIST numbers dataset and it consists of 28 x 28 pixel images of different variety of clothing with the numbers representing the following type of clothing.

#Loading required libraries

import keras

from keras.datasets import fashion_mnist

import numpy as np

import matplotlib.pyplot as plt

from sklearn.preprocessing import StandardScaler

from sklearn.cluster import KMeans

#Loading the dataset

(X_train,y_train), (X_test,y_test) = fashion_mnist.load_data()

The Fashion MNIST data set

Each image is 28 x 28 pixel with different variety of clothing, shoes and bags

Data pre-processing

We reshape the data into 2D array from a 3D array by flattening the features into 1 dimension. Each image of 28 x 28 array is now a single array of 784 features. So our dataset becomes 60000, 784 array from the previous 60000,28,28.

#Reshaping X to a 2D array for PCA and then k-means

X = X_train.reshape(-1,X_train.shape[1]*X_train.shape[2]) #We will only be using X for clustering. No need of y. print ("The shape of X is " + str(X.shape))

print ("The shape of y is " + str(y.shape)) #We will be using y only to check our clustering

Output:

The shape of X is (60000, 784)

The shape of y is (60000,)

PCA (Principle Component Analysis)

We will now use PCA on our dataset to reduce our dimensions. We will pick our retained variance of 0.98 (value selected through trial and error) and use it on our dataset.

from sklearn.decomposition import PCA

# Make an instance of the Model variance = 0.98 #The higher the explained variance the more accurate the model will remain, but more dimensions will be present pca = PCA(variance)



pca.fit(Clus_dataSet) #fit the data according to our PCA instance print("Number of components before PCA = " + str(X.shape[1]))

print("Number of components after PCA 0.98 = " + str(pca.n_components_))

#dimension reduced from 784 Output:

Number of components before PCA = 784

Number of components after PCA 0.98 = 420

We now transform the data according to the PCA instance

Clus_dataSet = pca.transform(Clus_dataSet) print(“Dimension of our data after PCA = “ + str(Clus_dataSet.shape)) Output:

Dimension of our data after PCA = (60000, 420)

We can also inverse transform the data to view how the data changes because of the PCA.

approximation = pca.inverse_transform(Clus_dataSet) #image reconstruction using the less dimensioned data

plt.figure(figsize=(8,4)); n = 500 #index value, change to view different data # Original Image

plt.subplot(1, 2, 1);

plt.imshow(X[n].reshape(X_train.shape[1], X_train.shape[2]),

cmap = plt.cm.gray,);

plt.xlabel(str(X.shape[1])+’ components’, fontsize = 14)

plt.title(‘Original Image’, fontsize = 20); # 196 principal components

plt.subplot(1, 2, 2);

plt.imshow(approximation[n].reshape(X_train.shape[1], X_train.shape[2]),

cmap = plt.cm.gray,);

plt.xlabel(str(Clus_dataSet.shape[1]) +’ components’, fontsize = 14)

plt.title(str(variance * 100) + ‘% of Variance Retained’, fontsize = 20);

The data on the left is the original (784 components) , the one on the right is inverse transformed from the PCA (420 components) . Both have 28 x 28 dimensions.

K-Means Clustering

Building the k-means models we need values for the following parameters.

init: Initialization method of the centroids. Value will be: “k-means++”. k-means++: Selects initial cluster centers for k-mean clustering in a smart way to speed up convergence.

n_clusters: The number of clusters to form as well as the number of centroids to generate. Value will be: 10 ( we have 10 classes according to INDEX, might not be best but good enough for our context)

n_init: Number of time the k-means algorithm will be run with different centroid seeds. The final results will be the best output of n_init consecutive runs in terms of inertia. Value will be: 35 based on our inertia results (might not be the best but good enough for our context)

So now we have our model and we fit it to our dataset. Try using the mentioned parameters first to check the results with this article. You can change them later and see different results.

k_means = KMeans(init = “k-means++”, n_clusters = 10, n_init = 35)

k_means.fit(Clus_dataSet)

Now the model has been fit on our Image dataset on which we had performed PCA . The data is now divided into 10 clusters. Now we have to check what type of image ended up in each of these clusters and see if there is any pattern to it. Remember the distribution will be similar but the number(label) of the clusters might be different when you execute this.

Visualization and analysis

After the fit, we use the following code to visualize our clusters.

G = len(np.unique(k_means_labels)) #Number of labels #2D matrix for an array of indexes of the given label

cluster_index= [[] for i in range(G)]

for i, label in enumerate(k_means_labels,0):

for n in range(G):

if label == n:

cluster_index[n].append(i)

else:

continue #Visualisation for clusters = clust plt.figure(figsize=(20,20));

clust = 3 #enter label number to visualise

num = 100 #num of data to visualize from the cluster

for i in range(1,num):

plt.subplot(10, 10, i); #(Number of rows, Number of column per row, item number)

plt.imshow(X[cluster_index[clust][i+500]].reshape(X_train.shape[1], X_train.shape[2]), cmap = plt.cm.binary);



plt.show()

As you can see below this particular cluster ( 3 in my case) does a great job of clustering Ankle Boots.

Cluster 3 seems to hold Ankle Boots (slightly short ones) , Sneakers and some sandals

The code for the following bar-graph visualization is available on the link at the end. This uses the category labels from the y_train of the dataset and checks the quantity of a certain category in a given cluster. The graph will be same (if you used the same parameters) while the number(label) of the clusters might be different when you execute it. E.g. in mine a cluster labelled 0 seems to be a cluster of Sandals, in yours this cluster might be labelled 4.

Note: Cluster Names are ABOVE the bargraph.

Bar graph of clusters 0~ 3

Cluster 0 seems to have mostly Sandals.

Cluster 1 seems random but mostly has only upper body clothes. (T-shirt, Pullover, Dress, Coat and Shirt)

Cluster 2 also has upper body clothes but less varied.(Pullover,Shirt and Coat)

Cluster 3 seems to have mostly Ankle Boots few Sneakers and sandals. Basically all are shoes.

Bar graph of clusters 4 ~ 7. Cluster 6 and 7 both seems to have bags.

Similarly we can observe clusters 4 ~7.In both the clusters 6 and 7 (in my case) there seem to be majority of Bags . But upon using the visualization of the above clusters (shown below) we see a sensible pattern. One cluster has the bag handles lifted (among other things), the other does not.

Cluster 6 seems to have bags with handles up.

Cluster 7 seems to have bags with handles downs.

Bar graph of clusters 8 and 9 which seem to have shoes.

The clusters 8 and 9 shown below seem to be shoes. Cluster 8 has a few sandals and mostly Sneakers, Cluster 9 seems to have a few sandals and mostly Ankle Boots.

Cluster 8 seems to have Sneakers and Sandals. Both are shoes.

Cluster 9 seems to have mainly Ankle Boots and a few Sandals. Both are shoes.

3D Visualization of the clusters

We will be visualizing the clusters in 3D using plotly. Plotly is an advanced visualization library for python. Use the following code to obtain a 3D scatter plot of the clustered data. We will using only be 3 features from the 420 features in our dataset. This visualization helps to understand how well the clusters have formed and how far out a single cluster is spread into other clusters.

#install these if you haven’t

!pip install chart_studio

!pip install plotly import plotly as py

import plotly.graph_objs as go

import plotly.express as px #3D Plotly Visualization of Clusters using go layout = go.Layout(

title='<b>Cluster Visualisation</b>',

yaxis=dict(

title='<i>Y</i>'

),

xaxis=dict(

title='<i>X</i>'

)

) colors = ['red','green' ,'blue','purple','magenta','yellow','cyan','maroon','teal','black']

trace = [ go.Scatter3d() for _ in range(11)]

for i in range(0,10):

my_members = (k_means_labels == i)

index = [h for h, g in enumerate(my_members) if g]

trace[i] = go.Scatter3d(

x=Clus_dataSet[my_members, 0],# 0 is a component among the 420 components. Feel free to change it

y=Clus_dataSet[my_members, 1],# 1 is a component among the 420 components. Feel free to change it

z=Clus_dataSet[my_members, 2],# 2 is a component among the 420 components. Feel free to change it

mode='markers',

marker = dict(size = 2,color = colors[i]),

hovertext=index,

name='Cluster'+str(i),



) fig = go.Figure(data=[trace[0],trace[1],trace[2],trace[3],trace[4],trace[5],trace[6],trace[7],trace[8],trace[9]], layout=layout)



py.offline.iplot(fig)

Output:

3D plotly scatter plot Visualization of the k-means clustered f-MNIST dataset with PCA. Each color represents a different cluster.

Since the data-set (with PCA) actually has 420 dimensions, this visualization only shows 3 of those features in the scatter plot. The component to be plotted can be changed in the code to get an idea of this in term of different components.

(A further PCA can be done to reduce all the 784 components to 3 components to fully represent it in a 3D plot but a lot of the information will be lost while doing so)

Conclusion

The clustering does seem to group similar items together. A cluster either contains upper-body clothes(T-shirt/top, pullover, Dress, Coat, Shirt) or shoes (Sandals/Sneakers/Ankle Boots) or Bags. The clustering however performs poorly on trousers and seems to group it together with dresses.

A cluster with trousers and dresses

The data visualization of the dresses and trousers in the aforementioned cluster

But it distinguishes these two types of bags quite well. The pattern can be interpreted by us. One cluster has bags with raised handles, the other has the bags with unraised handles.

2 clusters both with bags. But the algorithm was able to distinguish between the handle being raised and not raised

Similarly in the two clusters shown below both have a majority of Ankle boots. But the ankle boots seems to be of different sizes and the smaller ones do look similar to the sneakers which are in the same cluster.

Two clusters both with maximum ankle boots . The one on the left has shorter ones compared to the one on the right

Looking at the above results we can see the k -means algorithm distinguishes its clusters based on a reasonable pattern. Thus we can draw a rough conclusion that K-means clustering method after performing a Principal Component Analysis can get a decent result on classifying images without labels.

Click here for the link to the code

References

[1] A. Ng, Machine Learning by Stanford University. (n.d.). Retrieved from https://www.coursera.org/learn/machine-learning

[2] S Joel Franklin , K-Means Clustering for Image Classification(2020), Medium

[3] Michael Galarnyk, PCA + Logistic Regression (MNIST)(2018), github