How to cluster images based on visual similarity
By: Gabe Flomo
Tags: Image Feature Extraction, Transfer Learning, Feature Extraction, Clustering, Machine Learning

The Model

Now we can load the VGG model and remove the output layer manually. This means that the new final layer is a fully-connected layer with 4,096 output nodes. This vector of 4,096 numbers is the feature vector that we will use to cluster the images.

Now that the final layer is removed, we can pass our image through the predict method to get our feature vector.

Heres the all the code in a single function

feature extraction pipeline

Now we can use this feature_extraction function to extract the features from all of the images and store the features in a dictionary with filename as the keys.

Wall time: 56.2 s

Dimensionality Reduction (PCA)

Since our feature vector has over 4,000 dimensions, your computer will thank you if you reduce the number of dimensions from 4,000 to a much smaller number. We can't simply just shorten the list by slicing it or using some subset of it because we will lose information. If only there was a way to reduce the dimensionality while keeping as much information as possible.

Enter the realm of principle component analysis.

I'm not going to waste time explaining what PCA is because there are already tons of articles explaining it, which I’ll link here.

Simply put, if you are working with data and have a lot of variables to consider (in our case 4096), PCA allows you to reduce the number of variables while preserving as much information from the original set as possible.

The number of dimensions to reduce down to is up to you and I'm sure there's a method for finding the best number of components to use, but for this case, I just chose 100 as an arbitrary number.

print(f"Components before PCA: {f.shape[1]}")

print(f"Components after PCA: {pca.n_components}") Components before PCA: 4096

Components after PCA: 100

Now that we have a smaller feature set, we are ready to cluster our images.

You’ll define a target number k, which refers to the number of centroids you need in the dataset. A centroid is the imaginary or real location representing the center of the cluster.

This algorithm will allow us to group our feature vectors into k clusters. Each cluster should contain images that are visually similar. In this case, we know there are 10 different species of flowers so we can have k = 10.

kmeans.labels_

[6, 6, 8, 6, 6, 5, 4, 6, 5, 6, 4, 6, 6, 3, 3, 5, 6, 6, 4, 4, 8, 1,

3, 8, 4, 2, 8, 4, 2, 6, 9, 7, 4, 4, 0, 5, 4, 9, 8, 5, 9, 5, 3, 6,

5, 1, 3, 9, 6, 5, 0, 1, 3, 9, 6, 7, 4, 6, 4, 5, 8, 5, 3, 6, 5, 4,

6, 5, 2, 1, 4, 3, 9, 5, 4, 6, 2, 4, 5, 0, 5, 1, 2, 9, 5, 4, 8, 1,

7, 1, 3, 5, 4, 8, 5, 4, 6, 9, 5, 9, 5, 8, 1, 4, 9, 8, 5, 4, 5, 6,

4, 1, 8, 9, 4, 6, 5, 7, 5, 6, 4, 8, 1, 4, 5, 5, 8, 6, 5, 2, 4, 8,

5, 1, 1, 6, 6, 7, 8, 1, 9, 1, 6, 4, 8, 3, 6, 1, 0, 0, 8, 1, 3, 4,

9, 9, 0, 4, 0, 6, 4, 9, 0, 3, 5, 0, 3, 9, 9, 4, 9, 5, 0, 9, 5, 4,

5, 1, 8, 3, 6, 4, 5, 2, 6, 6, 9, 5, 0, 3, 1, 3, 5, 4, 5, 0, 9, 4,

2, 1, 0, 9, 4, 9, 1, 2, 6, 1, 6, 0]

Each label in this list is a cluster identifier for each image in our dataset. The order of the labels is parallel to the list of filenames for each image. This way we can group the images into their clusters.

# view the filenames in cluster 0

groups[0] output:

['0035.png',

'0051.png',

'0080.png',

'0149.png',

'0150.png',

'0157.png',

'0159.png',

'0163.png',

'0166.png',

'0173.png',

'0189.png',

'0196.png',

'0201.png',

'0210.png']

All we have left to do is to view a cluster to see how well our model did by inspecting the clusters.

Cluster 0

Cluster 1

Cluster 2

Here we can see that our model did pretty well on clustering the flower images. We can even see that cluster 2 and cluster 0 both have yellow flowers yet, the type of flowers in each cluster are different species.

Conclusion

Here is the whole process in one file.

Hope you all learned something new, leave a comment if you have any questions or had an aha moment :)

References