A step-by-step guide for clustering images | by Erdogan Taskesen | Towards Data Science

For the detection and exploration of natural groups or clusters of images by carefully pre-processing images, utilizing well-known feature extraction approaches, and evaluation of the goodness of the clustering. A theoretical background followed by a hands-on tutorial.
Many computer vision tasks rely on (deep) neural networks, and aim to predict “what’s on the image”. However, not all tasks require supervised approaches or neural networks. With an unsupervised approach, we can aim to determine natural groups or clusters of images without being constrained to a fixed number of (learned) categories. In this blog, I will summarize the concepts of unsupervised clustering, followed by a hands-on tutorial on how to pre-process images, extract features (PCA, HOG), and group images with high similarity taking into account the goodness of the clustering. I will demonstrate the clustering of the MNIST dataset, the 101 objects dataset, the flower dataset, and finally the clustering of faces using the Olivetti dataset. All results are derived using the Python library clustimage.
Image recognition is a computer vision task for which the recognition part can be separated into supervised and unsupervised approaches. In the case of the supervised task, the goal can be to classify the image or an object on the image. Nowadays there are various pre-learned models for the classification tasks. The common theme is that all supervised models require a learning step where ground truth labels are used to learn the objects for the model. Some models can readily recognizable hundreds of objects and this number is steadily increasing but the majority of domain-specific objects will likely remain unknown in pre-learned models. In such cases, the transition towards unsupervised clustering approaches seems inevitable. Building high-quality labeled datasets is a laborious task and favors the use of clustering approaches.
In general, unsupervised machine learning is the task of inferring a function to describe the hidden structure from “unlabeled” data.
Unsupervised tasks are broader than only clustering, and can also be used for various other tasks such as data exploration, outlier detection, and feature extraction. In this blog, I will focus on unsupervised clustering approaches with application in image recognition. However, clustering of images requires multiple steps, and each of them can influence the final clustering result. Let’s start with a schematic overview and work our way into the rabbit hole.
Clustering of images is a multi-step process for which the steps are to pre-process the images, extract the features, cluster the images on similarity, and evaluate for the optimal number of clusters using a measure of goodness. See also the schematic overview in Figure 1. All these steps are readily implemented in the python package clustimage which only needs the path locations or the raw pixel values as an input.
I will describe each of these steps in the following sections but let’s start with a brief background on unsupervised clustering and some terminology.
With unsupervised clustering, we aim to determine “natural” or “data-driven” groups in the data without using apriori knowledge about labels or categories. The challenge of using different unsupervised clustering methods is that it will result in different partitioning of the samples and thus different groupings since each method implicitly impose a structure on the data. Thus the question arises; What is a “good” clustering? Figure 2A depicts a bunch of samples in a 2-dimensional space. Intuitively we may describe it as a group of samples (aka the images) that are cluttered together. I would state that there are two clusters without using any label information. Why? Because of the distances between the dots, and the relatively larger “gap” between the cluttered samples.
With this in mind, we can convert our intuition of “clusters” into a mathematical statement such as; the variance of samples within a so-called-cluster should be small (within variance σW, red and blue), and at the same time, the variance between the clusters should be large (between variance, σB) as depicted in Figure 2B. The distance between samples (or the intrinsic relationships) can be measured with a distance metric (e.g., Euclidean distance), and stored in a so-called dissimilarity matrix. The distance between groups of samples can then be computed using the linkage type (for hierarchical clustering).
The most well-known distance metric is the Euclidean distance. Although it is set as the default metric in many methods, it is not always the best choice.
Understand the mathematical properties of metrics so that it fits the statistical properties of the data and aligns with the research question.
A schematic overview of various distance metrics is depicted in Figure 3 [1]. In the case of image similarity, the Euclidean distance is recommended when features are extracted using Principal Component Analysis (PCA) [2] or Histograms of Oriented Gradients (HOG).
The process of hierarchical clustering involves an approach of grouping samples into a larger cluster. In this process, the distances between two sub-clusters need to be computed for which the different types of linkages describe how the clusters are connected (Figure 4).
Briefly, Single linkage between two clusters is the proximity between their two closest samples. It produces a long chain and is therefore ideal to cluster spherical data but also for outlier detection. Complete linkage between two clusters is the proximity between their two most distant samples. Intuitively, the two most distant samples cannot be much more dissimilar than other quite dissimilar pairs. It forces clusters to be spherical and have often “compact” contours by their borders, but they are not necessarily compact inside. Average linkage between two clusters is the arithmetic mean of all the proximities between the objects of one, on one side, and the objects of the other, on the other side. Centroid linkage is the proximity between the geometric centroids of the clusters. Choose the metric and linkage type carefully because it directly affects the final clustering results. With this in mind, we can start preprocessing the images.
Before we can extract features from the images, we need to perform some preprocessing steps to make sure that images are comparable in color, value range, and image size. The preprocessing steps are utilized from open-cv and pipelined in clustimage.
colorscale: Conversion of the image into e.g. grayscale (2-D) or color (3-D).scale: Normalize all pixel values between the minimum and maximum range of [0, 255].dim: Resize each image to make sure that the number of features is the same.
colorscale: Conversion of the image into e.g. grayscale (2-D) or color (3-D).
scale: Normalize all pixel values between the minimum and maximum range of [0, 255].
dim: Resize each image to make sure that the number of features is the same.
pip install clustimage
After preprocessing the images, we can start extracting features from images using the pixel value information. There are many approaches to extract features but I will focus on PCA and HOG features as these are well-established techniques that can generalize very well across different types of objects. I will briefly recap PCA and HOG but for more details I recommend reading other articles and blogs.
With PCA we can reduce dimensionality and extract Principal Components (PC) where most of the variance is seen. It is utterly important that all images must have the same width and height, and also be preprocessed similarly because the pixel values will form the feature space. Suppose we have 100 grayscaled 2D images of 128x128 pixels. Each image will be flattened and form a vector of size 16384. The vectors can now be stacked and form a new NxM array, where N is 100 samples and M are the 16384 features. The feature extraction will take place on the NxM array. Within clustimage, we can either specify the exact number of PCs to be extracted or we can set the minimum amount of explained variance that the PCs should contain in total. In the latter case, the number of PCs will be automatically determined.
HOG is a feature descriptor to extract features related to the direction and orientation of edges from image data. In general, it is a simplified representation of the image that contains only the most important information, such as the number of occurrences of gradient orientation in localized portions of an image. A summary is as follows:
The HOG descriptor focuses on the structure or the shape of an object. HOG features contain both edge and direction information.The complete image is broken down into smaller regions (localized portions) and for each region, the gradient orientation is calculated.Finally, the HOG would generate a Histogram for each of these regions separately. The histograms are created using the gradient orientations of the pixel values, hence the name Histogram of Oriented Gradients.
The HOG descriptor focuses on the structure or the shape of an object. HOG features contain both edge and direction information.
The complete image is broken down into smaller regions (localized portions) and for each region, the gradient orientation is calculated.
Finally, the HOG would generate a Histogram for each of these regions separately. The histograms are created using the gradient orientations of the pixel values, hence the name Histogram of Oriented Gradients.
Not all applications are useful when using HOG features as it “only” provides the outline of the image. For example, if the use-case is to group different generic objects, HOG features can do a great job but a deeper similarity within the object may be difficult as the details can be lost. Nevertheless, If an increase of HOG features is desired, you can decrease the pixels per cell (e.g. 4,4).
At this point, we pre-processed the images, extracted the features, and with the goal in our mind, we choose the linkage type and can start clustering the images to find groups that are similar based on the distance metric. However, a clustering approach does not provide information about the separability, goodness or the optimal number of clusters. To evaluate the clustering, there are various approaches for which the Silhouette score and Davies–Bouldin (DB) are the most well-known methods. Each comes with its properties for which a summary is as follows:
Davies–Bouldin index (dbindex): Intuitively it can be described as a measure of the within-cluster distances, and between cluster distances. The score is bounded between [0, 1], lower is better. Note that, since it measures the distance between clusters centroids it is restricted to using the Euclidean distances.
Silhouette score: The silhouette value is a measure of how similar a sample is to its cluster (cohesion) compared to other clusters (separation). The score is bounded between [-1, 1], where a high value indicates that the object is well matched to its cluster and poorly matched to neighboring clusters. Thus higher scores are better. In contrast to the DBindex, the Silhouette score is a sample-wise measure, i.e., measures the average similarity of the samples within a cluster and their distance to the other objects in the other clusters. The silhouette score can be used in combination with any distance metric.
For the evaluation of clusters, the clusteval library is utilized in clustimage and contains three evaluation methods: Silhouette, DBindex, and the Derivatives method. The evaluation approaches should be used in combination with clustering methods, such as agglomerative, k-means, or dbscan which are also included in clustimage. Here again, it is important to understand the mathematical properties of the methods so that it matches with the statistical properties of the (expected) clusters. As an example, DBscan in combination with the Silhouette evaluation can detect clusters with different densities and shapes while k-means assumes that clusters are convex shaped.
Be aware that the cluster evaluation approaches can easily be fooled as scores can gradually improve by an increase of the number of clusters.
It can be wise to set the search range for the number of clusters to find the local minima/maxima.
MNIST is a well-known handwritten digits dataset that is great to examine and showcase the performance of methods. Let’s load the dataset and run clustimage to detect the clusters. In this scenario, I will extract features using PCA and leave all other parameters to the defaults. The default settings are as follows: for the preprocessing, the grayscale is set to False (setting it to True would not change the results as the images are readily grayscaled), and the image dimensions are reduced to 128,128 if needed. PCA is used for feature extraction where the number of Principal Components is selected that cover 95% of the variance. Clusters are detected using the agglomerative approach with Euclidean distance metric and ward linkage. The clustering and the optimal number of clusters are evaluated with the Silhouette-score. The search range for the number of clusters is bounded between [3, 25]. For visualization purposes, tSNE is performed to reduce the high dimensional space and scatter the samples relative to each other.
As you may notice, there are a lot of parameters for which all functions are readily pipelined in clustimage to cluster the images and determine the optimal number of clusters. The output is stored in results and in the object itself. Let’s have a look at the results. The dimensionality is reduced to 29 features (Figure 6) and stored in the feat key. The explained variance can be plotted with cl.pca.plot(). The coordinates of the embedding are stored in xycoord, and the cluster labels can be found in labels. To get a bit more intuition about the performance, we can make various plots such as the goodness of the clustering, the consistency of similarly detected samples per cluster, and a scatter plot.
We evaluated the hierarchical clustering using the Silhouette score for which an optimum of 10 clusters is detected (highest relative score in Figure 7A). The dataset also contains 10 digits [0 to 9] but we need to examine whether the clusters do also represent the digits. In Figure 7B we can see the coefficient values per sample. Many samples have a high value which means that the clustering configuration is appropriate. If there would have been many points with low or negative values, then the clustering configuration may have too many or too few clusters. Thus overall, both graphs indicate a good clustering of the samples.
cl.clusteval.plot() cl.clusteval.scatter()
To investigate the distribution of samples in clusters, we can plot the dendrogram with cl.plot_dendrogram(). The dendrogram gives insights into the distribution of samples across the 10 clusters, and the distances between clusters seem more or less evenly distributed without strong outliers or other effects that can disturb the clustering results.
With the functionality cl.plot_unique() we can plot the centroid image per cluster. In addition, all images per cluster can also be averaged by setting the following parameter: cl.plot_unique(img_mean=True) . The latter option helps to get more intuition whether the images per cluster are alike. As an example, if we look at cluster 7 in Figure 9, the centroid of the cluster seems to be digit number 3 whereas the average image seems more like a 9? This is an indication that the cluster may behold images that are not exactly alike.
With the scatterplot function cl.scatter(zoom=5) we can scatter the samples in the embedded 2D tSNE space and get an intuition of the distribution of the samples relative to each other. Note that the samples (digit images) are colored on the cluster labels that are determined in the high dimensional space. The 2D tSNE embedding is only for visualization purposes in this case. For each cluster, the centroid is determined in the 2D space and the image closest to the centroid is plotted. In general, we see a clear separation of the clusters for which some digits may need further inspection, such as clusters 4 and 5. To see the cluster labels, we can set the zoom parameter to None, cl.scatter(zoom=None).
To deeper examine the images in the clusters, we can use the plot functionality cl.plot(cmap=’binary’, labels=[4,5]) . Here we only plot the clusters of interest, such as clusters 4 and 5. Cluster 4 seems to contain the digits 1 and 9 and cluster 5 the digits 1, 4, and 9.
With clustimage, it is also possible to extract features using Histogram of Oriented Gradients by setting the method to hog. Because the image sizes of the MNIST are small, it also requires reducing the number of pixels per cell to compute the hog features. Let’s cluster again the MNIST dataset but now using the HOG features.
With pixels_per_cell:(2,2) the following results are achieved.
Fourteen clusters are detected based on the maximum Silhouette score. Note that the second local optimum is at 10 clusters though. When we compare the clustering results (Figure 12B) to the clustering results using PCA (Figure 10), the clusters using HOG features appear less distinctive based on a visual examination. We can deeper examine the clusters when plotting the unique images per cluster using the image closest to the centroid, and the average image per cluster. Nonetheless, the results seem reasonable.
Clustering the MNIST dataset is fun but now we will cluster two real-world datasets. The first dataset contains a large number of different objects, and the second dataset is relatively small but contains different subtypes of flowers.
A large dataset containing 101 objects is the Caltech101 dataset. The dataset contains 9144 real-world images belonging to 101 categories. About 40 to 800 images per category. The size of each image is roughly 300 x 200 pixels and can be downloaded at the Caltech website. As an input to clustimage, we can simply provide the path location where all images are stored. All images in subdirectories will be recursively collected.
An optimum of 63 clusters is detected (Figure 14A) for which the centroid image for the clusters is shown in Figure 14C which is less than the 101 known input objects. Nevertheless, some clusters are very well separated from the bulk (Figure 14B) and contain images with high similarity (Figure 14D). The clusters that are more centered in the 2-D space may not be as pure as demonstrated in Figure 14D because of the nature of the tSNE algorithm. Overall, some objects tend to cluster very well whereas others may require further fine-tuning of model parameters or maybe even a different feature extraction approach.
The flower dataset contains 210 images and is clustered based using the HOG method. In the case of the flower dataset, the HOG features seem to produce better results compared to using PCA features based on a visual inspection (results not shown) even though that colors are lost during the feature extraction approach. If the number of images becomes larger, it is also possible to set the method to pca-hog that will perform a PCA on the HOG features and thereby can significantly reduce the number of features saving memory and still retain the results.
An optimum of 6 clusters is detected (Figure 15A) for which the centroid image for the clusters is shown in Figures 15B and C. The HOG features for the centroid images are shown in Figure 15D.
Clustering of faces is also possible with the clustimage library. This however may require an additional step, namely the extraction of faces from the images. The extraction of faces from the images can be performed using the cl.extract_faces() functionality that in turn uses the haar cascades of openCV, more specifically, the haarcascade_frontalface_default.xml. In this example, we will be loading the well-known Olivetti faces dataset with 400 facial images. For this dataset, there is no need to extract the faces because the faces are readily cropped.
When we run clustimage, an optimum of 18 clusters is detected (Figure 16A), and the tSNE embedding with the cluster labels is shown in Figure 16B. We do see some separation of the clusters for which each cluster can now be inspected using the cl.plot() functionality.
Besides clustering functionalities, the clustimage library also contains a functionality to search for images. With the find functionality, an input image can be provided and similar images will be returned. Two approaches are implemented:
Based on the k-nearest neighbor.
Based on significance after probability density fitting.
For both approaches, the adjacency matrix is computed across all images using the specified distance metric (default Euclidean). In the case of the k-nearest neighbor approach, the k-nearest neighbors are selected, and logically it will always result in a hit. In the case of density fitting, the adjacency matrix is used to estimate the best fit for the loc/scale/arg parameters across the tested distributions [‘norm’, ‘expon’, ‘uniform’, ‘gamma’, ‘t’]. The fitted distribution forms the similarity distribution of samples. For each new unseen input image, the probability of similarity is computed across all images, and the images are returned that are P<alpha in the lower bound of the distribution. This approach will only return the significant hits. In the case that both k and alpha parameters are specified, the union of detected samples is taken. Let’s use the flower-dataset to find a similar image using an input image from a path location.
In the example above we aim to detect similar images for two input images. For demonstration purposes, these images were also in the fit_transform functionality itself. Or in other words, we should be able to detect at least the identical image. Figure 17 demonstrates the images for which the similarity was significant with P<0.05. Note that the results can slightly change because of the stochastic component in fitting the distribution.
I touched on the concepts of unsupervised clustering and how to go from raw input images towards the clustering of highly similar images. Clustering of images requires multiple steps and various methods. Taking different steps and/or methods will yield different grouping since it implicitly imposes a structure on the data, and thus the partitioning of the samples. The clustimage package can help to use well-known feature extraction, evaluation, and clustering methods. The results can easily be explored by the various plotting functionalities. Furthermore, it can also be used for the detection of significantly similar images for a new unseen image. If you aim to find (near-)identical images (photos), I would recommend reading this blog, which is all about the detection of Duplicate Images Using Image Hash Functions using the undouble library. It has become a lengthy article but the bottom line is to always make sure that the mathematical properties of the methods match the statistical properties of the data.
Be Safe. Stay Frosty.
Cheers E.
If you found this article helpful, help support my content by signing up for a Medium membership using my referral link or follow me to access similar blogs.
clustimage Colab notebook
clustimage Github/Documentation
clusteval Github/Documentation
undouble Github/Documentation
Let’s connect on LinkedIn
Follow me on Github
Maarten Grootendorst, 9 Distance Measures in Data Science, Towards Data Science, Feb 2021.Kimberly L. Elmore et al, Euclidean Distance as a Similarity Metric for Principal Component Analysis, AMS, 01 Mar 2001Sergei Evgenievich Ivanov et al, The Recognition and Classification of Objects Based on the Modified Distance Metric, Procedia Computer Science 136 (2018)
Maarten Grootendorst, 9 Distance Measures in Data Science, Towards Data Science, Feb 2021.
Kimberly L. Elmore et al, Euclidean Distance as a Similarity Metric for Principal Component Analysis, AMS, 01 Mar 2001
Sergei Evgenievich Ivanov et al, The Recognition and Classification of Objects Based on the Modified Distance Metric, Procedia Computer Science 136 (2018)