Cluster-based Image Segmentation -Python | by Mathanraj Sharma | Towards Data Science

Recently I was working on an Image classification task where first I wanted to capture the region of interest from the image before feeding it into the model. I tried a technique called cluster-based image segmentation which helped me to improve my model performance by a certain level. Let us see what it is and some sample codes to do cluster segmentation, you can find the Jupyter Notebook at the bottom.
Imagine that you are going to cross the road, what you do before you cross the road?
First, you see both sides of the road to determine the approaching vehicles and other environmental objects, then you do some amazing estimation of approaching speed and decide on when and how to cross the road. All these happens within a fraction of time, how amazing isn’t it.
Our brain captures the Images of both sides of the roadIt detects the vehicles and other objects on the road == Object DetectionNot only detect before that it determines the shape of every object it detects == Image Segmentation
Our brain captures the Images of both sides of the road
It detects the vehicles and other objects on the road == Object Detection
Not only detect before that it determines the shape of every object it detects == Image Segmentation
How amazing our brain is, by determining the shapes of different objects, it able to detect the multiple objects in the same snapshot.
Let me explain furthermore, assume we have our Image Classification model which is able to classify the apple and orange with more than 95% accuracy. When we input an Image that contains both apple and orange the prediction accuracy will go down. As the number of objects in the image increases the classification models' performances will go down. That is where the object localization comes into play.
Before we detect objects in an image and classify it, the model needs to understand what is in the image, this is where Image Segmentation helps. It creates a pixel-wise mask for the objects in an image which helps models to understand the shape of objects and their position in the image at a more granular level.
Image segmentation is broadly categorized into two main categories.
Semantic SegmentationInstance Segmentation
Semantic Segmentation
Instance Segmentation
In the first image, we can see that detected objects all are men. In semantic segmentation, we consider all those pixels belong to one class, so we represent them all by one color. On the other hand in instance segmentation, those pixels belong to the same class but we represent different instances of the same class with different colors.
Based on the approach we use segmentation can be divided into many narrower categories.
Region-Based Segmentation
Edge Detection based Segmentation
Cluster-based segmentation
CNN based Segmentation and etc.
I am only going to give an example of cluster-based segmentation in this article as I promised at the top.
Recall your understanding of clustering algorithms. Clustering algorithms are used to group closer the data points that are more similar to each other, from other group data points.
Now think of an image that holds apple and orange. Most of the pixel points in apple should be red/green, which is different from the pixel values of orange. If we can cluster these points we can distinguish each object from one another right. That’s how the cluster-based segmentation works. Let’s see some code samples now.
from skimage.io import imreadfrom skimage.color import rgb2grayimport numpy as npimport matplotlib.pyplot as plt%matplotlib inlinefrom scipy import ndimage# Scaling the image pixels values within 0-1img = imread('./apple-orange.jpg') / 255plt.imshow(img)plt.title('Original')plt.show()
As it is visible to our naked eye there are five color segments in the Image
The green part of ApplesThe orange part of OrangesGray Shadows at bottom of the Apples and orangesBright Yellowish part of Apple’s top and right partsWhite Background
The green part of Apples
The orange part of Oranges
Gray Shadows at bottom of the Apples and oranges
Bright Yellowish part of Apple’s top and right parts
White Background
Let us see whether we can cluster them using our KMeans algorithm from scikit-learn
# For clustering the image using k-means, we first need to convert it into a 2-dimensional arrayimage_2D = img.reshape(img.shape[0]*img.shape[1], img.shape[2])# Use KMeans clustering algorithm from sklearn.cluster to cluster pixels in imagefrom sklearn.cluster import KMeans# tweak the cluster size and see what happens to the Outputkmeans = KMeans(n_clusters=5, random_state=0).fit(image_2D)clustered = kmeans.cluster_centers_[kmeans.labels_]# Reshape back the image from 2D to 3D imageclustered_3D = clustered.reshape(img.shape[0], img.shape[1], img.shape[2])plt.imshow(clustered_3D)plt.title('Clustered Image')plt.show()
Wow, that works!!! We are able to cluster all five parts. This is how the cluster segmentation works.
There are many advanced techniques like Mask R-CNN to do more granular level segmentation. Let us see those topics in some other article. I hope you now have some level of Understanding of Image Segmentation.
You can find the notebook for the above example at https://github.com/Mathanraj-Sharma/sample-for-medium-article/blob/master/cluster-based-segmentation-skimage/cluster-based-segmentation.ipynb