Image Segmentation with Classical Computer Vision-Based Approaches | by Yağmur Çiğdem Aktaş | Towards Data Science

Image segmentation is the process of subdivision a digital image into multiple segments (objects). The goal of segmentation is to change the representation of an image into something more meaningful and easier to analyze.We can assemble different segmentation methods as shown below.
Classical Computer Vision-Based Approaches
ThresholdingRegion-Based MethodsEdge / Boundary Based Methods
Thresholding
Region-Based Methods
Edge / Boundary Based Methods
AI-Based Approaches
Clustering Based MethodsNeural Networks
Clustering Based Methods
Neural Networks
In this post, I will review Classical Computer Vision-Based Approaches, you can reach to my Clustering Based Methods post, and stay tuned for Neural Networks post!
Thresholding is the simplest way to segment images by dividing the image pixels into different groups concerning their intensity values. Usually, it’s used to obtain Binary images segmenting the image in two parts. The main idea is to select a threshold T, change the pixel values lower than this threshold value to 0 (background pixel) and the higher ones to 1 (foreground pixel). If you want to keep the image data type, you can select 0 for background pixels and the maximum value you can have with that data type as foreground pixel values. For example, if you work with uint8 data type images, the maximum value that your image pixels can take is 255, so assigning 255 to foreground pixels and 0 for background pixels is an applicable way too.
Global ThresholdingThe threshold value is chosen at the beginning and it can’t be changed during the process. You can manually choose that threshold value or use the Otsu thresholding method to obtain a threshold automatically according to the histogram of your image.
Basic Thresholding
Threshold value T is chosen manually.
You can apply Basic Thresholding very easily using OpenCV’s threshold function with cv2.THRESH_BINARY option.
import cv2import numpy as npimg = cv2.imread("dog.jpg", cv2.IMREAD_GRAYSCALE)_, segmented1 = cv2.threshold(img, 127,255,cv2.THRESH_BINARY)print(segmented1)cv2.imshow("Segmented Output Image", segmented1)cv2.waitKey(0)
Here, 127 is the threshold value, 255 is the value to be assigned for foreground pixels.
OpenCV’s threshold function returns a segmented image with the same data type of input image. So if you would like 1 for foreground pixels and 0 for background pixels, you would see a false result like below:
_, segmented1 = cv2.threshold(img, 127,1,cv2.THRESH_BINARY)print(segmented1)cv2.imshow("Segmented Output Image", segmented1)cv2.waitKey(0)
Even if the segmentation is right so that we see 1 and 0 in image pixels, where are our foreground pixels (1), in the output image?
That is a very important point to know if we want to work closer with OpenCV’s functions. As I said, cv2.threshold function gives us a segmented image with the same data type as our input image. Since our “dog.jpg” image is from uint8 data type, so the pixel range is between 0 and 255, when we have an output image only having 0 and 1 in 0 and 255 range, both 0 and 1 corresponds to black pixels! To do it properly, we need to change our output image data type to float32:
_, segmented2 = cv2.threshold(img, 127,1,cv2.THRESH_BINARY)segmented2 = segmented2.astype(dtype='f')print(segmented2)cv2.imshow("Segmented Output Image", segmented1)cv2.waitKey(0)
Here is the right output again!
Note that both images having 0 or 1 values and 0 or 255 values are binary since they have only 2 types of pixel values.
Otsu Thresholding
It is an automatic threshold deciding method which tries to minimize the following equation to obtain the optimal T.
Po: the probability of foreground pixels according to the threshold TPb: the probability of background pixels according to the threshold T
σ2o: variance of foreground pixelsσ2b: variance of background pixels
You can apply Otsu Thresholding very easily using OpenCV’s threshold function with cv2.THRESH_OTSU option. This time, we can use the first return value of the function to see what the selected threshold value is. That option doesn’t exist in THRESH_BINARY since it’s us who are giving the threshold already. 😉
thresh, segmented2 = cv2.threshold(img, 127,1,cv2.THRESH_OTSU)segmented2 = segmented2.astype(dtype='f')print(thresh)print(segmented2)cv2.imshow("Segmented Output Image", segmented2)cv2.waitKey(0) cv2.destroyAllWindows()
Local Thresholding
In Local Thresholding, the threshold value is chosen not globally but locally for a given size of the area. For that purpose, we take the mean of the pixels staying in that area, we subtract a constant value C from that mean and we obtain our threshold!
The idea behind Local Thresholding is that an image can be exposed to different lights in different areas. A general threshold for the whole image may not work well for that kind of exceptional region. So, Global Thresholding is not the best way for that kind of image, and picking different thresholds for different light conditions gives us better results.
In this problem, we need to decide on the value of n, C, and the mean calculation method. There are two different common ways to calculate mean when it comes to Local Thresholding. You can choose the standard mean calculation where we sum up all the pixel intensities and divide it to the pixel number in the area or you can use Gaussian Mean (or weighted mean), where the intensities of the nearest pixels to the center of the area are more contributed — weighted for the calculation.
Choosing n is another subject: a too big n may result in a similar output with Global Thresholding since it doesn’t split the image into enough small regions, and a too-small n may result that the area doesn’t consist of pixel diversity.
We can apply Adaptive Thresholding with OpenCV cv2.adaptiveThreshold function easily:
segmented4 = cv2.adaptiveThreshold(img,255,cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY,15,2)print(segmented4)cv2.imshow("Segmented Output Image", segmented4)cv2.waitKey(0)segmented5 = cv2.adaptiveThreshold(img,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY,15,2)print(segmented5)cv2.imshow("Segmented Output Image", segmented5)cv2.waitKey(0)
255 is the value for foreground pixels, ADAPTIVE_THRESH_MEAN_C is for choosing the standard mean method for local threshold calculation, and ADAPTIVE_THRESH_GAUSSIAN_C for Gaussian Mean. See that we still use cv2.THRESH_BINARY! That means, after choosing locally our threshold values, we apply the simple threshold method to decide whether a pixel will be foreground or background pixel. 15 is the size of area n, so we have a 15x15 window for each local region and 2 is the constant value C.
With that usage, some output results with different parameters are shown below:
Since with increasing C, threshold decreases, we see more foreground pixels in that output.
It may not be so clear what Adaptive Threshold does here because the results coming from Global Thresholding were quite well-segmented too. Now let’s look at some images with light differences in different regions or having marked different colors:
Now we can see Local Thresholding works far way better than Global Thresholding for these images. For books example, the output of Adaptive Thresholding may seem more noisy and complicated at first, but if you look closer you will see that there are some parts lost in Global Thresholding that we cant see in the lecture anymore, which is not the case for Adaptive Thresholding.
Eventually, which one of the thresholding methods to choose is up to you and your image type!
That’s all for Image Segmentation with Thresholding, you can find the related codes in my GitHub link, now let’s continue with the next topic ✋
In this post, we will review Image Segmentation methods based on the “Region” approach where the neighborhood and connection relations between the pixels are used. If you don’t have any idea about Neighborhood, Connection, and Connected Components terms, you can click to see my brief post.
Region Growing
One of the Region-Based methods is Region growing, where we select seed points and let that seeds grow following the pixels conforming to a set of rules. There are a different set of rules which determines the process of growing and the condition of stop, here we will examine 2 different approaches.
First approach:
Starting with that seed pixel, we look at the 4-connection or 8-connection pixels to the seed. 4 or 8, it’s up to your choice.Among those pixels having the possibility of being added in the region, the pixel having the least distance to the mean intensity of the region is chosen. If that distance is lower than a determined “similarity threshold” then we add that pixel in the region finally.When a new pixel is added into the region, now it’s our new seed, so 1. and 2. steps will be repeated for that seed point until any possible pixel compatible with these rules doesn’t exist in the neighborhood of seed (so it can’t grow anymore) or all the pixels in the image are already classified.
Starting with that seed pixel, we look at the 4-connection or 8-connection pixels to the seed. 4 or 8, it’s up to your choice.
Among those pixels having the possibility of being added in the region, the pixel having the least distance to the mean intensity of the region is chosen. If that distance is lower than a determined “similarity threshold” then we add that pixel in the region finally.
When a new pixel is added into the region, now it’s our new seed, so 1. and 2. steps will be repeated for that seed point until any possible pixel compatible with these rules doesn’t exist in the neighborhood of seed (so it can’t grow anymore) or all the pixels in the image are already classified.
I don’t think OpenCV has a built-in function for the region growing algorithm but I prepared an implementation to do some experiments following the rules I explained above. First, let’s look at some results with 2D arrays to see better what is going on, before trying a real image:
So, for the segmented image, we start with an empty image with the same size as our input and we see how the region grows while assigning a unique label for each seed region where 10 is the label for [1,1] seed and 30 is for [7,7] seed.
You can keep track of the seed pixels too, but before doing that I suggest you comment out segmented output to see better.
Now let’s look at a real image example with this approach:
On the left side, we see the input image, I want to segment the circle in the center so I choose some seed points in that area like [[245,245],[240,195], [300,245], [230,300]]. Here is the result with 8-connectivity and 10 as the threshold. Note that the segmented image is a blank image with 255 everywhere (that was 0 for 2D experiments to visualize it less noisy), so the black lines are the only parts we segmented.
It is not a good output, right?
Because even if that image is very simple to segment, having only 0 and 255 values when the seed reaches the boundary of the object (circle here), it is a huge possibility that the seed can’t go further since its neighbor pixels are all 0 and other neighbors are already classified. Let’s think about how we can improve that approach:
We can add all the neighbors of a seed compatible with the region, not the minimum distant one.
When the seed stops growing, we can search for another pixel having the compatibility conditions in the region but not directly the neighbor of the seed.
Let’s check the result with the second version of the implementation:
This one is far better than the first output right?
You can try both of the implementations and do your own experiments with the multiple seeds option by clicking my Github link.
Note that seed point choice is very important since they are the start points. It can be the center of the object you want to segment for example. The seed point selection may be done even automatically, where you need to calculate the histogram of the image and return n number of peak points. Then choose 1 of the pixel having that peak point value as seed for each peak point.
Peak point: If you have a gray level image in the 0–255 range and you see the histogram peaks (has a majority) at 200, 100, and 10 you can choose 3 seed points having 200, 100, and 10 as intensity values. In the end, you will have 3 segmented regions.
Additionally, here is a very good visualization of the Region Growing algorithm, where the method is being applied on an RGB image. https://www.youtube.com/watch?v=VaR21S8ewCQ
!!! Note that in RGB images, since we have 3 channels rather than 1 like in Gray images, the condition to be added in the region changes from being the closest to the region mean to have the least distance to the region mean. and the distance between two RGB pixels is:distance = sqrt((R2-R1)^2 +(G2-G1)^2 +(B2-B1)^2)
Split / Merge
The Split & Merge method is the opposite of Region Growing. We start with the whole image taking it as 1 region and
Split the region into 4 sub-regionIf any region in that 4 sub-region is not homogenous, split it into 4 sub-region again. If any neighbor sub-regions are homogenous, merge them.Repeat the 2. step until all the regions are homogenous.
Split the region into 4 sub-region
If any region in that 4 sub-region is not homogenous, split it into 4 sub-region again. If any neighbor sub-regions are homogenous, merge them.
Repeat the 2. step until all the regions are homogenous.
How to check if a region is homogenous?
There are different ways to check if a region is homogenous or not. You can choose one of them and apply it as the split condition to split the regions if they satisfy the condition and as the merge condition to merge them if the condition satisfies by neighbor regions.
If the grayscale level of pixels are in a given range (or all of are the same), the region is homogenous
If the mean of the region is higher than the mean of the whole image, the region is homogenous
If the variance of the region is lower than a determined threshold, the region is homogenous. Where the variance of the region is
with
The Split and Merge method is usually presented by a Tree structure. We see example output below:
Note that Region-Based Segmentation methods are frequently used in Medical Image Segmentation and they are still one of the important base methods for Image Segmentation.
As we already saw in Spatial Linear Filtering, there are different types of kernels that provide us the edges of an image. Edge Based Image Segmentation is not anything more than that! We need to track the edges of the image, when we obtain the edge map we usually fill the holes staying in those edges using Morphology, and it’s done! If you don’t have any idea about “First Derivative and Second Derivative Kernels” which we use to detect edges, you can take a look at that previous post I mentioned before go further.
Additional to that Kernel Types, there are some “Edge Detectors” where instead of using only an edge detection kernel, we have some additional steps to obtain more improved results. One of the very common Edge Detector is Canny Edge Detector which is explaining very well and detailed in the following post:
towardsdatascience.com
Let’s try to see some results using Edge Detector & Hole Filling together, with Scipy’s built-in functions that time :
"""Edge Based Segmentation """""" edge detection with canny """edges = canny(img)fig, ax = plt.subplots(figsize=(4, 4))ax.imshow(edges, cmap=plt.cm.gray)ax.axis('off')ax.set_title('Canny detector')""" region - hole filling """fill_holes = ndi.binary_fill_holes(edges)fig, ax = plt.subplots(figsize=(4, 3))ax.imshow(fill_holes, cmap=plt.cm.gray, interpolation='nearest')ax.axis('off')ax.set_title('Filling the holes')
Note that, working with too big images, or big objects may corrupt your result image since the edges are not closed mostly. If the edge is not closed, the filling hole algorithm doesn’t work properly too. Let’s check the same image without resizing:
Keep that small tricks in your mind and good luck with your own trials! You can reach the code from that Github link.
I tried to collect the classical methods in one post to give you an insight into each briefly. I hope the codes and explications are helpful for those who start to research this area! If you want to go further with Classical Methods, I suggest you continue with the Watershed Algorithm and Extraction of Connected Components. Both are using morphological operations in some parts and are worth taking a look at!
To go further with more state of art methods, click on my Clustering Based Methods post where I explain segmentation with Clustering, a subgroup of AI-Based Image Segmentation.
Thanks ✋
[1] [2] [3] : Images from https://en.wikipedia.org/wiki/Split_and_merge_segmentation
In this post, all the images used for making experiments with different code parts are taken from unsplash.com