Semantic Image Segmentation using Fully Convolutional Networks | by Arun Kumar | Towards Data Science

Humans have the innate ability to identify the objects that they see in the world around them. The visual cortex present in our brain can distinguish between a cat and a dog effortlessly in almost no time. This is true not only with cats and dogs but with almost all the objects that we see. But a computer is not as smart as a human brain to be able to this on its own. Over the past few decades, Deep Learning researchers have tried to bridge this gap between human brain and computer through a special type of artificial neural networks called Convolutional Neural Networks(CNNs).
After a lot of research to study mammalian brains, researchers found that specific parts of the brain get activated to specific type of stimulus. For example, some parts in the visual cortex get activated when we see vertical edges, some when we see horizontal edges, and some others when we see specific shapes, colors, faces, etc. ML researchers imagined each of these parts as a layer of neural network and considered the idea that a large network of such layers could mimic the human brain.This intuition gave rise to the advent of CNN, which is a type of neural network whose building blocks are convolutional layers. A convolution layer is nothing but a set of weight matrices called kernels or filters which are used for convolution operation on a feature matrix such as an image.
Convolution:2D convolution is a fairly simple operation, you start with a kernel and ‘stride’ (slide) it over the 2D input data, performing an element-wise multiplication with the part of the input it is currently on, and then summing up the results into a single output cell. The kernel repeats this process for every location it slides over, converting a 2D matrix of features into another 2D matrix of features.The step size by which the kernel slides on the input feature matrix is called stride. In the below animation, the input matrix has been added with an extra stripe of zeros from all four sides to ensure that the output matrix is of the same size as the input matrix. This is called (zero)padding.
Image segmentation is the task of partitioning a digital image into multiple segments (sets of pixels) based on some characteristics. The objective is to simplify or change the image into a representation that is more meaningful and easier to analyze.Semantic Segmentation refers to assigning a class label to each pixel in the given image. See the below example.
Note that segmentation is different from classification. In classification, complete image is assigned a class label whereas in segmentation, each pixel in an image is classified into one of the classes.
Having a fair idea about convolutional networks and semantic image segmentation, let’s jump into the problem we need to solve.
Severstal is among the top 50 producers of steel in the world and Russia’s biggest player in efficient steel mining and production. One of the key products of Severstal is steel sheets. The production process of flat sheet steel is delicate. From heating and rolling, to drying and cutting, several machines touch flat steel by the time it’s ready to ship. To ensure quality in the production of steel sheets, today, Severstal uses images from high-frequency cameras to power a defect detection algorithm.Through this competition, Severstal expects the AI community to improve the algorithm by localizing and classifying surface defects on a steel sheet.
A defective sheet must be predicted as defective since there would be serious concerns about quality if we misclassify a defective sheet as non-defective. i.e. high recall value for each of the classes is needed.We need not give the results for a given image in the blink of an eye. (No strict latency concerns)
A defective sheet must be predicted as defective since there would be serious concerns about quality if we misclassify a defective sheet as non-defective. i.e. high recall value for each of the classes is needed.
We need not give the results for a given image in the blink of an eye. (No strict latency concerns)
Our task is to
Detect/localize the defects in a steel sheet using image segmentation andClassify the detected defects into one or more classes from [1, 2, 3, 4]
Detect/localize the defects in a steel sheet using image segmentation and
Classify the detected defects into one or more classes from [1, 2, 3, 4]
To put it together, it is a semantic image segmentation problem.
The evaluation metric used is the mean Dice coefficient. The Dice coefficient can be used to compare the pixel-wise agreement between a predicted segmentation and its corresponding ground truth. The formula is given by:
where X is the predicted set of pixels and Y is the ground truth.Read more about Dice Coefficient here.
We have been given a zip folder of size 2GB which contains the following:
train_images —a folder containing 12,568 training images (.jpg files)
test_images — a folder containing 5506 test images (.jpg files). We need to detect and localize defects in these images
train.csv — training annotations which provide segments for defects belonging to ClassId = [1, 2, 3, 4]
sample_submission.csv— a sample submission file in the correct format, with each ImageId repeated 4 times, one for each of the 4 defect classes.
More details about data have been discussed in the next section.
The first step in solving any machine learning problem should be a thorough study of the raw data. This gives a fair idea about what our approaches to solving the problem should be. Very often, it also helps us find some latent aspects of the data which might be useful to our models.Let’s analyze the data and try to draw some meaningful conclusions.
train.csv tells which type of defect is present at what pixel location in an image. It contains the following columns:
ImageId: image file name with .jpg extension
ClassId: type/class of the defect, one of [1, 2, 3, 4]
EncodedPixels: represents the range of defective pixels in an image in the form of run-length encoded pixels(pixel number where defect starts <space> pixel length of the defect). e.g. ‘29102 12’ implies the defect is starting at pixel 29102 and running a total of 12 pixels, i.e. pixels 29102, 29103,........., 29113 are defective. The pixels are numbered from top to bottom, then left to right: 1 corresponds to pixel (1,1), 2 corresponds to (2,1), and so on.
train_df.ImageId.describe()count              7095unique             6666top       ef24da2ba.jpgfreq                  3Name: ImageId, dtype: object
There are 7095 data points corresponding to 6666 steel sheet images containing defects.
Number of train and test imagesLet’s get some idea about the proportion of train and test images and check how many train images contain defects.
Number of train images :  12568Number of test images :  5506Number of non-defective images in the train_images folder: 5902
There are more images in the train_images folder than unique image Ids in train.csv. This means, not all the images in the train_images folder have at least one of the defects 1, 2, 3, 4.
Sizes of train and test imagesLet’s check if all images in train and test are of the same size. If not, we must make them of the same size.
{(256, 1600, 3)}{(256, 1600, 3)}
All images in train and test folders have the same size (256×1600×3)
Let’s see how train data is distributed among various classes.
Number of images in class 1 : 5150 (77.258 %)Number of images in class 2 : 897 (13.456 %)Number of images in class 3 : 801 (12.016 %)Number of images in class 4 : 247 (3.705 %)
The dataset looks imbalanced.
The number of images with class 3 defect is very high compared to that of other classes. 77% of the defective images have class 3 defects.
Class 2 is the least occurring class, only 3.7 % of images in train.csv belong to class 2.
Note that the Sum of percentage values in the above analysis is more than 100, which means some images have defects belonging to more than one class.
Number of labels tagged per image
Number of images having 1 class label(s): 6239 (93.594%)Number of images having 2 class label(s): 425 (6.376%)Number of images having 3 class label(s): 2 (0.03%)
The majority of the images (93.6%) have only one class of defects.
Only 2 images (0.03%) have a combination of 3 classes of defects.
The rest of the images (6.37%) have a combination of 2 classes of defects.
No image has all 4 classes of defects.
Before we move ahead to training deep learning models, we need to convert the raw data into a form that can be fed to the models. Also, we need to build a data pipeline, which would perform the required pre-processing and generate batches of input and output images for training.
As the first step, we create a pandas dataframe containing filenames of train images under the column ImageId, and EncodedPixels under one or more of the columns Defect_1, Defect_2, Defect_3, Defect_4 depending on the ClassId of the image in train.csv. The images that do not have any defects have all these 4 columns blank. Below is a sample of the dataframe:
I would train my models on 85% of train images and validate on 15%.
(10682, 5)(1886, 5)
Let’s visualize some images from each class along with their masks. The pixels belonging to the defective area in the steel sheet image are indicated by yellow color in the mask image.
Our deep learning model would take steel sheet image as input (X) and return four masks (Y)(corresponding to 4 classes) as output. This implies, for training our model we would need to feed batches of train images and their corresponding masks to the model.I have generated masks for all the images in the train_images folder and stored them into a folder called train_masks.
The below code is data pipeline for applying pre-processing, augmentation to input images and generating batches for training.
I have used a hybrid loss function which is a combination of binary cross-entropy (BCE) and dice loss. BCE corresponds to binary classification of each pixel (0 indicating false prediction of defect at that pixel when compared to the ground truth mask and 1 indicating correct prediction). Dice loss is given by (1- dice coefficient).BCE dice loss = BCE + dice loss
There are several models/architectures that are used for semantic image segmentation. I have tried two of them in this case study: i)U-Net and ii) Google’s DeepLabV3+.
This model is based on the research paper U-Net: Convolutional Networks for Biomedical Image Segmentation, published in 2015 by Olaf Ronneberger, Philipp Fischer, and Thomas Brox of University of Freiburg, Germany. In this paper, the authors build upon an elegant architecture, called “Fully Convolutional Network”. They have used this for segmentation of neuronal structures in electron microscopic stacks and few other biomedical image segmentation datasets.
5.1.1. Architecture The Architecture of the network is shown in the image below. It consists of a contracting path (left side) and an expansive path (right side). The expanding path is symmetric to the contracting path giving the network a shape resembling the English letter ‘U’. Due to this reason, the network is called U-Net.
The contracting path follows the typical architecture of a convolutional network. It consists of the repeated application of two 3x3 convolutions (unpadded convolutions), each followed by a rectified linear unit (ReLU) and a 2x2 max pooling operation with stride 2 for downsampling. At each downsampling step, the number of feature channels is doubled. In this path, the model captures the important features (similar to defects in steel sheet) from the image and discards the unimportant features, reducing the resolution of the image at each convolution+maxpool layer.  In the expansive path, every step consists of an upsampling of the feature map followed by a 2x2 convolution (“up-convolution”) that halves the number of feature channels, a concatenation with the correspondingly cropped feature map from the contracting path, and two 3x3 convolutions, each followed by a ReLU. The cropping is necessary due to the loss of border pixels in every convolution. At the final layer, a 1x1 convolution is used to map each 64- component feature vector to the desired number of classes(4 in our case).  In order to localize precisely, high-resolution features from the contracting path are cropped and combined with the upsampled output and fed to a successive convolution layer which will learn to assemble more precise output.
Instead of using 64 filters in the first layer, have used only 8 filters (number of filters in the subsequent layers also changes accordingly). This results in a less complex model, which is faster to train when compared to the model with 64 filters.
The original size of the steel sheet images is 256x1600. Large-sized images consist of more pixels and hence require more number of computations for convolution, pooling, etc. I have resized the images to half-size (128x800) due to computational resource constraints.
I have added a small dropout = 0.2 after each convolution block to avoid the model from overfitting.
The code for U-Net model is available in my GitHub repository.
5.1.2. Training I have trained the model using Keras Adam optimizer with the default learning rate for 50 epochs. The loss function that the optimizer tries to minimize is bce_dice_loss, defined earlier in section 4.4.
I have used Keras model checkpoint to monitor the validation dice coefficient as the training progresses and save the model with the best validation dice coefficient score. TensorBoard has been used to dynamically plot the loss and score while training.
5.1.3. Training plots
5.1.4. TestingThe figure below shows a sample image from validation data alongside its ground truth mask and predicted mask.
Since Kaggle requires us to submit predictions on original size and not on half size images, I have rebuilt the model with input size = (256, 1600, 3) and loaded it with the weights of the model trained on 128×800 images. I have taken this liberty because CNNs are fairly invariant to different input sizes.
The Dice Coefficient score was pretty good when I uploaded the predictions of this model on Kaggle. I got a score of 0.80943 in Private leaderboard and 0.81369 in Public leaderboard.
DeepLab is a state-of-the-art semantic segmentation model designed and open-sourced by Google in 2016. Since then, multiple improvements have been made to the model, including DeepLab V2, DeepLab V3, and the latest DeepLab V3+.DeepLab V3+ is based on the paper Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation, published in 2018 by Google.
5.2.1. ArchitectureSimilar to U-Net discussed earlier, DeepLab V3+ is also an encoder-decoder architecture. The major difference is that it uses Atrous convolution instead of simple convolution. We would learn more about Atrous convolution later in this section.
The encoder module encodes multi-scale contextual information by applying atrous convolution at multiple scales, while the simple yet effective decoder module refines the segmentation results along object boundaries.
Atrous ConvolutionAtrous convolution is a generalized form of standard convolution operation that allows us to explicitly control filter’s field-of-view in order to capture multi-scale information. In the case of two-dimensional signals, for each location i on the output feature map y and a convolution filter w, atrous convolution is applied over the input feature map x as follows:
where the atrous rate r determines the stride with which we sample the input signal. Note that standard convolution is a special case in which rate r = 1. The filter’s field-of-view is adaptively modified by changing the dilation/atrous rate value.
Depth-wise Separable ConvolutionDepth-wise separable convolution drastically reduces computation complexity by dividing a standard convolution into two sub-parts — i. Depth-wise convolutionii. Point-wise convolution.
The first part is depth-wise convolution that performs a spatial convolution independently for each input channel. It is followed by a point-wise convolution (i.e., 1×1 convolution), which is employed to combine the output from the depth-wise convolution.
Let us understand this with the help of an example. Suppose we have an image of size 12×12 composed of 3 channels. We want to apply a convolution of 5×5 on this input and get an output of 8×8×256.
In the first part, depth-wise convolution, we give the input image a convolution without changing the depth. We do so by using 3 kernels of shape 5×5×1.
The point-wise convolution is so named because it uses a 1×1 kernel or a kernel that iterates through every single point. This kernel has a depth of however many channels the input image has; 3 in our case. Therefore, we iterate a 1×1×3 kernel through our 8×8×3 image, to get an 8×8×1 image.
To get 8×8×256 output image, we need to simply increase the number of 1×1×3 kernels to 256.
Encoder ArchitectureDeepLab V3+ encoder uses Xception architecture with the following modifications —
We add more layers in the middle flow
All the max pooling operations are replaced by depth-wise separable convolutions with striding
Extra batch normalization and ReLU are added after each 3×3 depth-wise convolution.
The output of the encoder is a feature map of size 16 times smaller than the input feature map. This is compensated by the decoder which has a provision to up-sample the encoder feature map by 4x twice (refer to the model architecture diagram).
5.2.2. TrainingI have trained the model using Keras Adam optimizer with the default learning rate for 47 epochs. The loss function that the optimizer tries to minimize is bce_dice_loss, defined earlier in section 4.4
As in the case of U-Net, I have saved the weights of the model with the best validation dice_coefficient.
5.2.3. Training plots
5.2.4. TestingThe figure below shows some sample images from validation data alongside their ground truth mask and predicted mask.
Rebuilding the model with original input size(256, 1600,3) and loading the weights of the model trained on half size did not work well in this case. I had to use a different strategy —I used the trained model to predict on 128×800 images and resized the predicted masks to 256×1600. This approach worked very well for DeepLab V3+.
My final submission was DeepLab V3+, which gave a decent score in both Private and Public.
This Kaggle competition was a popular one and many people have solved this problem using different approaches. However, most of them have used some or the other variant of U-Net or similar encoder-decoder architectures.
I have used simple U-Net as my first cut solution, which gives a decent performance on test data, thanks to the train on half-size and predict on full-size strategy.
I have implemented DeepLab V3+, which is a state of the art technique for semantic image segmentation from scratch. It helped me improve my score from 0.809 to 0.838.
Some hyper-parameter tuning can be done with U-Net.
Other image segmentation architectures like U-Net++, SegNet and Mask R-CNNs can be tried.
The idea of Transfer Learning with various backbones trained on large datasets can be utilized.
Thank you for reading such a long blog, I appreciate your patience. I thoroughly enjoyed writing it, hope you enjoyed reading it too.
I have skipped most of the code as I didn’t want to overwhelm the readers with code. Please refer to my GitHub repository for the complete Keras code.
If you have any queries, suggestions or discussions, please feel free to drop them in the comments section below. I will try to address them to the best of my knowledge.You can connect with me on LinkedIn, here’s my profile.
https://www.kaggle.com/c/severstal-steel-defect-detection/https://arxiv.org/abs/1505.04597v1https://arxiv.org/abs/1802.02611v3https://www.analyticsvidhya.com/blog/2019/02/tutorial-semantic-segmentation-google-deeplab/https://arxiv.org/abs/1610.02357https://github.com/MLearing/Keras-Deeplab-v3-plus/blob/master/model.pyhttps://www.appliedaicourse.com/
https://www.kaggle.com/c/severstal-steel-defect-detection/
https://arxiv.org/abs/1505.04597v1
https://arxiv.org/abs/1802.02611v3
https://www.analyticsvidhya.com/blog/2019/02/tutorial-semantic-segmentation-google-deeplab/
https://arxiv.org/abs/1610.02357
https://github.com/MLearing/Keras-Deeplab-v3-plus/blob/master/model.py
https://www.appliedaicourse.com/
Some rights reserved