Semantic Segmentation: A Deep Dive
By: Dig Labs
Tags: Pets, Image Processing, Computer Vision, Semantic Segmentation, Machine Learning

by Vamshi Kumar Bogoju

Now that we’ve laid the fundamentals for computer vision, let’s dive deep into one of the most important aspects of computer vision: Segmentation.

Understanding the scene is one of the most important aspects of computer vision. To address this challenge, image segmentation is used.

As discussed in the previous article, the segmentation task has 2 variants:

● Semantic Segmentation: all objects of the same class are classified as a single instance, represented with the same color

● Instance Segmentation: each unique object is represented as different instances with different colors

Here, we’ll deep dive into semantic segmentation. Semantic segmentation is the process of labeling the image at pixel level to its corresponding class. In semantic segmentation, if the objects which belong to one class then their corresponding pixels will be categorized to the same class.

Notable applications of semantic segmentation are:

● Medical Image Analysis — segmentation is used for image guided interventions, radiotherapy, or improved radiological diagnostics

● Autonomous Driving — recognition of roads, pedestrians, other cars, and sidewalks is required for accuracy.

● Robotics — Robot Vision to enable accurate job completion.

Currently, deep neural networks and the availability of data have made the work of segmentation easy and accurate. However, before the introduction of neural networks, conditional random fields are one of the methods used for segmentation.

Conditional random fields (CRFs):

CRFs are the probabilistic framework for labeling and segmentation of structured data. They take into account the relationship between the neighboring pixels, i.e. nearby pixels are more likely to have the same label as that of the current pixel.

Every pixel in the image belongs to one of the classes or the labels. For CRFs, these are treated as states. To segment the image, a loss function which is a sum of two costs, is minimized.

● Unary cost: cost of assigning a label or state to a pixel

● Pairwise Cost: cost of assigning pair labels for pair of pixels

Depending on the way the pair pixels are considered, CRFs are divided. Grid CRF considers immediate neighbors as pair of pixels, whereas dense CRF considers pairs of pixels in an image.

The Deep Learning Era:

In the early stages of deep learning, segmentation is done by using superpixels. Here, classification is performed on each superpixel.

Later for segmentation, Fully Convolution Networks[2] (FCNs) are being used. FCN consists of an encoder and decoder network. The encoder part of the network downsamples the input image by using series of convolutions to produce a feature vector. The decoder part upsamples the feature vector to the size of the image using transposed convolution.

Sometimes few features from the input might be lost while downsampling. To address this kind of problems, skip connections are added between encoder and decoder layers. In this article, one of the architectures of similar kind, U-Net[3], as above will be discussed below.

U-Net:

This architecture consists of two paths: contracting (left) and expansive (right). In the contracting path, the input image will be downsampled, whereas on the right path the downsampled image will be upsampled using up-convolution.

In the left path, there is a series of two 3x3 unpadded convolutions (unpadded convolutions), each followed by a rectified linear unit (ReLU) and a 2x2 max pooling operation with stride 2 for downsampling. On the other hand, the expansive path consists of an upsampling of the feature map followed by a 2x2 convolution (“up-convolution”) that halves the number of feature channels, a concatenation with the correspondingly cropped feature map from the contracting path, and two 3x3 convolutions, each followed by a ReLU.

Source: UNet++: A Nested U-Net Architecture for Medical Image ….” https://arxiv.org/abs/1807.10165

U-Net uses various data augmentations to compensate the low availability of training data. In the paper, elastic deformations like linear transformations, rotations are employed. The loss function used in this network is pixel wise soft-max over the final feature map combined with cross entropy loss function.

Various other loss functions can also be used depending on the application like:

● Dice loss

● IoU- balanced loss

● Boundary loss

● Weighted cross entropy

Few other networks which can be used for image segmentataion are :

● UNet ++[4]

● Seg-Net[5]

● Mask-Rcnn[6]

As illustrated above, semantic segmentation provides a critical aspect of computer vision, enabling scene understanding for additional analysis and intelligence. At DIG labs, we apply semantic segmentation to isolate stool and analyze it to provide health-related tips and recommendations. Check out our research areas here!

[1] “Feedforward Semantic Segmentation ….” Feedforward Semantic Segmentation With Zoom-OutFeatures.

[2] “Fully Convolutional Networks for Semantic Segmentation.” https://arxiv.org/abs/1411.4038.

[3] “U-Net: Convolutional Networks for Biomedical Image ….” https://arxiv.org/abs/1505.04597.

[4] “UNet++: A Nested U-Net Architecture for Medical Image ….” https://arxiv.org/abs/1807.10165.

[5] “SegNet: A Deep Convolutional Encoder-Decoder Architecture ….”https://arxiv.org/abs/1511.00561.

[6] “Mask R-CNN.” https://arxiv.org/abs/1703.06870.