Computer Vision: Instance Segmentation with Mask R-CNN | by Renu Khandelwal | Towards Data Science

This is the fourth part in the series on Computer vision journey. In this article we will explore Mask R-CNN to understand how instance segmentation works with Mask R-CNN and then predict the segmentation for an image with Mask R-CNN using Keras
Part 1- CNN, R-CNN, Fast R-CNN, Faster R-CNN
Part 2 — Understanding YOLO, YOLOv2, YOLO v3
Part 3- Object Detection with YOLOv3 using Keras
What is instance segmentation and how is different from semantic segmentation?
Semantic Segmentation detects all the objects present in an image at the pixel level. Outputs regions with different classes or objects
Semantic segmentation groups pixels in a semantically meaningful way. Pixels belonging to a person, road, building, fence, bicycle, cars or trees are grouped separately.
Instance Segmentation is identifying each object instance for every known object within an image.
Instance segmentation assigns a label to each pixel of the image. It is used for tasks such as counting the number of objects
Instance segmentation requires
Object detection of all objects in an image . Here the goal is to classify individual objects and localize each object instance using a bounding box
Segmenting each instance. Here the goal is to classify each pixel into a fixed set of categories without differentiating object instances
Mask R-CNN extends Faster R-CNN.
What’s different in Mask R-CNN and Faster R-CNN?
Mask R-CNN has an additional branch for predicting segmentation masks on each Region of Interest (RoI) in a pixel-to pixel manner
Faster R-CNN is not designed for pixel-to-pixel alignment between network inputs and outputs.
Faster R-CNN has two outputs
For each candidate object, a class label and a bounding-box offset;
Mask R-CNN has three outputs
For each candidate object, a class label and a bounding-box offset;
Third output is the object mask
What’s similar between Mask R-CNN and Faster R-CNN?
-Both Mask R-CNN and Faster R-CNN have a branch for classification and bounding box regression.
-Both use ResNet 101 architecture to extract features from image.
-Both use Region Proposal Network(RPN) to generate Region of Interests(RoI)
How does Mask R-CNN work?
Mask R-CNN model is divided into two parts
Region proposal network (RPN) to proposes candidate object bounding boxes.
Binary mask classifier to generate mask for every class
Image is run through the CNN to generate the feature maps.
Region Proposal Network(RPN) uses a CNN to generate the multiple Region of Interest(RoI) using a lightweight binary classifier. It does this using 9 anchors boxes over the image. The classifier returns object/no-object scores. Non Max suppression is applied to Anchors with high objectness score
The RoI Align network outputs multiple bounding boxes rather than a single definite one and warp them into a fixed dimension.
Warped features are then fed into fully connected layers to make classification using softmax and boundary box prediction is further refined using the regression model
Warped features are also fed into Mask classifier, which consists of two CNN’s to output a binary mask for each RoI. Mask Classifier allows the network to generate masks for every class without competition among classes
Mask R-CNN uses anchor boxes to detect multiple objects, objects of different scales, and overlapping objects in an image. This improves the speed and efficiency for object detection.
Anchor boxes are a set of predefined bounding boxes of a certain height and width. These boxes are defined to capture the scale and aspect ratio of specific object classes you want to detect.
To predict multiple objects or multiple instances of objects in an image, Mask R-CNN makes thousands of predictions. Final object detection is done by removing anchor boxes that belong to the background class and the remaining ones are filtered by their confidence score. We find the anchor boxes with IoU greater than 0.5. Anchor boxes with the greatest confidence score are selected using Non-Max suppression explained below
IoU computes intersection over the union of the two bounding boxes, the bounding box for the ground truth and the bounding box for the predicted box by algorithm
When IoU is 1 that would imply that predicted and the ground-truth bounding boxes overlap perfectly.
To detect an Object once in an image, Non-Max suppression considers all bounding boxes with IoU >0.5
What if I have multiple bounding boxes with IoU greater than 0.5?
Non-Max Suppression will remove all bounding boxes where IoU is less than or equal to 0.5
Pick the bounding box with the highest value for IoU and suppress the other bounding boxes for identifying the same object
For this we use MatterPort Mask R-CNN. The Mask R-CNN model used in this example is pre-trained on the COCO data set.
Step 1: Clone the Mask R-CNN repository
git clone https://github.com/matterport/Mask_RCNN.gitcd Mask_RCNN$ python setup.py install
Step 2: Download the pre-trained weights for COCO model from MatterPort. Place the file in the Mask_RCNN folder with name “mask_rcnn_coco.h5”
Step 3: Import the required libraries
# import the necessary packagesfrom mrcnn.config import Configfrom mrcnn import model as modellibfrom mrcnn import visualizeimport mrcnnimport numpy as npimport colorsysimport argparseimport imutilsimport randomimport cv2import osfrom matplotlib import pyplotfrom matplotlib.patches import Rectangle%matplotlib inline
Step 4: We Create a myMaskRCNNConfig class that inherits from Mask R-CNN Config class.
As I am using CPU hence setting the GPU_COUNT=1
COCO dataset has 80 labels so we set the NUM_CLASSES to 80 + 1 (for background)
class myMaskRCNNConfig(Config): # give the configuration a recognizable name NAME = “MaskRCNN_inference”  # set the number of GPUs to use along with the number of images # per GPU GPU_COUNT = 1 IMAGES_PER_GPU = 1  # number of classes (we would normally add +1 for the background # but the background class is *already* included in the class # names) NUM_CLASSES = 1+80
Step 5: Create an instance of the myMaskRCNNConfig class
config = myMaskRCNNConfig()
Step 6 :Initialize Mask R-CNN model for “inference” using the Config instance that we created
print(“loading  weights for Mask R-CNN model...”)model = modellib.MaskRCNN(mode=”inference”, config=config, model_dir=’./’)
Step 7: Load the weights for the Mask R-CNN. These are the pre-trained weights for COCO data set
model.load_weights(‘mask_rcnn_coco.h5’, by_name=True)
Step 8: Define 80 classes that the coco model and 1 for background(BG)
class_names = [‘BG’, ‘person’, ‘bicycle’, ‘car’, ‘motorcycle’, ‘airplane’, ‘bus’, ‘train’, ‘truck’, ‘boat’, ‘traffic light’, ‘fire hydrant’, ‘stop sign’, ‘parking meter’, ‘bench’, ‘bird’, ‘cat’, ‘dog’, ‘horse’, ‘sheep’, ‘cow’, ‘elephant’, ‘bear’, ‘zebra’, ‘giraffe’, ‘backpack’, ‘umbrella’, ‘handbag’, ‘tie’, ‘suitcase’, ‘frisbee’, ‘skis’, ‘snowboard’, ‘sports ball’, ‘kite’, ‘baseball bat’, ‘baseball glove’, ‘skateboard’, ‘surfboard’, ‘tennis racket’, ‘bottle’, ‘wine glass’, ‘cup’, ‘fork’, ‘knife’, ‘spoon’, ‘bowl’, ‘banana’, ‘apple’, ‘sandwich’, ‘orange’, ‘broccoli’, ‘carrot’, ‘hot dog’, ‘pizza’, ‘donut’, ‘cake’, ‘chair’, ‘couch’, ‘potted plant’, ‘bed’, ‘dining table’, ‘toilet’, ‘tv’, ‘laptop’, ‘mouse’, ‘remote’, ‘keyboard’, ‘cell phone’, ‘microwave’, ‘oven’, ‘toaster’, ‘sink’, ‘refrigerator’, ‘book’, ‘clock’, ‘vase’, ‘scissors’, ‘teddy bear’, ‘hair drier’, ‘toothbrush’]
Step 9: Function for drawing boxes for objected detected in the image
# draw an image with detected objectsdef draw_image_with_boxes(filename, boxes_list):     # load the image     data = pyplot.imread(filename)     # plot the image     pyplot.imshow(data)     # get the context for drawing boxes     ax = pyplot.gca()     # plot each box     for box in boxes_list:          # get coordinates          y1, x1, y2, x2 = box          # calculate width and height of the box          width, height = x2 - x1, y2 - y1          # create the shape          rect = Rectangle((x1, y1), width, height, fill=False, color='red', lw=5)          # draw the box          ax.add_patch(rect)     # show the plot     pyplot.show()
Step 10: We finally make the prediction and draw bounding boxes around the detected objects
Loading the image and then converting it to a numpy array
from keras.preprocessing.image import load_imgfrom keras.preprocessing.image import img_to_arrayimg = load_img(‘donuts.jpg’)pyplot.imshow(img)img = img_to_array(img)
we now make the prediction
# make predictionresults = model.detect([img], verbose=0)
Result is a dictionary for the image that we passed into the detect() function.
Dictionary has keys for the bounding boxes, masks, class and the scores. Each key points to a list for multiple possible objects detected in the image.
The keys of the dictionary are
‘rois‘: Regions-of-interest (ROI) for detected objects.
‘masks‘: Masks for the detected objects.
‘class_ids‘: Class integers for the detected objects.
‘scores‘: Confidence Probability for each predicted class.
Step 11: Visualize the results by drawing bounding box around the region of interest(rois)
# visualize the resultsdraw_image_with_boxes('donuts.jpg', results[0]['rois'])
Drawing the mask
# get dictionary for first predictionfrom mrcnn.visualize import display_instancesr = results[0]# show photo with bounding boxes, masks, class labels and scoresdisplay_instances(img, r[‘rois’], r[‘masks’], r[‘class_ids’], class_names, r[‘scores’])
To find the number of object and classes
classes= r['class_ids']print("Total Objects found", len(classes))for i in range(len(classes)):    print(class_names[classes[i]])
Code for prediction using Mask R-CNN available at github
Mask R-CNN paper