Building a Mask R-CNN from scratch in TensorFlow and Keras
By: Franciska Rajki
Tags: TensorFlow, Object Detection, Mask R Cnn, Machine Learning, Keras

During training we simply use the rpnloss as the sum of the classloss + deltaloss. We could use loss weights for these if necessary.

Now we choose the anchors we predict as foregrounds, and move them with the corresponding predicted deltas. So we will have NumberOfForegrounds * 4 coordinates, these are our ROIs. We cut these regions from the featuremap, and resize them to the same size: these are the proposals. These will be the inputs for our class, and boxrefinement head, and our maskhead. (These NN models will need to have a fixed inputshape, for this we can always pad our porposals with zeros.)

Image by author.

As we have at maximum two objects on are input images, there is a high chance, that our RPN moved different anchors to overlap with the same object. We will use non-max supression to filter out highly overlapping proposals, as they are unnecessary. We will only keep the bests.

3.The last steps are the “head” models, these can be trained pararelly.

After we have our proposals, we need a NN which will further refine the bounding box coordinates, and also predicts the classlabel. So not only will it predict if a region is foreground or background, but if it is foreground, it will classify it (heart or star in my toydatasets case). Just like in case of the RPN, only the foreground proposals contribute to the box loss, and an arbitrarily fixed number of proposals classlabel contribute to the classloss.

We not only predict one set of boxrefinement coordinates to each proposals, but three. Three is the number of classes +1 for the background.

So in our case we predict 3*4 coordinates to each proposal. We will choose the proper boxrefinement coordinates based on the class we predicted in this same network.

During training we simply use the chloss as the sum of the classloss + deltaloss. We could use loss weights for these if necessary. We use a new proposalcount because we use non-max supression on the RPN proposals, in order not to have very overlapping proposals. Because of this, we will only have a very few (<5) foreground boxes left, and we want to have a balanced training. (approximately 5FG-5BG proposals)

The maskhead is a Convolutional Network with one upsampling layer at the end. It predicts a mask only to the received ROI (not the whole image).

The last convolutional layers filter number is equal to the number of classes. When checking the predicted mask, we need to use the filter, corresponding to the classlabel. So if our labels are: 0:star, 1:heart, and our input image has one heart on it, we will take the first axis from the filter dimension.

As I mentioned before, these head models can be trained paralelly, however, you can’t predict a mask, without the classprediction. (As in case of predictions, you don’t have the ground truth classlabels, so you will need the predicted classlabel to choose the proper mask.)

This is it, we have a functioning Mask R-CNN. Hopefully this article helped in understanding the basics, so that you can implement your own, even improved Mask R-CNN.

You can find the whole implementation, and the toydataset generation files at:

https://github.com/rajkifranciska/maskrcnn-from-scratch

If this article was helpful, please cite me as:

@misc{rajki_2021,

title={Building a Mask R-CNN from scratch in TensorFlow and Keras}, journal={Medium},

author={Rajki, Franciska},

year={2021}, month={Mar}}