Understanding Regions with CNN features (R-CNN)
By: Parth Rajesh Dedhia
Tags: Artificial Intelligence, Computer Vision, Object Detection, R Cnn, Deep Learning

Student

I am well versed with the famous Object Classification algorithms like VGG, AlexNet, ResNet, InceptionNet, MobileNet, (and all their variants) to name some. I was amazed by the architecture of such methods and would like to expand my knowledge in this domain further. However, I had one question in my mind, that these models are only able to tell whether an image contains an object or no. However, I would like to work on models that could also tell where that object is, in an image?

Teacher

The section of deep learning you wish to venture into has been also categorized as follows: Image Localisation, Object Detection, and Image Segmentation.

Object Detection: Predicting bounding boxes of multiple objects of multiple classes. May contain many objects belonging to the same class as well.

Image Localization: Predicting bounding box of only a single object, of a single class, in an image

Image Segmentation: To create a pixel-wise mask for each of the objects in an image.

Among the three, we will be talking about Object Detection. Before diving into R-CNN architecture details, I would like you to know the concept of Intersection Over Union (IoU). The area of intersection between the two boxes, divided by the union of the area occupied by the two boxes, is defined as the Intersection Over Union. In the image below, the upper half shows a visual representation of the formula.

Intersection over Union (IoU) — image by author, created with the help of draw.io

Let the blue box, in the lower half of the above image, be the prediction of the object bounding box, and the yellow one be another prediction. So, the IoU is 1 when two boxes completely overlap, and the IoU is 0 when they don’t overlap at all. All other cases fall between 0 and 1, and the exact value could be determined by using the bounding box coordinates. While training and evaluating the performance, one of the boxes can be labeled as ground truth, and if the IoU is below a particular threshold, then the prediction can be considered as false positive. Keep this explanation in mind, we will need it in a while.

Also, the performance measurement used in the network is mean Average Precision (mAP). The mean Average Precision’s nature is just like accuracy, the more the mAP, the better. Before jumping to R-CNN, we shall have a look at a model popular before R-CNN.

One of the first papers on object detection using Convolution Neural Network was OverFeat. They used a sliding window approach to detect objects present in different locations of an image. They also used an image pyramid so that smaller objects could be detected in enlarged images and larger objects can be detected in smaller images. The gif below will give you a visual idea about how the sliding window is used.

Sliding Window Approach for Object Detection — gif by author, Photo by Sonnie Hiles on Unsplash

The authors of OverFeat had used AlexNet as their base architecture and provided two networks, one fast and another, an accurate one. They had replaced the fully connected layers of CNN with a 1 x 1 Convolutional Layer (a nice idea) which reduce their computation and bumped the speed during training and testing. They also have a regression layer that takes input from the Pool5 layer and passes them through two dense layers and then the final output has 4 units that determine the bounding box.

Note: I have over-simplified the explanation of OverFeat paper and there are many subtle details about the architecture and refer to the paper for those details

Student

I did not get time to look deep into the OverFeat paper, but based on the gif you showed, doesn’t it look like many of the sub-section of images selected by the window won’t be detecting any objects? Like these regions would be selected by the detector, send through the entire CNN architecture, and still not produce any objects and increase the computational cost? Won’t it be better if we could somehow just give the model a hint that these places may have some object and the model will only tell me whether there is an object there or no at a particular location?

Teacher

The problem that you highlighted is indeed a caveat in OverFeat. The R-CNN paper does somewhat similar to what you intuitively felt should be done. It uses a Selective Search algorithm to detect possible locations of an object in an image and sends only those sections of images (around 2000 region proposals) to the AlexNet network. They replaced the last Softmax layer with an SVM layer and trained only the SVM layer for classification on Pascal VOC 2007(details about training will be explained later). Moreover, the Pascal VOC dataset has 20 classes, but they trained the SVM for 21 classes, where the extra class corresponds to the background class where there is no object. With this method, they were able to achieve an mAP of 44.7%. Have a look at the architecture!

R-CNN Network architecture — paper

Student

Those are really good results for object detection on the Pascal VOC, as they were much better than the contemporary models of that time. However, I had this one query, since the ImageNet dataset comes from a different distribution as compared to the Pascal VOC dataset, shouldn’t the authors have fine-tuned the model? Based on the literature I am aware of, we generally fine-tune a model if the dataset distribution changes. And also, convolution networks take input of fixed size i.e., 227 x 227 in the case of AlexNet, so how did they apply that constrained on the regions proposed by the selective Search Algorithm?

Teacher

The authors first warped the images to 227 x 227 size and then passed them to the input and that’s when they got an mAP of 44.7% after training the SVM. While warping the images the objects in the image may be stretched, elongated, squashed, etc and they may even hinder the performance. Hence the authors first fine-tuned the entire AlexNet network for classification. After that, the authors replaced the Softmax layer with the SVM layer and trained only this layer. Due to this fine-tuning, the mAP increased from 44.7% to 54.2%.

There one more detail about the data used for fine-tuning and SVM training. The pre-trained network is trained on the ImageNet dataset. While fine-tuning the model the authors used the Pascal VOC 2007 dataset. During fine-tuning i.e., with the softmax layer at the top, the images containing IoU ≥ 0.5 were used as ground-truth positive box and rest as a negative class (background, no object). However, for training the SVM, they used only images that are ground-truth as training and with an IoU < 0.3 as negative class. The remaining grey zone proposals were ignored. The authors also replaced the SVM with 21 way Softmax and the mAP dropped from 54.2% to 50.9%.

Note: Details about selecting positive and negative samples for Softmax and SVM are provided in the Appendix-B of the paper

Student

This is the first time I came across such a method for fine-tuning. Are there any other peculiarities in the RCNN paper?

Teacher

Yes, there are. The authors have done an exhaustive ablation study about their architecture. Most of the discussion lies around the necessity of the Dense Layers and fine-tuning the network. Let’s look at it this way.

Without Fine-tuning

The authors extracted features from the Pool5 layer of the Network, then from the fc6 layer, and then from the fc7 layer. For each of these extracted features, they trained the SVM layer separately and produced the mAP. Have a look at the image below.

mAP without fine-tuning — paper

From the last column in the above image, we see that removing both the FC layers did not harm the performance much. Moreover, the features from fc6 layers are better predictors than the ones from the fc7 layer. Here’s what the authors have said after this observation.

This means that 29%, or about 16.8 million, of the CNN’s parameters can be removed without degrading mAP. More surprising is that removing both fc7 and fc6 produces quite good result seven though pool5 features are computed using only 6% of the CNN’s parameters.

With Fine-tuning

The authors have now fine-tuned the network in various ways. First, when they used the features directly from Pool5 and sent it to SoftMax for fine-tuning the entire network. In this case, they saw a jump of 3% to the Pool5 features used without fine-tuning. Next, they also did the same for the fc6 and fc7 layers and saw an increase to 53% and 54.2% respectively.

mAP with fine-tuning — paper

You can compare the mAP of the results obtained after tuning to those without tuning shared in the image above. Fine-tuning has considerably improved the results. However, there is something special about the 4th row, let’s discuss that !!

Using Bounding Box Regressor

The authors have added a bounding box regressor to the output of the pool 5 layers. After doing that they saw a considerable improvement in the performance of 58.5% mAP. This is an almost 14% hike in mAP to the model which wasn’t fine-tuned. This bounding box regression was trained with l2 loss and was penalized the same with lambda = 1000.

Note: Details about the bounding box regressor are in Appendix-C of the paper.

The authors also tried replacing the AlexNet with VGG-16 and found a more 8% increase in mAP while using the bounding box regressor.

mAP with different model architectures — paper

I would now refer you to read the paper for a deeper dive into the paper and implementation details. They have also provided a detailed explanation of the ILSVRC2013 detection dataset and also compared their models with other models.

Student

That was a pretty exhaustive analysis done by the authors. This paper may be a bit old, but the way the analysis of model architecture is performed is noteworthy. These strategies, of experimentally verifying the contribution of a layer to the improvement in accuracy, could be used while designing any other types of models for even some other tasks.

Teacher

Yes, they have performed a detailed and comprehensive analysis of their model and also compared the same with the contemporary models. Just on a final note, and for definition sake, a model like R-CNN is called two-stage model. Such models have a separate algorithm/model (selective search) for detecting the possible location of the object and a separate model for detecting the presence of the object (AlexNet). And the training of this model (including the bounding box regressor) is a three-stage process, where the first stage, the model is fine-tuned, in the second stage, the SVM classifier is trained and in the third stage, the bounding box regressor is trained.