Airplanes Detection for Satellite using Faster RCNN | by Shubhankar Rawat | Towards Data Science

Object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class (such as humans, buildings, or cars) in digital images and videos.
Object detection is one of those fields that have witnessed great success.It is used in many areas like face detection(used by Facebook to recognize people), tumor detection(used in medical fields), etc.
Ever since the inception of deep learning in computer vision, tasks like object detection have become comparatively easier and efficient.The deep learning models provide better accuracy, less time consumption, less complexity, overall better performance than the earlier computer vision approaches.Deep learning provided outstanding results over the traditional computer vision methods for object detection, leading to the wide use of deep learning models.
One of the best performing object detection(deep learning) algorithms include :1. RCNN (Region-based Convolution Neural Network)2. Fast RCNN3. Faster RCNN4. SSD (Single Shot Multibox Detector)5. YOLO (You Look Only Once)
In this article, I will be using Faster RCNN(FRCNN) to demonstrate the eminent use of object detection.
The dataset is taken from kaggle, you can find it hereIt contains 400 training images and 100 test images. The bounding boxes are saved in both XML and CSV files.
The dataset contains computer-generated satellite images of planes.
As you can see from the dataset, some images do not contain highly differentiable objects(airplanes) from the background, i.e., the airplanes are somewhat similar to the background in terms of color, texture, and appearance. For example, consider the following image :
How many airplanes can you see??Can you even see an airplane???!!Well, this is an image from the training set and you can see how it is really hard to figure out the airplanes in the image(even for a human).
Here is the image by the way, with the bounding boxes, showing that there were 2 airplanes. (Did you figure it out??)
But will there be an effect on the model performance due to such images where it is hard to distinguish between background and object?If you have a strong computer vision background, then you must be knowing that initial approaches for object detection first converted the image to grayscale then looked for edges or gradient changes ... All this stuff would be highly affected by the appearance of the object in the image. If the image and the object were quite similar(like in the above image) then these approaches will fail. This condition is called clutter, where it becomes difficult to distinguish between the object and the background.But will it affect the deep learning models as well???What these neural nets are searching for is how they can distinguish the object from the background(recognition), so that they can enclose them in a bounding box(localization). These are called features that are responsible for object detection.Such features are hard to find if there is less variation between the object and the background.So, this dataset is a good example, for testing the performance of deep learning model-FRCNN.
You can find the python code for this problem here
To begin with, 400 images is no place to be. For deep learning models to perform at their best we need loads of data.
It is a well-known fact that the performance of deep learning models is proportional to the amount of training data. More the data, the better the performance.Models like FRCNN require huge amounts of training data to produce good results. However, there are very few (if not none) datasets for object detection which have ample training data.In this dataset also we have only 400 images, which are not enough to produce good results.So, to tackle this data augmentation comes to rescue.Again this dataset is a good example to demonstrate how data augmentation is essential for deep learning models to produce good results.
Data augmentation is a technique to artificially create new training data from existing training data.If you have been working with deep learning models for computer vision(be it image classification or localization or detection etc) then you must have encountered the term data augmentation, and realize the importance of it.
Data augmentation generates new images for training and hence improving the model performance.Why does data augmentation work?? Modern deep learning algorithms, such as the convolutional neural network, or CNN, can learn features that are invariant to their location in the image. Nevertheless, augmentation can further aid in this transform invariant approach to learning and can aid the model in learning features that are also invariant to transforms such as left-to-right to top-to-bottom ordering, light levels in photographs, and more.
There are various data augmentation techniques, the following are used in this approach :1. Horizontal flip2. Scaling3. Translation4. Rotation 5. Shearing6. Various combinations of above
Data augmentation is different for object detection when compared to image classification. In image classification, you only need to change the image accordingly, but in object detection, you need to change and adjust the bounding boxes as well. For example, if you are flipping the image horizontally then you also need to flip the bounding boxes.
Keeping the bounding box constraint in mind I implemented data augmentation and produced 4400 images in total.Still not a high number of images but because there are only 100 images to test for and only one class to detect, these many will do.But I highly recommend increasing the training data as high as possible without creating unnecessary data(which often happens during data augmentation).
Now that we have some data to train on, let’s implement the model.
I recommend you to download the code files from the GitHub repository mentioned above, to make the best of this article.
Faster RCNN is a great deep learning model that performs extremely well, which we are about to figure out.
First things first, organizing the data.The training set images were moved to a folder named train_images and the test set images were moved to folder test_images.An empty folder results _imgs was created, this folder will contain all the output images with the bounding boxes.
All the required libraries and packages were installed by executing the following line in the command prompt(with the current folder as the working directory):
pip install -r requirements.txt
Annotate file(annotate.txt) was created, which looks like :
It contains the folder name of the train set images, with the image name and the bounding box coordinates and the class of the object.Note that the bounding box coordinates have the order: xmin, ymin, xmax, and ymax.
The entries in the annotate file will vary depending upon how the data augmentation is implemented.
The model was run by executing the following line in the command prompt :
python train_frcnn.py -o simple -p annotate.txt
I have run the code for 800 epochs with 20 iterations per epoch, one can change these numbers for better accuracy, like increasing the number of epochs.
It is highly recommended to use GPU for running the above code as training can take a lot of time, especially for a high number of epochs.
After the training has completed, it is time to test the model.To get the output images, the following command was executed :
python test_frcnn.py -p test_images
As the command is executed one can see output images with the bounding boxes being saved in the results_imgs folder.
Here are some of the results I attained :
The above images show how FRCNN excels in its task of object detection.The model performance can be enhanced by generating more quality images using data augmentation and increasing the number of epochs.
Now, let us look at the results obtained when no data augmentation is done, keeping the number of epochs and iterations the same.
As one can see that the results are not as good as the previous ones, where data augmentation was implemented. The model fails to identify the airplanes correctly rather identifies non-airplane objects as airplanes. That is, the number of false positives and false negatives is high.
This shows how data augmentation helps in reducing the error and increasing the performance of the deep learning models.
Object detection is a great field to explore.I have seen many object detection models used in various places like for self-driving cars, mobile phones, security, etc but using it for satellites was somewhat new to me.
Also, Faster RCNN lives up to its name and produces great results.
My sole purpose in this article was to share the use of Faster RCNN and not how to implement it, so if you face any errors running the above code or any doubts, feel free to ask in the comment section.