Understanding Object Detection. Detect image objects with Deep Learning... | by Vincent Tatan | Towards Data Science

Imagine Google Photos: for all of the pictures you have, how do you label those by objects. Do you want to tag them one by one?
How about automated cars driving? How do they detect pedestrians, cars, traffic lights, and impending obstacles?
Enter Object Detection algorithms
In recent years, image classification has gained huge traction especially with CNN and disruptive applications (e.g: self driving cars). In 2015, the ImageNet competition produced RESNet which performed better than human’s.
While image classifications (e.g: CNN stacks) worked really well to classify images, it still has limitations to detect multiple objects of interest within an image and understanding how they are placed within the image.
Object detections took one step further from image classifications, drawing one or many bounding boxes of objects of interest.
In this tutorial, we are going to learn about object detections methods with R-CNN and YOLO to detect objects fast.
towardsdatascience.com
WHERE is the object?
To localize objects, we need to identify anchors, the focus of objects of interests. This comes from the region proposal and is used to customize horizontal/vertical elongated boxes. For every position, there is an anchor.
We create a model to generate regional proposals which mark regions of interests. We assign bounding boxes and annotate picture segments that represent those particular region proposals. This includes multiple approaches such as: sliding window (brute force), selective search (clustering), and region proposal network (deep learning).
WHAT is in the image?
We extract all visual features to bounding boxes and evaluate each bounding box on the region proposals. This includes kernels and histogram functions.
For every shape we will calculate the regressor to check how deviate the ground truth with the anchor. We also talk about Region proposal which maintains object of interests represented by the backbone architecture (e.g: CNN/VGG16/etc)
We seek the possible bounding boxes which capture the anchors within Intersection over Union (IOU). This allows us to reduce all overlapped boxes to a single representative bounding box.
R-CNN selects a huge number of regions by proposing selective search to extract regions from images (aka. region proposals). The selection search will 1) generate sub-segmentation to generate candidate regions, 2) use a greedy algorithm to combine similar regions, and 3) use generated regions with final candidate regions.
The warped regions are fed to CNN Classifiers which extract feature maps to feed into Support Vector Machines (SVM). SVM classifies if the object is present within the warped regions. Then the regressor will tune and set up the bounding box to represent the region proposals.
Despite its effectiveness in determining representative regions, R-CNN is still inefficient because it takes a huge amount of time to train with all 2000 warped regions. The selective algorithm could also lead to generations of bad candidates which slow down the process.
You need to have multiple CNN classifiers. In the 2014 paper. This uses faster selected search. Fast R-CNN is also called double shot because it runs both region proposals and CNN directly.
Since the Fast RNN region proposal might be wasteful, we can actually run the region proposal to use feature maps as the inputs. This is faster and more direct because feature maps already include objects of interest.
SSD focuses on the speed by calculating the bounding box and image classification from each convolutional layer. SSD uses VGG19 with backbone architecture with whatever architecture we want. We can use RESnet but it defeats the purpose due to its complexity because it is meant for speed. Another choice is MobileNet which is used for small object detection programs.
You can take a look at tf2_detection_zoo for a complete list of open source algorithms.
You Only Look Once (YOLO) is similar to SSD but rather than calculating the bounding box by feature maps, YOLO uses the darknet as CNN stack backbone and aggregates the last three layers YOLO uses dark net as the backbone.
Transformer based detectors (Facebook). Implemented to detect objects in images as well.
What about Mask RCNN? Mask is image segmentation. It is another application of object detection. This shows what exactly the segmentation is.
This tutorial shows how to classify objects with images. You will get some practical experience and develop intuition for the following concepts:
Out of box detection with pre-trained model from Tensorflow Object Detection library with MSCoco Dataset
Transfer Learning by Modifying existing object detection architecture with a novel class outside out of box detection. This training will run in eager mode (TF2)
It takes ~1 hour to run through this colab with GPU. Please feel free to open the colab below and get started :)
drive.google.com
We are using TensorFlow Object Detection API (ODAPI) library to iterate over different object detection configurations with TF backend.
TF-Slim is a lightweight library to define, train, and evaluate TF models. This library is developed by Google AI Brain Team and at the moment of writing is still in development so use it at your own risk (might have fast deprecations).
pycocotools provides a huge image dataset for object detections and image segmentations. COCO dataset
!pip install tf_slim!pip install pycocotools
We’ll use “MobileNet SSD” (model zoo) to attempt our out-of-the-box model.
Any model exported using the export_inference_graph.py tool can be loaded by specifying the model name and path.
model_name = ‘ssd_mobilenet_v1_coco_2017_11_17’detection_model = load_model(model_name) # a helper function to access http://download.tensorflow.org/models/object_detection
Call the models
Generate bounding box with masks (shapes for annotations)
model_fn = model.signatures[‘serving_default’]output_dict = model_fn(input_tensor)if 'detection_masks' in output_dict:# Reframe the the bbox mask to the image size.detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(output_dict['detection_masks'], output_dict['detection_boxes'],image.shape[0], image.shape[1])detection_masks_reframed = tf.cast(detection_masks_reframed > 0.5, tf.uint8)output_dict['detection_masks_reframed'] = detection_masks_reframed.numpy()
Show the images with IPyImage display function:
display(IPyImage(open(anim_file, ‘rb’).read()))
Notice that the annotation is calling raccoons as cat/dog/bird/baseball gloves. This means that the model trained on top of the coco dataset does not know how to identify raccoons.
We can use our prepared train set to train our model to identify raccoons. An easy way to do this is to annotate using the colab_utils tool from object_detection.utils .
We can use pre trained models to finetune based on a novel class. In this case, we will use RetinaNet architecture and avoid wasting time during experimentation and fine tuning. Tensorflow has object detections under tensorflow/models/research/object_detection.
In this cell we build a single stage detection architecture (RetinaNet) and restore all but the classification layer at the top (which will be automatically randomly initialized).
The goal is to load pipeline config and build a detection model. Since we are working off of a COCO architecture which predicts 90 class slots by default, we override the `num_classes` field here to be just one.
Given that we only annotate 10 images. We do not need to create a batch input and could just run the model through the test image.
#more code in the colab.image, shapes = detection_model.preprocess(tf.zeros([1, 640, 640, 3]))prediction_dict = detection_model.predict(image, shapes)_ = detection_model.postprocess(prediction_dict, shapes)print(‘Weights restored!’)optimizer = tf.keras.optimizers.SGD(learning_rate=learning_rate, momentum=0.9)train_step_fn = get_model_train_step_function(detection_model, optimizer, to_fine_tune)
With the right model and right annotations for training. These 10 images should be sufficient to mark the image as raccoon
In Microsoft coco there are ~1000 images with 80 objects. But there are times when we want to train the model on objects which never exist.
Personally I found labelimg open sourced by an Amazon engineer to be the most useful tool for labeling. But if you do not want to use it, we always have Tensorflow helper functions to help you label and annotate.
With transfer learning you don’t need much data, in my tutorial, you can run this within 12 images. But of course it depends on the use cases. If you need to run SSD at night with dark images, then you might need to upload many images on that as well.
The more data you have the better, however as more data is inputted, you will find diminishing returns. Most importantly is to have the data that represents what you need. This is the case with siamese networks where the same configurations with the same parameters and weights work for multiple labels.
Most of the time, we will use microservice to launch. This means we will dish up a server, give API gateway to connect with other services, and give the results whenever requests come. For the type of the model, it really depends whether you prioritize accuracy (R-CNN) or speed (SSD). There is a really cool paper which I refer to here
R-CNN, Fast R-CNN, Faster R-CNN, YOLO
MLSS Tutorial as the Base Code credit to Rony Kalfarisi
arxiv.org
Vincent fights internet abuse with ML @ Google. Vincent uses advanced data analytics, machine learning, and software engineering to protect Chrome and Gmail users.
Apart from his stint at Google, Vincent is also Georgia Tech CS MSc Alumni, a triathlete, and a featured writer for Towards Data Science Medium to guide aspiring ML and data practitioners with 1M+ viewers globally.
Lastly, please reach out to Vincent via LinkedIn, Medium or Youtube Channel