How to Get Started With Facebook’s Detectron2 | by Rob | Towards Data Science

Among many different techniques for object detection, Facebook came up with its model: Detectron2. This model, similarly to Yolo models, is able to draw bounding boxes around objects and inference with a panoptic segmentation model, in other words, instead of drawing a box around an object it “wraps” the object bounding its real borders (Think of it as the smart snipping tool from photoshop.)
The purpose of this guide is to show how to easily implement a pretrained Detectron2 model, able to recognize objects represented by the classes from the COCO (Common Object in COntext) dataset. This guide is meant to provide a starting point for a beginner in computer vision, it aims at explaining what are the first steps to implement a pre-trained model, and its final goal is to spike your interest into learning more, and arranging your thoughts in this overwhelming field. I provide a Google Colab Notebook that you can clone and use as you wish or you can follow up and code along!
To get started its core dependencies must be installed:
PyTorch
Cython
Detectron2
OpenCV
Those tasks can take several minutes, and since you need a GPU to run this notebook, I advise you to clone my Colab notebook and write everything from Google Colab, this will also help you become more confident with command line snippets and file management in the cloud!
!pip install -U torch==1.4+cu100 torchvision==0.5+cu100 -f https://download.pytorch.org/whl/torch_stable.html!pip install cython pyyaml==5.1!pip install -U ‘git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'!git clone https://github.com/facebookresearch/detectron2 detectron2_repo!pip install -e detectron2_repo#!pip install opencv-python Google Colab already comes with OpenCV
Once those dependencies are installed you will most probably need to restart your runtime/kernel. Now we need to import some libraries we will need to import, process and infers predictions, among those libraries we have ......
Since I am using Google Colab, the function cv2.imshow() is not available, and I need to use a "patched" version of it if you are running this notebook on your local machine to get rid of it.
# import some common librariesimport numpy as npimport cv2import randomfrom google.colab.patches import cv2_imshow # On your local machine you don’t need it.# import some common detectron2 utilitiesfrom detectron2 import model_zoofrom detectron2.engine import DefaultPredictorfrom detectron2.config import get_cfgfrom detectron2.utils.visualizer import Visualizerfrom detectron2.data import MetadataCatalog
In this guide I am using an image from my google drive, you can mount your Google Drive and use an image of your choice:
from google.colab import drivedrive.mount(‘/content/drive’)
Let’s now get a picture for which we want to run our model:
im = cv2.imread(“./drive/My Drive/turin.png”)cv2_imshow(im)
Once our image is loaded, we need to load our model and a little bit of configuration. First, we load the configurations from Detectron, and then we apply to it a custom configuration coming from COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml. Then the threshold is set, hence the sensitivity of the model while detecting objects (play with these parameters!). The next step consists of loading the checkpoints relative to this model, in other words, the model itself in .pklformat. The next step consists in instantiating the model itself, think like the same you do in scikit-learn when for example you want to make a regression:
reg = LinearRegression() reg.fit(X, y) reg.predict(X_test) However, instead of passing to our predictor an array X_train we pass our image, which corresponds as well in an array (a 3D array to be precise: one axis for the width, another for the height and the last one representing the colors of the image).
The reason we are using a pre-trained model, instead of training our model can be explained by the first two lines from the description file detectron2/MODEL_ZOO:
“This file documents a large collection of baselines trained with detectron2 in Sep-Oct, 2019. All numbers were obtained on Big Basin servers with 8 NVIDIA V100 GPUs & NVLink. The software in use was PyTorch 1.3, CUDA 9.2, cuDNN 7.4.2 or 7.6.3.”.
However, of course, you can use a baseline model to further train it to recognize a specific object, this process is known as Transfer Learning.
cfg = get_cfg()cfg.merge_from_file(model_zoo.get_config_file(“COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml”))cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set threshold for this modelcfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(“COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml”)predictor = DefaultPredictor(cfg)outputs = predictor(im)
If we inspect the outputs generated by the model we are presented with a massive dictionary (please, refer to the Colab Notebook), this dictionary contains, the number of objects detected (num_instances), info about the image and the coordinates of the image where the box should be drawn, together with the labels (pred_classes) for each box (instance). Let's take a look at the first box:
[1314.3333, 485.8268, 1345.7896, 539.3832]
There are 4 values, each of them representing respectively x, y and width, and height of the rectangle. (Sometimes, in OpenCV, when iterating over things such as detected object you do a loop like: for x, y, w, h in ...)
Now we can visualize the image along with the boxes showing predicted objects in the image, to achieve this we must use a specific object that let us merge the predictions with the original image, this object is called Visualizer, once instantiated the Visualizer with our image and the metadata, we can call the draw_instance_predictions method with the outputs as argument:
v = Visualizer(im, MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.5) # Scaling the image 1.5 times, for big images consider a value below 0v = v.draw_instance_predictions(outputs[“instances”].to(“cpu”))cv2_imshow(v.get_image())
Computer vision is a field which behind the obstacles one can have at learning it, also has some entry barriers: often it is difficult to have access to specific hardware, a model can take hours to train, and our small laptops can often freeze, hence it is hard to follow the “classical” trial and error learning path that maybe we followed while experimenting with some machine learning algorithms while cracking scikit-learn!
Since (hopefully) you have now a basic idea of the workflow, from how to import an image to how to display it, as long as how to load a model from detectron2 library, the next step could be trying to retrain it, using (I suggest) some cloud providers, it can be Google Colab, Paperspace Gradient, Azure Notebooks or anything else. Another good forward step could be to try implementing the Detectron2 “rival” Yolo, and maybe implement it in an iOS app using CoreMLtools.