Detecting pedestrians and bikers on a drone with Jetson Xavier | by Jaiyam Sharma | Towards Data Science

Drones are one of the coolest technologies every maker and enthusiast wants to lay their hands on. At the same time as drones are becoming common, AI is rapidly advancing and we are now in a state where object detection and semantic segmentation are possible right onboard the drone. In this blog post, I will share how to perform object detection on images taken by a drone.
A few basics about the problem at hand
First, it is important to realize that we cannot realistically run object detection on general purpose embedded hardware such as the raspberry pi and special purpose hardware built for AI inferencing is required.
Second, if you run commonly used object detection models, such as YOLO or SSD trained on COCO and Pascal VOC datasets, they won’t do well at all. This is because the view of an object from a height is quite different from that on the ground. Thus, the distribution of the inference data will be very different from that encountered by the model during training, which will cause it to fail.
Some solutions
As mentioned in the title In this post, I will be using the highest end embedded processor for autonomous robots available at the time of writing, Jetson AGX Xavier from Nvidia. If you want to use a Jetson TX2 or nano, I will provide some suggestions for improving their performance towards the end of the post.
To remedy the problem of differing data distribution, researchers at Stanford University have released a dataset called Stanford Drone dataset, which contains several videos taken from drones along with labels for each frame of each video. There are six classes being detected: Biker, Car, Bus, Cart, Skater and Pedestrian.
Any decent object detection model trained the Stanford dataset should do a good job of detecting these six objects. In this post, we will be using RetinaNet, a very good object detection model released by Facebook (FAIR), which shapes the loss function in such a way that the model learns to focus on hard examples during training and thus learns much better. More details about RetinaNet can be found here.
Although the Detectron 2 model zoo by FAIR has several models which can be downloaded and used, they are all trained on COCO and Pascal datasets, which as we discussed are not useful for our task. It would be a big task to train a model from scratch on the Stanford dataset. Fortunately, I found that Nvidia provided a model trained on this dataset as a part of one of their webinars on DeepStream SDK. I am not trying to endorse Nvidia in this blog post, but if you want to get started without training your own model, the quickest way would be to register for the webinar and download the model as a part of the webinar resources. The file you are looking for is stanford_resnext50.onnx which is about 150 MB in size.
Once you have this file, you can use the C++ API of retinanet-examples repository from GitHub to convert the onnx file to an engineplan file which is compiled specifically for the Jetson device you are using. Here is a walkthrough of these steps:
git clone https://github.com/NVIDIA/retinanet-examples.gitcd retinanet-examples/extras/cppapimkdir build && cd buildcmake -DCMAKE_CUDA_FLAGS="--expt-extended-lambda -std=c++11" ..makecp /path/to/onnx/file . #copy onnx file to this directory./export stanford_resnext50.onnx engine.plan
After engine.plan has been created successfully, you can use the infer utility to test out the performance of the model on an image:
./infer engine.plan image.jpg
This will write a file called detections.png which will have the bounding boxes of the objects detected in the image. I gave an image from one of the videos in the Stanford dataset as input to the model and here is what it spat out:
We can see that the model has detected several pedestrians walking on the sidewalk. This would not be possible with Pascal or COCO trained models. The aspect ratio of this image looks odd because the infer utility resizes the image to 1280x1280, which is the input size of the model.
Now that we can detect pedestrians in a single image, it is straightforward to extend this to videos by editing the infer.cpp file in cppapi directory. I came up with this:
To use this script, save it as infervideo.cpp in the cppapi directory and edit CMakeLists.txt to add the lines to add the infervideo executable and link it to retinanet and other libraries.
Once you are done, you can switch over to the build directory and invoke cmake and make as before:
cmake -DCMAKE_CUDA_FLAGS="--expt-extended-lambda -std=c++11" ..make
Once the targets have been built you will see a new executable in the build directory called infervideo, which can be used as:
./infervideo engine.plan input.mov output.mp4
This will create a new video called output.mp4 which will show the bounding boxes for each object. If you want to perform object detection on a live video stream from the drone, you can simply provide the gstreamer pipeline of your camera as the second argument to the script and it will handle cameras as well.
I used the script above to run inference on a video taken at the IIT Delhi campus in New Delhi, my alma mater. I used a low threshold of 0.2 for drawing bounding boxes, which is why there are some false positives in the video below.
If you run the script provided above on the Xavier, you will find that each frame of the video takes 150 ms for inference. This is painfully slow on the Xavier and would be even slower on the smaller Jetsons like TX2 or nano. Here are some things we can do to improve the performance:
The engine we created in this post was for fp16 precision. You can run this model on integer precision, which will improve its performance significantly. For this, you can use a small subset of the data from the Stanford dataset and the trtexec utility from tensorRT to create an INT8 calibration file and provide that file to export utility in our build directory.
In practice, any realtime object detection pipeline does not perform full inference on each frame, but usually mixes it up with computationally inexpensive trackers such as the Kalman filter or optical flow. You can use opencv’s KalmanFilter class to keep track of objects across frames and perform inference only once per 4 or 5 frames. If the drone does not have a sudden jerky motion between inferences, this will work well in practice.
The model we are using is very large as it takes in 1280x1280 images. You could train the model on lower resolution images or even a custom dataset to significantly improve the latency and throughput of the model. Instructions for training the model are on retinanet-examples repository, but this is best done on a good x86 workstation with a CUDA enabled GPU.
This blog post is meant for anyone who having trouble deploying a retinanet model on Jetson Xavier and to chronicle my efforts towards getting a good object detection pipeline running on a drone. While we get significantly better results than would be possible with a Coco/Pascal model, there are many improvements to be made to make the model run in realtime on a Jetson device. Please use this as a starting point for your own projects and if you have any other suggestions to improve the performance, please do comment below.