Announcing YOLTv4: Improved Satellite Imagery Object Detection | by Adam Van Etten | Towards Data Science

Preface: Though CosmiQ Works (and its associated blog: The DownLinQ) has unfortunately been shut down, there remains much to be done in the geospatial analytics domain. Accordingly, this blog details work performed independently of IQT and in my spare time.
In a number of previous blogs [e.g. 1, 2, 3] and academic papers [e.g. 4, 5, 6] we’ve demonstrated the striking efficacy of adapting YOLO to detect objects in satellite imagery. Recall that YOLO is a leading deep learning object detection framework, designed to detect objects in imagery. YOLO maxes out at image sizes of a few thousand pixels in size, far too small to handle large scale satellite imagery which can exceed 100 million pixels. Therefore we built YOLT (and extended YOLT with SIMRDWN) to optimize this object detection framework for satellite images of arbitrarily large size.
We are now pleased to announce that an improved version, YOLTv4, has been released that is both faster and more performant. The code is open-sourced at github.com/avanetten/yoltv4. In the sections below we detail this new repository and provide example results.
1. Introduction
YOLTv4 is designed to rapidly detect objects in aerial or satellite imagery in arbitrarily large images that far exceed the ~600×600 pixel size typically ingested by deep learning object detection frameworks. This repository is built upon the impressive work of AlexeyAB’s YOLOv4 implementation, which improves both speed and detection performance compared to YOLOv3 (which is implemented in SIMRDWN). We use YOLOv4 insead of “YOLOv5”, since YOLOv4 is endorsed by the original creators of YOLO, whereas “YOLOv5” is not; furthermore YOLOv4 appears to have superior performance.
YOLTv4 therefore combines the impressive YOLOv4 backbone with the pre- and post-processing scripts of SIMRDWN to create an improved, lighter-weight alternative SIMRDWN. Recall that SIMRDWN allows users to select from a multitude of backbones (e.g. YOLO, SSD, Faster-RCNN, etc.). While this flexibility is quite useful, it does balloon the code base; furthermore multiple studies [5, 6] have shown that YOLO backbones outperform other methods for satellite imagery object detection. Therefore, in YOLTv4 we opt to forge a simpler codebase and focus only on the YOLOv4 backbone.
Below, we provide examples of how to use this repository with the open-source Rareplanes dataset.
In this section we demonstrate how to train and test YOLTv4 using the Rareplanes dataset to localize aircraft.
2.1. Installation
YOLTv4 is built to execute within a docker container on a GPU-enabled machine. The docker command creates an Ubuntu 16.04 image with CUDA 9.2, python 3.6, and conda.
Clone this repository (e.g. to /yoltv4/).Download model weights to yoltv4/darknet/weights (e.g. see: https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.conv.137).Install nvidia-docker.Build docker file. nvidia-docker build -t yoltv4_image /yoltv4/dockerSpin up the docker container (see the docker docs for options). nvidia-docker run -it -v /yoltv4:/yoltv4 -ti — ipc=host — name yoltv4_gpu0 yoltv4_imageCompile the Darknet C program. First Set GPU=1 CUDNN=1, CUDNN_HALF=1, OPENCV=1 in /yoltv4/darknet/Makefile, then:cd /yoltv4/darknet  make
Clone this repository (e.g. to /yoltv4/).
Download model weights to yoltv4/darknet/weights (e.g. see: https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.conv.137).
Install nvidia-docker.
Build docker file. nvidia-docker build -t yoltv4_image /yoltv4/docker
Spin up the docker container (see the docker docs for options). nvidia-docker run -it -v /yoltv4:/yoltv4 -ti — ipc=host — name yoltv4_gpu0 yoltv4_image
Compile the Darknet C program. First Set GPU=1 CUDNN=1, CUDNN_HALF=1, OPENCV=1 in /yoltv4/darknet/Makefile, then:cd /yoltv4/darknet  make
2.2. Train
A. Prepare Data
Make YOLO images and labels (see yoltv4/notebooks/prep_data.ipynb for further details).Create a txt file listing the training images.Create file obj.names file with each desired object name on its own line.Create file obj.data in the directory yoltv4/darknet/data containing necessary files. For example /yoltv4/darknet/data/rareplanes_train.data: classes = 30train = /local_data/cosmiq/wdata/rareplanes/train/txt/train.txtvalid = /local_data/cosmiq/wdata/rareplanes/train/txt/valid.txtnames = /yoltv4/darknet/data/rareplanes.namebackup = backupPrepare config files. See instructions here, or tweak /yoltv4/darknet/cfg/yoltv4_rareplanes.cfg.
Make YOLO images and labels (see yoltv4/notebooks/prep_data.ipynb for further details).
Create a txt file listing the training images.
Create file obj.names file with each desired object name on its own line.
Create file obj.data in the directory yoltv4/darknet/data containing necessary files. For example /yoltv4/darknet/data/rareplanes_train.data: classes = 30train = /local_data/cosmiq/wdata/rareplanes/train/txt/train.txtvalid = /local_data/cosmiq/wdata/rareplanes/train/txt/valid.txtnames = /yoltv4/darknet/data/rareplanes.namebackup = backup
Prepare config files. See instructions here, or tweak /yoltv4/darknet/cfg/yoltv4_rareplanes.cfg.
B. Execute Training
Execute. cd /yoltv4/darknet  time ./darknet detector train data/rareplanes_train.data cfg/yoltv4_rareplanes.cfg weights/yolov4.conv.137 -dont_show -mjpeg_port 8090 -mapReview progress (see Figure 1 below):
Execute. cd /yoltv4/darknet  time ./darknet detector train data/rareplanes_train.data cfg/yoltv4_rareplanes.cfg weights/yolov4.conv.137 -dont_show -mjpeg_port 8090 -map
Review progress (see Figure 1 below):
2.3. Test
A. Prepare Data
Make sliced images (see yoltv4/notebooks/prep_data.ipynb for further details).Create a txt file listing the training images.Create file obj.data in the directory yoltv4/darknet/data containing necessary files. For example /yoltv4/darknet/data/rareplanes_test.data:  classes = 30  train =  valid = /local_data/cosmiq/wdata/rareplanes/test/txt/test.txt  names = /yoltv4/darknet/data/rareplanes.name  backup = backup/
Make sliced images (see yoltv4/notebooks/prep_data.ipynb for further details).
Create a txt file listing the training images.
Create file obj.data in the directory yoltv4/darknet/data containing necessary files. For example /yoltv4/darknet/data/rareplanes_test.data:  classes = 30  train =  valid = /local_data/cosmiq/wdata/rareplanes/test/txt/test.txt  names = /yoltv4/darknet/data/rareplanes.name  backup = backup/
B. Execute Testing
Execute (proceeds at >80 frames per second on a Tesla P100): cd /yoltv4/darknettime ./darknet detector valid data/rareplanes_test.data cfg/yoltv4_rareplanes.cfg backup/ yoltv4_rareplanes_best.weightsPost-process detections, stitch images back together, and make plots:
Execute (proceeds at >80 frames per second on a Tesla P100): cd /yoltv4/darknet
time ./darknet detector valid data/rareplanes_test.data cfg/yoltv4_rareplanes.cfg backup/ yoltv4_rareplanes_best.weights
Post-process detections, stitch images back together, and make plots:
time python /yoltv4/yoltv4/post_process.py \ — pred_dir=/yoltv4/darknet/results/rareplanes_preds_v0/orig_txt/ \ — raw_im_dir=/local_data/cosmiq/wdata/rareplanes/test/images/ \ — sliced_im_dir= /local_data/cosmiq/wdata/rareplanes/test/yoltv4/images_slice/ \ — out_dir= /yoltv4/darknet/results/rareplanes_preds_v0 \ — detection_thresh=0.25 \ — slice_size=416 \ — n_plots=8
Outputs will look something like the figures below.
YOLTv4 is highly performant and quite rapid. On a single Tesla P100 GPU inference proceeds at ~2 square kilometers a second even at the highest resolution 30 cm satellite imagery. At this rate a simple 4-gpu cluster could process the entire WorldView-3 collect in approximately real time. So feel free to give the code a spin to find cars, boats, infrastructure projects, aircraft, buildings, or whatever objects of interest suit your fancy.