The SpaceNet Challenge Off-Nadir Buildings: Introducing the winners
By: Nick Weir
Tags: Machine Learning, Geospatial, Computer Vision, Spacenet, Deep Learning

Off-nadir imagery of Atlanta, GA. Imagery courtesy of DigitalGlobe.

SpaceNet’s mission is to accelerate geospatial machine learning and is supported by the SpaceNet LLC member organizations.

The SpaceNet Challenge: Round 4, Off-Nadir Building Footprint Extraction hosted by TopCoder is complete! Congratulations to cannab, selim_sef, MaksimovKA, number13, and XD_XD for taking home the top 5 prizes. Check out this GitHub repository for their solution algorithm code. Their solutions represented a 1.5-fold improvement over our initial baseline model’s performance. This is the first post in a series where we’ll dig into the competitors’ solutions to find out how they applied machine learning to identify buildings in overhead imagery taken at look angles from 7 to 54 degrees off-nadir.

The Challenge

In Round 4 of the SpaceNet Challenge we asked competitors to identify building footprints from 27 different satellite collects taken during a single pass of a WorldView 2 satellite over Atlanta. The data comprised both South-facing and North-facing collects with look angle ranging from 7 degrees off-nadir to a whopping 54 degrees off-nadir. For more details about the challenge (or if you’re unfamiliar with that terminology), check out these earlier posts describing the competition:

Introducing the SpaceNet Off-Nadir Imagery Dataset

A Baseline Model for the SpaceNet Off-Nadir Building Detection Challenge

Challenges with the SpaceNet 4 off-nadir satellite imagery

We had 238 registrants and 21 competitors beating our baseline model in this extremely challenging competition.

Key takeaways

Before we get into the nitty-gritty, here are our main takeaways from running and scoring the challenge:

Building detection in “ideal” imagery is still far from perfect

In these perfectly cloudless, high-resolution Worldview 2 images over Atlanta, even when we restrict to the images looking nearly straight down (look angle of <25 degrees off-nadir, the “nadir” set), the best algorithm still missed about 21% of buildings.

The winning algorithm correctly identified about eight out of ten buildings in images taken at nadir.

Furthermore, about one out of every six buildings the winner predicted in the nadir set did not match up to a building on the ground (i.e. was a “false positive”).

For every six predictions the winner made from nadir images, roughly five corresponded to buildings.

There’s still a lot of room for improvement in automated analysis of overhead imagery!

It is very hard to identify buildings at very off-nadir look angles

Performance at look angle >40 degrees off-nadir was an average of 30% worse than <25 degrees off-nadir. The winners only positively identified about 50% of the buildings in these very off-nadir images:

The winning algorithm correctly identified about five out of ten buildings in images taken very off-nadir.

…and about one in three of their predictions were false positives:

For every six predictions the winner made from very off-nadir images, about four corresponded to buildings.

If you want to use algorithms to identify objects in very off-nadir imagery, be aware of these performance limitations.

Big ensembles of models help, but the juice may not always be worth the squeeze

The winning algorithm, an ensemble of 28 convolutional neural networks (CNNs) with a gradient boosting machine (GBM) filter to remove bad buildings yielded only a 5% improvement over the #5 algorithm, which comprised only 3 CNNs. This came at a substantial computing cost: the winning algorithm takes more than 10 times longer to identify buildings in new images than the 5th place competitor’s algorithm! With a difference of 6.5 vs. 68 seconds per square km of prediction, this may represent the difference between a deployable solution and an algorithm that is only academically relevant.

Let’s dive into the details!

Winners’ algorithm performance

A summary of the winners’ results alongside the baseline model are in the table below.

Competitors’ scores in the SpaceNet 4: Off-Nadir Building Detection Challenge compared to the baseline model. Each score represents the SpaceNet Metric for the entire image set (Overall) or subsets of the imagery with specific look angles: Nadir, 7–25 degrees; Off-nadir, 26–40 degrees; Very off-nadir, >40 degrees.

The results were remarkably close — cannab’s winning solution represents only a 5% improvement over XD_XD’s 5th place solution. Much of the competitors’ improvements over the baseline came in the off-nadir and very off-nadir image subsets, where the winning solutions represented 79% and 170% improvements over the baseline model. By contrast, the algorithm with the best performance in the nadir set only improved on the baseline by 32%:

Improvement in competitor algorithm over baseline, stratified by look angle. Nadir: 7–25 degrees, Off-nadir: 26–40 degrees, Very off-nadir: >40 degrees.

So, how did the competitors achieve these improvements? This will be the topic of the remainder of this post and several deep dives to come.

A summary of the winners’ algorithms

Every top 10 competitor who submitted a solution used convolutional neural networks (CNNs) to identify building footprints. Each competitor produced a binary output image with each pixel labeled as 1 (contains a building) or 0 (does not contain a building), which they then converted to polygons outlining individual buildings. The intricacies of the competitors’ algorithms, summarized below, varied dramatically.