SpaceNet 6: Exploring Foundational Mapping at Scale
By: Jake Shermeyer
Tags: Data Science, Mapping, AI, Computer Vision, Geospatial

Preface: SpaceNet LLC is a nonprofit organization dedicated to accelerating open source, artificial intelligence applied research for geospatial applications, specifically foundational mapping (i.e., building footprint & road network detection). SpaceNet is run in collaboration by co-founder and managing partner, CosmiQ Works, co-founder and co-chair, Maxar Technologies, and our partners including Intel AI, Amazon Web Services (AWS), Capella Space, Topcoder, IEEE GRSS, the National Geospatial-Intelligence Agency and Planet.

Introduction

In this third blog (1, 2) of our SpaceNet 6 post-challenge series, we 1)demonstrate that zbigniewwojna’s 1st place algorithm performs well at scale and 2)quantify the value of multiple SAR revisits to the same location. For both of these experiments we are ultimately interested in how well a model performs across a broad area, rather than on a tile by tile basis. Although tile specific metrics are certainly informative of a model’s performance, what we really want to know is “how accurately can this AI algorithm extract buildings across an entire city.” This is important to quantify so we can begin to understand how AI algorithms would perform in a real-world setting where analyzing vast areas is a baseline requirement.

Additionally, the SAR data in the SpaceNet 6 dataset features 204 individual image strips and frequent revisits of the same geographic area. As such, we can also test if model outputs from multiple revisits can be aggregated together to improve the quality of our building footprint predictions. This will inform how many revisits or how much SAR data is required to optimize performance when mapping static objects like buildings. In the figure below we display the distribution of revisits over Rotterdam which ranges from 1 to 30.

The density of aerial SAR collects over Rotterdam in the SpaceNet 6 dataset ranging from 1 to 30. The median number of image strips for the whole area of interest is approximately 15.

In summary, this blog will examine:

How well would our 1st place algorithm perform in a real world setting when we want to map buildings across a broad area? How do multiple revisits to the same area affect the quality of building footprint predictions? Is there an optimal number of revisits? How much SAR data is required to optimize performance?

The Workflow

To achieve our analysis goals, we construct the following workflow:

I. Pre-Processing: We rework the SpaceNet 6 training and testing data a bit to ensure that we can achieve the best results at scale. We slightly increase the tile size to 1,200 pixels² and intelligently tile to ensure that each tile has as little blank area as possible. The tiles also include a 20% horizontal overlap which will reduce edge effects and improve stitching quality. We then split the data again to match the same training and testing areas that are present in the SpaceNet 6 dataset.

II. Training and Inference: We train SpaceNet 6 winner zbigniewwojna’s algorithm once more on the larger tiles and then perform inference on our restructured test set (1,200 pixels² tiles).

III. Post-Processing: The final stage of post-processing is aggregating our predicted tiles (model outputs) together to evaluate the effectiveness of the algorithm at city scales. We achieve this by:

Merging all tile predictions together that belong to the same SAR image strip. Expand the extent of each SAR image to the full scale of our testing area, padding with 0's. For each of the 204 expanded images, calculate the number of times a pixel is mapped as a building or a non-building. If a pixel is mapped as a building ≥ 50% of the time based upon the number of observations for that pixel, we then classify that pixel as a building. Otherwise, we classify it as a non-building. We then finalize our binarization process and convert the pixel masks back to vectorized geojson format building footprints using the mask_to_poly_geojson function from Solaris.

The Results

Performance differences based upon the scoring format. We compare tiling specific scores vs. the full test area score- when we stitch all tiles back together and ensemble them based upon revisit rate. The overall score represents the SpaceNet Metric (x 100). We also report model precision (ratio of false predictions) and recall (ratio of missed ground truth polygons).

In the table above we report the performance differences when scoring on individual tiles, versus when we score on the full testing area by merging all of our tiled predictions into a contiguous area. The performance on the larger 1200 pixel² tiles was surprisingly identical to the original 900 pixel² SpaceNet 6 tiles. Both tiled formats produced an overall SpaceNet score of ~42.4 .We hypothesized that these larger tiles would produce a slightly better overall score as there were fewer edge-effects and as a result fewer smaller structures, however this was not the case.