Multi-Resolutional Search Ranking Pipeline for Satellite Imagery
Channel: COSPAR2021 Workshops
Date: 02/02/2021

Authors: Rajeev Godse, Anirudh Koul, Meher Kasam, Siddha Ganju

Abstract: NASA has tracked satellite imagery of the Earth for several decades. Since NASA Worldview provides browsable access to many unlabelled images, earth scientists looking for specific phenomena like “hurricane” or “wildfire” typically have to manually sift through petabytes worth of data in order to find similar images for their research projects. Tasks involving unlabeled datasets can also utilize embeddings from off-the-shelf CNNs to retrieve similar images. Such content-based image retrieval suffers from two issues: (1) sub-par performance for objects which occupy a small area w.r.t to the full image (2) the inability to distinguish between counts of objects.

To that end, how do we ensure a high-performance embedding-based search in multi-class, multi-label datasets? -- by developing a multi-resolutional search ranking pipeline. The search pipeline consists of three stages which iteratively improve the similarity ranking of the results. The first stage takes a query image as input, tiles it at multiple resolutions, and finds the nearest neighbors at each resolution, thus creating a candidate list of similar images. Our tile generating method splits the image into multiple, equally sized, overlapping squares of the given resolution size. Choosing to overlap tiles reduces the chances of partial or occluded objects, thereby increasing the utility of the candidates. The next stage, stage two, prunes the candidate list by aggregating neighboring candidates, as observed by their location and time. Stage three takes the pruned candidate list as input and scores each candidate, generating a final ranking based on the sum of reciprocal percentile rank.

The pipeline shows sizable benefits in recognizing phenomena on the multi-class, multi-label MS COCO dataset. Compared to a traditional whole-image, metric-based search, the multiresolutional approach shows an increase in F1 score in the first 50 ranks, and showed a less steep F1 dropoff. The pipeline has shown an increased ability to retrieve matches with the same number of multiple objects. Additionally, it has shown a marked increase in locality awareness between multiple objects in images.