Serverless Querying of Remote Sensing Data at Scale
By: Raghu K Ganti
Tags: Geospatial, Data Science, Ai Solutions, Raster To Vector

With the advances in Cloud combined with cloud native formats for raster data, IBM Research and The Weather Company jointly explored the efficacy of using spatiotemporally indexed raster data on Cloud Object Storage (an S3 compatible Object storage) for querying varying regions and time frames. We will examine in the rest of this article the performance of querying 10 years of MODIS NDVI (vegetation index) data for the entire Earth.

Before we deep dive into querying and its performance, we note that today’s MODIS data (among other remote sensing data) typically is distributed in scientific formats such as netCDF, HDF, and GeoTiff files. These formats are great for distribution, but not good for querying data at rest. Popularity of querying data at rest (in object stores) has gained significant traction in the last few years and with big data formats such as Parquet and ORC, indeed querying data at rest is feasible and at performant query times as well as attractive price point. Can we follow a similar approach for Raster data and do we have all the “pieces” in place to leverage this query at rest primitive?

Figure 2: Pieces in place (source: Unsplash, credit: @hpzworkz)

Given our years of expertise in this space and the development of key technologies such as Cloud Optimized GeoTiff (COG) , GDAL’s Virtual File System, and Object storage range reads, we felt that the data format pieces are in place and with the right transformations and layout of the raster data, we should be able to query this data using a simple Python API without having a large database or noSQL store stood up.

Ingestion and layout on object store

For the ingest process that transforms the scientific format to the COG format with the virtual file system index, we use simple transforms using a wrapper on the Python API to GDAL. For larger scale ingests, we deploy this in a Kubernetes service. For more details on the ingest step, the reader can refer to our related article here since it is out of scope of this article. We followed the process in the article and laid out 10 years of all of MODIS NDVI data as a virtual dataset on Cloud Object Store (using Cloud Code Engine, a serverless Kubernetes service). Now, we are ready to do some query performance testing! The total volume of data in the object storage is 2TB.

The layout on Object Storage is a Hive style layout with each “key” holding multiple TIF images and a index.vrt key holding the index information.

modis/processed_images/product=MOD13Q1.006/layer=MOD13Q1_250m_16_days_NDVI/date=20200609/

Query performance

In our experience, end users and developers are most interested in queries such as: given a region and timeframe, can we retrieve the corresponding raster images? We characterize the performance of retrieval of images from the 2TB dataset for different types of queries using a Watson Studio Python notebook with 8vCPUs and 16GB RAM. All the query times are measuring the time taken to query object storage and then write the TIFF images to the local storage and are averaged over 10 runs each.

The first query is a simple point and radius for a given day, where we increase the radius from 1km to 9km and measure the time taken to retrieve the images to the notebook. The results are shown below (Figure 3).

Figure 3: Query time vs Radius for a point in Amazon

The average query times did not change much when varying the radius from 1 to 10km and took anywhere between 400ms and 500ms! The NDVI TIFF images are visualized in Figure 4 for 1km and 9km radius and the reader can see the zoomed out view of the Amazon river.