Large Scale Satellite Data Processing
Tags: Satellite, Satellite Technology, Spatial Analysis, Big Data, Remote Sensing

Large Scale Satellite Data Processing

With more amounts of spatial data coming in than ever, we look to increase satellite data processing methods and efficiency.

Source: Techspot

Spatial Data — What is it?

Spatial data, or geospatial data, can be thought of as any data containing location information on the Earth’s surface. Spatial data is present in any field or industry, especially in today’s “real-time” data-driven world. There are a few common terminologies that better explain the language of spatial data.

Vector data is any data point that contains points, lines, and/or polygons. It can be thought of as our “man-made” map view of the world, consisting of road networks, administrative boundaries, etc. Raster data, referred to as imagery, is any pixelated data, such as satellite images. Often, raster data are photos taken from any areal device like satellites or drones. The resolutions can greatly vary depending on device precision and other technological and areal factors. Geographic Coordinate System (GCS) is a projection of the Earth’s 3D surface to define locations on Earth. It uses latitude, longitude, degrees (angels), and axes.

High-Resolution Earth Imagery is Now Made Publicly Available

Source: EOSDIS Imagery

There have been huge advancements in remote sensing technologies within the last decade alone. This has paved the way for petabytes of high-resolution satellite imagery being made publicly available. The use-case of this data applies to any industry — especially fields like atmospheric science, agriculture, ecology, soil science, etc. NASA EOSDIS provides public access to over 17 petabytes, and growing, satellite imagery data. The European Space Agency (ESA) launched the Sentinel-1A satellite which collects over 5 petabytes of data within the first two years of its launch alone.

These advancements, however, exposed vulnerabilities with processing drastic amounts of data and consequently paved the way for us to build new systems of spatial data processing — including SpatialHadoop, Simba, GeoSpark, RasDaMan, and more. These systems, however, focus solely focus on processing raster or image data. They perform poorly on queries processing both vector and raster data simultaneously. Additionally, these methods require the conversion of raster-to-vector or vector-to-raster — which proves to be extremely inefficient when processing large amounts of data.

So what’s a solution that can handle both vector and raster data? One answer is zonal statistics.

Zonal Statistics

Zonal statistics is the operation that processes a combination of vector and raster data. It computes statistics over a raster layer, for each polygon in a vector layer. All values are aggregated from the raster layer which overlaps with a set of polygons from the vector layer.

For example, let’s say we are interested in finding the average temperature of U.S. states. We have polygons of U.S. states from the vector layer vector and temperature data from the raster layer. Zonal statistics can compute the average temperature (or other statistics) for each state in the U.S. This computation of processing vector and raster layers simultaneously is called the scanline method.

The Scanline Method

Figure 1

The scanline method works directly for the zonal statistics problem by finding intersections between horizontal scanlines and respective polygon boundaries, as seen in Figure2(c). In the latest version, the scanline method can now process multiple adjacent polygons at a time, further reducing processing complexities. This means, for example, if we wanted to find the average temperature of all the states in the west — we can now compute this query in one computation, as opposed to computing the average separately per state.

The Algorithm

Figure 2

Input : a set of polygons (vector layer), raster layer

: a set of polygons (vector layer), raster layer Output: desired aggregation (e.g. sum, minimum, maximum, count, average of pixels inside the polygon)

Step 1: As shown in Figure2(b), first calculate the Minimum Bounding Rectangle (MBR) of the input polygon and map its two corners to the raster layer to locate the range of rows/scanlines to process in the raster layer. To include multiple polygons, we extend this step to all queried polygons.

Step 2: As shown in Figure2(c), compute the intersections of each scanline with its polygon boundaries. Each scanline is converted to the vector space and stores its y-coordinates in a sorted list. Then, each polygon is scanned for its corresponding range of scanlines — which are then used to compute intersections with the polygon.

Step 3: As shown in Figure 2(d), the pixels that lie inside the polygons are processed. Rather than processing one polygon at a time, this step processes one scanline at a time, speeding up computation significantly.

Pros of the Scanline Method

This algorithm overcomes the limitation of raster and vector-based methods. First, it only requires minimal intermediate storage for the intersection points. Second, it accesses the pixels that are inside the polygon which improves disk IO for very large raster layers. Thirds, it does not require any complicated point-in-polygon tests which make it faster than the vectorization methods. Finally, the scanline method is IO-bound which makes it optimal from the processing perspective since it requires one scan over the raster data to process all polygons.

Use-Cases

Source: ESA — Average Temperature in Arctic 1997–2008

As I mentioned above, the use-cases for processing spatial data are endless. As we are slowly but surely moving towards a more “green” and environmentally friendly global community, it is important to be able to process and extract insights from Earth imagery. Processing both vector data and satellite data simultaneously allows us to not only apply spatial data insights to our societies, but it allows us to foresee trends on the Earth’s surface. We can better monitor vegetation, temperature, ocean level changes, etc.

Visit my GitHub repository to view a full-stack application using the scanline method in the backend to process multiple polygons through a user-interface.