Turn Amazon S3 into a spatio-temporal database!
By: David Moten
Tags: Search, Geospatial, Java, Programming, S3

Turn Amazon S3 into a spatio-temporal database!

OK, full of promise that title, what exactly is possible with S3? Quite a bit, as it turns out. In fact, this article describes how to search n-dimensions using S3. David Moten Jul 2, 2019·8 min read

S3 for storage

S3 is Amazon’s virtually unlimited storage offering. You can store files of any size in S3 Buckets and they will be stored redundantly on multiple devices across multiple facilities in a region. For files that you want to interact with frequently standard storage costs are about 3c (US) per GB per month. If you mark your data for storage in S3 Glacier then the cost is 0.4c per GB per month and you can bring it back to standard status in a few hours if required.

I store vessel movement data from a big chunk of the world’s surface in S3, and I configure older data to move to Glacier automatically. One day may have as many as 27 million records and in Comma Separated Values (CSV) format take up 2.4GB. In fact I store the raw data (AIS NMEA) in S3 and a Java Lambda trigger creates a CSV version.

Random access to S3

One of the things I love about cloud services is discovering sweet spots in terms of cost and capability and I wondered if I could quickly and cheaply query this data set in S3.

It turns out that files in S3 are randomly accessible. I can specify a byte range using a Range HTTP request header and that’s all that will be returned (without a latency penalty).

Random access enables index lookups. If I know approximately where in a file my records are by using an index then I can grab only that chunk of the file and extract the information I want.

How do GIS systems perform spatial queries?

A window query in 2, 3 or more dimensions is a search for data within a box of the same number of dimensions. For example if I have spatio-temporal data for the whole US then a window query might be to find records in a particular suburb of Chicago over lunchtime on a particular day.

Fig 1. A spatio-temporal window query (researchgate.net)

How do GIS systems perform window queries on 2D spatial and spatio-temporal data?

Spatial queries are commonly done with R-Trees which is a reasonably complex data structure. Searching tree structures represented in flat files can mean making a number of random access reads as the tree is traversed. More importantly the reads are serial in nature in that I don’t know the next read location till I’ve completed the previous read. More discrete serial reads means more delays especially if I’m reading a flat file in S3 and experience latency with each read.

Mapping 3 or more dimensions to the Hilbert Curve

At 3 or more dimensions a popular technique for indexing is to map the multi-dimensional region to a single dimension using a space-filling curve. One good choice for that space-filling curve is the Hilbert Curve. Chop your 3D domain up into a regular grid of 1024x1024x1024 cells then a Hilbert curve made up of a single wiggly line will visit all 1m points.

As an interlude, here are a couple of 2D examples of that wiggly line called the Hilbert Curve.

Fig 2. 2D Hilbert Curve, 4 bits

Fig 3. 2D Hilbert Curve, 8 bits

Here’s a 3 dimensional animation of the Hilbert Curve:

The Hilbert curve has some nice locality properties because it makes no big jumps, the next value on the curve is only ever one unit away. This means that if two points have indexes that are close to each other then they will be geometrically close to each other as well. The converse is not guaranteed.

A sparse index

Now that all the data points are mapped to one dimension I can sort them in that dimension and then use simple one-dimension database index lookup techniques. When you index a column in a database table you end up with the ability to find the exact rows in each table corresponding to a value in that column. However, there can be as many entries in the index as rows in the table. If you use what’s called a sparse index then you have the positions of some rows but you might have to do a bit of extra reading after them to find others. A sparse index can be much smaller than a full index, say 0.1% of the size of a full index.

I’m beating around the bush here, so let’s get to the punch line. If I sort my 27 million records (2.4GB of CSV) in ascending value of a Hilbert index with one million points (10 bits) and then create a small sparse index file (440K) I’ve got the ability to do fast window queries via random access on my data. I can even do the search in parallel because I know in advance where my records are (approximately). Ideally I’ve got queries in mind that are quite small in space and time though I found queries unconstrained in time can work pretty well too.

A library to help

There are some curly issues to deal with here in terms of translating a search box into ranges on the Hilbert curve (perhaps coarsening the ranges if there are too many of them) and from there to accessing byte ranges pointed to by the index. Fortunately these issues are handled for you by a java library called sparse-hilbert-index which I knocked up and published on GitHub.

Let’s try an example. My CSV input data looks like this:

mmsi,messageId,time,lat,lon,speedKnots,heading,course,navigationStatus,rateOfTurn,source,specialManoevreIndicator,timeSecondsOnly,isUsingRAIM,class

4124607775,18,1543283257000,-66.97455333333333,33.95234833333333,8.7,67.3,67,,,NORAIS2,,35,N,N,

4124607775,18,1543283257000,-66.97455333333333,33.95234833333333,8.7,67.3,67,,,NORAIS2,,35,N,N,

538006789,1,1543283259000,-39.57300333333333,33.18371833333333,12.7,85.1,84,UNDER_WAY_USING_ENGINE,-13,NORAIS2,0,39,Y,N,A

...

I create a sorted data file and a Hilbert index by running this code (with thanks to commons-csv):

Viewing search statistics

13 minutes later on an i5 laptop the sort of 27 million CSV records is complete and the index file is created.

I then deploy input-sorted.csv (2.4GB) and input-sorted.csv.idx to an S3 bucket and try querying the data like below. First I’ll count the number of records in the Sydney region for an hour. I’m going to make things easier and suppose the data is in a publicly accessible bucket so I don’t need to authenticate (but only do this if your data is not sensitive).

Load the index file (which you can cache locally if you want):

Perform a search:

Loading the index file (440K) took 500ms.

The search above found 2389 records in 169ms over a not-very-zippy corporate internet connection. 3347 records were read (a hit ratio of 0.71). Some level of wasted effort is a consequence of the sparse index but of course we don’t need a GIS system on the server!

Interestingly enough if we query the dataset for the Sydney (Australia) region for the whole time dimension (24 hours) then the performance is still reasonable: 6720ms to return 36940 records with a hit ratio of 0.55. 37 chunks (sections of the S3 object pointed to by index entries) were read (partially) with an average time to first byte (TTFB) of 114ms (thus the latency of our internet connection is responsible for 4200ms of the elapsed time). Being closer to S3 (say in EC2 or Lambda) can offer a significant time saving but another trick up our sleeve is to read concurrently from S3.

Use concurrency!

In the above search command you’ll see that the concurrency parameter was set to 1. I’ve found that an optimal number over a mediocre internet connection for this dataset on S3 is 8; that is, 8 chunks are retrieved simultaneously, parsed, filtered and merged. The query that took 6720ms (mainly because of the unconstrained time dimension) now takes 839ms!

Search from AWS

I did test runs from EC2 (t2.large) as well and as expected the time to first byte came down, to about 50ms. With concurrency the the all-day query went down to 380ms. With latency down I presume seek times start to play a stronger role and the sweet spot for the concurrency level seems to be about 4.

Streaming API

The api used to retrieve records from searches offers streaming functionality via the RxJava library.

Other formats

Bear in mind that CSV is not the most efficient method of storing and retrieving data. More targeted binary formats may make your queries many times faster.

Tuning

The number of entries in the sparse index has a large effect on the hit ratio ( a measure of how many records you read to find what you requested). If you can cache index files locally or put them on faster storage then you may go for a larger number of entries in each index file. Experimentation is the way to go. Create multiple indexes for the same file of various sizes and see what suits your querying patterns best.

Generally speaking if your queries are tailored so that they don’t return a lot of records then your response time should be of the order mentioned above. Searching with unconstrained dimensions (like time in the query earlier) may work but I’d suggest having only one dimension unconstrained.

Viewing search statistics

When you perform a search you can specify the withStats option to view various metrics about the search:

index

.search(bounds)

.withStats()

.concurrency(1)

.url(url)

.last()

.forEach(System.out::println);

Produces:

WithStats [elapsedMs=169, recordsFound=2389, recordsRead=3347, hitRatio=0.7138, bytesRead=392261, timeToFirstByteMsTotal=94, timeToFirstByteMsAverage=94.0000, chunksRead=1]

What about 2 dimensions?

I’ve run through an example with 3 dimensions but a sparse Hilbert index will work well for 2 dimensions as well. So S3 can become your 2D spatial database as well.

Not just S3

Note that Azure Blob Storage and Google Cloud Storage also offer random access via the HTTP Range header. You can use those storage options as well with the sparse-hilbert-index library.

What about Athena?

That’s a good question! Especially as AWS offer Athena on CSV files (and other formats) in S3 buckets that can can do a full scan of a 2GB CSV file in 1.5 seconds!

The sparse-hilbert-index approach may appeal when you consider the costs of running many many indexed searches across a lot of data compared to full scans. Athena costs are low ($US5/TB data scanned for queries) but may become significant at some scale. In some cases the consequent power consumption from doing a lot of full scan searches may also be ethically challenging (and also rather hard to calculate). I think it’s hard to compete with Athena on big file search but there may be some edge cases that favour sparse-hilbert-index!

To add fuel to the fire, Athena supports the Parquet format which can be indexed such that every page has min-max statistics. If you sort the data on the field you want to query (in our case we would add a calculated Hilbert index column) then Athena can in theory do indexed lookups itself (untested). Athena still has to look at the statistics for every page (1Mb by default) so it’s not quite as efficient theoretically as sparse-hilbert-index that knows exactly what pages to search. Note that as of June 2019 Athena does not support indexed Parquet formats for faster access (1). When or if support arrives it will be worth experimenting with!

The take home