How We Scale Geospatial Calculations using Shapely and Rtree
By: Vaibhav Gupta
Tags: Programming, Geospatial, Data Science, Innovation, Spark

At Zendrive, we help drivers drive safer. We do this by collecting driving behavior data through our partners and analyzing a driver’s risk to calculate a risk score based on the trips taken by the driver.

We get the trips data for the driver, including spatial information as GPS points. We then perform spatial operations, such as finding the state or zip-code for a location, on the data, and then use the results to calculate the risk score of the driver.

Fig 1. Geospatial calculations required to analyse drivers’ risk

Our Method for Performing Spatial Computations

But first, a couple of terms.

POSTGIS — PostGIS is an extension to the PostgreSQL object-relational database system which allows GIS (Geographic Information Systems) objects to be stored in the database. See POSTGIS docs. OSM DB — We load the open street map(OSM) data into a postgres database and use POSTGIS to perform spatial calculations. We will be referring to this postgres database containing OSM data as OSM DB for the rest of the article. Spark — Apache Spark is a fast and general-purpose cluster computing system (See Apache docs). In Spark, computation is carried out using executors, which are worker nodes in charge of running individual tasks in a spark job.

For risk modelling at Zendrive, some of the calculations we do are:

Calculate the distance of trip that was taken on a highway.

If a trip spans across multiple zip-codes, we like to calculate the distance travelled in each zip-code.

Other similar spatial calculations.

To do these, we use spark jobs. Spark executors make a connection with the OSM DB, run a spatial query, and get the results for further processing.

Fig 2. Using spark to query OSM DB for spatial operations

Limitations of OSM DB

The method described for spatial computation has an issue with scaling. OSM DB, being a RDBMS database, limits the number of concurrent connections made to it. For our use case, this number stands around 300–350.

Opting for a larger database to increase the number of connections would lead to the database being idle most of the time, since our spark jobs are on demand. This limits the number of executors used in our spark job and becomes a bottleneck in on-demand scaling for our spark jobs.

These batch jobs can take several hours to process and with the increase in traffic, this time is going up linearly. We have to find a scalable way of doing spatial calculation, and also make sure that time and resources needed to run our pipeline don’t increase significantly.

Scaling this Approach

Our approach to solve the problem was to remove the root cause. If we could remove our dependency on OSM DB, then we can scale up. But how?

Our solution was simple. Could we somehow do all the computations in the spark executor itself, without needing to make connections with the OSM DB? The answer is yes.

This is where Shapely comes in. From their docs,

Shapely is a BSD-licensed Python package for manipulation and analysis of planar geometric objects.

…

The first premise of Shapely is that Python programmers should be able to perform PostGIS type geometry operations outside of an RDBMS.

Shapely helps us perform spatial calculations locally using Python instead of using OSM DB. But since Shapely does not support indexing operations, we use Rtree, which provides a number of advanced spatial indexing features for indexing our spatial data.

Shapely assumes all the data in a 2D coordinate plane. We need to transform our data into a coordinate reference system on which Shapely can perform geometric operations (i.e. epsg:3857).

In our case, we converted the OSM DB data present in epsg:4326 to epsg:3857 using Pyproj (Coordinate transformation library in Python). You may need similar transformation if you want to use Shapely for spatial computations.

Shapely is just the tool for computation, so we need to convert the geo-data present in OSM DB into Shapely objects upon which spatial operations can be done. The idea was to have local geo-data in each executor so that it can perform calculations independently, and completely remove the OSM DB dependency.

Fig 3. Approach to remove the OSM DB dependency for spatial calculations

This Causes Another Problem

For smaller tables in OSM DB (like states, or zip-codes), we can directly convert the data into Shapely objects and load it into our spark executors. But we cannot afford to load bigger tables like North America Line (containing highway data) completely into our executors. This could lead to out-of-memory errors.

We need to find a way to minimize the amount of data loaded into each executor.

Solution: Divide and Rule

To solve this problem, we used the technique of data storage used for digital maps. For rendering a map on your screen, the whole world is divided into a specific number of tiles based on the zoom level. When you request the data for a region, only the tiles for that region are downloaded.

Similar to this, we stored the highway data for each tile instead of storing complete data. Now, we can only download the tiles data required for processing of the trips in that executor instead of all North America data.

Fig 4. Tile-based map rendering (Maptiler)

But, merely dividing the data into tiles and downloading only the required tiles for computation will not give any advantage. Why?

As depicted in Figure 5, our trips data is distributed across spark executors and each executor needs to load tiles required for processing each trip present in it. Since different trips in an executor require different tiles for processing, we end up downloading a large number of tiles in each executor leading to the same problem.

Fig 5. Trip data distributed across spark executors

So we tried various techniques to solve this issue:

1. Load, Process and Erase

In this technique, we can load the tile data for one trip, process, and then erase the tile data for that trip. This way, only one tile data will be present in the executor at any moment and we have successfully solved the data loading problem.

But, as is apparent, we would need to download the same tile data again and again in the same executor. Furthermore, one tile may need to be downloaded across multiple executors. This will increase the time taken by our spark job.

2. Repartition data in spark smartly

A good technique to solve the problem would be to group tiles of the same region into one executor, as shown in Figure 6. This way each executor will download the tiles data of the respective region and a smaller total number of tiles will be downloaded.

Moreover, the trips in an executor now belong to the same region and end up requiring the same tile data for getting processed, so we could reuse the same data without downloading it multiple times.

Fig 6. Repartitioning the trip data to improve performance

To achieve this, we thought of repartitioning our trips data based on the tiles from which the trip starts. We find the starting tile for trips from each trip’s start location and repartition the trip data across executors. Now, the trips belonging to the same region are present in the same executor. We will also save the tile data downloaded once into an executor so that it can be reused by other trips in the same executor.

This solves our memory overload problem by downloading less trips. Further, using the same tile data for processing multiple trips reduces the time taken by downloading the same tile multiple times. We’re able to hit two birds with one stone.

Fig 7. Advantage of re-partitioning data based on tiles

Maintaining the Quality of Our Code Structure and Readability

Using this combination of Shapely and Rtree solves our problem. But, using this combination causes us to compromise the structure and readability of our code.

To improve readability, we thought of using geopandas. What is geopandas? From their docs:

GeoPandas is an open source project to make working with geospatial data in python easier. GeoPandas extends the datatypes used by pandas to allow spatial operations on geometric types. Geometric operations are performed by shapely.

Geopandas uses geodataframes (very similar to pandas dataframes) which provides a very clean way to perform spatial calculations. The geometric operations are done in Shapely so we needn’t worry about any change in our results. Further, it has an in-built Rtree index (sindex) which can be used for indexing data. This was a perfect fit for us.

Below is a code snippet to find the state for any location. Indexing and handling of data becomes simple using geopandas.

# state_gdf is the geodataframe containing state information

+------------+---------+

|state | geometry|

+------------+---------+

def get_state_for_point(self, x, y):

#x, y are coordinates in epsg:3857

point_geom = Point(x, y)

possible_matches_index = list(state_gdf.sindex.intersection(point_geom.bounds))

possible_matches = state_gdf.loc[possible_matches_index]

precise_matches = possible_matches.loc[possible_matches.contains(point_geom)]

if precise_matches.empty:

return None

else:

return precise_matches.iloc[0].state

With 160 executors in place, Shapely and Rtree worked smoothly whereas OSM DB couldn’t provide connections to handle these many executors.

Scalability

As the calculations take place locally, this method has no constraint on the number of executors. We can scale up our spark cluster according to our needs.

Earlier we used to run our pipeline using 80 executors. To test the new approach, we scaled up our spark cluster to use double the number of executors. With 160 executors in place, Shapely and Rtree worked smoothly whereas OSM DB couldn’t provide connections to handle these many executors.

Moreover, we ran our pipeline with the same number of executors as was used with OSM DB and found that the time taken by Shapely and Rtree is comparable to the time taken by OSM DB.

OSM DB (80 executors, 10 gb)

1486.45 sec Shapely + Rtree (80 executors, 10 gb)

1703.88 sec Shapely + Rtree (160 executors, 8 gb)

1057.63

This solution also helped us in discovering the right geo-match for significant trips which were not getting mapped to zipcodes earlier, while also cutting the cost of a dedicated OSM DB.

Needless to say, we’re quite happy with the result.

Author Bio

Ashutosh Goel is a final year student at BITS Pilani, pursuing a dual major in Computer Science and Economics. He currently works as an intern in the Customer Products team at Zendrive. Ashutosh likes to work on large-scale distributed systems. Python and spark are his favourite tools.