Picking the right tool for geospatial data enrichment (part 2) | by Bart Grasza | Towards Data Science

If you didn’t read the introduction, you can find it in part 1.
As a reminder, tools selected in part 1 of the series were: Geopandas, PostGIS and BigQuery. Let’s talk about their pros and cons.
GeoPandas
Geospatial extension for Pandas that under the hood uses Shapely (Python library to manipulate and analyze geometric objects)
Supports most important spatial functions, enough to solve even quite complex problems
Relatively easy to master if you are familiar with Pandas
Slower than PostGIS
Similarly to Pandas, operates on single core, but together with Dask (dask.org) can be parallelized
Built-in plotting function (directly in jupyter notebook) that will be enough for very simple data visualization/validation cases
Supports most common input/output formats
PostGIS
PostGIS extends standard (PostgreSQL) SQL language with spatial functions
Takes time to get familiar with some of the concepts but writing actual queries is quite simple
Can solve almost any geospatial-like problem, supports even functions like Nearest Neighbor
Many data loading and converting functions
Fast but query execution is limited to one core
Like any RDB, can crunch larger than memory queries, but when dealing with real “Big Data”, it might not be enough
Has many associated open source tools that will help you e.g. visualize your data (QGIS being my favorite)
Great documentation and online community
When linked together with PostgreSQL procedural language (PL/pgSQL), gives quite a powerful pipeline to process queries without “leaving” database
BigQuery’s geospatial functions
Processes queries in Google Cloud and if you know SQL then you will feel at home
Part of Google Cloud ecosystem where it’s easy to load, transform and export data using various APIs
Can crunch gigabytes within seconds
Supported input formats: CSV with geospatial column in WKT format and GeoJSON as a column in JSON format (new-line delimited)
Has inbuilt plotting tools, Big data ready
Charges per data storage and query running time which can greatly reduce your costs even if you’re crunching through terabytes of data
Supports joins between BigQuery data and Cloud SQL (which can also use PostgreSQL as an engine) data in single query
Less functions that PostGIS, but all essential manipulations and analysis related ones are already there
No procedural language like PL/pgSQL but (BigQuery in general) can mimic it to some extent using (SQL) WITH function
You are familiar with Pandas and SQL isn’t your strong side
Want to quickly explore / preprocess your geospatial data and you’re not planning to do complex transformations
Your geospatial dataframe doesn’t go into millions of records (Can process more but will make you wait for the results ; Dask can mitigate that problem to some point)
Your use case is rather simple and you want to get going asap
Pick PostGIS if:
Geopandas isn’t enough. Either because your problem was to difficult for Geopandas or it made you wait too long for results
You might end up manipulating more (larger than memory) records
You need complete set of geospatial functions
You have more time to familiarize yourself with a bit more complex tool than Geopandas
Pick BigQuery if ... you have to process Big Data! 🙂
Ok, we’re finally ready to do some real work. Imagine following problem:Your company has an app that each day generates large number of locations (longitude and latitude) where users interacted with your app. You were given 3 separate cases to work on.
Given user’s coordinates:1. Assign it to the city’s district2. Find the distance (in meters) to the nearest shopping mall, bus station and hospital (straight line distance)3. Generate aggregate statistics like e.g. count of number of app interactions per district
Notice that in the task 1 the number of shapes (city districts) won’t probably exceed 100 records in the database.Second task is derivative of first one where the output of the computation has as many records as the number of city districts.As for the third task, there might be more than 100.000 POI (Points of interest) you’d need to check against location of your users, but the actual POIs will not change that often and can be updated e.g. on a weekly basis.
All of this is actually great news because it means that the geospatial data preprocessing can take place e.g. directly on your laptop with Geopandas or PostGIS.
What kind of preprocessing steps would be required before merging with our users locations? Of course it highly depends on your case, but let’s talk about most common ones.
First you need to get actual geo-data (e.g. shapefile or GeoJSON representing city’s districts) and import the file to your tool (let’s assume it’s on your laptop). With Geopandas is one-liner:
Notice in the example above I used bbox (bounding box) parameter to “throw away” all data that is outside of defined area. Function read_file() can do more, e.g. load files directly from internet and in ZIP format (details on https://geopandas.org/io.html).
Next step is is to clean the data and change SRID if needed.
# Changing SRID (projection) in Geopandas# (where "df" is you geodataframe)df.to_crs("EPSG:4326")# Changing SRID in PostGISUPDATE my_tableSET geometry=ST_Transform(geometry)
Then, a good practice (especially if the data was downloaded from the internet) is to check if shapes are valid (e.g. polygon that lines don’t cross itself).
# Checking if shapes are valid in Geopandas# (where "geometry" is your geometry column)df.geometry.is_valid.all()# If invalid you can try to a quick fixdf.geometry.buffer(0.00001)# Checking if shapes are valid in PostGIS# Returns number of valid and invalid geometriesSELECT count(1), ST_IsValid(geometry)FROM my_tableGROUP BY ST_IsValid(geometry)
Sometimes buffer function won’t fix it or you can’t use it because your polygons/multipolygons are hollow inside (and using buffer would remove them). In that case you’ll need to take additional steps like e.g. exporting to PostGIS and trying to fix in there:
# Geopandas supports saving geometry directly to PostGIS but from my experience this is safer solution:# 1. Create additional column with geometry converted to WKTdf['geom_wkt'] = df.geometry.apply(lambda x: x.wkt)# 2. Save dataframe to table "my_table" but export geometry in WKT format instead of actual "geometry" columnconnection_str='postgres://<username>:<password>@localhost:<port>/<db_name>'df.drop('geometry', axis=1).to_sql(    'my_table', con=connection_str, if_exists='replace',    dtype={'geometry': Geometry})# 3. (Inside PostreSQL database) fix the geometry# Function ST_GeomFromEWKT translates WKT back to geometry# Function ST_MakeValid fixes geometryINSERT INTO my_table_fixedSELECT ST_MakeValid(ST_GeomFromEWKT(geom_wkt)), <other columns>FROM my_table# 4. Load it back into Geopandas dataframesql = 'SELECT * FROM my_table_fixed'new_df = pd.read_postgis(sql=sql, con=connection_str)
Apart from learning how to fix the geometry, now you know how fast and easy can be moving data between these two systems. So in any case you’d be missing a function in Geopandas, you can always do it to PostGIS.
Lastly comes Q&A process where you’d e.g. visualize the data to check if everything is ok. In Geopandas you can use simply use .plot() :
If that’s not enough, you might need to save the data back to SHP or GeoJSON and load them using external application like QGIS.
In Geopandas we need to transform our users dataframe to geodataframe:
import geopandas as gpd# Load users data into Pandas dataframeusers_df = <load function># Convert to geodataframeusers_gdf = gpd.GeoDataFrame(users_df,     geometry=gpd.points_from_xy(        users_df.longitude, users_df.latitude))
Now the data is ready to be merged with your users locations. For our Task 1 it would simply be:
# Perform spatial joingpd.sjoin(users_gdf, geo_dataframe, how="inner", op='intersects')
Similar operation (Task 1) in PostGIS:
INSERT INTO merged_tableSELECT * FROM geo_table g, users_table uST_Intersects(    ST_SetSRID(ST_MakePoint(u.longitude, u.latitude), 4326),    g.geometry)# In this line :# ST_SetSRID(ST_MakePoint(u.longitude, u.latitude), 4326)# we are converting our users coordinates to POINT and setting SRID
Task 2 is equally simple:
INSERT INTO count_interactions_by_districts_tableSELECT COUNT(1) user_count, geo_table.district_nameFROM geo_table g, users_table uST_Intersects(    ST_SetSRID(ST_MakePoint(u.longitude, u.latitude), 4326),    g.geometry)GROUP BY geo_table.district_name
Task 3 is a bit more challenging:
WITH poi_with_point as (    SELECT *, ST_SetSRID(        ST_MakePoint(            u.longitude, u.latitude), 4326)::geography AS geog    FROM poi    WHERE p.poi_type='shopping mall')SELECT p.id, p.longitude, p.latitude,        min(ST_Distance(p.geog, u.geog)) shopping_mall_min_distance FROM users u, poi_with_point p WHERE ST_DWithin(p.geog, u.geog, 8000)GROUP by p.id, p.longitude, p.latitude
Where:
The top WITH part of the query is generating “temporary view” called poi_with_point. It extends original table poi (that contains all of our POI locations) by adding POINT column called “geog”. It also filters out all POIs that aren’t shopping malls.
ST_DWithin can significantly speed up your query execution. It prevents checking every user’s location against every shopping mall by adding maximum distance of (arbitrary) 8 km between them:
Things get interesting when the size of our users database is gigabytes in size.
With GeoPandas (by default on a single thread) will take forever and you’d need to process it in batches if dataset exceeds your working memory. If you happen to have Dask cluster running on multiple workers then you might consider using it but it probably still won’t be an optimal solution. I’ve explained batching and running on cluster in my other article “Simplest way to enrich your Pandas dataframe” and both solutions require only few lines of code.Hint: for batching in Dask cluster, use map_partitions function and in each partition separately: load your geospatial data and perform e.g. spatial join operation with “mapped” chunk of user data.
PostGIS will be able to crunch through the data but execution time might be sub-optimal. If your PostGIS server is not part of the production environment and you don’t mind waiting the you should go for it. If not our last option is ... BigQuery
First we will make few assumptions:1. You have registered Google Cloud account and activated Google Cloud Storage (GCS)2. Your users data is already loaded into BigQuery table3. You have installed google “toolbelt” on your laptop and commands “gsutil“ and “bq” are available in your terminal4. You have permissions to create datasets and tables in BigQuery5. You have set (in terminal, using google “toolbelt”) your default projectTip: Keep GCS and and all BigQuery datasets in the same region, otherwise you might not be able to load it or perform JOIN.
As we already know from, BigQuery can import geography data in GeoJSON or WKT formats. We already learned how to convert to WKT so let’s save as CSV and transfer to BigQuery. First we will save it to CSV:
# Assuming your geography is already in WKTdf.to_csv('output_file.csv')
Loading directly from laptop to BigQuery is possible but has limitations so we’re going to send it to GCS first:
# (In your terminal)gsutil cp output_file.csv gs://your_bucket_name_in_gcs
Create dataset, table that will store your data and populate it from GCS:
# (In your terminal)# Create datasetbq --location=US mk --dataset your_dataset# Create tablebq mk --table your_dataset.geo_table \id:INTEGER,geom_wkt:STRING,<other columns># Load CSV into BigQuery tablebq load \--source_format=CSV \your_dataset.geo_table \gs://your_bucket_name_in_gcs/output_file.csv \id:INTEGER,geom_wkt:STRING,<other columns>
BigQuery syntax is almost identical to PostGIS:
# Task 1 in BigQueryINSERT INTO merged_tableSELECT * FROM     your_users_dataset.users_table u,     your_dataset.geo_table gWHERE ST_Intersects(    ST_GEOGPOINT(u.longitude, u.latitude),    g.geometry)
Task 2 is almost identical in BigQuery and PostGIS so I will skip the snippet.
As for our 3rd Task, Notice that the resulting number of rows is very small, no more than the number of districts. In that case (or similar where you know that data will fit into your working memory of e.g. your laptop) you can use BigQuery Python client and download result directly into Pandas dataframe:
from google.cloud import bigqueryclient = bigquery.Client()sql = """SELECT COUNT(1) as user_count, g.district FROM     your_users_dataset.users_table u,     your_dataset.geo_table gWHERE ST_Intersects(    ST_GEOGPOINT(u.longitude, u.latitude),    g.geometry)GROUP BY g.district"""df = client.query(sql).to_dataframe()
The goal of the series was to show how to process or enrich your data with external geospatial information. I wanted to show you how you can mix and match available tools, easily transfer data between them and which one will be the most useful to quickly do your job.