Migrating from AWS Glue to BigQuery for ETL
By: Daniel Voyce
Tags: Etl, Data, Vertex Ai, Bigquery, Aws Glue

Photo by Andrew Ruiz on Unsplash

I documented our journey with AWS Glue pretty heavily https://tymlez-au.medium.com/

Why migrate?

I had been interested in AWS Glue ever since I saw the keynote on it at AWS re:invent a few years back, it seemed to have a really nice balance of “tools you know” vs “no setup”, and for a while it worked nicely.

It kind of filled the missing gap of Orchestration & Serverless that these tools often have.

Unfortunately in order to do this it has to wipe out some of the USPs that those tools have, and in doing so it creates complications.

AWS Glue served us well for our batch ingestions but it became clear very quickly that it wouldn’t scale to real time ingestion in it’s current version despite AWS Kinesis being a supported input.

We needed an alternative solution that was less painful.

Our Glue Pipeline

AWS Meter Data setup at TYMLEZ

In a nutshell out glue pipeline took in batch data that our Behind the Meter technology and Smart Meter readings dumped into S3 (more on this later as its important) and ran it through various scripts to ETL the data into what we require.

You can read more about the specific setup here: https://tymlez-au.medium.com/smart-meter-data-etl-systems-at-tymlez-5643e232dfbb

Orchestration was done serially via AWS Glue Triggers (e.g. when this step is done run the next step), unfortunately this meant that everything required deployment to use them each time, this slows down the development process massively and due to other problems (discussed here: https://tymlez-au.medium.com/aws-glue-is-a-mess-886cc1d13ca9) there was no easy way to set up an easy local environment to run these items manually.

In short — it was producing more hassles than it was solving.

Why GCP?

Simply — because of BigQuery. There is nothing on the market like Google BigQuery when it comes to processing huge amounts of data easily.

It comes down to simplicity, our workflow is quite standard when it comes to reading from devices, a device emits JSON every x minutes — we need to take that JSON and write it to our data warehouse.

AWS Flow:

Device -> “Processor*” -> Kinesis -> AWS Glue Table -> Athena

Getting Kinesis working with AWS Glue Tables was a near impossibility (and trying to develop this locally even harder).

GCP Flow:

Device -> “Processor*” -> BigQuery

*Processor is some form of processing capability like EC2, Lambda, Cloud Functions etc

The GCP flow just works, it works so easily that I can include the code right here that I use to ingest for Dev:

Once the data is in BigQuery then we just use a standard ELT pattern to mangle the data into a bunch of new tables as required and set them to run via cron.

Our pipeline now looks like this:

CREATE OR REPLACE TABLE data_clean AS ( SELECT x FROM data…) CREATE OR REPLACE TABLE data_business AS ( SELECT x FROM data_clean…) CREATE OR REPLACE TABLE prediction_input AS ( SELECT x FROM data_business…) CREATE OR REPLACE MODEL usage_forecast OPTIONS(model_type=’ARIMA_PLUS’...

Notice that as part of this step with SQL we can also train a model directly from within BigQuery? Not only have we ETL’d our data into the required tables we need for BI and other reporting — we have also generated an ARIMA+ model for forecasting of the data based on a number of different inputs. Additionally we can separately use the same data to train an AutoML model for testing.

ML Pipeline — Sagemaker vs Vertex.Ai

Sagemaker is great, it allows for really clean deployment of models via the SAGEMAKER.invoke_endpoint request. It handles a load of things like creating endpoints easily for trained models to allow for predictions to be run against.

GCP has Vertex.AI which is similar to this although it is much less mature and doesnt support endpoint creation for AutoML models, at the moment it is much simpler to simply batch process the days prediction models and make them available in a BigQuery or other table which can then be queried via a static API.

This again can be done via Cron very simply by batch creating the days prediction_input based off previous days and running the batch prediction against the retrained model.

No need for step machines, triggers or anything — its all nicely idempotent too if things need updating intra-day.

Data Visualisation — Quicksight vs Data Studio

AWS has Quicksight as its BI tool, however I have never got on with it, it always seems too restrictive. Sometimes you want to just throw together a really quick dashboard to produce some exec or board level insights from data without spending hours on producing it.

Google has Data Studio, which in my opinion is a really nicely honed product that just feels like a true part of the Google Suite of products.

I can create a dataset and share it with anyone in our business to allow for specific BI dashboards to be created.

Who can then drag and drop charts and hook it up to easily understandable representations of data:

I particularly like the Blend Data interface which abstracts a lot of the underlying joins that need to be done in order to get the majority of the data out from the system and easily lets you join several tables to get the results you need:

Conclusion

Our journey with AWS Glue was a bit of a struggle once we started to dig deeper into the streaming functionality of it, the orchestration of so many layers added a huge overhead that we weren’t expecting and whilst most of that is handled within the AWS suite of products, there are just too many benefits to switching our pipelines over to GCP and BigQuery to be ignored.

Next steps are to finalise our deployment by using Cloud Composer (Airflow) to orchestrate the creation of each of the tables and provide a monitoring dashboard to help us detect failures and act on them.

Update — 13/09/2021

I will say that AWS got in touch with me after my previous article and I got on a call with the AWS Glue product team, in their words I had “hit pretty much every sharp edge possible” (seems to be a running theme with me — perhaps I should switch careers to QA engineer?), they have a bunch of exciting updates in the pipeline that should solve many of these items, I look forward to seeing them because I do still maintain that for less complicated batch processing of data AWS Glue is an excellent product with great visibility.

About the author

Dan Voyce — CTO

Dan leads the technology strategy, architecture and development at TYMLEZ.

A start-up veteran of over 20 years, with a specialty of building high performance development teams and leading them to successfully deliver solutions for some of the largest names in Australia and the UK.