Sync AWS RDS Postgres to Redshift using AWS DMS
By: Axel Furlan
Tags: Postgres, Redshift, AWS, Data, Data Engineering

The problem

When we first started to get to know AWS Redshift, we fell in love for the fast aggregated query processing. This strong advantage meant sky-rocketing our productivity and speed when performing statistical studies or simply data-extractions. So, of course, we turned it into our main data warehouse for all the data sources we managed (Adwords, Salesforce, FBAds, Zendesk, etc.) — all of this, using Stitch as our main ETL tool.

RDS Postgres instance vs Redshift on the company’s everyday aggregated query performance time.

Stitch holds a nice subscription plan of $100, offering process capacity for 5M rows and $20 per additional million rows. Stitch logs and billing invoices tell us we barely reached $180 on a very busy month using all the data sources mentioned above. Most of those months were just plain $100 (no additional million rows used).

Our company develops, maintains and sells its core SaaS product, IncreaseCard. Hosted in the AWS Cloud, our production database resides in AWS RDS, storing between 10 and 20 million new rows every month, and considerably growing.

So typically, using Stitch to sync our production database to a Redshift warehouse, would turn out to be painfully expensive for a third-world startup company. Also considering our country’s currency lost a lot of its value against US dollars recently.

Argentina’s Congress — Photo by Sander Crombach on Unsplash

We also had another restriction. The company’s developing its second major product, IncreaseConciliación. This one uses the NoSQL Apache Cassandra database to store and process its huge data. The problem is, both products must be synced in order for Conciliación to use transactions extracted by Card. In other words, we had to build a data-lake accessible for consumption by any service to perform syncing operations on-demand.

Many things to take into account, right?

Testing solutions

Being a small team of 2 people, the mighty “Data Team”, we get it easy to try and test new things, especially architectures.

We started by using AWS Data Pipeline, a UI based service to build ETLs between a bunch of data sources. Although the process of building an ETL was rather easy, there were a bunch of workarounds that we had to take in order for it to be effective — remember that we have to update every change whether it be an insertion, a deletion or an update. Since you can’t use code here, it became unmaintainable quickly. Furthermore, there isn’t much detailed documentation or clear examples for this service IMO.

We tried to set a Lambda to consume every 15 minutes the Wal log of a replication slot of our Postgres database, and send it to a Kinesis Firehose data stream. It seemed to be all safe and sound until a production process updated more rows than usually expected. We found out that, in these cases, the way the records came from the logical decoding were huge rows full of chunks of changes of the tables involved, causing the function to die of lack of memory every time it tried to load it. We solved this by setting true to the property write_in_chunks of the logical decoding plugin, we used (wal2json), enabling us to partition the incoming JSON log. Long story short, the function could still be terminated unsuccessfully due to not enough time to process the huge transaction. No bueno.

What you came here for

Ok so, let’s head to it.