Sync AWS RDS Postgres to Redshift using AWS DMS | by Axel Furlan | Towards Data Science

Disclaimer: this post assumes some understanding of programming.
When we first started to get to know AWS Redshift, we fell in love for the fast aggregated query processing. This strong advantage meant sky-rocketing our productivity and speed when performing statistical studies or simply data-extractions. So, of course, we turned it into our main data warehouse for all the data sources we managed (Adwords, Salesforce, FBAds, Zendesk, etc.) — all of this, using Stitch as our main ETL tool.
Stitch holds a nice subscription plan of $100, offering process capacity for 5M rows and $20 per additional million rows. Stitch logs and billing invoices tell us we barely reached $180 on a very busy month using all the data sources mentioned above. Most of those months were just plain $100 (no additional million rows used).
Our company develops, maintains and sells its core SaaS product, IncreaseCard. Hosted in the AWS Cloud, our production database resides in AWS RDS, storing between 10 and 20 million new rows every month, and considerably growing.So typically, using Stitch to sync our production database to a Redshift warehouse, would turn out to be painfully expensive for a third-world startup company. Also considering our country’s currency lost a lot of its value against US dollars recently.
We also had another restriction. The company’s developing its second major product, IncreaseConciliación. This one uses the NoSQL Apache Cassandra database to store and process its huge data. The problem is, both products must be synced in order for Conciliación to use transactions extracted by Card. In other words, we had to build a data-lake accessible for consumption by any service to perform syncing operations on-demand.
Many things to take into account, right?
Being a small team of 2 people, the mighty “Data Team”, we get it easy to try and test new things, especially architectures.
We started by using AWS Data Pipeline, a UI based service to build ETLs between a bunch of data sources. Although the process of building an ETL was rather easy, there were a bunch of workarounds that we had to take in order for it to be effective — remember that we have to update every change whether it be an insertion, a deletion or an update. Since you can’t use code here, it became unmaintainable quickly. Furthermore, there isn’t much detailed documentation or clear examples for this service IMO.
We tried to set a Lambda to consume every 15 minutes the Wal log of a replication slot of our Postgres database, and send it to a Kinesis Firehose data stream. It seemed to be all safe and sound until a production process updated more rows than usually expected. We found out that, in these cases, the way the records came from the logical decoding were huge rows full of chunks of changes of the tables involved, causing the function to die of lack of memory every time it tried to load it. We solved this by setting true to the property write_in_chunks of the logical decoding plugin, we used (wal2json), enabling us to partition the incoming JSON log. Long story short, the function could still be terminated unsuccessfully due to not enough time to process the huge transaction. No bueno.
Ok so, let’s head to it.
Our current architecture consists of the following:
A DMS (Database Migration Service) instance replicating on-going changes to Redshift and S3.
The Redshift source endpoint.
An S3 bucket used by DMS as a target endpoint.
A Lambda that triggers every time an object is created in the S3 bucket mentioned above.
(Optional) An SNS topic subscribed to the same event of object creation. This enables you to subscribe anything to that topic, for example, multiple lambdas. For more info, click this link.
For this post to be more “democratic”, I’ll divide it into 2 sections. The first one will be the steps to replicate changes directly to Redshift. The second one, building the S3 data-lake for other services to use. Feel free to read one or the other, or even better, both 😄.
Since I don’t consider myself smarter than people that write AWS documentation, I’ll copy-paste some of their instructions below 🤷‍♂.
Find your RDS instance, look up the parameter group that this instance applies. Either duplicate or modify the parameter group directly with the following.
1.- Set the rds.logical_replication static parameter to 1. As part of applying this parameter, we also set the parameters wal_level, max_wal_senders, max_replication_slots, andmax_connections. These parameter changes can increase WAL generation, so you should only set therds.logical_replication parameter when you are using logical slots.
2.- Reboot the DB instance for the static rds.logical_replication parameter to take effect.
After rebooting, we should be good to go. A good way to test if everything’s gucci is by running the following line on your database console.
increasecard=> show wal_level; wal_level----------- logical(1 row)
First, we must create the resources that the task is going to use. Create the endpoints (source and target) and the replication instance (basically an EC2 instance that’s in charge of all the dirty work).
The process of creating a replication instance is super straightforward, so I won’t go over it, just make sure to use a VPC that has access to the source and target you intend to use.
For the source endpoint, tick the option that says something like “Select a RDS instance” and effectively, select your source database and fill the fields for the credentials.
For the target endpoint, select Redshift and fill in all the textboxes with Host, Port and credentials.
Great, now we have everything we need, let’s create the task that’s going to migrate and replicate the stuff.
In DMS, press the Create Task button, put it a fancy name and start selecting all the resources we created earlier. For Migration type choose Migrate existing data and replicate ongoing changes, this performs a full migration first and then starts replicating on-going CRUD operations on the target database.
On the selection section, select whatever schema/table you want to use, in my case I’ll just replicate every change within a schema — also, check if the tables have primary keys, DMS may mess up if they don’t have a PK. Use % character as a wildcard (i.e “all tables containing the word ‘foo’ ” would be table = %foo%.
The task can also have transformation rules, it enables you to, for example, change a schema or table name on the target destination.
Ready to unleash it? Hit that create task button and wait for it to start — or start it if you chose not to start when created. The task now is going to fully load the tables into your destination and then start replicating changes. You should see Load complete, replication ongoing as its status when finished migrating.
Aaaaaand...
Voila, done!. Your Redshift warehouse is now enriched by your production RDS data, good job!. At this point, if anything went wrong you can always enable the logging feature of the DMS task to see what happened 😄, just take into account Cloudwatch billing.
You can also check some useful charts that the DMS task presents to us.
Next section is going to be about the creation of the S3 data-lake for other services to consume database changes. If you’re leaving now, thanks for reading and I hope this post was useful to you.
Same as before, create a task that instead of Redshift as a target destination, it’s an S3 bucket.
For the target endpoint, previously create an S3 bucket, and simply put the name of the bucket and the corresponding role to access it. Take into account the service role should have the following policy.
{    "Version": "2012-10-17",    "Statement": [       {            "Effect": "Allow",            "Action": [                "s3:PutObject",                "s3:DeleteObject"            ],            "Resource": [                "arn:aws:s3:::<name-of-bucket>"            ]        },        {            "Effect": "Allow",            "Action": [                "s3:ListBucket"            ],            "Resource": [                "arn:aws:s3:::<name-of-bucket>"            ]        }    ]}
On the additional parameters section, put the following addColumnName=true;, including the previous parameters or others that you may want to use.
In this case, choose to Replicate data changes only as migration type, we don’t want to infest our S3 bucket of previous data.
For more info about using S3 as a target for DMS, click here.
Ok, the moment of truth, run your little Frankenstein baby.
The state of the task should be Ongoing replication. If otherwise, check the logs related to the replication instance for errors in the Cloudwatch service.
From now on, if you’re using a staging database instance (which I hope you are), create, update and delete some rows in order for changes to be replicated by the task. This will hopefully conclude in a csv file in the S3 bucket you specified in your destination endpoint.
This file should have something like:
Op, id, field..fieldsI, 1, ...D, 2U, 3, ...
If you’re not, what here we call as salame, you’d realize I stands for Insert, D for delete and U for Update.
The moment you’ve been waiting for (or not). This is where we get to code the function that’s going to parse the CSV document that’s being created on our bucket.
For this part, I suggest you use Serverless, a nice tool to easily deploy Lambda functions using your AWS CLI credentials. It’s as simple as writing a .yml file, hitting serverless deploy and voila.
Now, let’s create our Lambda function and later add the S3 bucket event to trigger the function on every object creation. Also, you may want to add a prefix if you’ve specified that the files are going to be created within a specific folder, just put folder/ on the prefix textbox, and you’re good to go.
Let’s code part of the function that’s going to receive and extract the data from the file uploaded in the bucket.
So, what we first want to do is build our handler function, you know, the usual main() but Lambdably speaking.
This main function is going to receive as a parameter the JSON event that Cloudwatch handles to it, basically saying “Yo, there’s a new file created in the bucket, here’s its key to access it.”
Here’s the sample event that AWS gives us.
And below I paste the Python code that I use to get the final content of the file created in S3.
Now you have the data of the CSV in the file_content variable. If you have experience with parsers, this should be a piece of cake, if that’s not the case for you, I recommend you check this link.
From now on, it’s on your hands. Use a python driver to handle the CRUD operations, processing every row of the CSV impacting the changes on Redshift. For this case, I recommend using functions like execute_values() (psycopg2) to minimize execution time as explained here. Use Lambda environment variables to handle credential secrets for the function to use, remember it’s not a good idea to hardcode them.
What I present to you is just one of the thousand possibilities to achieve the goal of synchronizing databases. If you don’t mind spending some hundreds 💸 on services that handle the ETL process for you, go for it. No really, just go for it.
This architecture serves us with 2 main positive side effects IMO. First one, having tracking information about the changes in your production database, this can never be needless. Nowadays with services like AWS Athena and Glue you can query that data directly via the console. The second one is the ability to connect/trigger any procedure via S3 bucket object creation event — in our case, the IncreaseConciliación team replicating changes in their own Cassandra database.
We’ve come to an end, hopefully, you’ve enjoyed my first article 😄