Amazon Redshift CI/CD — How we did it and why you should do it too
By: Doron Vainrub
Tags: Postgres, Big Data, Redshift, Cicd, Data Engineering

Database’s CI/CD

But Redshift SQL Scripts were just pushed to Gerrit (our VCS) without testing nor versioning and were manually applied by the analysts (most of the time skipping the Redshift instance in DEV, generating a discrepancy between both DEV and PROD environments).

ETL Projects’ Structure

Each ETL project corresponds to a Redshift schema, therefore, each project contains both Spark code and Redshift Scripts.

The Scripts folder is divided into tables and views folders, where each folder contains the scripts that create the relevant database object.

For example:

As you can see, in addition to not having a proper CI/CD pipeline for the scripts, all table fixes are appended to the same file without an easy way of keeping track of changes.

The journey begins

The application code’s CI/CD pipeline was already robust so we thought about integrating Redshift’s scripts into the same CI/CD process.

We came up with a list of requirements and goals for this to happen:

1. Database code is version controlled

Database code changes should be tracked in the same version control system as application code. Some immediate benefits this will give us:

Easier to keep track of changes.

Single source of truth.

Every change to the database is stored, allowing easy audit if any problems occur.

Prevent deployments where the database is out of sync with the application.

Changes can be easily rolled back.

New environments can be created from scratch by applying these scripts.

All database changes are migrations, therefore, each code change needs a unique identification.

We will need to keep track of which migrations have been applied to each database.

2. Database Code Validation

Once database code is checked in, a series of automated tests are immediately triggered. That way, analysts can get immediate feedback on SQL code, just as they do with application code. Some of the things we want to automatically test are:

Check all tables/views/functions are following the naming conventions set by the team and that they align with the requirements of the organization.

Check correct dependencies between tables and views (column names, views referencing table columns, etc.).

Validate Redshift/PSQL Syntax.

Run unit-test for complex views.

Test code changes in a clean instance before testing them on a real one to make sure if the deployment will work as intended.

In addition, everybody gets their own database instance so the same tests can also be triggered locally while developing to get feedback even faster.

3. CI/CD pipeline

After validating the database changes, an immutable artifact should be created together with the application code so automated DB deployments can be consistent, repeatable and predictable as application code releases. All this pipeline should be integrated into our current Jenkins pipeline.

4. Automatic Deployments

Analysts used to have Redshift permissions to run DDL queries manually, these permissions should be removed and automatic DB deployments should be allowed instead.

The automated deployment should be applied first on our Redshift DEV instance so the database changes and ETLs can be tested before deploying the changes to production.

In addition, this should allow us to keep both DEV and PROD instances in sync.