Top 3 ways to back up your Matillion Data Engineering efforts
By: D Ma
Tags: AWS, Analytics, Big Data, Cloud Computing, Software Development

We have 3 main elements here:

Matillion ETL (EC2 instance)

Redshift DW

Tableau Server (EC2 instance)

With Redshift, we don’t have problems, it creates a snapshot itself. With Tableau Server we can do backups of content using TSM script or we can do EC2 snapshots. But in the case of multiple clusters, it could be challenging.

The final piece is Matillion ETL. It is running on top of the EC2 instance. We also can do the snapshot of content using Matillion API. However, in the case of EC2 instance loss, it will take lots of time to apply all configuration for the new instance. As a result, the best practice here is to have a snapshot of the instance.

In this blog post, we will talk about multiple options of Matillion ETL backups.

1. Using Matillion build-in functionality

Since Matillion ETL release 1.24 we are able to turn on automatic EBS snapshots. Go to Admin → Manage Backups.

Matillion EBS Backups

The downside of this solution, you aren’t able to delete old snapshots and you have to do this manually or set up another process that will delete older snapshots. We don’t want to accumulate EBS snapshots, because everything in the cloud costs money.

You can find more information about backups for AWS in Matillion documentation.

2. Using Cloud Watch Events

Amazon EBS emits notifications based on Amazon CloudWatch Events for a variety of volume, snapshot, and encryption status changes. With CloudWatch Events, you can establish rules that trigger programmatic actions in response to a change in volume, snapshot, or encryption key state. For example, when a snapshot is created, you can trigger an AWS Lambda function to share the completed snapshot with another account or copy it to another region for disaster-recovery purposes.

In case, if you like to be AWS native and you are operating with AWS tools, you might use Cloud Watch Events for creating for EBS snapshots.

Go to Cloud Watch -> Events → Rules.

Create the Rule

Add a new rule and choose the frequency for the rule. For example, in my case I want to run it every day at 22 pm:

Enter the schedule

There is a useful cheatsheet for cron expressions:

Then choose the volume id from your EC2:

At step 2, you can give a name for your rule

Next day, we can check the result by going to EC2->Snapshots and we will find our snapshot:

However, Cloudwatch events can only be used to create the EBS snapshots and not delete the snapshots. Also, for the target, it expects to provide an EBS volume Id and does not support tag. As a result, we can’t easily delete snapshots.

Creating rules with Cloudwatch Events is simple. But next question for us, how would we delete old snapshots? We might think about EBS Lifecycle Manager that allows us to create policy and delete old snapshots. Although Lifecycle Manager only supports tags and CloudWatch doesn’t create tags for use. We could leverage Lambda, but this is an advanced case.

3. Using AWS Backups

AWS Backup is a fully managed backup service that makes it easy to centralize and automate the backup of data across AWS services in the cloud and on premises. Using AWS Backup, you can configure backup policies and monitor backup activity for your AWS resources in one place. AWS Backup automates and consolidates backup tasks that were previously performed service-by-service, and removes the need to create custom scripts and manual processes. With just a few clicks on the AWS Backup console, you can create backup policies that automate backup schedules and retention management.

Create a Backup Plan:

Back up rule configuration:

Add tag:

Backup plan “MatillionETLBackupDaily” creation successful. You can now add additional schedule rules and assign resources to the Backup plan by selecting the Backup plan.

You can assign resources to this Backup plan after the plan has been created:

Assign resources:

As a result, we create the backup plan that will do daily snapshots for EBS instance and it has a lifecycle policy, that will delete the older snapshots and keep our cost low for storing snapshots.

Summary

We reviewed 3 methods of Matillion ETL backups. I’ve found that AWS Backups is the best choice for our cases. There are other methods that exists with AWS CLI or Lambda.