Cross-account data migration using AWS Database Migration Service
Tags: AWS, Data Migration Service, Cloud, Redshift, Aurora Serverless

AWS Database Migration Service is a convenient solution for more advanced data migrations. In this post I’ll describe how to setup cross-account database migration tasks from Aurora Serverless MySQL to Amazon Redshift from one AWS Account to another AWS account and region using AWS Database Migration Service (DMS).

The Challenge

For a recent project I had to migrate existing database tables with millions of records from an Aurora Serverless MySQL cluster in one AWS Account — in region eu-central-1 (Frankfurt) — into an existing Redshift cluster that resides in another AWS account — in region eu-west-1 (Ireland).

Both the Aurora MySQL cluster (Source) and the Redshift cluster (Target) resided inside a private subnet in a VPC for security reasons and are therefore not accessible from the public internet.

The data migration only needed to happen once or twice, keeping the target database continuously in sync with the source was out-of-scope for this project.

Furthermore, the data inside the source database was pretty straight-forward; it didn’t contain complex types or structures, mostly booleans, integers and (var)char type columns.

Handling more advanced data types comes with additional challenges and is not taken into consideration for this post.

The Solution

The goal of the solution was to prevent manual data export, data mapping and data import and rely upon native AWS services to perform the data migration task with the option for easy task repetition as well.

As a solution I relied upon the AWS Database Migration Service (DMS) which allows to define data migration tasks — with support for schema, table and column filtering & mapping between the source and target database. For the cross-account — inter-region — aspect of the data migration I made use of a VPC Peering connection, which allowed me to create a private network connection between private subnets in both the source and target VPC’s.

On a higher level, this resulted in the following steps to achieve this solution:

Create a VPC Peering connection from the target VPC to the source VPC

Create a private DMS Replication Instance (EC2) inside the Target VPC

Create a DMS Replication Source Endpoint for Aurora MySQL

Create a DMS Replication Target Endpoint for Amazon Redshift

Create the DMS Replication Task

Execute the replication task to perform the migration

The implementation will take approximately 30 minutes in total. Understandably complexity will vary on a per use case basis as well.

Prerequisites

Before starting the steps below, please make sure to note down the following details:

Account IDs of the target and source AWS accounts

VPC IDs of the target and source AWS accounts

CIDR ranges of both the target and source VPCs

The Aurora MySQL database endpoint details: server name, port, user name, password

The Redshift database endpoint details: server name, port, user name, password and database name

Creating the VPC Peering Connection

The VPC Peering connection connects the VPC in the target account with the VPC in the source account so that private communication between becomes possible over IPv4. One of the main prerequisites for the VPC Peering connection is that CIDR blocks between both the VPCs are not allowed to overlap.

Creating the VPC Peering connection in most scenario’s is pretty straightforward:

Login into the AWS Console on your Target Account and navigate to “AWS Services > VPC” In the menu click on “Peering Connections” Press the “Create Peering Connection” button to create a new peering connection and fill in the details and press “Create Peering Connection”

AWS VPC Peering connection form

Enter details about the local and destination VPC:

VPC (Requester): is the ID of the VPC in your local (Redshift) account. Once entered, the associated CIDR blocks will be displayed

is the ID of the VPC in your local (Redshift) account. Once entered, the associated CIDR blocks will be displayed Account: select “Another account” and enter the Account ID of the Aurora MySQL account.

select “Another account” and enter the of the Aurora MySQL account. Region: select “Another Region” and select the region of the Aurora MySQL Account

select “Another Region” and select the region of the Aurora MySQL Account VPC ID (Accepter): enter the VPC ID of the VPC where Aurora MySQL is located in

After creation the accepter will receive a VPC peering request under their Account and must accept the VPC peering request:

Login into the other AWS Account and navigate to Amazon VPC Choose Peering Connections from the menu Select the peering connection with pending status and choose Actions > Accept Request

Make sure to enable DNS Resolution on both sides of the peering connection so that we’re able to target the source and target database by their IPv4 but their DNS names instead:

Select the peering connection and then Actions > Edit DNS Settings For the Requester: check “Allow accepter VPC (vpc-XXX) to resolve DNS of requester VPC (vpc-YYY) hosts to private IP” For the Accepter: check “Allow requester VPC (vpc-YYY) to resolve DNS of accepter VPC (vpc-XXX) hosts to private IP”

As a last step, modify the Route tables associated with the subnets in which your source and target databases reside. This can be done directly after creating the Peering Connection as prompted by AWS or later:

Navigate to Route Tables Select the route tables associated with the database and choose Actions > Edit Routes Press “Add route” and enter the destination VPC’s CIDR range under Destination and as Target choose “Peering Connection” and select the connection we’ve just created

Route tables > VPC peering connection

The VPC Peering connection between the source and target VPC has been established. Now it’s time to setup the AWS DMS service.

Setting up the AWS DMS Replication Instance

The DMS Replication Instance is an Amazon EC2 instance that resides in the VPC of the target (Redshift) database and is used to execute Replication Tasks to migrate data from the source endpoint to the target endpoint. For access to the source database it will use the newly created VPC Peering connection.

To create the Replication Instance:

Navigate to AWS DMS in the AWS Console Give the instance a useful name and description. For this project the default dms.t3.medium Instance class was enough capacity to execute our migration. Make sure to select the VPC where the target (Redshift) database resides under VPC For security reasons I’ve disabled the Publicly accessible checkbox. We don’t have access to the instance and it uses the private VPC peering connection to access the database so no need for a public EC2 instance.

AWS DMS Replication Instance

I left all the other details on their default settings. Now press “Create” and the instance will be created.

Instance creation can take up to 15 minutes

Once created, the private instance will have a private IPv4 address associated. Note down this address and lets modify both the Aurora MySQL and Redshift Security Groups to allow access from the Replication Instance to the databases:

Navigate to the Security Group associated with the database Press Actions > Edit inbound rules Select under Type dropdown either “MYSQL/Aurora” or “Redshift” Enter the private IPv4 address under Source followed by /32 (e.g. 10.0.10.100/32 ). Provide a logical Description

Note that as a Source you should enter the name of the Replication Instance Security Group ID (e.g. “SG-XXX”) instead of its IPv4 address, but this is not allowed for cross-region access.

The Replication Instance should now have proper access to both the databases.

Creating the DMS Database Endpoints

In this step we’ll create both the Source (Aurora MySQL) and the Target (Amazon Redshift) DMS Endpoints to be used by the Migration Task.

Create the Source Endpoint and pay attention to the following details:

Select Source endpoint

For Source engine select “Amazon Aurora MySQL”

select “Amazon Aurora MySQL” Enter the database credentials under Access to endpoint database > provide access information manually

I left Endpoint-specific settings to their defaults

Test the connection (optional)

To create the Target Endpoint:

Select Target endpoint

Select “Amazon Redshift” as the Target Engine

Provide the connection details

When your Redshift cluster uses a specific (non-default) KMS master key for data encryption, please ensure to select the correct key under KMS master key

As an optional step, test the connection as well.

Now that both DMS Endpoints have been created it’s time for our final step; creating the DMS Database migration task.

Creating the DMS Database Migration Task

For the last step, let’s create the Database Migration Task so that we’re actually able to execute the migration. Head to Database migration tasks in the menu and press Create Task.

Enter the following information under Task Configuration:

Select the Replication Instance we’ve just created Select the Source database endpoint we’ve just created Select the Target database endpoint we’ve just created Select “Migrate existing data” for Migration Type as we only need to copy data once or twice and we don’t need to replicate ongoing changes. When you do, it comes with additional requirements such as “binary logging” enabled on the MySQL Aurora Cluster.

Replication Task configuration

Use the Task Settings Wizard to configure the task. Of course JSON mapping is also an option, but I found the Wizard pretty useful for most task configurations.

For Target table preparation mode I selected “Truncate” so that I always have an exact copy of the source data in my target database when I have to re-execute the replication task.

I selected “Truncate” so that I always have an exact copy of the source data in my target database when I have to re-execute the replication task. Choose “Enable CloudWatch logs” to have the replication instance log task information in CloudWatch.

Replication Task Settings

Under Table mappings you are able to select a source scheme, whitelist/blacklist tables and apply schema, table and column based transformations such as renaming the Aurora MySQL schema to the desired Redshift schema (e.g. “public”).

First let’s apply the Selection rules where we select which tables to include from the database Schema. I used this to select only a sub-set of tables to migrate to the Target.

Then configure Transformation rules to apply schema, table and/or column level transformations. I used this to migrate tables from the source schema to my Redshift’s “public” schema, rename tables and add the “_aurora_prod” suffix to avoid naming conflicts and improve identification and as well to exclude some columns— for example columns which contain sensitive information.

Replication task Transformation Rules

Under Migration task startup configuration I selected “Manually later” and then created the task.

Now that the task has been created it should appear on the Database migration task dashboard.

Executing the Migration Task

To execute the migration task, select it from the overview and select Actions > Start. The migration task will now start.

Select the migration tasks’ details or press the Quick view and compare button to get insights in the Migration Task’s:

Details

Status and progress

Table statistics: mapping, state, rows to be migrated, etc.

CloudWatch metrics

Mapping Rules (also useful to copy mapping rules as JSON when creating new instances)

Once finished it will say “Load complete” and the progress says “100 %”