Building an Event-Driven Data Pipeline to Copy Data from Amazon S3 to Azure Storage | by Yi Ai | Towards Data Science

It may be a requirement of your business to move a good amount of data periodically from one public cloud to another. More specifically, you may face mandates requiring a multi-cloud solution. This article covers one approach to automate data replication from AWS S3 Bucket to Microsoft Azure Blob Storage container using Amazon S3 Inventory, Amazon S3 Batch Operations, Fargate, and AzCopy.
Your company produces new CSV files on-premises every day with a total size of around 100GB after compression. All files have a size of 1–2 GB and need to be uploaded to Amazon S3 every night in a fixed time window between 3 am, and 5 am. Your business has decided to copy those CSV files from S3 to Microsoft Azure Storage after all files uploaded to S3. You have to find an easy and fast way to automate the data replication workflow.
To accomplish this task, we can build a data pipeline to copy data periodically from S3 to Azure Storage using AWS Data Wrangler, Amazon S3 Inventory, Amazon S3 Batch Operations, Athena, Fargate, and AzCopy.
The diagram below represents the high-level architecture of the pipeline solution:
Create a VPC with private and public subnets, S3 endpoints, and NAT gateway.
Create an Azure Storage account and blob container, generate a SAS token, then add a firewall rule to allow traffic from AWS VPC to Azure Storage.
Configure daily S3 Inventory Reports on the S3 bucket.
Use Athena to filter only the new objects from S3 inventory reports and export those objects’ bucket names & object keys to a CSV manifest file.
Use exported CSV manifest file to create an S3 Batch Operations PUT copy Job that copies objects to a destination S3 bucket with lifecycle policy expiration rule configured.
Setup an Eventbridge rule, invoke lambda function to run Fargate task that copies all objects with the same prefix in destination bucket to Azure Storage container.
Setup an AWS account
Setup an Azure account
Install the latest AWS-CLI
Install AWS CDK-CLI
Basic understanding of AWS CDK
Basic understanding of Docker
We use CDK to build our infrastructure on AWS. First, let’s create a source Bucket to receive files from external providers or on-premise and set up daily inventory reports that provide a flat-file list of your objects and metadata.
Next, create a destination bucket as temporary storage with lifecycle policy expiration rule configured on prefix /tmp_transition. All files with the prefix (eg. /tmp_transition/file1.csv) will copy to Azure and will be removed by lifecycle policy after 24hours.
Use the following code to create S3 buckets.
Next, we need to create VPC with both public and private subnets, NAT Gateway, an S3 endpoint, and attach an endpoint policy that allows access to the Fargate container to which S3 bucket we are copying data to Azure.
Now define your VPC and related resources using the following code.
While creating NAT Gateway, an Elastic IP Address will create in AWS. We will need the IP address to set up the Azure Storage Firewall rule in step3.
To simplify managing resources, we can use the Azure Resource Manager template (ARM template) to deploy resources at our Azure subscription level.
I will assume you already have an Azure Subscription setup. We will use Cloud shell to deploy a Resource Group, Azure Storage account, a container, and Firewall rule to allow traffic from a specific IP address.
Click on the Cloud Shell icon in the Azure Portal's header bar, and it will open the Cloud Shell.
Run the following command to deploy:
az group create --name examplegroup --location australiaeastaz deployment group create --resource-group examplegroup --template-uri https://raw.githubusercontent.com/yai333/DataPipelineS32Blob/master/Azure-Template-DemoRG/template.json  --parameters storageAccounts_mydemostroageaccount_name=mydemostorageaccountaiyi --debug
Once the template has been deployed, we can verify the deployment by exploring the Azure portal's resource group. All resources deployed will be displayed in the Overview section of the Resource group.
Let’s create a Firewall rule for our Storage Account:
Firstly, go to the storage account we just deployed.
Secondly, click on the settings menu called Firewalls and virtual networks.
Thirdly, check that you’ve selected to allow access from Selected networks.
Then, for granting access to an internet IP range, enter AWS VPC’s public IP address (step 2) and Save.
We will then generate Shared Access Signatures (SAS) to grant limited access to Azure Storage resources.
Run below command in Cloudshell:
RG_NAME='examplegroup'ACCOUNT_NAME='mydemostorageaccountaiyi' ACCOUNT_KEY=`az storage account keys list --account-name=$ACCOUNT_NAME --query [0].value -o tsv`BLOB_CONTAINER=democontainerSTORAGE_CONN_STRING=`az storage account show-connection-string --name $ACCOUNT_NAME --resource-group $RG_NAME --output tsv`SAS=`az storage container generate-sas --connection-string $STORAGE_CONN_STRING -n $BLOB_CONTAINER --expiry '2021-06-30' --permissions aclrw --output tsv`echo $SAS
We will get the required SAS and URLs that grant (a)dd (d)elete (r)ead (w)rite access to a blob container democontainer.
se=2021-06-30&sp=racwl&sv=2018-11-09&sr=c&sig=xxxxbBfqfEppPpBZPOTRiwvkh69xxxx/xxxxQA0YtKo%3D
Let’s move back to AWS and put SAS to AWS SSM Parameter Store.
Run following command in local terminator.
aws ssm put-parameter --cli-input-json '{  "Name": "/s3toblob/azure/storage/sas",  "Value": "se=2021-06-30&sp=racwl&sv=2018-11-09&sr=c&sig=xxxxbBfqfEppPpBZPOTRiwvkh69xxxx/xxxxQA0YtKo%3D",  "Type": "SecureString"}'
Now, let’s move up to lambda functions. We will create three lambda functions and one lambda layer:
fn_create_s3batch_manifest and DataWranglerLayer
fn_create_batch_job
fn_process_transfer_task
This lambda function uses AWS Data Wrangler’s Athena module to filter new files in the past UTC date and save files list to a CSV manifest file.
Copy the following code to CDK stack.py. download awswranger-layerzip file from here.
then create ./src/lambda_create_s3batch_manifest.py with the following code:
In the above coding, we use Athena query to create Glue Database, Table and add a partition to that table every day. Then lambda executes except query to return the difference between the two date partitions.
Note that start_query_execution is asynchronous, hence no need to wait for the result in Lambda. Once the query is executed, the result will save to s3_output=f"s3://{os.getenv('DESTINATION_BUCKET_NAME')}/csv_manifest/dt={partition_dt}" as a CSV file.
In this section, we will create a lambda function fn_create_batch_job and enable Amazon S3 to send a notification to trigger fn_create_batch_job when a CSV file is added to an Amazon S3 Bucket /csv_manifest prefix. Put following code to CDK stack.py:
Create ./src/lambda_create_batch_job.py with the following code:
Lambda fn_create_batch_job function create S3 Batch Operation Job, copy all the files listed in CSV manifest to S3 Destination Bucket /tmp_transition prefix .
S3 Batch Operations is an Amazon S3 data management feature that lets you manage billions of objects at scale. To start an S3 Batch Operation Job, we also need to set up an IAM role S3BatchRole with the corresponding policies:
We will create an Eventbridge custom rule that tracks an S3 Batch Operations job in Amazon EventBridge through AWS CloudTrail and send events in Completed status to the target notification resource fn_process_transfer_task .
Lambda fn_process_transfer_task will then start a Fargate Task programmatically to copy files in /tmp_transition prefix to Azure Storage Container democontainer .
Create ./src/lambda_process_s3transfer_task.py with the following code:
Now, We have set up the Serverless part. Let’s move up to the Fargate task and process the data replication.
We will create:
An ECR image with AzCopy was installed. AzCopy is a command-line utility that you can use to copy blobs or files to or from a storage account.
An ECS Cluster with a Fargte task.
Let’s getting started.
Build ECS, ECR, and Fargate stack.
Build ECS, ECR, and Fargate stack.
2. Build a Docker image and install Azcopy there.
Note that to use AzCopy transfer files from AWS, we will need to set up AWS Credentials in the container. We can retrieve AWS credentials using:
curl http://169.254.170.2/$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
3. Push Docker image to ECR
eval $(aws ecr get-login --region ap-southeast-2 --no-include-email)docker build . -t YOUR_ACCOUNT_ID.dkr.ecr.ap-southeast-2.amazonaws.com/YOUR_ECR_NAMEdocker push YOUR_ACCOUNT_ID.dkr.ecr.ap-southeast-2.amazonaws.com/YOUR_ECR_NAME
Great! We have what we need! You can find the full solution CDK project in my Github Repo. Clone the repo and deploy the stack:
cd CDK-S3toblob pip install -r requirements.txtcdk deploy
Once the stack has been successfully created, navigate to the AWS CloudFormation console, locate the stack we just created, and go to the Resources tab to find the deployed resources.
Now it’s time to test our workflow; go to the S3 source bucket demo-databucket-source . Upload as many files in different folders (prefix). Wait 24 hours for the next inventory report generated; then, you will see the whole pipeline start running, and files will eventually be copied to Azure democontainer .
We should see the logs of the Fargate task like the below screenshot.
We can also monitor, troubleshoot, and set alarms for ECS resources using CloudWatch Container Insights.
In this article, I introduced the approach to automate data replication from AWS S3 to Microsoft Azure Storage. I walked you through how to use CDK to deploy VPC, AWS S3, Lambda, Cloudtrail, Fargte resources, showing you how to use the ARM template deploy Azure services. I showed you how to use the AWS Wrangler library and Athena query to create a table and querying the table.
I hope you have found this article useful. You can find the complete project in my GitHub repo.