Building an ETL pipeline with Airflow and ECS | by Daniel Da Costa | Towards Data Science

ETL is an automated process that takes raw data, extracts and transforms the information required for analysis, and loads it to a data warehouse. There are different ways to build your ETL pipeline, on this post we’ll be using three main tools:
Airflow: one of the most powerful platforms used by Data Engineers for orchestrating workflows.
AWS ECS/Fargate: a container management service that makes it easy to run, stop, and manage your containers.
AWS s3: AWS simple storage service.
The architecture that we will be building follows the schema bellow:
First of all, you will need to have an ETL script ready to deploy. In this post, we will be using a Python script that reads from S3, transforms the data, and save it back to S3. You can check the full code here.
Since we will be using a container-based application, we need to build our Docker Image containing all prerequisites to run the ETL job and configuration. Here is an example of my Dockerfile:
FROM python:3.7-slimENV APP_DIR /ml-pipelineRUN mkdir -p ${APP_DIR}WORKDIR ${APP_DIR}COPY . .RUN apt-get updateRUN apt-get install libgomp1RUN pip install -r requirements.txtENV READ_BUCKET=ml-sls-deploy-prdENV READ_DATA_PATH=dataENV READ_MODELS_PATH=modelsENV WRITE_BUCKET=ml-sls-deploy-prd-resultsENV WRITE_DATA_PATH=resultsCMD ["python", "main.py"]
To upload your image to your ECS repository, you will need to store it in a registry like Docker Hub, for example. I used Amazon ECR, AWS's fully managed container service. First, you will need to create a Repository on ECR.
Second, you will need to push your container image to the repository, using the following set of commands.
ECS is a container orchestration service provided by AWS. ECS has two launch types that can define how the compute resources will be managed:
The traditional EC2 launch type, where it utilizes EC2 underneath to power its compute resources.
Fargate removes the responsibility of provisioning, configuring, and managing the EC2 instances by allowing AWS to manage the EC2 instances.
The Fargate billing is described by Nathan Peck in EC2 or AWS Fargate?:
With the AWS Fargate launch type, billing is based on how many CPU cores, and gigabytes of memory your task requires, per second. You only ever pay for what your task uses, no more paying for EC2 capacity that goes unused.
One of the use cases of Fargate is for small workloads, a small test environment; that's exactly what we are looking for! If you want to dig deep on which launch type of ECS to choose you can check this post: ECS or Fargate?
The main steps for creating your cluster are:
Create a Cluster: Choose Network only. This configuration is built using Fargate Tasks: the Fargate launch type allows you to run your containerized applications without the need to manage EC2 instances, you pay for running tasks. When you run a task, Fargate launches the containers for you.
Task Definition: The creation of your container blueprint. You’ll need to create a Task Role: IAM Role that tasks can use to make API requests to authorized AWS services; Since our container is reading and writing to/from s3, it will need these permissions. You will also need to create a Task Execution Role: an IAM that helps pulling images from your docker register, we are using ECR here.
Add a Container: You’ll need to deploy your container to ECS Fargate. You may use here our Docker Image created in section ETL.
We won’t go through all the steps on how to create your own ECS cluster, since there are many tutorials on how to do it on the internet. I recommend that you follow this tutorial.
Nowadays, Airflow is used to solve a variety of data ingestion, preparation, and consumption problems. The platform allows you to build you own ETL for integrating data between disparate systems, you can use it to build simple workflows or even to orchestrate complex ML workflows. Airflow also counts on a huge open source community!
One of the main benefits of using Airflow is that is designed as a configuration-as-code: Airflow pipelines are defined in Python, allowing us to build pipelines in a very simple way.
In this post, we will be using Docker for deploying airflow on our local computer. You will just need to set up your Dockerfile as follows:
FROM puckel/docker-airflowWORKDIR /airflowRUN pip install boto3
We will need to install the boto3 library inside our container so that we can configure our AWS credentials in Airflow.
After building the docker image, we will create a volume that maps the directory on our local machine where we’ll hold DAG definitions and the locations where Airflow reads them on the container with the following command:
docker run -d -p 8080:8080 -v /path/to/dags/on/your/local/machine:/usr/local/airflow/dags {your-image}
You can go now to your localhost:8080 to access your Airflow UI!
OBS: If you'd like to host it directly on the cloud you may use AWS's new service: Amazon Managed Workflows for Apache Airflow. You can check more about it here.
You will need do set yours AWS credentials through the Airflow UI. Go to Admin > Connection > Create
Create a connection called aws_credentials of type Amazon Web Services. Put your AWS_ACCESS_KEY_ID on Login and your AWS_SECRET_ACCESS_KEY on Password:
We will be creating a DAG with the following characteristics:
schedule_interval: Our DAG will be run manually through the Airflow UI interface,
ECS Operator: We will be using this operator to run our task defined in AWS ECS.
Environment Variables: We will need to define some local variables containing information from our ECS Cluster and Task Definition.
These environment variables are created directly on the Airflow UI. Go to Admin > Variables and create the following variables:
Once everything is set up, you will just have to run your DAG!
Congratulations! You have just created your ETL pipeline using Airflow and ECS.
Running an ETL pipeline is often troublesome. We need to provision and manage the backend infrastructure, making it as reliable as possible. The architecture described in this post makes it easy to run ETL workflows as quickly as possible and sufficiently reliable for some use cases.
You can check the full code in here!