Running Airflow with Docker on EC2 + CI/CD with GitLab | by Ammar Chalifah | Towards Data Science

I’ve been learning data science for a year, yet until several weeks ago I still found myself completely clueless on the engineering side. Data analysis and artificial intelligence are stuff on the top of the pyramid, which requires strong data pipeline foundation. With this post, I will share my lessons learned while trying to built a simple data pipeline using Airflow, an open source orchestration tool. This orchestrator is deployed on top of Amazon EC2, utilizing GitLab’s CI/CD to ease up development process. Okay, one by one, let’s talk about definition of each terms.
Based on its website, Airflow is a “platform created by the community to programmatically author, schedule and monitor workflows.” Airflow usually used by Data Engineers to orchestrating workflows and pipelines. Let’s say a Data Engineer need to move data from a source to another destination four times a day (e.g. to keep the freshness of the data). Instead of doing that task manually, they can automate the script to run on a particular schedule using Airflow. Airflow can also be used to manage dependencies, monitor workflow executions, and visualize workflows.
Airflow needs to be installed in a machine. Instead of using local computer, it is better to install it in a cloud-based host to improve reliability.
Amazon EC2 (stands for Elastic Compute Cloud) is a rented cloud-based computers offered by Amazon Web Services. EC2 will serve as the host, which Airflow will be deployed on. EC2 is just a computer, like our local laptop/computer, but the machine will be up 99% of the time and managed by AWS team. We can ‘tunnel’ our way inside the command line interface of EC2’s machine by using SSH. Later on, we will learn how we can use SSH and git to deploy codes from a remote repository to EC2.
Airflow won’t be installed directly on EC2 local machine, but installed inside a Docker container. Docker is a set of platform that creates virtual OS called containers. We use Docker to reduce setups, as all setups will be managed by the maintainer of the Docker images (in this case, Airflow contributors). A Docker image contains application code, libraries, tools, dependencies and other files needed to make an application run. We will pull an image from Docker image registry, build it, start the container, and run Airflow inside.
GitLab will be used as our repository and DevOps lifecycle tool. There are practically countless way we can utilize CI/CD in GitLab, but for the sake of simplicity we will only use the ‘CD’ part. Every time user merge any branch into the main branch, the deployment script will be triggered. Deployment pipeline consists of SSH tunneling into our EC2, pulling main branch from GitLab repository into EC2, write several variables to a file, and rebuild + restart the Docker container.
Before diving into the details, check this repository, which contains the complete codebase and summary of the steps that we will take to successfully create a continuous deployment pipeline of Airflow. Okay, let’s start!
Access AWS Console, then prepare an EC2 instance. I recommend using instance with ubuntu OS with lower-mid performance. Download the SSH key of your EC2 instance (usually ends with .pem suffix), and write down the IP address of your instance. Then, configure your instance to open all ports (at least open port 8080 to access the Airflow webserver). Lastly, access the EC2 by using the SSH key to install Docker and docker-compose from command line interface.
ssh -i ssh-key.pem ubuntu@EC2_ADDRESS
Guide for EC2 provision: https://www.youtube.com/watch?v=YB_qanudIzA
Sign up on GitLab (https://gitlab.com) and create a new project. Clone the contents from the github repository to our new GitLab repository. After the repository is filled with the codes, we need to configure GitLab runner and set repository variables. GitLab runner is used to execute CI/CD scripts which we defined in gitlab-ci.yml . Repository variables are used to store sensitive informations (such as username, password, IP address of your EC2, and SSH key of the EC2). Repository variables are loaded as environment variables in CI/CD pipelines, where it could be deployed to the server as variables/files.
Follow instructions in GitLab runner configuration guide to configure our runner. For me, I used EC2 instance with ubuntu OS to install my runner. You don’t have to configure your own runner if you activate GitLab’s free trial by giving your credit card information, as GitLab will provide a shared runner for you.
GitLab Repository Variables
There are six repository variables that we need to set. Go to Settings > CI/CD > Variables, click Expand, and start adding variables.
The first two are _AIRFLOW_WWW_USER_PASSWORD and _AIRFLOW_WWW_USER_USERNAME, which will be used to log in to the Airflow server once it is deployed. Choose any arbitrary value for these two variables.
The second pair is EC2_ADDRESS (IP address of your EC2 instance) and SSH_KEY_EC2 (SSH key of your EC2 instance, usually ends with .pem suffix). These values will be used to access your EC2 machine. Set SSH_KEY_EC2 as file instead of variable.
The last pair is GITLAB_PASSWORD and GITLAB_USERNAME, used for credentials to pulling commits from GitLab repository.
Start by looking at the directory structure below.
The three directories (dags, logs, and plugins) are basic Airflow directories. We leave logs and plugins (almost) empty, because those folders will be filled after deployment. Scripts that we wish to automate are written inside dags, which we will cover later.
During any push to GitLab’s repo, the scripts in .gitlab-ci.yml (which define CI/CD pipeline) will be triggered. The script execute series of commands to deploy the repository to EC2. Inside EC2, we will build the Docker container containing Airflow by using docker-compose up on the docker-compose.yaml.
Before getting more complex, please refer to this guide to understand what GitLab’s CI/CD is.
The CI/CD pipeline is defined in .gitlab-ci.yml.
Because I’m using GitLab runner that executes pipelines using Docker, the default image is specified at the top. I used ubuntu:latest as my default Docker image.
As stated earlier, we won’t utilize the continuous integration aspect of CI/CD. We only use deployment that only triggered in main branch. That means all push to other branches beside main won’t results in deployment.
deploy-prod:  only:   - main  stage: deploy  before_script: ...  script: ...
deploy-prod is the name of our pipeline, which will only run in main branch. The stage of this pipeline is deployment stage (in contrast to other stages such as testing or building). Commands are specified inside before_script and script.
before_script:- ls -la- pwd- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'- eval $(ssh-agent -s)- mkdir -p ~/.ssh- chmod 700 ~/.ssh- cat $SSH_KEY_EC2- echo "$(cat $SSH_KEY_EC2)" >> ~/.ssh/ssh-key.pem- chmod 400 ~/.ssh/ssh-key.pem- cat ~/.ssh/ssh-key.pem- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config- apt-get update -y- apt-get -y install rsync
before_script consists of several commands that prepare our runner for deployment. The commands are used to install open-ssh, evaluate ssh-agent, dump SSH key from repository variable to file, change permission of the SSH key, set ssh config, and install rsync.
ssh -i ~/.ssh/ssh-key.pem ubuntu@$EC2_ADDRESS \'git config --global --replace-all user.name "Full Name"&& git config --global --replace-all user.email "email@email.com"'
The script section are divided into three sections. Each sections are started with ssh tunneling into the EC2. The first section is used to set git config to use your full name and email as the global config.
ssh -i ~/.ssh/ssh-key.pem ubuntu@$EC2_ADDRESS \'if [ -d airflow-pipeline ];then cd airflow-pipeline&& git status&& git restore .env&& git pull --rebase&& git status&& sed -i s:%AIRFLOW_UID%:'"$(id -u)"':g .env&& sed -i s:%AIRFLOW_GID%:0:g .env&& sed -i s:%_AIRFLOW_WWW_USER_USERNAME%:'"$_AIRFLOW_WWW_USER_USERNAME"':g .env&& sed -i s:%_AIRFLOW_WWW_USER_PASSWORD%:'"$_AIRFLOW_WWW_USER_PASSWORD"':g .envelse git clone https://'"$GITLAB_USERNAME"':'"$GITLAB_PASSWORD"'@gitlab.com/gitlab-group/gitlab-project.git&& cd airflow-pipeline&& ls -la&& chmod 777 logs&& sed -i s:%AIRFLOW_UID%:'"$(id -u)"':g .env&& sed -i s:%AIRFLOW_GID%:0:g .env&& sed -i s:%_AIRFLOW_WWW_USER_USERNAME%:'"$_AIRFLOW_WWW_USER_USERNAME"':g .env&& sed -i s:%_AIRFLOW_WWW_USER_PASSWORD%:'"$_AIRFLOW_WWW_USER_PASSWORD"':g .env&& docker-compose up airflow-init;fi'
The second section is used to pulling changes from main branch in GitLab repository and writing environment variables into .env file. Here we use conditions based on existence of the directory on the EC2 home directory. If we have cloned GitLab repository before, we only executed: restore all changes from .env file, pulling changes from remote GitLab repository, and writing environment variables to .env. If we haven’t, we start by cloning the repository, then write environment variables to .env file, and starting the Docker container.
ssh -i ~/.ssh/ssh-key.pem ubuntu@$EC2_ADDRESS \'cd airflow-pipeline &&if [ docker ps | grep -q keyword ];then docker-compose down && docker-compose up -d --build;else docker-compose up -d --build;fi;'
The latest section is used to restart the Docker to incorporate new changes.
CI/CD pipeline can be monitored on CI/CD > Pipelines on GitLab.
If everything went smoothly, your Airflow webserver will be ready. Go to https://{EC2_ADDRESS}:8080 on browser to log in to the user interface. Log in with Airflow username and password that you set on the repository variables before.
Later, any script you wish to automate can be written inside a DAG (or directed acyclic graph, a directed graph with no directed cycles. Tasks inside a DAG are framed as a node of a graph, where dependencies can be defined. Any changes in the repository could be deployed by pushing changes to GitLab repository, merge to main branch (or push to main branch directly), and wait until CI/CD pipeline succeeded.
You can define many things inside a DAG, such as data ingestion tasks, data querying task, send automated email, scrape data, make API call, etc.
It is not recommended to log Airflow activity locally. However, due to time and cost constraint, I log all activities locally, especially under /logs directory. Besides logs, DAG execution also create caches in /dags/__pycache__/. Hence, I ignored those files by using .gitignore to safely pull changes without causing conflicts.
If anything goes wrong, you can always inspect what happens inside EC2 by using the SSH key.
ssh -i ssh-key.pem ubuntu@EC2_ADDRESS
Delete unused files, unused Docker images, or make any adjustments from inside!
[1] https://airflow.apache.org/[2] https://aws.amazon.com/ec2/[3] https://www.docker.com/[4] https://about.gitlab.com/[5] https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html [6] Code references from Bunga Amalia K for setting up GitLab runner