Let’s Build An ELT Pipeline Pt.2: Setting Up Airflow + Docker
By: Jonathan Duran
Tags: Data Engineering, Docker, Postgres, Airflow, Python

Photo by Sebastian Herrmann on Unsplash

This is part two of my ELT project series. Please refer to the introductory article for an overview of the project.

The focus of this article is entirely on configuring Airflow and Postgres inside of Docker containers. There are quite a bit of nuance in setting things up correctly, but for Docker newbie’s, this should be a great hands-on learning experience.

A Word To All The Aspiring Data Engineers

I cannot stress enough how important it is for aspiring data engineers to learn technologies like Docker. Personally, every time I have felt like giving up, or felt inadequate for this field, it has been in the middle of configuring and installing tools locally. There is nothing more frustrating than having the passion to work with data, only to spend 80% of your project time exploring the dark side of google (anything beyond the first page) in hopes of fixing installation/configuration issues.

Now it is unlikely that as a junior data engineer you will find yourself in a role with expectations of building something from scratch. But unfortunately for your personal projects, if you cannot figure out how to install tools or learn how different technologies interconnect, you will not make it to the finish line. Tools like Docker allow us to set up projects involving multiple technologies (like this one) with less frustration, getting us to the fun part of working with the data much quicker.

The Plan

Let’s start with Airflow…

At a bare minimum, Airflow needs three components.

A metadatabase to store information related to DAGs, variables, connections, etc. A webserver that produces the user interface and also communicates with the metadatabase. The scheduler, which works with Executors to handle all tasks.

These requirements translate into three docker containers: A Postgres image for the metadatabase, and two copies of the same Airflow image for the webserver and scheduler.

Next, we have to sort out DBT. There are a few options that will allow us to schedule DBT jobs.

First, we can create our project on DBT cloud and schedule jobs to run whenever we need it. The problem with this approach is that it makes our Airflow and DBT project completely independent from each other. In other words, our DBT run does not depend on the success of the Airflow tasks involved in the earlier stages of our pipeline. We need an alternative method that keeps our two projects in sync when it comes to errors in the pipeline.

Our second option is to run our DBT project from the inside (or alongside) our Airflow project. We could either use a plugin to run DBT as an Airflow task or use Airflow’s BashOperator to trigger DBT runs. This method solves the independence issue from our first option but creates additional problems. Having both Airflow and DBT in the same directory produces a messy project structure. The original idea was to have the analyst handle the DBT code, while engineers dealt with the Airflow code, however, this decoupling fails with this particular design and will make changes to the codebase unnecessarily difficult.

Our final and best option is to use Airflow’s DockerOperator. We can create our DBT project somewhere far from Airflow, and turn it into a Docker image. The DockerOperator allows us to execute a command inside of a docker container. That is perfect for our needs! Analysts can continue to develop, improve, and test their models away from the Airflow codebase, while the production version of DBT runs from a Docker image by Airflow’s DockerOperator. All we have to do to update our production DBT project is rebuild our Docker image with the new changes.

Great, option three it is!

I will leave the DBT setup for a later article, as for now, let’s focus on Airflow and Docker.

DockerFile

I am assuming you already have Docker installed on your host machine.

There already exists a pretty popular Docker image for Airflow from Puckel, however, we need to build on top of it in order to allow our DBT-Docker-In-A-Docker plan to work.

Create the DockerFile inside your project directory.

$ touch DockerFile

/my-airflow-project/DockerFile

FROM puckel/docker-airflow:1.10.4 USER root RUN curl -sSL https://get.docker.com/ | sh

RUN pip install --upgrade pip && \

pip install docker==4.1.0

We switch to the root user to avoid any permission-related issues when installing Docker inside our containers.

Docker Compose

Let’s set up our services. Recall that we need three containers: Postgres, Airflow webserver, and Airflow scheduler.

/my-airflow-project/docker-compose.yaml

version: "3.8" services:

postgres:

image: postgres:9.6

env_file:

- .env

ports:

- "5432:5432"

volumes:

- ./db/postgres:/var/lib/postgresql/data webserver:

build: .

restart: always

entrypoint: ./scripts/webserver-entrypoint.sh

depends_on:

- postgres

env_file:

- .env

ports:

- "8080:8080"

volumes:

- ./dags:/usr/local/airflow/dags

- ./scripts:/usr/scheduler-entrypoint.sh

- ./logs:/usr/local/airflow/logs scheduler:

build: .

restart: always

entrypoint: ./scripts/scheduler-entrypoint.sh

depends_on:

- postgres

- webserver

env_file:

- .env

ports:

- "8793:8793"

volumes:

- ./dags:/usr/local/airflow/dags

- ./logs:/usr/local/airflow/logs

- ./scripts:/usr/local/airflow/scripts

- ./data:/usr/local/airflow/data

- /var/run/docker.sock:/var/run/docker.sock

A few things to point out. We will need a .env file to pass environment variables to our container, as well as entrypoint files to run the appropriate commands from our Airflow containers.

We also mount a couple of directories that need to be shared between the host and Docker. For instance, Airflow’s metadata needs to be persisted, and mounting the directory where Postgres keeps its data does just that.

Environment Variables

Creating a .env file will allow us to safely pass in sensitive information as environment variables into our containers. To be honest, the only real sensitive information in this project will be your AWS credentials, which will be stored in Airflow connections and not the .env file.

/my-airflow-project/.env

# Environment variables # Postgres

POSTGRES_USER=airflow

POSTGRES_PASSWORD=airflow

POSTGRES_DB=airflow

POSTGRES_PORT=5432 # Airflow

AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow

AIRFLOW__CORE__EXECUTOR=LocalExecutor

AIRFLOW__CORE__FERNET_KEY=<we will cover how to generate this>

AIRFLOW__CORE__LOAD_EXAMPLES=False # AWS

S3_BUCKET_NAME=<your-s3-bucket-name>

REDSHIFT_TABLE_NAME=<your-redshift-table-name>

REDSHIFT_SCHEMA=<your-redshift-schema_name>

REDSHIFT_DB=<your-redshift-db-name>

Entrypoints

Creating bash scripts for each Airflow container allows for commands to be executed inside the container during build time. Each file will live inside of a “scripts” folder in our project root.

/my-airflow-project/scripts/scheduler-entrypoint.sh

#! /usr/bin/env bash airflow initdb

airflow scheduler

/my-airflow-project/scripts/webserver-entrypoint.sh

#! /usr/bin/env bash airflow initdb

airflow webserver

Recall that these scripts are called inside our docker-compose.yaml for each container under the entrypoint option.

Volumes

The only directories we need to be persisted between our host machine and Docker containers are the dag folder from Airflow, the metadata from Postgres, the Unix socket the Docker daemon listens on, the scripts folder containing the Airflow entrypoints, the logs from both Airflow containers, and the data folder used to store the API responses.

Taking a look at our project directory tree before creating the containers ...

|-- my-airflow-project/

|-- dags/

|-- data/

|-- scripts/

| |-- scheduler-entrypoint.sh

| |-- webserver-entrypoint.sh

|-- .env

|-- .gitignore

|-- docker-compose.yaml

|-- Dockerfile

We build our containers with Docker compose in detached mode using the command line.

$ docker-compose up -d

This should take a few minutes. You should see a new /db/postgres/ and /logs/ directory added to the project (Remember, we mounted this directory in order to persist the metadatabase).

|-- my-airflow-project/

|-- dags/

|-- data/

|-- db/

|-- postgres/

|-- logs/

|-- scripts/

| |-- scheduler-entrypoint.sh

| |-- webserver-entrypoint.sh

|-- .env

|-- .gitignore

|-- docker-compose.yaml

|-- Dockerfile

Wrap Up

That’s it for this article. I strongly recommend creating a GitHub repository at this point. The project in its current state can be used as a starting point for any future Airflow work.

⚠️ If adding this project to a remote repository, remember to add the db/ and logs/ directory, and .env file to the .gitignore. These files contain sensitive information such as credentials, secrets, and keys.

Next, we will focus on creating Airflow tasks to pull the collisions data into our local data/ directory and subsequently loading it into our data lake and warehouse.

🚨 Next Article Update: Let’s Build An ELT Pipeline: Pt.3 Automating Data Downloads

Thanks for reading!