Let’s Build An ELT Pipeline Pt. 3: Automating Data Downloads
By: Jonathan Duran
Tags: Data Engineering, Airflow, Python, Programming, API

If you haven’t already, please read my previous article showing how to set up our project for this section.

Tasks

Let’s break down our pipeline.

We start by downloading our data using the NYC collisions data API and schedule this process to run daily. The next step is loading the data into an S3 bucket, which we use as a data lake. Once our data lake is updated, the next destination for the raw collisions data is the data warehouse. I chose Redshift for this project, but you are welcome to explore alternatives such as Snowflake, BigQuery, etc. DBT is up next and used to create models from the raw data and fed back into the data warehouse as views. Finally, data analysts and other data users can query the transformed collisions data to build analysis, dashboards, and reports.

We consider every step in the pipeline a task that we must put in the proper execution order. This collection of dependent tasks are what we refer to as DAGs, and we must ensure that no downstream components depend on any upstream ones to avoid circular dependencies.

This article will only focus on the API-related, “check API” and “download data” tasks.

Defining Our DAG

Before coding our tasks, we must instantiate our DAG object globally. This object can take in a slew of optional keyword arguments, and readers should familiarize themselves with the documentation to determine which are needed.

For this project, we will stick with the mandatory dag_id and the optional start_date and schedule_interval parameters.

Let’s start by creating the nyc_collisions_pipeline.py file in the dags/ directory.

$ cd dags

$ touch nyc_collisions_pipeline.py

/my-airflow-project/dags/nyc_collisions_pipeline.py

# Import Airflow modules

from airflow import DAG # Import Python libraries

from datetime import datetime # Initiate our dag definition

dag = DAG(

dag_id="nyc_collisions_pipeline",

start_date=datetime(2020, 12, 19),

schedule_interval="@daily"

)

We use the CRON preset “@daily” to tell our scheduler that our tasks should run once a day at midnight. The scheduler_interval parameter is quite flexible and can take in any CRON expression right for the job.

One thing to keep in mind about the start_date is Airflow waits until the end of the first period to trigger a task. For our project, the Airflow scheduler triggers our first DAG run on December 20, 2012 (start_date + scheduler_interval). The scheduler triggers the DAG every day at midnight after the initial run.

Check API

The data for this project comes from NYC Open Data, and, as with all third-party data sources, things can go unexpected. The data delivered by the API may change, be delayed, or not exist every time Airflow triggers our DAG.

For this reason, we will run Airflow’s HTTP sensor to check for the existence of data. The sensor polls an HTTP endpoint until a condition is met. In our case, we check if the API response contains data in the crash_date field for a given date. The response check is accomplished by using a lambda function that returns a boolean if the crash_date field is found inside the response object content.

# Import HTTP Sensor object

from airflow.sensor.http_sensor import HttpSensor # Check if the collisions api has data for a given date

is_collisions_api_available= HttpSensor(

task_id="is_collisions_api_available",

method="GET",

http_conn_id="nyc_collisions_api",

endpoint="resource/h9gi-nx95.json?crash_date={{ ds }}",

response_check=lambda response: "crash_date" in response.text,

poke_intervals=5,

timeout=20,

dag=dag

)

We poll the API for the condition defined in the response_check every five seconds and timeout after 20 seconds in the event that the condition isn’t met.

Airflow uses Jinja Templating, providing us a way to access the collision data API across different days dynamically. We tie the date of the data from the API to our execution date by passing the “{{ ds }}” template variable into our crash_date query parameter. The ds template variable renders into the current day’s date stamp.

For the HTTP sensor to work, we have to create the Airflow connection to the API in the Airflow UI. To do this, we must first run our Docker containers to access the UI running on our localhost.

Make sure you are in the project root /my-airflow-project/ before running docker-compose.

$ cd /my-airflow-project

$ docker-compose up -d

Once everything is up and running, navigate to http://localhost:8080 in your web browser.

In the navigation bar, click on the Admin dropdown menu to access Connections.

There will be a handful of out-of-the-box connections already created but ignore those for now and click on Create.

Keep in mind, the Conn Id field must match the http_conn_id parameter in our HttpSensor object. Click save and congratulations! the code for our first task is done!

Download Data

We will show off our Linux skills for the second task and use Airflow’s BashOperator to download the collision’s data. More specifically, we can use the curl command to pull in data from the NYC Open Data API into a file in the data directory created in the previous article.

# Import Airflow's BashOperator object

from airflow.operators.bash_operator import BashOperator

fetch_collisions_data = BashOperator(

task_id="fetch_collisions_data",

bash_command="curl -o /usr/local/airflow/data/{{ ds }}.json \

--request GET \

--url

dag=dag

) # Download collisions data for a given datefetch_collisions_data = BashOperator(task_id="fetch_collisions_data",bash_command="curl -o /usr/local/airflow/data/{{ ds }}.json \--request GET \--url https://data.cityofnewyork.us/resource/h9gi-nx95.json?crash_date={{ ds }}",dag=dag

Notice the BashOperator allows us to use template variables to dynamically access data from different dates.

Dependencies

Great! We have our two tasks. However, they do not form a DAG just yet. The final step is to establish a relationship between them. We set the is_collisions_api_available as the upstream task and fetch_collisions_data as the downstream task.

# Define the dependencies

is_collisions_api_available >> fetch_collisions_data

Go ahead and navigate back to the Airflow webserver UI, and switch on the nyc_collisions_pipeline DAG. Click on the DAG and make sure you are in the Tree View while the DAG is running to view your triggered tasks (You will have to keep refreshing your browser). The collisions data will appear inside your /data folder as long as your tasks are successful.

In my case, the nyc_collisions_pipeline DAG ran from December 19th to the 25th and was successful from the 19th to the 21st.

What about the remaining days? Taking a closer look at the Tree View shows us that the is_collisions_api_available task failed to meet the response check condition, and so the fetch_collisions_data task never ran (the power of dependency!). In fact, for some reason data was never updated for the days where our DAG failed at the time of this writing (we can, at any point in the future, re-run any failed tasks).

This level of oversight over our pipelines is why Airflow is one of my favorite tools!

Wrap Up

This was fun! Even if we stopped the project now, you would still leave with an invaluable skill. I recommend reading Airflow documentation and playing around with the different types of parameters available for the DAG object.

In the next article, the focus will be on loading the collisions data into a data lake and warehouse and providing the AWS resources needed for the two.

GitHub

The full code for this article is available here.

🚨 Please make sure you are on the airflow-http-data branch in the repository.