Data Engineering — How to Set Dependencies Between Data Pipelines in Apache Airflow | by Nicholas Leong | Towards Data Science

Greetings my fellow readers, it’s me again.That guy who writes about his life experiences and a little tad bit about data.Just a little bit.Article after article, I always start with how important is data in a strong organisation. How large companies are using data to impact their business, impact our society and in turn, making them profits. Data can be used to save lives, just take a look at this article on how data is used to predict cancer cells. So please, let’s skip the small talk about how important is data and how it’s always right, just like your wife.
In God we trust. All others must bring data. — W.Edwards Demming
Before the world began, there was only darkness. Then somehow, someone said let there be light and suddenly all life came to life. Similarly, before there were any data, there was only darkness. Then somehow, data engineers built amazing pipelines to feed data into data warehouses and suddenly, all the data scientists be coming up with their models. The foundation of the data available heavily depends on the structure of the pipelines written by the engineers. Hence, it is important that we use the right tools to optimise our pipelines.
In my previous article, I talk about how CRONJOBS are no longer viable in scheduling pipelines for a proper data warehouse. Instead, engineers should look into well documented workflow management systems that provide huge benefits like monitoring, setting dependencies and not to mention a nice UI to work with. The WMS that I have chosen is Apache Airflow simply because after researching between all the WMS available, my company thinks that Apache Airflow is the best fit currently for our warehouse. Did I also mention that twitter is using Apache Airflow for their data warehousing as well ? I also guided readers into setting up their first pipeline, talking about the Basics of Apache Airflow and how it works. This article is an extension to that because I will be talking about setting dependencies between your pipelines and why is it so important for your data warehouse.
Every data warehouse differs from one another. As engineers, the best we can do is to build efficient pipelines to fit the PROs and CONs of the data warehouse itself. Every data warehouse gets data from a data source, and that might potentially be your master database or a clone of that. Depending on your company, you might run ETL pipelines to ingest data into your data warehouse. However, having Google Bigquery as my company’s data warehouse, we think that it is more speed and cost efficient to run ELT pipelines since Google BQ handles transformations very well. It also takes the load off of our slave of the master db. I believe this is not unique to our company, its a pretty common practice in the data space right now. The usual agenda is pure data extractions of raw tables starting midnight for X hours, leading to transformations of those tables for another X hours to complete the full pipeline. Here’s a visualisation I made to represent that.
The reason that the pipeline is designed like this is because the first level of transformed tables originate and depends on the completion of the extraction of the raw tables. The second level of transformed tables then depend on these first level transformed tables. Hence, it is important that we set dependencies between these tasks. We do not want to execute a transformation of a table if one of it’s origin tables has not been extracted/transformed yet. This would result in incorrect data, which is really what data engineers are blamed for.
As mentioned, we will be working with Apache Airflow as our main WMS. By doing so, we can monitor our workflow efficiently, being able to track the tasks that failed, if there is any. If you have no idea on how to operate airflow then the following will look like puzzles to you, please read the basics of Apache Airflow first.
Sensors are pre-built in airflow. As the title suggests, they sense for the completion of a state of any task in airflow, simple as that. We will be using sensors to set dependencies between our DAGS/Pipelines, so that one does not run until the dependency had finished. There is no need to write any custom operator for this. Here’s the default format of sensors:
check_dependency =ExternalSensor.ExternalTaskSensor(task_id='name of the sensor',external_dag_id='name of dag to be checked',external_task_id= 'name of task to be checked',execution_delta = time difference between execution times,timeout = maximum running time of sensors)
We write this block of code as one of our tasks in our DAG. 2 things to note:
execution delta can be tricky, it depends on the execution time of the task, not the run time. Here’s an example of that.
Here, the execution time was 9pm, 2019–08–30 (not to be mistaken with the run time which is also named ‘Started’ here : 9pm, 2019–08–31). If we had set the execution time (setting of ‘start_date’ in the sensor’s DAG) of the sensor to be on 10pm, 2019–08–30, the execution delta, in this case, would be timedelta(hours=1).
The timeout parameter is necessary. When your pipelines scale, there will be many sensors running to check for completions. If no timeout is set and some of our dependencies fail, the sensors will run indefinitely and cause your airflow to hang. This is because airflow only allows a certain maximum number of tasks to be run on an instance and sensors are considered as tasks. If you somehow hit that number, airflow will not process further tasks. Hence, we need to set the timeout parameter for the sensors so if our dependencies fail, our sensors do not run forever.
In this example, say Table 1 is a result of a few joins from raw data sources. We will need to create 4 sensors to sense for the completion of the extraction dags before running the transformation of Table 1.
To clear all doubts, let’s assume that all of the extraction dags :
are run daily
run on 9pm
We will write the following code in the transformation DAG for Table 1:
check_data_source_1 =ExternalSensor.ExternalTaskSensor(task_id='check_data_source_1',external_dag_id='dag of data source 1',external_task_id= 'last task of the dag',execution_delta = timedelta(hours=1),timeout = 300)check_data_source_2 =ExternalSensor.ExternalTaskSensor(task_id='check_data_source_2',external_dag_id='dag of data source 2',external_task_id= 'last task of the dag',execution_delta = timedelta(hours=1),timeout = 300)check_external_data_source_1 =ExternalSensor.ExternalTaskSensor(task_id='check_external_data_source_1',external_dag_id='dag of external data source 1',external_task_id= 'last task of the dag',execution_delta = timedelta(hours=1),timeout = 300)check_external_data_source_2 =ExternalSensor.ExternalTaskSensor(task_id='check_external_data_source_2',external_dag_id='dag of external data source 2',external_task_id= 'last task of the dag',execution_delta = timedelta(hours=1),timeout = 300)transform_table_1 = /* code for transformation of table 1 */check_data_source_1check_data_source_2check_external_data_source_1check_external_data_source_2transform_table_1.set_upstream([check_data_source_1,check_data_source_2,check_external_data_source_1,check_external_data_source_2])
With the above code written, the task ‘transform_table_1’ shall not proceed before the completion of the four dependencies we set. Once the sensors start, they will sense for the completion of the dependencies for 5 minutes. If the mentioned dependencies fail to complete, the sensors then appear as failure and up for retry.
Here’s the complete code for the DAG.
You are one more step ahead in achieving clean data. Even if you are not a fellow data engineer, knowledge like these are beneficial to all data practitioners. Furthermore, sensors can be used for all airflow tasks.
Wanna send an email after a query is ran? Use sensors
Wanna scan through a certain shared google drive after a new file is inserted? Use sensors
The possibilities are endless.
Ending note: Pardon my absence recently, I am working on Apache Beam to implement streaming pipelines. Interesting Stuff!Once its up I’ll write about it !
Thanks for reading! If you want to get in touch with me, feel free to reach me on nickmydata@gmail.com or my LinkedIn Profile. You can also view the code in my Github.