Best Practices for Airflow Developers | Towards Data Science

Apache Airflow is one of the most popular open-source data orchestration frameworks for building and scheduling batch-based pipelines. To master the art of ETL with Airflow, it is critical to learn how to efficiently develop data pipelines by properly utilizing built-in features, adopting DevOps strategies, and automating testing and monitoring. In this blog post, I will provide several tips and best practices for developing and monitoring data pipelines using Airflow. As always, I will explain the underlying mechanisms of Airflow to help you understand the “why” behind each tip.
(New to Airflow? Read the beginner’s guide to Airflow first.)
(Looking for more Airflow tips? Check out Apache Airflow Tips and Best Practices.)
Macros
Airflow has powerful built-in support for Jinja templating, which lets developers use many useful variables/macros, such as execution timestamp and task details, at runtime. An important use case of macros is to ensure your DAGs are idempotent, which I explain in detail in my previous blog post. Since macros allow users to retrieve runtime information at task run level, another great use case of macros is for job alerts, which I will demonstrate with examples in a later section.
However, not all operator parameters are templated, so you need to make sure Jinja templating is enabled for the operators that you plan to pass macros to. To check which parameters in an operator take macros as arguments, look for the template_fields attribute in the operator source code. For example, as of today, the most recent version of PythonOperator has three templated parameters: ‘templates_dict’, ‘op_args’, and ‘op_kwargs’:
template_fields = ('templates_dict', 'op_args', 'op_kwargs')
In order to enable templating for more parameters, simply overwrite thetemplate_fields attribute. Since this attribute is an immutable tuple, make sure to include the original list of templated parameters when you overwrite it.
DAG factory
Even building the simplest DAG in Airflow still requires writing~30 lines of Python code, in addition to knowledge of Airflow basics. If a large group of non-engineer members in the data organization build and deploy pipelines daily (most of which are very simple workflows with minimal dependency between tasks), then these employees need to invest a lot of time learning to write low-level scripts using the apache-airflow library. In this case, creating a high-level wrapper on top of Airflow’s native Python library (aka a DAG factory) will allow them to build simple data pipelines using fewer lines of code and without having in-depth knowledge of Airflow, which in turn saves time and resources.
As an example, below is a DAG factory class that returns a DAG that runs all SQL scripts in a folder at a given schedule:
If someone with zero knowledge on Airflow wants to schedule a couple of SQL queries to run daily, now they only need to put those SQL files in a folder and write the 2-line code below:
import DagFactorydag = DagFactory('jon_snow_sql_dag', 'Jon Snow', '2020-02-19', '@daily', __file__).sql_dag()
The DagFactory here is a very straightforward implementation, but you can enrich it based on your organization’s use cases; for example, adding a data lineage and qualify check, or allowing custom task dependency.
How does this work behind the scenes? Though the file that defines DagFactory is present in Airflow’s DAG folder, no actual DAG exists until a DagFactory instance is initialized with parameters in a DAG file. In the example above, only when dag is created in the global namespace, Airflow is able to pick it up, as it only recognizes DAG objects inglobals().
Automated Tests
After workflow files are uploaded to Airflow’s DAG folder, the Airflow scheduler will try to quickly compile all the files and validate all DAG definitions, e.g. checking whether there are any loops in task dependencies. Therefore, even if your IDE does not report any compiling errors, Airflow might still reject your DAGs at runtime. To catch Airflow exceptions ahead of time before deployment, you need a pytest function to ensure all DAG files are valid:
Option #1: Use importlib
# DAG_FOLDER_PATH: the relative path to the DAG folder# DAG_FOLDER_NAME: the name of the DAG folderdef validate_dags_1():  for dag_file in DAG_FOLDER_PATH.iterdir():    if dag_file.is_file() and dag_file.name.endswith('.py'):                   import_path = f'{DAG_FOLDER_NAME}.{dag_file.name[:-3]}'                importlib.import_module(import_path)
Option #2: Use DagBag
from airflow.models import DagBagdef validate_dags_2():  dagbag = DagBag()  assert len(dagbag.import_errors) == 0, f'DAG failures: {dagbag.import_errors}'
Continuous Deployment (CD)
An important but easy way to boost your team’s Airflow development efficiency is by adopting Continuous Deployment (CD) in your engineering workflow, which means whenever a set of changes is committed to the Airflow repository and passes automated tests (if any), these changes will automatically be deployed to Airflow’s DAG folder. There are a lot of DevOps automation tools that can help you achieve this easily:
Jenkins
Drone
GitHub Actions
Real-time failure alerts
If there are tons of Airflow DAGs running at high frequency on your Airflow cluster, users would love to get notified whenever a task fails rather than manually checking task status. Luckily, Airflow supports a handy parameter: on_failure_callback, which will trigger a user-provided callback function with a context dictionary full of task run information. For example, below is a callback function that sends a detailed Slack alert upon task failure:
To set up the failure alert for a DAG, set callback function in default_args:
default_args = {    'owner': 'Xinran Waibel',    'start_date': datetime.datetime(2020, 2, 20),    'on_failure_callback': slack_failure_msg}
(TL;DR) Here is a digest of key takeaways from this blog post:
Use macros to build idempotent DAGs and provide relevant error information in job alerts. Remember you can use thetemplate_fields attribute to enable templating for any operator parameters.
Create a DAG factory to allow users to create DAGs with efficiency.
Test DAG definitions using importlib or DagBag so that you can deploy workflows with confidence later.
Implement CD for Airflow repository using automation tools like Jenkins.
Make good use of on_failure_callback to send real-time failure alerts.
Happy learning and see you next time!
Want to learn more about Data Engineering? Check out my Data Engineering 101 column on Towards Data Science: