Is Apache Airflow 2.0 good enough for current data engineering needs? | by Anna Geller | Towards Data Science

Several months ago I wrote an article discussing the pros and cons of Apache Airflow as a workflow management platform for ETL and data science. Due to the recent major upgrade, I want to give an update on what has changed since then in Airflow 2.0. To get a full picture, you may want to have a look at the previous article first:
towardsdatascience.com
Table of contents
· The strengths of Airflow 2.0 as opposed to the previous versions ∘ The new UI looks fresh and modern ∘ The scheduler is no longer a bottleneck ∘ Airflow finally has a full-fledged REST API ∘ Smart sensors ∘ The entire project has been restructured · New features which didn’t convince me ∘ TaskGroup feature ∘ Data sharing between tasks via TaskFlow abstraction· The weaknesses of Airflow that has not been yet addressed in the new release ∘ The confusing scheduling logic ∘ Still no versioning of your data pipelines ∘ Configuration overload ∘ Local development· Conclusion
After installing Airflow 2.0, you will be able to access UI that looks somewhat better as compared to the previous versions. The main benefit of a new UI is the Auto-refresh feature — you no longer have to constantly refresh your browser to get the updated status of your workflow’s progress. Instead, the UI is refreshing automatically and you can deactivate it with a single click, as demonstrated in the image below.
You no longer have to take a coffee break after triggering a DAG run to see the progress in your workflow. The scheduler appears to be light-years faster as compared to any previous Airflow version. According to a performance benchmark performed by Astronomer [6], the scheduler is now (depending on the use case) up to 17 times faster than before.
The biggest (positive) surprise was when the scheduler restarted itself after not receiving a heartbeat for over 100 seconds.
Up until now, Airflow only had an experimental API with limited functionality. With the 2.0 release, you can now leverage a full REST API to create, read, update, or delete DagRuns, Variables, Connections, or Pools.
If you have Airflow running locally, you can access the Swagger UI by following the URL: http://localhost:8080/api/v1/ui/. It provides you with thorough documentation so that you can start building additional functionality on top of Airflow, even with programming languages other than Python.
Sensors are tricky, as they are constantly poking for status and may consume a lot of resources. With the new release, Airflow introduced changes to the sensors' logic to make it more resource-efficient and “smarter”. This feature may be helpful to those who want to keep all their workflows in a single place rather than relying on FaaS (such as AWS Lambda) for event-driven use cases.
Airflow has a welcoming community, which until now was allowing contributions to the library of operators, sensors, or hooks without thinking too much about the structure of those contributed modules. Over time, it led to a situation where airflow.contrib got so large that dependency management, as well as planning and testing of the next releases, have become challenging. As Jarek Potiuk (Airflow PMC member) nicely put it [3]:
Everything but a kitchen sink was thrown into a single “apache-airflow” package, no matter if you want to use it or not.
With Airflow 2.0, the modules have been restructured according to external systems that can be used with Airflow. This means that if you want to use AWS-related operators, but not the ones related to GCP and Kubernetes, you can install Airflow only with the Amazon provider subpackage:
pip install apache-airflow[amazon]
This change is significant, as it allows for a separation of concerns, faster release cycles for specific components, and a much cleaner organizational structure of where you can find code related to the specific external system. When migrating workflows from previous versions, make sure to use the correct imports. For instance, instead of:
from airflow.contrib.operators.aws_athena_operator import AWSAthenaOperator
you should use [4]:
from airflow.providers.amazon.aws.operators.athena import AWSAthenaOperator
There are some new abstractions that seem great on the surface, but which need to be treated with caution.
This new abstraction allows treating a group of tasks as a single task. It was motivated by the inefficiencies and bugs that many data engineers experienced when using SubDags. Even though this task grouping seems to be helpful in the UI, from the design perspective, it doesn’t address the actual problem of decoupling and separation of concerns that many data engineers were trying to solve using SubDags.
This is a feature many people were curious about, as it promises data sharing functionality, and should allow writing simple Python functions rather than Airflow operators. After reviewing this feature, it seemed to be rather a weakness of this release since the TaskFlow abstraction leaves more questions than it answers.
Here is a detailed overview of those features (TaskFlow and TaskGroup):
towardsdatascience.com
Nothing has changed in the scheduling logic. Airflow still starts scheduling the workflows at the end of the scheduling interval, which means that your DAG will not start immediately at the time of the scheduled period, but only when the execution_date reaches start_date + schedule_interval.
What does it mean for you as an end-user? It means that when you see the Last Run “2020–12–28” at exactly 1 AM for a scheduled job, as shown in the image below, it doesn’t necessarily mean that the workflow actually ran at exactly 1 AM on that day — it’s the jobs’ execution date which is the start timestamp of the workflow’s scheduling period. The job’s execution is triggered at the end of this period. In this example, the job was actually triggered one day later. Only for manually triggered jobs, the Last Run field will indicate the real timestamp of a job run.
If you want to learn more about the entire scheduling logic, this article provides an excellent explanation:
towardsdatascience.com
Storing metadata changes about workflows helps analyze what has changed over time. With Airflow 2.0, you still can’t track the version history of your workflows. If you are interested, here is a proposal of how it was planned to be implemented.
Airflow was built to satisfy the needs of many different groups of people. It can be leveraged as a tool to orchestrate your containerized workloads on Kubernetes, to use several different technologies as a message broker of your choice and it’s trying to do it all in a semi-flexible way in a single product. The downside of this approach is added complexity and the number of things you need to configure to customize the tool to your needs. As a new user, you need to learn many components and a wide range of Airflow’s vocabulary to leverage the tool in the right way.
For instance, you need to know all the nuances related to the previously mentioned scheduling logic (incl. potential traps with catchup), the syntax of how to write the Airflow code, the logic of executors, message brokers and Celery queues, operators, hooks, XComs, metadata database, and how to put all those different components together and maintain them.
You could say that this is true for any tool. But Airflow simply has so much of a specific (not intuitive) vocabulary and components that it may result in a significant barrier of entry. You can’t be an effective user unless you learn all the components and vocabulary of the tool. Doing that can require a considerable amount of training which is expensive and could be avoided when using a platform with more intuitive abstractions. In fact, the headache of managing Airflow is likely the reason why the workflow orchestration space is currently exploding with new more user-friendly solutions.
How Airflow could fix that? It could be beneficial to abstract away some internal components that are not relevant to the end-users these days. To give an example: why do you, as an end-user, have to care about the start and execution date? Isn’t it enough to define your schedule and deploy your DAG, and let the scheduling platform handle the rest?
Maintaining Airflow and all its components requires full-time resources only to manage the platform and underlying infrastructure. There are several vendors providing commercial Airflow offerings. Companies involved financially in this open-source project are providing either fully managed service for Airflow, consulting, training services, or a combination of all of them.
Astronomer — offers a managed service with some extra functionality built on top of the open-source version, enterprise support, training, and consulting, and has several Airflow PMC members as employees to contribute back to the open-source project,
Polidea — similarly to Astronomer, it has several PMC members as employees, but they seem to be mainly focused on maintaining the codebase and offering consulting services,
Google — offers a fully-managed service: Cloud Composer service on GCP
Amazon — offers a fully managed service with the extra benefit of elastic (i.e. automatically scaling) worker node capacity: Managed Workflows for Apache Airflow. More about it in this article:
towardsdatascience.com
Local development with the open-source version of Airflow is difficult because very often your local environment is completely different from what is used in production. When using a default local installation, you will likely experience many weird errors. To make this process slightly less painful, the astro CLI from Astronomer helps with the local development [1]. Their CLI allows you to initialize a local Airflow environment that spins up Docker containers including a Postgres database. If you use their commercial offering, you can deploy your DAGs to production by leveraging another CLI command. With a dockerized environment, you can ensure dev and prod environment parity that helps avoid any surprises when moving your DAGs from development to production.
Overall, with Airflow 2.0 the project has reached a maturity level that allows it to be used for production workloads. The scheduler is orders of magnitude faster and more reliable as compared to the previous versions. Thanks to the separation of code for different providers into subpackages, you can also expect faster release cycles.
Thank you for reading!
References & additional resources
[1] Start guide with astro CLI— Astronomer.io
[2] Airflow 2.0: DAG Authoring Redesigned — Polidea
[3] Airflow 2.0 Providers — Polidea
[4] Example of a new Provider package — Airflow on Github
[5] Apache Airflow 2.0 is here — Documentation
[6] Airflow’s scheduler benchmark — Astronomer.io
[7] Release notes — showing a full list of features added in the Airflow 2.0 release.