A Gentle Introduction To Understand Airflow Executor | by Chengzhi Zhao | Towards Data Science

Apache Airflow is a prominent open-source python framework for scheduling tasks. There are many new concepts in the Airflow ecosystem; one of those concepts you cannot skip is Airflow Executor, which are the “working stations” for all the scheduled tasks. Airflow is generally user-friendly to the end-users, and getting a good understanding of the Airflow Executor is critical to personal use, as well as the production Airflow environment. In this article, we are going to discuss details about what’s Airflow executor, compare different types of executors to help you make a decision.
There are many Airflow executors to choose, let’s put those choices aside, and first focus on what do the Airflow executors do in Airflow ecosystem. The discipline “Executor,” fittingly, is a mechanism that gets tasks executed. The worker is the node or processor which runs the actual tasks. The Airflow scheduler won’t run any tasks but handle tasks over to the Executor. The Executor acts as a middle man to handle resource utilization and how to distribute work best.
Although an Airflow job is organized at the DAG level, the execution phase of a job is more granular, and the Executor runs at the task level. As the images are shown below, if a DAG ends up with six tasks, the Airflow scheduler will assign each task separately to the Executor. Whether those tasks finished in parallel or sequentially, it is determined by the executor type.
As for now, Airflow (version 1.10.10) is a single scheduler system. In this case, all of the tasks are scheduled by only one scheduler. The scheduler controls when tasks will be sent to the Executor and monitor the state of tasks from executors.
The life cycle of a task from the scheduler to Executor includes the following steps:
Before the scheduler sends the command on which task the Executor is going to run, depending on the types of executors, the resource of executors themself keeps idle or unavailable.Once the scheduled time arrives, the Airflow scheduler sends the command to the Executor.After received signals from the scheduler, the Executor starts allocating resources and put tasks into the queue. When work is available, it will pick up the tasks from the queue to execute it. Meanwhile, the scheduler is poking the tasks every once a while (called heartbeat) to get the current state of the task, then update its state in backend DB.Once the tasks finished, and the scheduler received the Completed state from the Executor, the resource allocated for running the task got clean up.
Before the scheduler sends the command on which task the Executor is going to run, depending on the types of executors, the resource of executors themself keeps idle or unavailable.
Once the scheduled time arrives, the Airflow scheduler sends the command to the Executor.
After received signals from the scheduler, the Executor starts allocating resources and put tasks into the queue. When work is available, it will pick up the tasks from the queue to execute it. Meanwhile, the scheduler is poking the tasks every once a while (called heartbeat) to get the current state of the task, then update its state in backend DB.
Once the tasks finished, and the scheduler received the Completed state from the Executor, the resource allocated for running the task got clean up.
Majority types of the Airflow executor instrument tasks to run in a distributed manner, those tasks are running on either multi-processing or multiple worker nodes. An Airflow production environment usually has hundreds of DAGs, which include thousands of tasks to run. With the capability to run various tasks in parallel at such a large scale, the Airflow executor shines on the intensive workload.
One of the reasons you have various types of executors is you have the option to choose based on your requirement and infrastructure. Airflow provides executors “toolkit,” different kinds of executors give the flexibility to have Airflow integrate smoothly with your environment.
Airflow is extensible, which supports various executors. Initially, Airflow only has SequentialExecutor, LocalExecutor, CeleryExecutor, and MesosExecutor available. In the last two years, since Airflow 1.9.0, Airflow gets more attention, and more executors have been contributed to the community, those executors include DaskExecutor, KubernetesExecutor, DebugExecutor.
The difference between the Executors is the mechanism on where the executors are running the tasks. In my opinion, not all executors are equally to be considered, some you may even want to skip unless you have specific reasons you have to use them. Below is the description of each Executor and what should be the consideration when you tend to choose one of them.
SequentialExecutor is the default executor out of the box. As the name suggested, tasks will get executed sequentially. Even you have a branch operator, and the tasks are still going to be executed one by one in order of ’branch_a’, ‘branch_b’, ‘branch_c’, ‘branch_d’
LocalExecutor is excellent for testing Airflow multiple jobs in parallel, performing tasks for a small scale production environment. LocalExecutor runs the task on the same node as the Airflow scheduler but different processors. There are other executors which use this type while distributing the actual work; for example, KubernetesExecutor would use LocalExecutor within each pod to run the task.
CeleryExecutor is the most mature option for Airflow as most of the early Airflow adoption is using CeleryExecutor. It requires infrastructure support — Celery and Celery’s backend (Redis or RabbitMQ) additionally. However, you’d have much better help with this option from the Airflow community since a lot of companies are running under this option.
MesosExecutor is one of the early community contributions. However, Since Kubernetes is adopted widely than Mesos, the Airflow Community is also discussing retiring MesosExecutor. Unless your company is running Mesos, and you don’t see migration to Kubernetes in the next few years, and you want to use Mesos to manage your Airflow executor resource, you might want to choose this option. Otherwise, you might want to avoid choosing MesosExecutor.
Dask.org inspires DaskExecutor. There is some discussion about removing DaskExecutor as well due to lack of usage, which DaskExecutor failed on Airflow master for months, but no one noticed. I personally like Dask; it’s unfortunate for less usage with Airflow and Dask. Due to less usage and support, you might want to avoid choosing DaskExecutor, unfortunately.
KubernetesExecutor was introduced in version 1.10.0 and contributed by Bloomberg. This contribution sets a milestone for Airflow to integrate with the Kubernetes ecosystem. Although the first few minor version is buggy, the recent ones are more stable. If your company has Kubernetes widely adopted, KubernetesExecutor might be the best choice for the Airflow executor. Another beautiful thing about KubernetesExecutor is you can prepare different docker images for your tasks, and it gives you more flexibility here.
DebugExecutor was introduced in 1.10.8. It might not be classified as an executor; the purpose of this DebugExecutor is to run with IDE. It is similar to SequentialExecutor to run one task at a time, and it supports to work with sensors.
To summary, CeleryExecutor and KubernetesExecutor would be an excellent selection for your production environment. LocalExecutor is also a preference to be considered for the production environment. If you have a light workload or majority of the tasks are running in cloud services like AWS or Azure service, Airflow Executor acts as the middle man to talk to different services without actually running them, LocalExecutor is also viable choice to consider. SequentialExecutor and DebugExecutor are for local testing purposes. You probably would skip them on production. MesosExecutor and DaskExecutor you might want to avoid them due to ambivalence about their future road map in the Airflow ecosystem.
Most of the configurations on the Airflow Executor are controlled by airflow.cfg file. Different functional sections organize the file in brackets. For executor selection, you’d see under the core section, SequentialExecutor is chosen as the default. It allows you to run Airflow without setting up too many dependencies. SequentialExecutor can work directly with SQLite, which should be installed with the Python installation. As we discussed above, you can choose different types of executors, but each requires additional setup in the AirflowAirflow.cfg file.
[core]executor = SequentialExecutor
The LocalExecutor is also easy to set up, and it does require the metadata database to be MySQL or PostgreSQL instead of SQLite. Once the LocalExecutor is set up, 90% of the functionality of the Airflow executor is unveiled. The other 10% of functionality the Executor is to run Airflow in a distributed manner.
CeleryExecutor has its configuration section — [celery] . There are two main components: Celery and Celery backend. Celery is an asynchronous task queue. With Celery, Airflow can scale its tasks to multiple workers to finish the jobs faster. More setup can be found at Airflow Celery Page
KubernetesExecutor is the beloved child in Airflow due to the popularity of Kubernetes. If you have Kubernetes in your environment, setting up KubernetesExecutor in Airflow won’t be too much work, I have covered the basic set up in my previous article: Explore Airflow KubernetesExecutor on AWS and kops
It is a critical step for Airflow infrastructure to set up proper Executor. If you want Airflow to handle not only the scheduling part but also run tasks on your worker node, Airflow executor provides more extra potential here with its distributed capability. I hope this article can give you some basic ideas on Airflow Executor. Cheers!