Traps and Misconceptions of Running Reliable Workloads in Apache Airflow
Channel: Apache Airflow
Date: 02/10/2023

Presented by Bartosz Jankiewicz at Airflow Summit 2023.

Reliability is a complex and important topic. I will focus on both reliability definition and best practices. I will begin by reviewing the Apache Airflow components that impact reliability. I will subsequently examine those aspects, showing the single points of failure, mitigations, and tradeoffs. The journey starts with the scheduling process. I will focus on the aspects of Scheduler infrastructure and configuration that address reliability improvements. It doesn’t run in a vacuum therefore I’ll share my observations on the reliability aspect of Scheduler infrastructure.