Airflow: Save Tons of Money by Using Deferrable Operators - Kaxil Naik, Astronomer
Channel: The Linux Foundation
Date: 2022-10-08

Airflow: Save Tons of Money by Using Deferrable Operators - Kaxil Naik, Astronomer

Apache Airflow 2.2 introduced the concept of Deferrable Tasks that uses Python's async feature. All the Airflow sensors and poll-based operators can be hugely optimized to save tons of money by freeing up worker slots when polling. This session will cover the following topics: - Introduction to the concept of deferrable operator - When to use them & Why? - Writing Custom deferrable operators & Sensors - Advantages over Smart Sensors and reschedule mode for sensor