Love for writing deferrable operators Why and how to defer
Channel: Apache Airflow
Date: 2022-06-15

Session presented by Ankit Chaurasia at Airflow Summit 2022

Have you faced a scenario where 100 worker slots are available to run the Tasks, but you have 100 DAGs waiting on a Sensor that’s currently running but idle, waiting for something to happen? Ultimately, you got frustrated as you could not run anything else - even though your entire Airflow cluster was essentially idle. This is exactly where the concept of Deferrable Operators is very useful.

This talk aims to give a brief introduction to solving this problem using Deferrable or Async Operators and how to implement it for your use case.