Airflow at Bloomberg: Leveraging dynamic DAGs for data ingestion
Channel: Apache Airflow
Date: 30/10/2023

Presented by Ivan Sayapin & Gabby Clavell at Airflow Summit 2023.

Bloomberg’s Data Platform Engineering team powers some of the most valuable business and financial data on which Bloomberg clients rely. We recently built a configuration-driven system that allows non-engineers to onboard alternative datasets into the company’s ecosystem. This system uses Apache Airflow to orchestrate the data flow across different applications and Bloomberg Terminal functions. We are unique in that we have over 1500 dynamic DAGs tailored for each dataset’s needs (which very few Airflow users have).