Christian Trebing - Get in control of your workflows with Airflow
Channel: EuroPython Conference
Date: 02/08/2016

Christian Trebing - Get in control of your workflows with Airflow
[EuroPython 2016]
[20 July 2016]
[Bilbao, Euskadi, Spain]
(https://ep2016.europython.eu//conference/talks/get-in-control-of-your-workflows-with-airflow)

Airflow (https://github.com/airbnb/airflow) is an open source Python
package from Airbnb to control your workflows.

This talk will explain the concepts behind Airflow, demonstrating how
to define your own workflows in Python code and how to extend the
functionality with new task operators and UI blueprints by developing
your own plugins. You'll also get to hear about our experiences at
Blue Yonder,  using this tool in real-world scenarios.

-----

Whenever you work with data, sooner or later you stumble across the
definition of your workflows. At what point should you process your
customer's data? What subsequent steps are necessary? And what went
wrong with your data processing last Saturday night?

At Blue Yonder we use Airflow (https://github.com/airbnb/airflow), an
open source Python package from Airbnb to solve these problems. It can
be extended with new functionality by developing plugins in Python,
without the need to fork the repo. With Airflow, we define workflows
as directed acyclic graphs and get a shiny UI for free. Airflow comes
with some task operators which can be used out of the box to complete
certain tasks. For more specific cases, tasks can be developed by the
end user. Best of all: even the configuration is done completely in
Python!

This talk will explain the concepts behind Airflow, demonstrating how
to define your own workflows in Python code and how to extend the
functionality with new task operators and UI blueprints. You'll also
get to hear about our experiences using this tool in real-world
scenarios.