Federico Marani - Feeding data to AWS Redshift with Airflow
Channel: EuroPython Conference
Date: 21/10/2017

"Feeding data to AWS Redshift with Airflow
[EuroPython 2017 - Talk - 2017-07-13 - Anfiteatro 1]
[Rimini, Italy]

Airflow is a powerful system to schedule workflows and define them as a collection of interdependent scripts. It is the perfect companion to do extract/transform/load pipelines into data warehouses, such as Redshift.

This talk will introduce some of the basis of Airflow and some of the concepts that are data pipeline specific, like backfills, retries, etc. Then there will be some examples on how to integrate this, along with some lessons learned there. 

At the end, there will be a part dedicated to Redshift, how to structure data there, how to do some basic transformation pre-loading, how to manage the schema using SQLAlchemy and Alembic.

License: This video is licensed under the CC BY-NC-SA 3.0 license: https://creativecommons.org/licenses/by-nc-sa/3.0/
Please see our speaker release agreement for details: https://ep2017.europython.eu/en/speaker-release-agreement/