GitHub repository: GabrielAmazonas/airflow-pyspark-emr
Primary language: Python
Created: 2020-04-24T17:44:32Z
Description: This project demonstrate how to process data stored in a data lake fashion, transforming it into an OLAP optimized structure by using PySpark.  The PySpark Job runs on AWS EMR, and the Data Pipeline is orchestrated by Apache Airflow, including the infrastructure creation and the EMR cluster termination.