Package: gcp-airflow-foundations
gcp-airflow-foundationsAirflow is an awesome open source orchestration framework that is the go-to for building data ingestion pipelines on GCP (using Composer - a hosted AIrflow service). However, most companies using it face the same set of problemsLearning curve: Airflow requires python knowledge and has some gotchas that take time to learn. Further, writing Python DAGs for every single table that needs to get ingested becomes cumbersome. Most companies end up building utilities for creating DAGs out of configuration files to simplify DAG creation and to allow non-developers to configure ingestionDatalake and data pipelines design best practices: Airflow only provides the building blocks, users are still required to understand and implement the nuances of building a proper ingestion pipelines for the data lake/data warehouse platform they are usingCore reusability and best practice enforcement across the enterprise: Usually each team maintains its own Airflow source code and deploymentWe have written an opinionated yet flexible ingestion framework for building an ingestion pipeline into data warehouse in BigQuery that supports the following features:Zero-code, config file based ingestion - anybody can start ingesting from the growing number of sources by just providing a simple configuration file. Zero python or Airflow knowledge is required.Modular and extendable- The core of the framework is a lightweight library. Ingestion sources are added as plugins. Adding a new source can be done by extending the provided base classes.Opinionated automatic creation of  ODS (Operational Data Store ) and HDS (Historical Data Store)in BigQuery while enforcing best practices such as schema migration, data quality validation, idempotency, partitioning, etc.Dataflow job supportfor ingesting large datasets from SQL sources and deploying jobs into a specific network or shared VPC.Support ofadvanced Airflow featuresfor job prioritization such as slots and priorities.Integration withGCP data servicessuch as DLP and Data Catalog [work in progress].Well tested- We maintain a rich suite of both unit and integration tests.Installing from PyPIpipinstall'gcp-airflow-foundations'Full DocumentationSee thegcp-airflow-foundations documentationfor more details.Running locallySample DAGsSample DAGs that ingest publicly available GCS files can be found in the dags folder, and are started as soon Airflow is ran locally. In order to have them successfully run please ensure the following:Enable: BigQuery, Cloud Storage, Cloud DLP, Data Catalog API'sCreate a BigQuery Dataset for the HDS and ODSCreate a DLP Inspect template in DLPCreate a policy tag in Data CatalogUpdate the gcp_project, location, dataset values, dlp config and policytag configs with your newly created valuesUsing Service AccountCreate a service account in GCP, and save it ashelpers/key/keys.json(don't worry, it is in .gitignore, and will not be push to the git repo)Run Airflow locally (Airflow UI will be accessible athttp://localhost:8080):docker-compose upDefault authentication values for the Airflow UI are provided in lines 96, 97 ofdocker-composer.yamlUsing user IAMuncomment line 11 indocker-composer.yamlsend env var PROJECT_ID to your test projectAuthorize gcloud to access the Cloud Platform with Google user credentials:helpers/scripts/gcp-auth.shRun Airflow locally (Airflow UI will be accessible athttp://localhost:8080):docker-compose upDefault authentication values for the Airflow UI are provided in lines 96, 97 ofdocker-composer.yamlRunning testsRun unit tests./tests/airflow  "pytest tests/unitRun unit tests with coverage report./tests/airflow  "pytest --cov=gcp_airflow_foundations tests/unitRun integration tests./tests/airflow  "pytest tests/integrationRebuild docker image if requirements changed:docker-compose buildContributingInstall pre-commit hookInstall pre-commit hooks for linting, format checking, etc.Install pre-commit python lib locallypip install pre-commitInstall the pre-commit hooks for the repopre-commit install