Package: airflow-dbt-cta
airflow-dbtNOTE: this repository was forked fromhttps://github.com/gocardless/airflow-dbtin order to release an updated version to PyPi.This is a collection ofAirflowoperators to provide easy integration withdbt.fromairflowimportDAGfromairflow_dbt_cta.operators.dbt_operatorimport(DbtSeedOperator,DbtSnapshotOperator,DbtRunOperator,DbtTestOperator,DbtCleanOperator,DbtBuildOperator,)fromairflow.utils.datesimportdays_agodefault_args={'dir':'/srv/app/dbt','start_date':days_ago(0)}withDAG(dag_id='dbt',default_args=default_args,schedule_interval='@daily')asdag:dbt_seed=DbtSeedOperator(task_id='dbt_seed',)dbt_snapshot=DbtSnapshotOperator(task_id='dbt_snapshot',)dbt_run=DbtRunOperator(task_id='dbt_run',)dbt_build=DbtBuildOperator(task_id='dbt_build',)dbt_test=DbtTestOperator(task_id='dbt_test',retries=0,# Failing tests would fail the task, and we don't want Airflow to try again)dbt_clean=DbtCleanOperator(task_id='dbt_clean',)dbt_seed>>dbt_snapshot>>dbt_run>>dbt_build>>dbt_test>>dbt_cleanInstallationInstall from PyPI:pipinstallairflow-dbtIt will also need access to thedbtCLI, which should either be on yourPATHor can be set with thedbt_binargument in each operator.UsageThere are six operators currently implemented:DbtDocsGenerateOperatorCallsdbt docs generateDbtDepsOperatorCallsdbt depsDbtSeedOperatorCallsdbt seedDbtSnapshotOperatorCallsdbt snapshotDbtRunOperatorCallsdbt runDbtTestOperatorCallsdbt testDbtCleanOperatorCallsdbt cleanDbtBuildOperatorCallsdbt buildEach of the above operators accept the following arguments:envIf set as a kwarg dict, passed the given environment variables as the arguments to the dbt taskprofiles_dirIf set, passed as the--profiles-dirargument to thedbtcommandtargetIf set, passed as the--targetargument to thedbtcommanddirThe directory to run thedbtcommand infull_refreshIf set toTrue, passes--full-refreshvarsIf set, passed as the--varsargument to thedbtcommand. Should be set as a Python dictionary, as will be passed to thedbtcommand as YAMLmodelsIf set, passed as the--modelsargument to thedbtcommandexcludeIf set, passed as the--excludeargument to thedbtcommandselectIf set, passed as the--selectargument to thedbtcommandselectorIf set, passed as the--selectorargument to thedbtcommanddbt_binThedbtCLI. Defaults todbt, so assumes it's on yourPATHverboseThe operator will log verbosely to the Airflow logswarn_errorIf set toTrue, passes--warn-errorargument todbtcommand and will treat warnings as errorsTypically you will want to use theDbtRunOperator, followed by theDbtTestOperator, as shown earlier.You can also use the hook directly. Typically this can be used for when you need to combine thedbtcommand with another task in the same operators, for example runningdbt docsand uploading the docs to somewhere they can be served from.Building LocallyTo install from the repository:
First it's recommended to create a virtual environment:python3-mvenv.venvsource.venv/bin/activateInstall usingpip:pipinstall.TestingTo run tests locally, first create a virtual environment (seeBuilding Locallysection)Install dependencies:pipinstall.pytestRun the tests:pytesttests/Code styleThis project usesflake8.To check your code, first create a virtual environment (seeBuilding Locallysection):pipinstallflake8
flake8airflow_dbt/tests/setup.pyPackage managementIf you use dbt's package manager you should include all dependencies before deploying your dbt project.For Docker users, packages specified inpackages.ymlshould be included as part your docker image by callingdbt depsin yourDockerfile.Amazon Managed Workflows for Apache Airflow (MWAA)If you use MWAA, you just need to update therequirements.txtfile and addairflow-dbtanddbtto it.Then you can have your dbt code inside a folder{DBT_FOLDER}in the dags folder on S3 and configure the dbt task like below:dbt_run=DbtRunOperator(task_id='dbt_run',dbt_bin='/usr/local/airflow/.local/bin/dbt',profiles_dir='/usr/local/airflow/dags/{DBT_FOLDER}/',dir='/usr/local/airflow/dags/{DBT_FOLDER}/')Templating and parsing environments variablesIf you would like to run DBT using custom profile definition template with environment-specific variables, like for example profiles.yml using jinja:<profile_name>:outputs:<source>:database:"{{env_var('DBT_ENV_SECRET_DATABASE')}}"password:"{{env_var('DBT_ENV_SECRET_PASSWORD')}}"schema:"{{env_var('DBT_ENV_SECRET_SCHEMA')}}"threads:"{{env_var('DBT_THREADS')}}"type:<type>user:"{{env_var('USER_NAME')}}_{{env_var('ENV_NAME')}}"target:<source>You can pass the environment variables via theenvkwarg parameter:importos...dbt_run=DbtRunOperator(task_id='dbt_run',env={'DBT_ENV_SECRET_DATABASE':'<DATABASE>','DBT_ENV_SECRET_PASSWORD':'<PASSWORD>','DBT_ENV_SECRET_SCHEMA':'<SCHEMA>','USER_NAME':'<USER_NAME>','DBT_THREADS':os.getenv('<DBT_THREADS_ENV_VARIABLE_NAME>'),'ENV_NAME':os.getenv('ENV_NAME')})License & ContributingThis is available as open source under the terms of theMIT License.Bug reports and pull requests are welcome on GitHub athttps://github.com/gocardless/airflow-dbt.GoCardless ♥ open source. If you do too, comejoin us.