Package: airflow-dbt-dinigo
airflow-dbtThis is a collection ofAirflowoperators to provide easy integration withdbt.fromairflowimportDAGfromairflow_dbt.operators.dbt_operatorimport(DbtSeedOperator,DbtSnapshotOperator,DbtRunOperator,DbtTestOperator)fromairflow.utils.datesimportdays_agodefault_args={'dir':'/srv/app/dbt','start_date':days_ago(0)}withDAG(dag_id='dbt',default_args=default_args,schedule_interval='@daily')asdag:dbt_seed=DbtSeedOperator(task_id='dbt_seed',)dbt_snapshot=DbtSnapshotOperator(task_id='dbt_snapshot',)dbt_run=DbtRunOperator(task_id='dbt_run',)dbt_test=DbtTestOperator(task_id='dbt_test',retries=0,# Failing tests would fail the task, and we don't want Airflow to try again)dbt_seed>>dbt_snapshot>>dbt_run>>dbt_testInstallationInstall from PyPI:pipinstallairflow-dbtIt will also need access to thedbtCLI, which should either be on yourPATHor can be set with thedbt_binargument in each operator.UsageThere are five operators currently implemented:DbtDocsGenerateOperatorCallsdbt docs generateDbtDepsOperatorCallsdbt depsDbtSeedOperatorCallsdbt seedDbtSnapshotOperatorCallsdbt snapshotDbtRunOperatorCallsdbt runDbtTestOperatorCallsdbt testEach of the above operators accept the arguments inhere (dbt_command_config). The main ones being:profiles_dirIf set, passed as the--profiles-dirargument to thedbtcommandtargetIf set, passed as the--targetargument to thedbtcommanddirThe directory to run thedbtcommand infull_refreshIf set toTrue, passes--full-refreshvarsIf set, passed as the--varsargument to thedbtcommand. Should be set as a Python dictionary, as will be passed to thedbtcommand as YAMLmodelsIf set, passed as the--modelsargument to thedbtcommandexcludeIf set, passed as the--excludeargument to thedbtcommandselectIf set, passed as the--selectargument to thedbtcommanddbt_binThedbtCLI. Defaults todbt, so assumes it's on yourPATHverboseThe operator will log verbosely to the Airflow logswarn_errorIf set toTrue, passes--warn-errorargument todbtcommand and will treat warnings as errorsTypically you will want to use theDbtRunOperator, followed by theDbtTestOperator, as shown earlier.You can also use the hook directly. Typically this can be used for when you need to combine thedbtcommand with another task in the same operators, for example runningdbt docsand uploading the docs to somewhere they can be served from.A more advanced example:If want to run yourdbtproject other tan in the airflow worker you can use
theDbtCloudBuildHookand apply it to theDbtBaseOperatoror simply use the
providedDbtCloudBuildOperator:fromairflow_dbt.hooksimportDbtCloudBuildHookfromairflow_dbt.operatorsimportDbtBaseOperator,DbtCloudBuildOperatorDbtBaseOperator(task_id='provide_hook',command='run',use_colors=False,config={'profiles_dir':'./jaffle-shop','project_dir':'./jaffle-shop',},dbt_hook=DbtCloudBuildHook(gcs_staging_location='gs://my-bucket/compressed-dbt-project.tar.gz'))DbtCloudBuildOperator(task_id='default_hook_cloudbuild',gcs_staging_location='gs://my-bucket/compressed-dbt-project.tar.gz',command='run',use_colors=False,config={'profiles_dir':'./jaffle-shop','project_dir':'./jaffle-shop',},)You can either define the dbt params/config/flags in the operator or you can
group them into aconfigparam. They both have validation, but only the config
has templating. The following two tasks are equivalent:fromairflow_dbt.operators.dbt_operatorimportDbtBaseOperatorDbtBaseOperator(task_id='config_param',command='run',config={'profiles_dir':'./jaffle-shop','project_dir':'./jaffle-shop','dbt_bin':'/usr/local/airflow/.local/bin/dbt','use_colors':False})DbtBaseOperator(task_id='flat_config',command='run',profiles_dir='./jaffle-shop',project_dir='./jaffle-shop',dbt_bin='/usr/local/airflow/.local/bin/dbt',use_colors=False)Building LocallyTo install from the repository:
First it's recommended to create a virtual environment:python3-mvenv.venvsource.venv/bin/activateInstall usingpip:pipinstall.TestingTo run tests locally, first create a virtual environment (seeBuilding Locallysection)Install dependencies:pipinstall.pytestRun the tests:pytesttests/Code styleThis project usesflake8.To check your code, first create a virtual environment (seeBuilding Locallysection):pipinstallflake8
flake8airflow_dbt/tests/setup.pyPackage managementIf you use dbt's package manager you should include all dependencies before deploying your dbt project.For Docker users, packages specified inpackages.ymlshould be included as part your docker image by callingdbt depsin yourDockerfile.Amazon Managed Workflows for Apache Airflow (MWAA)If you use MWAA, you just need to update therequirements.txtfile and addairflow-dbtanddbtto it.Then you can have your dbt code inside a folder{DBT_FOLDER}in the dags folder on S3 and configure the dbt task like below:fromairflow_dbt.operators.dbt_operatorimportDbtRunOperatordbt_run=DbtRunOperator(task_id='dbt_run',dbt_bin='/usr/local/airflow/.local/bin/dbt',profiles_dir='/usr/local/airflow/dags/{DBT_FOLDER}/',dir='/usr/local/airflow/dags/{DBT_FOLDER}/')License & ContributingThis is available as open source under the terms of theMIT License.Bug reports and pull requests are welcome on GitHub athttps://github.com/gocardless/airflow-dbt.GoCardless ♥ open source. If you do too, comejoin us.