Package: airflow-dbt-python
airflow-dbt-pythonA collection ofAirflowoperators, hooks, and utilities to executedbtcommands.Read thedocumentationfor examples, installation instructions, and more details.InstallationRequirementsBefore usingairflow-dbt-python, ensure you meet the following requirements:Adbtproject usingdbt-coreversion 1.4.0 or later.An Airflow environment using version 2.2 or later.If using any managed service, like AWS MWAA, ensure your environment is created with a supported version of Airflow.If self-hosting, Airflow installation instructions can be found in theirofficial documentation.Running Python 3.8 or later in your Airflow environment.WarningEven though we don't impose any upper limits on versions of Airflow anddbt, it's possible that new versions are not supported immediately after release, particularly fordbt. We recommend testing the latest versions before upgrading andreporting any issues.NoteOlder versions of Airflow anddbtmay work withairflow-dbt-python, although we cannot guarantee this. Our testing pipeline runs the latestdbt-corewith the latest Airflow release, and the latest version supported byAWS MWAA.From PyPIairflow-dbt-pythonis available inPyPIand can be installed withpip:pipinstallairflow-dbt-pythonAs a convenience, somedbtadapters can be installed by specifying extras. For example, if requiring thedbt-redshiftadapter:pipinstallairflow-dbt-python[redshift]From this repoairflow-dbt-pythoncan also be built from source by cloning this GitHub repository:gitclonehttps://github.com/tomasfarias/airflow-dbt-python.gitcdairflow-dbt-pythonAnd installing withPoetry:poetryinstallIn AWS MWAAAddairflow-dbt-pythonto yourrequirements.txtfile and edit your Airflow environment to use this newrequirements.txtfile, or upload it as a plugin.Read thedocumentationfor more a more detailed AWS MWAA installation breakdown.In other managed servicesairflow-dbt-pythonshould be compatible with most or all Airflow managed services. Consult the documentation specific to your provider.If you notice an issue when installingairflow-dbt-pythonin a specific managed service, please open anissue.Featuresairflow-dbt-pythonaims to make dbt afirst-class citizenof Airflow by supporting additional features that integrate both tools. As you would expect,airflow-dbt-pythoncan run all your dbt workflows in Airflow with the same interface you are used to from the CLI, but without being a mere wrapper:airflow-dbt-pythondirectly communicates with internaldbt-coreclasses, bridging the gap between them and Airflow's operator interface. Essentially, we are attempting to usedbtas a library.As this integration was completed, several features were developed toextend the capabilities of dbtto leverage Airflow as much as possible. Can you think of a waydbtcould leverage Airflow that is not currently supported? Let us know in aGitHub issue!Independent task executionAirflow executesTasksindependent of one another: even though downstream and upstream dependencies between tasks exist, the execution of an individual task happens entirely independently of any other task execution (see:Tasks Relationships).In order to work with this constraint,airflow-dbt-pythonruns each dbt command in atemporary and isolated directory. Before execution, all the relevant dbt files are copied from supported backends, and after executing the command any artifacts are exported. This ensures dbt can work with any Airflow deployment, including most production deployments as they are usually runningRemote Executorsand do not guarantee any files will be shared by default between tasks, since each task may run in a completely different environment.Download dbt files from a remote storageThe dbt parametersprofiles_dirandproject_dirwould normally point to a directory containing aprofiles.ymlfile and a dbt project in the local environment respectively (defined by the presence of adbt_project.ymlfile).airflow-dbt-pythonextends these parameters to also accept an URL pointing to a remote storage.Currently, we support the following remote storages:AWS S3(identified by as3scheme).Remote git repositories, like those stored in GitHub (bothhttpsandsshschemes are supported).If a remote URL is used forproject_dir, then this URL must point to a location in your remote storage containing adbtproject to run. Adbtproject is identified by the prescence of adbt_project.yml, and contains all yourresources. All of the contents of this remote location will be downloaded and made available for the operator. The URL may also point to an archived file containing all the files of a dbt project, which will be downloaded, uncompressed, and made available for the operator.If a remote URL is used forprofiles_dir, then this URL must point to a location in your remote storage that contains aprofiles.ymlfile. Theprofiles.ymlfile will be downloaded and made available for the operator to use when running. Theprofiles.ymlmay be part of yourdbtproject, in which case this argument may be ommitted.This feature is intended to work in line with Airflow'sdescription of the task concept:Tasks don’t pass information to each other by default, and run entirely independently.We interpret this as meaning a task should be responsible of fetching all thedbtrelated files it needs in order to run independently, as already described inIndependent Task Execution.Push dbt artifacts to XComEach dbt execution produces one or moreJSON artifactsthat are valuable to produce meta-metrics, build conditional workflows, for reporting purposes, and other uses.airflow-dbt-pythoncan push these artifacts toXComas requested via thedo_xcom_push_artifactsparameter, which takes a list of artifacts to push.Use Airflow connections as dbt targets (without a profiles.yml)Airflow connectionsallow users to manage and store connection information, such as hostname, port, username, and password, for operators to use when accessing certain applications, like databases. Similarly, adbtprofiles.ymlfile stores connection information under each target key.airflow-dbt-pythonbridges the gap between the two and allows you to use connection information stored as an Airflow connection by specifying the connection id as thetargetparameter of any of thedbtoperators it provides. What's more, if using an Airflow connection, theprofiles.ymlfile may be entirely omitted (although keep in mind aprofiles.ymlfile contains a configuration block besides target connection information).See an example DAGhere.MotivationAirflow running in a managed environmentAlthoughdbtis meant to be installed and used as a CLI, we may not have control of the environment where Airflow is running, disallowing us the option of usingdbtas a CLI.This is exactly what happens when usingAmazon's Managed Workflows for Apache Airflowor MWAA: although a list of Python requirements can be passed, the CLI cannot be found in the worker's PATH.There is a workaround which involves using Airflow'sBashOperatorand running Python from the command line:fromairflow.operators.bashimportBashOperatorBASH_COMMAND="python -c 'from dbt.main import main; main()' run"operator=BashOperator(task_id="dbt_run",bash_command=BASH_COMMAND,)But it can get cumbersome when appending all potential arguments adbt runcommand (or other subcommand) can take.That's whereairflow-dbt-pythoncomes in: it abstracts the complexity of interfacing withdbt-coreand exposes one operator for eachdbtsubcommand that can be instantiated with all the corresponding arguments that thedbtCLI would take.An alternative toairflow-dbtthat works without thedbtCLIThe alternativeairflow-dbtpackage, by default, would not work if thedbtCLI is not in PATH, which means it would not be usable in MWAA. There is a workaround via thedbt_binargument, which can be set to"python -c 'from dbt.main import main; main()' run", in similar fashion as theBashOperatorexample. Yet this approach is not without its limitations:airflow-dbtworks by wrapping thedbtCLI, which makes our code dependent on the environment in which it runs.airflow-dbtdoes not support the full range of arguments a command can take. For example,DbtRunOperatordoes not have an attribute forfail_fast.airflow-dbtdoes not offer access todbtartifacts created during execution.airflow-dbt-pythondoes so by pushing any artifacts toXCom.UsageCurrently, the followingdbtcommands are supported:cleancompiledebugdepsdocs generatelsparserunrun-operationseedsnapshotsourcetestExamplesAll example DAGs are tested against the latest Airflow version. Some changes, like modifyingimportstatements or changing types, may be required for them to work in other versions.importdatetimeasdtimportpendulumfromairflowimportDAGfromairflow_dbt_python.operators.dbtimport(DbtRunOperator,DbtSeedOperator,DbtTestOperator,)args={"owner":"airflow",}withDAG(dag_id="example_dbt_operator",default_args=args,schedule="0 0 * * *",start_date=pendulum.today("UTC").add(days=-1),dagrun_timeout=dt.timedelta(minutes=60),tags=["example","example2"],)asdag:dbt_test=DbtTestOperator(task_id="dbt_test",selector_name="pre-run-tests",)dbt_seed=DbtSeedOperator(task_id="dbt_seed",select=["/path/to/first.csv","/path/to/second.csv"],full_refresh=True,)dbt_run=DbtRunOperator(task_id="dbt_run",select=["/path/to/models"],full_refresh=True,fail_fast=True,)dbt_test>>dbt_seed>>dbt_runMore examples can be found in theexamples/directory and thedocumentation.DevelopmentSee thedevelopment documentationfor a more in-depth dive into setting up a development environment, running the test-suite, and general commentary on working onairflow-dbt-python.TestingTests are run withpytest, can be located intests/. To run them locally, you may usePoetry:poetryrunpytesttests/-vvLicenseThis project is licensed under the MIT license. SeeLICENSE.