Package: prefect-dbt
Coordinate and use dbt in your dataflow with prefect-dbtVisit the full docshereto see additional examples and the API reference.With prefect-dbt, you can easily trigger and monitor dbt Cloud jobs, execute dbt Core CLI commands, and incorporate other services, like Snowflake, into your dbt runs!Check out the examples below to get started!Getting StartedBe sure to installprefect-dbtandsave a blockto run the examples below!Integrate dbt Cloud jobs with Prefect flowsIf you have an existing dbt Cloud job, take advantage of the flow,run_dbt_cloud_job.This flow triggers the job and waits until the job run is finished.If certain nodes fail,run_dbt_cloud_jobefficiently retries the specific, unsuccessful nodes.fromprefectimportflowfromprefect_dbt.cloudimportDbtCloudJobfromprefect_dbt.cloud.jobsimportrun_dbt_cloud_job@flowdefrun_dbt_job_flow():result=run_dbt_cloud_job(dbt_cloud_job=DbtCloudJob.load("my-block-name"),targeted_retries=5,)returnresultrun_dbt_job_flow()Integrate dbt Core CLI commands with Prefect flowsprefect-dbtalso supports execution of dbt Core CLI commands.To get started, if you don't have aDbtCoreOperationblock already saved,
set the commands that you want to run; it can include a mix of dbt and non-dbt commands.Then, optionally specify theproject_dir.Ifprofiles_diris unset, it will try to use theDBT_PROFILES_DIRenvironment variable.
If that's also not set, it will use the default directory$HOME/.dbt/.Using an existing profileIf you already have an existing dbt profile, specify theprofiles_dirwhereprofiles.ymlis located.fromprefectimportflowfromprefect_dbt.cli.commandsimportDbtCoreOperation@flowdeftrigger_dbt_flow()->str:result=DbtCoreOperation(commands=["pwd","dbt debug","dbt run"],project_dir="PROJECT-DIRECTORY-PLACEHOLDER",profiles_dir="PROFILES-DIRECTORY-PLACEHOLDER").run()returnresulttrigger_dbt_flow()Writing a new profileTo setup a new profile, firstsave and load a DbtCliProfile blockand use it inDbtCoreOperation.Then, specifyprofiles_dirwhereprofiles.ymlwill be written.fromprefectimportflowfromprefect_dbt.cliimportDbtCliProfile,DbtCoreOperation@flowdeftrigger_dbt_flow():dbt_cli_profile=DbtCliProfile.load("DBT-CORE-OPERATION-BLOCK-NAME-PLACEHOLDER")withDbtCoreOperation(commands=["dbt debug","dbt run"],project_dir="PROJECT-DIRECTORY-PLACEHOLDER",profiles_dir="PROFILES-DIRECTORY-PLACEHOLDER",dbt_cli_profile=dbt_cli_profile,)asdbt_operation:dbt_process=dbt_operation.trigger()# do other things before waiting for completiondbt_process.wait_for_completion()result=dbt_process.fetch_result()returnresulttrigger_dbt_flow()ResourcesIf you need help getting started with or using dbt, please consult thedbt documentation.InstallationTo useprefect-dbtwith dbt Cloud:pipinstallprefect-dbtTo use dbt Core (CLI):pipinstall"prefect-dbt[cli]"To use dbt Core with Snowflake profiles:pipinstall"prefect-dbt[snowflake]"To use dbt Core with BigQuery profiles:pipinstall"prefect-dbt[bigquery]"To use dbt Core with Postgres profiles:pipinstall"prefect-dbt[postgres]"!!! warning "Some dbt Core profiles require additional installation"According to dbt's [Databricks setup page](https://docs.getdbt.com/reference/warehouse-setups/databricks-setup), users must first install the adapter:

```bash
pip install dbt-databricks
```

Check out the [desired profile setup page](https://docs.getdbt.com/reference/profiles.yml) on the sidebar for others.Requires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2. For more information about how to use Prefect, please refer to thePrefect documentation.Saving credentials to blockNote, to use theloadmethod on Blocks, you must already have a block documentsaved through codeorsaved through the UI.!!! info "Registering blocks"Register blocks in this module to
[view and edit them](https://orion-docs.prefect.io/ui/blocks/)
on Prefect Cloud:

```bash
prefect block register -m prefect_dbt
```A list of available blocks inprefect-dbtand their setup instructions can be foundhere.dbt CloudTo create a dbt Cloud credentials block:Head over to yourdbt Cloud profile.Login to your dbt Cloud account.Scroll down to "API" or click "API Access" on the sidebar.Copy the API Key.Click Projects on the sidebar.Copy the account ID from the URL:https://cloud.getdbt.com/settings/accounts/<ACCOUNT_ID>.Create a short script, replacing the placeholders.fromprefect_dbt.cloudimportDbtCloudCredentialsDbtCloudCredentials(api_key="API-KEY-PLACEHOLDER",account_id="ACCOUNT-ID-PLACEHOLDER").save("CREDENTIALS-BLOCK-NAME-PLACEHOLDER")Then, to create a dbt Cloud job block:Head over to yourdbt home page.On the top nav bar, click on Deploy -> Jobs.Select a job.Copy the job ID from the URL:https://cloud.getdbt.com/deploy/<ACCOUNT_ID>/projects/<PROJECT_ID>/jobs/<JOB_ID>Create a short script, replacing the placeholders.fromprefect_dbt.cloudimportDbtCloudCredentials,DbtCloudJobdbt_cloud_credentials=DbtCloudCredentials.load("CREDENTIALS-BLOCK-NAME-PLACEHOLDER")dbt_cloud_job=DbtCloudJob(dbt_cloud_credentials=dbt_cloud_credentials,job_id="JOB-ID-PLACEHOLDER").save("JOB-BLOCK-NAME-PLACEHOLDER")Congrats! You can now easily load the saved block, which holds your credentials:fromprefect_dbt.cloudimportDbtCloudJobDbtCloudJob.load("JOB-BLOCK-NAME-PLACEHOLDER")dbt Core CLI!!! info "AvailableTargetConfigsblocks"The following may vary slightly depending on the service you want to incorporate.

Visit the [API Reference](cli/configs/base) to see other built-in `TargetConfigs` blocks.

If the desired service profile is not available, check out the
[Examples Catalog](examples_catalog/#clicredentials-module) to see how you can
build one from the generic `TargetConfigs` class.To create dbt Core target config and profile blocks for BigQuery:Save and load aGcpCredentialsblock.Determine the schema / dataset you want to use in BigQuery.Create a short script, replacing the placeholders.fromprefect_gcp.credentialsimportGcpCredentialsfromprefect_dbt.cliimportBigQueryTargetConfigs,DbtCliProfilecredentials=GcpCredentials.load("CREDENTIALS-BLOCK-NAME-PLACEHOLDER")target_configs=BigQueryTargetConfigs(schema="SCHEMA-NAME-PLACEHOLDER",# also known as datasetcredentials=credentials,)target_configs.save("TARGET-CONFIGS-BLOCK-NAME-PLACEHOLDER")dbt_cli_profile=DbtCliProfile(name="PROFILE-NAME-PLACEHOLDER",target="TARGET-NAME-placeholder",target_configs=target_configs,)dbt_cli_profile.save("DBT-CLI-PROFILE-BLOCK-NAME-PLACEHOLDER")Then, to create a dbt Core operation block:Determine the dbt commands you want to run.Create a short script, replacing the placeholders.fromprefect_dbt.cliimportDbtCliProfile,DbtCoreOperationdbt_cli_profile=DbtCliProfile.load("DBT-CLI-PROFILE-BLOCK-NAME-PLACEHOLDER")dbt_core_operation=DbtCoreOperation(commands=["DBT-CLI-COMMANDS-PLACEHOLDER"],dbt_cli_profile=dbt_cli_profile,overwrite_profiles=True,)dbt_core_operation.save("DBT-CORE-OPERATION-BLOCK-NAME-PLACEHOLDER")Congrats! You can now easily load the saved block, which holds your credentials:fromprefect_dbt.cloudimportDbtCoreOperationDbtCoreOperation.load("DBT-CORE-OPERATION-BLOCK-NAME-PLACEHOLDER")FeedbackIf you encounter any bugs while usingprefect-dbt, feel free to open an issue in theprefect-dbtrepository.If you have any questions or issues while usingprefect-dbt, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to star or watchprefect-dbtfor updates too!ContributingIf you'd like to help contribute to fix an issue or add a feature toprefect-dbt, pleasepropose changes through a pull request from a fork of the repository.Here are the steps:Fork the repositoryClone the forked repositoryInstall the repository and its dependencies:pip install -e ".[dev]"Make desired changesAdd testsInsert an entry toCHANGELOG.mdInstallpre-committo perform quality checks prior to commit:pre-commit installgit commit,git push, and create a pull request