Package: palm-dbt
Palm dbtdbt plugin forPalm CLIThis plugin adds dbt-specific commands for use with Palm CLIInstallingInstall this plugin along with palmpip install palm-dbtOr from sourcepython3 -m pip install .Configuring your projectTo configure your project to use the palm-dbt plugin, you will need a.palm/config.yamlthis can be created by runningpalm init, once you have your config file,
add the dbt-palm plugin with the following configuration:plugins:-dbtCheck dbt plugin versionCheck the version of palm-dbt inside a project in which you have configured palm with the dbt plugin:palm plugin versionsPalm-ing an existing dbt projectpalm-dbt ships with a command to containerize and convert your existing dbt project.For example, if you wanted to containerize your existing dbt project running on 0.21.0, you would run:palm containerize --version 0.21.0Adding palm dbt macrospalm-dbt uses the git branch name to set the schema for all commands via env vars.
This allows palm to clean up test data after each run, ensuring that your data
warehouse stays clean and free of development/test data.To enable this functionality, palm-dbt ships with 2 macros that handle schema naming
and cleanup:generate_schema_name -This macro overrides the dbt-core macroto auto-generate
a schema name based on your current git branch and PALM_DBT_ENV.drop_branch_schemas - This macro uses the branch named schema and the TEST database
to clean up any models generated by running dbt in development or test environments.
Calls to this macro are baked in to many of the palm dbt commands.See the sectionabout the palm dbt naming macrosbelow for more information.To install these macros, runpalm installfrom within a project that is configured
to use the palm-dbt plugin.Recommended (optional) protected branch configurationIn order to ensure your runs are idempotent, we recommend that you do not run
palm-dbt commands againstmain,masteror any other production-like branches
you may be using.To prevent palm running against specific branches, add the following config to
your project's.palm/config.yamlprotected_branches:-main-master# Any other branches you want to protectAbout the palm dbt branch naming macrosOne of the most painful parts of data testing is unfortunateshared mutable state.
palm-dbt provides a mechanism to eliminate this undesirable situation by namespacing
each run of dbt. for git branches other than main or master, palm will prefix the
calculated schema name with a formatted version of your branch name. In CI, this
will be additionally prefixed with "CI". For example:you open a branch FEATURE/DATA-100/update-widgetwhen youpalm runin your local env, the schemapublicwill be built asfeature_data_100_update_widget_public. The schemasaleswill be built asfeature_data_100_update_widget_sales.in CI the schemas will beci_feature_data_100_update_widget_public,ci_feature_data_100_update_widget_sales(respectively).in prod the schemas will be 'public' and 'sales' (respectively).Refs will automatically update as well. This way, you can use a single test
database and not worry about conflicts between developers, or between branches
for the same developer (like during hotfixes).palm-dbt and dbt depsIn palm-dbt we have determined that runningdbt depsbefore every command is
problematic for a few reasons:It takes time, slowing down development, CI, and every production run.If dbt hub or github have an outage, our dbt commands fail and remain broken
until the upstream error is resolvedIf you forget to run dbt deps, the resulting error messages can be quite confusing.To solve these problems, we have decided that runningdbt clean && dbt depsshould
happen in the Dockerfile, when the image is being built.To support this decision your projectmustdo the following:IncludeRUN dbt clean && dbt depsin the DockerfileInclude avolumeentry in the docker-compose.yaml for the dbt_modules directory
like this- /app/{{packages_dir}}, which will prevent the.:./appvolume from
blowing away the deps generated when the image was built.if you usepalm containerizethis will be done for you!Additionally, if you need to make changes to your deps you should usepalm buildto rebuild the image, which will update your deps!Typical palm-dbt workflowFrom a non-protected branch, runningpalm runwill:drop (if it exists) the namespaced schema in developmentcreate the namespaced schema in developmentseed and rundrop the namespaced schema in developmentWhy drop it? so your testing is atomic.Want to persist it? use the flag--persist