Package: dbt-invoke
dbt-invokedbt-invoke is a CLI (built withInvoke) for
creating, updating, and deletingdbtproperty files.Supported dbt resource types:modelsseedssnapshotsanalysesUnder the hood, this tool works by combining the power of thedbt lsanddbt run-operationcommands with dbt's built-inget_columns_in_querymacro.This methodology allows the tool to work onephemeral modelsandanalyses,
which other approaches, such as those based on listing data warehouse
tables/views, can miss.InstallationIf you are using dbt-core version 1.5.0 or later:pipinstalldbt-invokeIf you are using a dbt-core version earlier than 1.5.0:pipinstalldbt-invoke~=0.2UsageYou must have previously executeddbt run/dbt seed/dbt snapshoton the
resources for which you wish to create/update property files.If you have made updates to your resources, execute the appropriate command
(dbt run/dbt seed/dbt snapshot) before using this tool to
create/update property files.Property files will be created, updated, or deleted on a one-to-one basis in
the same paths as the resource files they represent (the only change being a.ymlfile extension).For example, given a resource file in the locationmodels/marts/core/users.sql, this tool will create, update, or delete a
property file in the locationmodels/marts/core/users.yml.If your dbt project defines properties for multiple resources per.ymlfile, see theMigrating to One Resource Per Property Filesection.Any newly generated property files are created with the correct resource
type, resource name, and columns.  A blank description field will be included
for each column and for the resource itself.For example, when generating a new property file for a modeluserswith
column namesuser_idandcreated_at, the following yaml will be
generated:version:2models:-name:usersdescription:''columns:-name:user_iddescription:''-name:created_atdescription:''When updating an already existing property file, new columns in the resource
will be added, and columns that no longer exist will be removed.You may fill in the blankdescriptionproperties and add other properties
(e.g.tests).  They will remain intact when updating existing property
files as long as the column/resource name to which they belong still exists.Creating/Updating Property Filesdbt-invokeproperties.update<options># OR, because 'update' is set as the default command:dbt-invokeproperties<options>The first time you run this command, you should be prompted to add a short
macro called_log_columns_listto your dbt project.You may accept the prompt to add it automatically.Otherwise, copy/paste it into one your dbt project's macro-paths yourself.To print the macro, at any time, rundbt-invoke properties.echo-macro.<options>primarily uses the same arguments as thedbt lscommand to
allow flexibility in selecting the dbt resources for which you wish to
create/update property files (rundbt ls --helpfor details).--resource-type--models--select--selector--exclude--project-dir--profiles-dir--profile--target--vars--bypass-cache--stateNotes:This tool supports only the long flags ofdbt lsoptions (for
example:--models, and not short flags like-m).Multiple values for the same argument can be passed as a comma separated
string (Example:--models modelA,modelB)Keep in mind that dbt may not support multiple values for certain
options.Two additional flags are made available.--log-levelto alter the verbosity of logs.It accepts one of Python's standard logging levels (debug, info, warning,
error, critical).--threadsto set a maximum number of concurrent threads to use in
collecting resources' column information from the data warehouse and in
creating/updating the corresponding property files. Each thread will run
dbt's get_columns_in_query macro against the data warehouse.Some examples:# Create/update property files for all supported resource typesdbt-invokeproperties# Create/update property files for all supported resource types, using 4 concurrent threadsdbt-invokeproperties--threads4# Create/update property files for all models in a models/marts directorydbt-invokeproperties--modelsmarts# Create/update property files for a 'users' model and an 'orders' modelsdbt-invokeproperties--modelsusers,orders# Create/update property files for a 'users' model and all downstream modelsdbt-invokeproperties--modelsusers+# Create/update property files for all seedsdbt-invokeproperties--resource-typeseed# Create/update a property file for a snapshot called 'users_snapshot'dbt-invokeproperties--resource-typesnapshot--selectusers_snapshot# Create/update property files when your working directory is above your dbt project directorydbt-invokeproperties--project-dirpath/to/dbt/project/directoryDeleting Property Filesdbt-invokeproperties.delete<options><options>uses the same arguments as for creating/updating property files,
except for--threads.Migrating to One Resource Per Property FileTwo conventions for dbt property files are:One resource per property fileMultiple resources per property filedbt-invoke abides by the "One resource per property file" convention.If your dbt project uses the "Multiple resources per property file"
convention, you can migrate to one resource per property file by using
dbt-invoke'sproperties.migratecommand as shown here:dbt-invokeproperties.migrate<options>Successfully migrated properties will be removed from existing
multiple-resource property files.At then end of migration, property files that are newly empty (other thanversion: 2) will be automatically deleted.<options>uses the same arguments as for creating/updating property
files, except for--threads.HelpTo view the list of available commands and their short descriptions, run:dbt-invoke--listTo view in depth command descriptions and available options/flags, run:dbt-invoke<command_name>--helpLimitationsdbt-invoke will try to preserve formatting and comments when updating
existing files. If you want to preserve line-breaks, use>or|on your
multiline strings, as recommendedhere.In order to collect or update the list of columns that should appear in
each property file, dbt'sget_columns_in_querymacro is run for each
matching resource. As of the time of writing,get_columns_in_queryuses a
SELECT statementlimited to zero rows.
While this is not typically a performance issue for table or incremental
materializations, execution may be slow for complex analyses, views, or
ephemeral materializations.This may be partially remedied by increasing the value of the--threadsoption indbt-invoke properties.update.dbt-invoke has not been tested across different types of data warehouses.