Package: dbt_coves
dbt-covesSponsorHosted VS Code, dbt-core, SqlFluff, and Airflow, find out more atDatacoves.com.What is dbt-coves?dbt-coves is a CLI tool that automates certain tasks fordbt, making life simpler for the dbt user.dbt-coves generates dbt sources, staging models and property(yml) files by analyzing information from the data warehouse and creating the necessary files (sql and yml).Finally, dbt-coves includes functionality to bootstrap a dbt project and to extract and load configurations from Airbyte.Supported dbt versionsVersionStatus< 1.0❌ Not supported>= 1.0✅ TestedFromdbt-coves1.4.0 onwards, our major and minor versions match those ofdbt-core.
This means we release a new major/minor version once it's dbt-core equivalent is tested.
Patch suffix (1.4.X) is exclusive to our continuous development and does not reflect a version match with dbtSupported adaptersFeatureSnowflakeRedshiftBigQuerydbt project setup✅ Tested✅ Tested✅ Testedsource model (sql) generation✅ Tested✅ Tested✅ Testedmodel properties (yml) generation✅ Tested✅ Tested✅ TestedNOTE: Other database adapters may work, although we have not tested them. Feel free to try them and let us know so we can update the table above.Here's the tool in actionInstallationpip install dbt-covesWe recommend usingpython
virtualenvsand create
one separate environment per project.Command ReferenceFor a complete list of options, please run:dbt-coves -hdbt-coves <command> -hEnvironment setupYou can configure different components:Set upgitrepository of dbt-coves projectdbt-coves setup gitSet updbtwithin the project (delegates to dbt init)dbt-coves setup dbtSet up SSH Keys for dbt project. Supports the argument--open_ssl_public_keywhich generates an extra Public Key in Open SSL format, useful for configuring certain providers (i.e. Snowflake authentication)dbt-coves setup sshSet up pre-commit for your dbt project. In this, you can configure different tools that we consider essential for proper dbt usage:sqlfluff,yaml-lint, anddbt-checkpointdbt-coves setup precommitModels generationdbt-coves generate <resource>Where<resource>could besources,properties,metadata,docsorairflow-dags.dbt-coves generate sourcesThis command will generate the dbt source configuration as well as the initial dbt staging model(s). It will look in the database defined in yourprofiles.ymlfile or you can pass the--databaseargument or set up default configuration options (see below)dbt-coves generate sources --database rawSupports Jinja templates to adjust how the resources are generated. See below for examples.Everydbt-coves generate <resource>supports--no-promptflag, which will silently generate all sources/models/properties/metadata without asking anything to the user.Source Generation Argumentsdbt-coves can be used to create the initial staging models. It will do the following:Create / Update the source yml fileCreate the initial staging model(sql) file and offer to flatten VARIANT(JSON) fieldsCreate the staging model's property(yml) file.dbt-coves generate sourcessupports the following args:See full list in helpdbt-coves generate sources -h--database#Databasetoinspect--schemas#Schema(s)toinspect.Acceptswildcards(mustbeenclosedinquotesifused)--select-relations#Listofrelationswhererawdataresides.Theparametermustbeenclosedinquotes.Acceptswildcards.--exclude-relations#Filterrelation(s)toexcludefromsourcefile(s)generation.Theparametermustbeenclosedinquotes.Acceptswildcards.--sources-destination#Wheresourcesymlfileswillbegenerated,default:'models/staging/{{schema}}/sources.yml'--models-destination#Wheremodelssqlfileswillbegenerated,default:'models/staging/{{schema}}/{{relation}}.sql'--model-props-destination#Wheremodelsymlfileswillbegenerated,default:'models/staging/{{schema}}/{{relation}}.yml'--update-strategy#Actiontoperformwhenapropertyfilealreadyexists:'update','recreate','fail','ask'(perfile)--templates-folder#Folderwithjinjatemplatesthatoverridedefaultsourcesgenerationtemplates,i.e.'templates'--metadata#Pathtocsvfilecontainingmetadata,i.e.'metadata.csv'--flatten-json-fields#ActiontoperformwhenJSONfieldsexist:'yes','no','ask'(perfile)--overwrite-staging-models#Flag:overwriteexistentstaging(SQL)files--skip-model-props#Flag:don't create model'sproperty(yml)files--no-prompt#SilentlygeneratesourcedbtmodelsProperties Generation ArgumentsYou can use dbt-coves to generate and update the properties(yml) file for a given dbt model(sql) file.dbt-coves generate propertiessupports the following args:--destination#Wheremodelsymlfileswillbegenerated,default:'{{model_folder_path}}/{{model_file_name}}.yml'--update-strategy#Actiontoperformwhenapropertyfilealreadyexists:'update','recreate','fail','ask'(perfile)-s --select#Filtermodel(s)togeneratepropertyfile(s)--exclude#Filtermodel(s)toexcludefrompropertyfile(s)generation--selector#Specifydbtselectorformorecomplexmodelfiltering--templates-folder#Folderwithjinjatemplatesthatoverridedefaultpropertiesgenerationtemplates,i.e.'templates'--metadata#Pathtocsvfilecontainingmetadata,i.e.'metadata.csv'--no-prompt#SilentlygeneratedbtmodelspropertyfilesNote:--select (or -s),--excludeand--selectorwork exactly asdbt lsselectors do. For usage details, visitdbt list docsMetadata Generation ArgumentsYou can use dbt-coves to generate the metadata file(s) containing the basic structure of the csv that can be used in the abovedbt-coves generate sources/propertiescommands.
Usage of these metadata files can be found inmetadatabelow.dbt-coves generate metadatasupports the following args:--database#Databasetoinspect--schemas#Schema(s)toinspect.Acceptswildcards(mustbeenclosedinquotesifused)--select-relations#Listofrelationswhererawdataresides.Theparametermustbeenclosedinquotes.Acceptswildcards.--exclude-relations#Filterrelation(s)toexcludefromsourcefile(s)generation.Theparametermustbeenclosedinquotes.Acceptswildcards.--destination#Wherecsvfile(s)willbegenerated,default:'metadata.csv'#SupportsusingtheJinjatags`{{relation}}`and`{{schema}}`#ifcreatingonecsvperrelation/tableinschema,i.e:"metadata/{{relation}}.csv"--no-prompt#SilentlygeneratemetadataMetadatadbt-coves supports the argument--metadatawhich allows users to specify a csv file containing field types and descriptions to be used when creating the staging models and property files.dbt-coves generate sources --metadata metadata.csvMetadata format:
You can download asample csv fileas referencedatabaseschemarelationcolumnkeytypedescriptionrawraw_airbyte_raw_country_populations_airbyte_dataYearintegerYear of country population measurementrawraw_airbyte_raw_country_populations_airbyte_datavariantAirbyte data columns (VARIANT) in Snowflakerawraw_airbyte_raw_country_populations_airbyte_ab_idvarcharAirbyte unique identifier used during data loadDocs generation argumentsYou can use dbt-coves to improve the standard dbt docs generation process. It generates your dbt docs, updates external links so they always open in a new tab. It also has the option to merge productioncatalog.jsoninto the local environment when running in deferred mode, so you can rundbt-checkpointhooks even when the model has not been run locally.dbt-coves generate docssupports the following args:--merge-deferred#Mergeadeferredcatalog.jsonintoyourgeneratedone.#Flag:novaluerequired.--state
# Directory where your production catalog.json is located
# Mandatory when using --merge-deferredGenerate airflow-dagsdbt-coves generate airflow-dagsTranslate YML files into their Airflow Python code equivalent. With this, DAGs can be easily written with somekey:valuepairs.The basic structure of these YMLs must consist of:Global configurations (description, schedule_interval, tags, catchup, etc.)default_argsnodes: where tasks and task groups are definedeach Node is a nested object, with it'snameas key and it's configuration as values.this configuration must cover:type: 'task' or 'task_group'operator: Airflow operator that will run the tasks (fullmodule.classnaming)dependencies: whether the task is dependent on another one(s)anykey:valuepair ofOperator argumentsAirflow DAG GeneratorsWhen a YML Dagnodeis of typetask_group,Generatorscan be used instead ofOperators.They are custom classes that receive YMLkey:valuepairs and return one or more tasks for the respective task group. Any pair specified other thantype: task_groupwill be passed to the specifiedgenerator, and it has the responsibility of returning N amount oftask_name = Operator(params).We provide some prebuilt Generators:AirbyteGeneratorcreatesAirbyteTriggerSyncOperatortasks (one per Airbyte connection)It must receive Airbyte'shostandport,airbyte_conn_id(Airbyte's connection name on Airflow) and aconnection_idslist of Airbyte Connections to SyncFivetranGenerator: createsFivetranOperatortasks (one per Fivetran connection)It must receive Fivetran'sapi_key,api_secretand aconnection_idslist of Fivetran Connectors to Sync. It can optionally receivewait_for_completion: trueand 2 tasks will be created for each sync: aFivetranOperatorand it's respectiveFivetranSensorthat monitors the status of the sync.AirbyteDbtGeneratorandFivetranDbtGenerator: instead of passing them Airbyte or Fivetran connections, they use dbt to discover those IDs. Apart from their parent Generators mandatory fields, they can receive:dbt_project_path: dbt/project/foldervirtualenv_path: path to a virtualenv in case dbt has to be ran with another Python executablerun_dbt_compile: true/falserun_dbt_deps: true/falseBasic YML DAG example:description:"dbt-covesDAG"schedule_interval:"@hourly"tags:-version_01default_args:start_date:2023-01-01catchup:falsenodes:airbyte_dbt:type:task_grouptooltip:"Syncdbt-relatedAirbyteconnections"generator:AirbyteDbtGeneratorhost:http://localhostport:8000dbt_project_path:/path/to/dbt_projectvirtualenv_path:/virtualenvs/dbt_160run_dbt_compile:truerun_dbt_deps:falseairbyte_conn_id:airbyte_connectiontask_1:operator:airflow.operators.bash.BashOperatorbash_command:"echo'Thisrunsafterairbytetasks'"dependencies:["airbyte_dbt"]Create your custom GeneratorYou can create your own DAG Generator. Anykey:valuespecified in the YML DAG will be passed to it's constructor.This Generator needs:aimportsattribute: a list ofmodule.classOperator of the tasks it outputsagenerate_tasksmethod that returns the set of"task_name = Operator()"strings to write as the task group tasks.classPostgresGenerator():def__init__(self)->None:""" Any key:value pair in the YML Dag will get here """self.imports=["airflow.providers.postgres.operators.postgres.PostgresOperator"]defgenerate_tasks(self):""" Use your custom logic and return N `name = PostgresOperator()` strings """raiseNotImplementedErrorairflow-dags generation argumentsdbt-coves generate airflow-dagssupports the following args:--yml-path --yaml-path#PathtothefoldercontainingYMLfilestotranslateintoPythonDAGs--dag-path#PathtothefolderwherePythonDAGswillbegenerated.--validate-operators#EnsureAirflowoperatorsareinstalledbytryingtoimportthembeforewritingtoPython.#Flag:novaluerequired--generators-folder#PathtoyourPythonmodulewithcustomGenerators--generators-params#ObjectwithdefaultvaluesforthedesiredGenerator(s)#Forexample:{"AirbyteGenerator":{"host":"http://localhost","port":"8000"}}--secrets-path#SecretfileslocationforDAGconfiguration,i.e.'yml_path/secrets/'#SecretcontentmustmatchtheYMLdagspecof`nodes->node_name->config`Extract configuration from Airbytedbt-coves extract airbyteExtracts the configuration from your Airbyte sources, connections and destinations (excluding credentials) and stores it in the specified folder. The main goal of this feature is to keep track of the configuration changes in your git repo, and rollback to a specific version when needed.Full usage example:dbt-coves extract airbyte --host http://airbyte-server --port 8001 --path /config/workspace/load/airbyteLoad configuration to Airbytedbt-coves load airbyteLoads the Airbyte configuration generated withdbt-coves extract airbyteon an Airbyte server. Secrets folder needs to be specified separately. You can usegit-secretto encrypt secrets and make them part of your git repo.Loading secretsSecret credentials can be approached in two different ways: locally or remotely (through a provider/manager).In order to load encrypted fields locally:dbt-coves load airbyte --secrets-path /path/to/secret/directory#Thisdirectorymusthave'sources','destinations'and'connections'foldersnestedinside,andinsidethemtherespectiveJSONfileswithunencryptedfields.#Namingconvention:JSONunencryptedsecretfilesmustbenamedexactlyastheextractedones.To load encrypted fields through a manager (in this case we are connecting to Datacoves' Service Credentials):--secrets-manager datacoves--secrets-url https://api.datacoves.localhost/service-credentials/airbyte--secrets-token <secret token>Full usage example:dbt-coves load airbyte --host http://airbyte-server --port 8001 --path /config/workspace/load/airbyte --secrets-path /config/workspace/secretsExtract configuration from Fivetrandbt-coves extract fivetranExtracts the configuration from your Fivetran destinations and connectors (excluding credentials) and stores it in the specified folder. The main goal of this feature is to keep track of the configuration changes in your git repo, and rollback to a specific version when needed.Full usage example:dbt-coves extract fivetran --credentials /config/workspace/secrets/fivetran/credentials.yml --path /config/workspace/load/fivetranLoad configuration to Fivetrandbt-coves load fivetranLoads the Fivetran configuration generated withdbt-coves extract fivetranon a Fivetran instance. Secrets folder needs to be specified separately. You can usegit-secretto encrypt secrets and make them part of your git repo.CredentialsIn order for extract/load fivetran to work properly, you need to provide an api key-secret pair (you can generate themhere).These credentials can be used with--api-key [key] --api-secret [secret], or specyfing a YML file with--credentials /path/to/credentials.yml. The required structure of this file is the following:account_name:# Any name, used by dbt-coves to ask which to use when more than one is foundapi_key:[key]api_secret:[secret]account_name_2:api_key:[key]api_secret:[secret]This YML file approach allows you to both work with multiple Fivetran accounts, and treat this credentials file as a secret.:warning:Warning: --api-key/secret and --credentials flags are mutually exclusive, don't use them together.Loading secretsSecret credentials can be approached via--secrets-pathflagdbt-coves load fivetran --secrets-path /path/to/secret/directoryField naming conventionAlthough secret files can have any name, unencrypted JSON files must follow a simple structure:Keys should match their corresponding Fivetran destination ID: two words automatically generated by Fivetran, which can be found in previously extracted data.Inside those keys, a nested dictionary of which fields should be overwrittenFor example:{"extract_muscle":{// Internal ID that Fivetran gave to a Snowflake warehouse Destination"password":"[PASSWORD]"// Field:Value pair},"centre_straighten":{"password":"[PASSWORD]"}}Run dbt commandsdbt-covesdbt<arguments>--<command>Run dbt commands on special environments such as Airflow, or CI workers, with the possibility of changing dbt project location and activating a specific virtual environment in which running commands.Argumentsdbt-coves dbtsupports the following arguments--project-dir# Path of the dbt project where command will be executed, i.e.: /opt/user/dbt_project--virtualenv# Virtual environment path. i.e.: /opt/user/virtualenvs/airflowSample usagedbt-covesdbt--project-dir/opt/user/dbt_project--virtualenv/opt/user/virtualenvs/airflow--run-smodel--vars\"{key:value}\"# Make sure to escape special characters such as quotation marks# Double dash (--) between <arguments> and <command> are mandatorySettingsdbt-coves will read settings from.dbt_coves/config.yml. A standard settings files could look like this:generate:sources:database:"RAW"# Database where to look for source tablesschemas:# List of schema names where to look for source tables-RAWselect_relations:# list of relations where raw data resides-TABLE_1-TABLE_2exclude_relations:# Filter relation(s) to exclude from source file(s) generation-TABLE_1-TABLE_2sources_destination:"models/staging/{{schema}}/{{schema}}.yml"# Where sources yml files will be generatedmodels_destination:"models/staging/{{schema}}/{{relation}}.sql"# Where models sql files will be generatedmodel_props_destination:"models/staging/{{schema}}/{{relation}}.yml"# Where models yml files will be generatedupdate_strategy:ask# Action to perform when a property file already exists. Options: update, recreate, fail, ask (per file)templates_folder:".dbt_coves/templates"# Folder where source generation jinja templates are located. Override default templates creating  source_props.yml, source_model_props.yml, and source_model.sql under this foldermetadata:"metadata.csv"# Path to csv file containing metadataflatten_json_fields:askproperties:destination:"{{model_folder_path}}/{{model_file_name}}.yml"# Where models yml files will be generated# You can specify a different path by declaring it explicitly, i.e.: "models/staging/{{model_file_name}}.yml"update-strategy:ask# Action to perform when a property file already exists. Options: update, recreate, fail, ask (per file)select:"models/staging/bays"# Filter model(s) to generate property file(s)exclude:"models/staging/bays/test_bay"# Filter model(s) to generate property file(s)selector:"selectors/bay_selector.yml"# Specify dbt selector for more complex model filteringtemplates_folder:".dbt_coves/templates"# Folder where source generation jinja templates are located. Override default template creating model_props.yml under this foldermetadata:"metadata.csv"# Path to csv file containing metadatametadata:database:RAW# Database where to look for source tablesschemas:# List of schema names where to look for source tables-RAWselect_relations:# list of relations where raw data resides-TABLE_1-TABLE_2exclude_relations:# Filter relation(s) to exclude from source file(s) generation-TABLE_1-TABLE_2destination:# Where metadata file will be generated, default: 'metadata.csv'docs:merge_deferred:truestate:logs/dbt_args:"--no-compile--selectfoo--excludebar"airflow_dags:yml_path:dags_path:generators_params:AirbyteDbtGenerator:host:"{{env_var('AIRBYTE_HOST_NAME')}}"port:"{{env_var('AIRBYTE_PORT')}}"airbyte_conn_id:airbyte_connectiondbt_project_path:"{{env_var('DBT_HOME')}}"run_dbt_compile:truerun_dbt_deps:falseextract:airbyte:path:/config/workspace/load/airbyte# Where json files will be generatedhost:http://airbyte-server# Airbyte's API hostnameport:8001# Airbyte's API portfivetran:path:/config/workspace/load/fivetran# Where Fivetran export will be generatedapi_key:[KEY]# Fivetran API Keyapi_secret:[SECRET]# Fivetran API Secretcredentials:/opt/fivetran_credentials.yml# Fivetran set of key:secret pairs# 'api_key' + 'api_secret' are mutually exclusive with 'credentials', use one or the otherload:airbyte:path:/config/workspace/loadhost:http://airbyte-serverport:8001secrets_manager:datacoves# (optional) Secret credentials provider (secrets_path OR secrets_manager should be used, can't load secrets locally and remotely at the same time)secrets_path:/config/workspace/secrets# (optional) Secret files location if secrets_manager was not specifiedsecrets_url:https://api.datacoves.localhost/service-credentials/airbyte# Secrets url if secrets_manager is datacovessecrets_token:<TOKEN># Secrets auth token if secrets_manager is datacovesfivetran:path:/config/workspace/load/fivetran# Where previous Fivetran export resides, subject of importapi_key:[KEY]# Fivetran API Keyapi_secret:[SECRET]# Fivetran API Secretsecrets_path:/config/workspace/secrets/fivetran# Fivetran secret fieldscredentials:/opt/fivetran_credentials.yml# Fivetran set of key:secret pairs# 'api_key' + 'api_secret' are mutually exclusive with 'credentials', use one or the otherenv_varFromdbt-coves 1.6.28onwards, you can consume environment variables in you config file using"{{env_var('VAR_NAME', 'DEFAULT VALUE')}}". For example:generate:sources:database:"{{env_var('MAIN_DATABASE','dev_database')}}"schemas:-"{{env_var('DEV_SCHEMA','John')}}"-"{{env_var('STAGING_SCHEMA','Staging')}}"Telemetrydbt-coves has telemetry built in to help the maintainers from Datacoves understand which commands are being used and which are not to prioritize future development of dbt-coves. We do not track credentials nor details of your dbt execution such as model names. The one detail we do use related to dbt is the anonymous user_id to help us identify distinct users.By default this is turned on – you can opt out of event tracking at any time by adding the following to your dbt-covesconfig.yamlfile:disable-tracking:trueOverride generation templatesCustomizing generated models and model properties requires placing
template files under the.dbt-coves/templatesfolder.There are different variables available in the templates:adapter_namerefers to the Adapter's class name being used by the target, e.g.SnowflakeAdapterwhen usingSnowflake.columnscontains the list of relation columns that don't contain nested (JSON) data, it's type isList[Item].nestedcontains a dict of nested columns, grouped by column name, it's type isDict[column_name, Dict[nested_key, Item]].Itemis adictwith the keysid,name,type, anddescription, whereidcontains an slugified id generated fromname.dbt-coves generate sourcesSource property file (.yml) templateThis file is used to create the sources yml filesource_props.ymlStaging model file (.sql) templateThis file is used to create the staging model (sql) files.staging_model.sqlStaging model property file (.yml) templateThis file is used to create the model properties (yml) filestaging_model_props.ymldbt-coves generate propertiesThis file is used to create the properties (yml) files for modelsmodel_props.ymlThanksThe project main structure was inspired bydbt-sugar. Special thanks toBastien Boutonnetfor the great work done.AuthorsSebastian Sassi@sebasuy--DatacovesNoel Gomez@noel_g--DatacovesBruno Antonellini --DatacovesAboutLearn more aboutDatacoves.⚠️dbt-coves is still in development, make sure to test it for your dbt project version and DW before using in production and please submit any issues you find. We also welcome any contributions from the communityMetrics