Package: dbtvault-generator
dbtvault-generatorGenerate DbtVault files from externalyml!Why this existsA good dev is a lazy dev. Let's do less work, the right way!DBTVault is a great tool, but due to quirks of DBT (looking at you, compile context) we don't necessarily have access additional variables we want to use...at least, without overloading themetaorvarsattributes. Seethis issuefor some details about the limitations and workarounds for the build context. Thedbtvaultdocs usedymlspecified inside DBTsqlfiles to specify its variables, or otherwise directly set the variables usingjinja. While this is a workable solution, it's still a workaround to the fact that additional attributes can't be consumed by thedbt compileoperation.Using env vars as a stopgap isn't a problem on small projects, but on more complicated codebases it can quickly lead to confusion. It becomes difficult to trace which variables belong where. Further, it hinders the development of other data vault management tools being built on top of DBT. Data Vault is a metadata-heavy method of architecting a data warehouse, as the modelling discipline associated with data vault makes it ripe for automation and abstraction. A standardised modelling output format independent of env vars permits the building of GUI tools to streamline the development and expansion of the data vault.This tool is a Python-based templating library designed to act around DBT anddbtvault, and allow externaldbtvaultparameterisation. It also seeks to leverage the uniformity of data vault specifications to automate certain low-level maintenance tasks such as code documentation. The primary purpose is to generate Data Vault architectures (trhoughdbtvault), generating SQL andschema.ymldirectly from specialiseddbtvault.ymlfiles.Watch this space for a GUI assistance tool in future.Requirementspython>=3.9dbt>=1.0.0available in the Python environmentA DBT project initialised and ready to go. Thedbtv-genCLI tool needs to execute from the same directory as you would normally execute DBT from.The DBT packagedbtvault>0.9.0installed in the DBT projectOverview Of FeaturesDepending on your set-up, installdbtvault-generatorinto your Python environment. A shim will be installed that addsdbtv-gento your CLI, and you're ready to go.The following assumes you havedbtvault.ymlfiles preconfigured (see below for details).Thedbtvault.ymlfile contains the metadata required for templating thedbtvaultlibraryYou can also use a root-leveldbtvault.ymlto specify global defaults.Generate SQL TemplatesIn your command line of choice, run the following command:dbtv-gensqlBy default, you will see the SQL files appear in the same directory as thedbtvault.ymlfile. Files can be prefixed automatically (shown), can be manually specified per-model, or ignored.The contents of the file will match the details specified indbtvault.yml.Generateschema.ymlDoc ConfigOnce the sql has been generated, executedbt runto instantiate them as views in your development environment. From here,dbt docs generatecan be executed to build DBT'scatalog.jsonartifact.dbtvault-generatorcan then use this to pre-populate aschema.ymlfile with basic documentation.dbtv-gendocsThe schema file will infer certain properties about the columns. This includes the data types of the columns, read from the information schema. Unfortunately we can't yet describe the columns automatically.In additionFor example, primary keys will automatically havenot_nullanduniquetests added for alerting of clashes. Foreign key columns will have arelationshipstest added, but this test is conditional on awhere 1 != 1condition to prevent it from triggering alerts. What this does do, however, is play nicely with other tools and packages that use the "relationship" test to automatically create foreign keys or similar metadata (e.g. on Snowflake) for downstream use by other tools.Instantiating the models is currently required to build the docs. In future versions, the docs will be built directly from the config options.BONUS! UsedbterdTo Generate ER Diagrams!The wonderfulDat Nguyenhas builta cool library calleddbterd.dbterdpicks up DBT's catalog and manifest artifacts and, using the relationship test, creates a.dbmlfile detailing the core relationships within your DBT catalog. The.dbmlformat is an open-source entity-relation spec that can either be consumed directly by a number of SaaS tools or by open-source tools likeDBML Renderer.Importantly,dbterduses therelationshipstest to identify foreign key relations...which you will have out of the box. You will need to rerundbt docs generateto add in the new relationships, butGo check outhis page on Githubfor further details...and leave him a star while you're there.Installation InstructionsFrom Pippipinstall--upgradedbtvault-generatorDirect From GitMany package managers allow installation of Python packages direct from version control repositories. Add this repo directly with e.g.poetryaddgit@github.com:Oracen/dbtvault-generator.gitFrom SourceFor now, using PDM as dependency manager as it seems to be better supported than Poetry. As such, give PDM a try! It's like Yarn for Python, or Poetry with a faster resolver and Poe built in.gitclonehttps://github.com/Oracen/dbtvault-generatorcddbtvault-generator
pdmbuildUsageOn command execution,dbtv-genscans your project directory for files nameddbtvault.yml, containing a root-level key calleddbtvault. The location of these files specify where any associated models will be generated, unless the target path is overriden. The only special file is an optionaldbtvault.ymllocated at the project root. This file's default attributes will be treated as the defaults for the entire project, and so serves as a good place for specifying prefixes etc.Eachdbtvault.ymlcontains 2 keys:defaults: the default options for each model. These default options can be overwritten on a per-model basismodels: the specification for each model to be generated in the parent folder of the current file.Thedefaultselement has the following structure:version:2dbtvault:defaults:use_prefix:trueprefixes:table:stage:stg_hub:hub_link:lnk_t_link:t_lnk_sat:sat_eff_sat:eff_sat_ma_sat:ma_sat_xts:xts_pit:pit_bridge:bridge_custom_macros:stage:custom_staging_macrotarget_path:""use_prefix: indicates whether to prepend the designated string to each table name as the model generates. Defaults tofalseprefixes: a mapping of prefixes to use per supported model type. Defaults are shown above, and are aligned withdbtvault's examplescustom_macros: sometimes the defaultdbtvaultmacros don't do exactly what you want and a custom macro needs to be substituted. For example, you may want to restrict your data to a small time range in the development environment. If specified, the custom macro will be substituted in for particular model structures. Defaults to{}(empty)target_path: The location where files will be created. It is advised to not set this on the default settings, but can be overriden on the objects attribute per-model. Defaults to""(empty).Themodelselement is a list ofmodelelements, each of whic has the following structure:version:2dbtvault:models:-name:clientsmodel_type:stageoptions:use_prefix:falsetarget_path:"raw_vault/staging"dbtvault_arguments:include_source_columns:falsesource_model:main_db:clientDataderived_columns:SOURCE:"!1"LOAD_DATETIME:"CRM_INGESTION_TIME"null_columns:LAST_TICKET_DATE:"lastTicketDate"hashed_columns:CLIENT_HASHKEY:"CLIENT_ID"CUSTOMER_NATION_HK:-"CUSTOMER_ID"-"NATION_ID"ranked_columns:one:"two"name: specifies the base name of the generated model. Ifuse_prefix = false, the generated file will have this name, otherwise it will have the appropriate prefix prepended depending on its typemodel_type: thedbtvault-compliant model type. This field drives the yml schema validation, and so should throw errors if the overall model object is incorrectly specified.options: this specifies the generation options for the model, and inherits values from the defaults. The choices are the same.dbtvault_arguments: Implements thedbtvaultmacro API. Choices depend on the type of model implemented.You can also use!include relative/path/to/file.ymlsyntax to push config out to other files. This is handy in e.g. keeping all DBTVault config in separate ymls to avoid overloading a single.ymlfile, streamlining PRs and maintenance. All included files will still require theversionKV-pair DBT looks for, but it will be removed from any included templates. Please note; to use this functionality you'll need to adddbtvault.ymlto your.dbtignorefile:# .dbtignoredbtvault.ymlDBT does not support the!includeconstructor pattern.For further information on how to specifydbtvaultmodels, please seethe DBTVault macros documentation.