Package: dbt-light
Lightweight data build tooldbt-lightis a library for developing transformations inside data warehouse. It handles the T in ELT processes by turning select queries templated withjinja2into tables/views without the need to write and apply ddl as it will automatically handle objects creation and addition of new columns.The purpose of DWH is to store history,dbt-lightdoes that by using highly configurable snapshots that implement SCD2. They can handle CDC full/delta loads, events and store relationships.Exampledbt-lighthas a simple API that allows object creation by its name in project directory:>>>fromdbt_light.modelimportModel>>>fromdbt_light.snapshotimportSnapshot>>>fromdbt_light.seedimportSeed>>>Model('customer_data').materialize()>>>Seed('customer_ref').materialize()>>>Snapshot('customer_snapshot').materialize()InstallationUsepip:pip install dbt-lightThen execute the module to create new dbt project:python -m dbt-lightThis will create ~/.dbt-light/profiles.yaml if it doesn't exist already. This file contains all dbt projects with paths to their folders as well as connection parameters to target db (only postgres is supproted now).Example ofprofiles.yamlfile:dbt-light_project:path:/home/user/dbt-light_projectadapter:postgresdbname:postgreshost:localhostpass:passport:5432user:postgresThis will also create template folder with dbt-light project in your current directory with the following structure:├──dbt-light_project
│├──models
│├──incremental_models
│├──snapshots
│├──seeds
│├──macrosConfigurationAll configuration is done in yaml files separately for each type of object (models, snapshots, seeds, sources) in project directory. There's no need to create these files for objects you don't use. There are three ways to define configuration properties: for all objects at the top of yaml file, for objects which name follows specific pattern withpattern_namekey and for specific object withnamekey. These methods can be combined, in this case more specific property will be chosen.For example, if there's amodels.yamlfile in project's directory such as:materialization:tablemodels:-pattern_name:"vw_.*"materialization:view-name:vw_modelmaterialization:tableAll models will be created as tables unless they follow the pattern "vw_.*" except for model with the name "vw_model".Basic classesThere are three main classes:Model,SnapshotandSeed. They can be created by passing object's name and if there are more than one projects defined inprofiles.yamlyou will also need to passdbt_projectparam with the name of the project. All object types havefull_refreshconstructor parameter which is False by default.These classes have methodmaterializeto load the object in database. By default, object (and its schema if not exists) gets created on the first run and insert (sometimes truncate before that) is performed on all consecutive runs. Iffull_refreshis passed as True thanDROP CASCADEwill be used which will also drop all objects that depend on it. Models withviewmaterialization are dropped with cascade every time even iffull_refreshis False.dbt-lightwill handle adding new fields without loading objects withfull_refreshset to True. Sofull_refreshcan be used to recreate objects to delete fields that are no longer used, by defaultnullwill be inserted into them. Or it can be used to recreate objects that have state like snapshots and incremental models but in this case all the history will be lost.Jinja variables can be used to reference another object by its name.dbt-lightwill interpolate schema when rendering the object.ModelsModelsconfiguration is specified inmodels.yamlbut some properties are implied. For example, target schema is defined by the subfolder name in models/incremental_models in which sql query resides. Whether model's incremental or not is defined by the parent folder name (models or incremental_models). By default, model will be created as table unless materialization property is defined as view inmodels.yaml.For incremental models there's a way to filter new rows added to it. Jinja variablemodel_existcan be used to determine whether model already exists and model itself can be referenced with jinja variablethis. Incremental models can have sequence key which gets created ifincr_keyproperty is defined for the model which can be used to generate integer surrogate keys. However, it is much better to use hash keys.For example:select*from{{some_model}}{%ifmodel_exist%}wherecustomer_idnotin(selectcustomer_idfrom{{this}}){%endif%}SnapshotsSnapshotsimplement SCD2 (Slowly Changing Dimensions) to identify how row changes over time. If you're building snapshot based on another object you can set propertymodelinsnapshots.yaml, that way you don't need to create .sql file for that snapshot. Because of that unlike models target schema is defined in configuration file with the keytarget_schemarather than implied by the subfolder name. Unique key or keys should be defined withkey_fieldsfor each snapshot.target_schemaandkey_fieldsare two only required properties. But .sql file should exist insnapshots/ormodelshould be defined otherwiseNoInputSpecifiedErrorwill be raised.Methodmaterializecan be used to load a snapshot as well as methodsdelta_calcanddelta_applyif there's a need to calculate and apply changes in separate steps. Methoddelta_calcwill create delta table containing all changes anddelta_applywill load changes from that table. In order to use these methodsdelta_schemaanddelta_tableshould be defined for that model otherwise temporary table is used when calculating delta which will not exist when callingdelta_apply. Delta table has some additional columns includingprocessed_dttm,hash_diff(if there are data columns) anddiff_typewhich indicate the type of change. There are four diff types:NO - new objectNV - new value of existing objectDO - deleted objectDU - direct update (when source timestamp hasn't changed but hash sum's changed)There are two ways of detecting changes:check(used by default) andtimestamp.Timestampwill detect changes by comparingupdated_at_fieldtimestamp which indicates if a row has changed in source system. It gives better performance but should be used only if timestamp is reliable. If changes were made butupdated_at_fielddidn't change, new version won't be created or if timestamp changed but no actual changes were made, new version will be created anyway.Checkstrategy detects changes by comparing hash sums of data columns. By default, all data columns are used but you can specify columns indata_fieldsor setignored_data_fieldsto exclude some of them. Hash sum will be stored in column with the namehash_diffbut you can change the name withhash_diff_field.dbt-lightuses md5 to calculate hash sum but you can calculate it in advance using your own method and ifhash_diff_fieldis in input table it will be used instead of calculating a new one.There can be no data columns at all. If you're modeling many-to-many relationship you may find it useful to create snapshot to see when relationship started and ended (when row was deleted from source). In this casehash_diff_fieldwill not be created.Checkstrategy can also be used withupdated_at_field. In this case you avoid some problems with unreliable timestamp that occur when usingtimestampstrategy. New version will not be created ifupdated_at_fieldchanged and data columns didn't. And if changes were made butupdated_at_fielddidn't change row will be updated directly without creating new version (DU diff type in delta table).Ifupdated_at_fieldis not specified thenprocessed_field(processed_dttmby default) is used to create date intervals using current timestamp. If you want to use other timestamp, for example, timestamp when a row was extracted from source to a staging area than you can specify it inprocessed_fieldproperty and it will be used instead of current datetime.updated_at_fieldcan also be used for this purpose but it'll be less clear asupdated_at_fieldis used for source timestamps.Rows that are absent in input table but exist in snapshot by default will get invalidated unlessinvalidate_hard_deletesis set to False.processed_fieldorupdated_at_fieldwill be used ifdeleted_flgis defined which tells us when exactly it was deleted from source.nullis used for effective records but you can specify a timestamp inmax_dttm. Fields that hold date intervals are calledeffective_from_dttmandeffective_to_dttmbut that can be changed withstart_fieldandend_field.
SeedsTimestampstrategy can be used to load data from CDC source. If you load data from table containing Change Data Capture events like Insert, Update, Delete,dbt-lightwill handle situations like multiple versions for the same row in one portion of data and deletions based on Delete events and not on the absence of a row. In order for it to work you must specifydeleted_flgwheredbt-lightwill search for Delete events. By default it will check for 'Delete' values but that can be changed withdeleted_flg_valproperty. You might also need to setinvalidate_hard_deletesto False so it won't invalidate rows based on the absence of it in data portion.deleted_flgcan be used not only for CDC data but for invalidating rows based on some value, for example, if you haveis_activecolumn which if set toNmeans it was deleted from source and should be invalidated in snapshot instead of creating new version.SeedsSeedsare csv files containing static data that rarely changes. Configuration is specified inseeds.yamlbut schema is implied by the subfolder inseeds/. By default comma is used as a delimeter but it can be changed indelimiterproperty.Column names and types should be specified under columns for each seed:seeds:-name:seedcolumns:-name:country_codetype:varchar(2)-name:country_nametype:varchar(32)SourcesSourcesare tables containing data loaded by your DI tools. They're are used to interpolate source schema in models that reference them. Each source hasname,schemaandtables. Optionallyquotingcan be specified either for source or specific table which is False by default.You can use {{ some_source.some_table }} to reference source in the example below:sources:-name:some_sourceschema:etl_stgtables:-name:some_tableTestsTestsare assertions that you make about your data. They're used instead of constraints to alert you that something went wrong and possibly make rollback. Tests are executed right after object is materialized and ifon_test_failiserror_with_rollbackrollback is done or not if it's justerror.Tests can be defined forModels,SnapshotsandSeedsincolumnskey:models:-name:some_tableon_test_fail:error_with_rollbackcolumns:-name:some_idtests:-not_null-unique-name:some_typetests:-accepted_values:values:["1type","2type"]-name:some_fktests:-relationships:to:other_modelfield:other_idAll existing tests are showed in the example above. Custom tests are not yet supported.MacrosYou can use jinja macros in models/snapshots. They're defined as usual jinja macros and stored inmacros/:{%macrosurrogate_key(bus_key,sur_key)-%}casewhen{{bus_key}}isnotnullthensource_system_cd||'_'||row_idelsenullendas{{sur_key}}{%-endmacro%}StatementFunctionstatementis a way to perform select query from template and return results back to jinja context. Statement will return either a list if only one column was selected or list of tuples where tuple represents a row:{%setcountry_codes=statement('select country_code from {{ some_seed }}')%}select*from{{some_model}}wherecountry_codein({%forccincountry_codes%}'{{ cc }}'{{', 'ifnotloop.last}}{%endfor%})