Package: dbt-unit-test
dbt-unit-test (dut)A tiny framework for testing reusable code inside of dbt modelsDBT(Data Buld Tool) is a great way to maintain reusable sql code. Withmacros, we can build parametrizable code blocks which can be used in many places while maintaining DRYness. DBT even allows us to package macros asmodulesto be maintained independently and even shared with the world as open-source code like thedbt-utilsproject.While reusable code has big advantages, it introduces a single point of failure, which requires careful testing.dbt-unit-testmakes it easy to write test cases for any code that might be reused in DBT models.Installation$ pip install dbt-unit-testor for the very latest version:$ pip install git+https://github.com/AgeOfLearning/dbt-unit-test.gitSetupFrom the root of your dbt project run:$ dut initThis will set up adbt_unit_test.ymlfile right next to yourdbt_project.yml.Now, you will need to create a dbt profile calledunit_test. This profile should use a database/schema on the system where the code to be tested is run. Ideally, at least the schema should be different than any schema that contains "real" DBT models.Quick Start$ dut runThis runs all the tests. You can select which tests to run with--tests.Anatomy of a TestThe init command installs an example unit test here (the one you just ran if you did the last session).:your_dbt_project_root/
    dbt_project.yml
    ...
    dbt_unit_test.yml
    unit_tests/
        example_test/
            expect.csv
            input.csv
            model.sqlThisexample_testis the basic setup of a unit test: A directory with 3 files:input.csv,model.sqlandexpect.csv. The files are always named like this. The directory distinguishes one test from another. Test directories can be organized within other directories, but must have unique names.Theinput.csvandexpect.csvfiles are both DBTseeds. They represent data referenced by the dbt model inmodel.sqland the data that this model is expected to produce.Themodel.sqlfile is just a dbt model. It should contain code that is used elsewhere like a DBT macro or a UDF.These files are co-located in a single directory so that each test represents a discret unit of code an all of its components are easy to find. Thedut runcommand will place these files in the appropriate locations (models/,data/) so that the they can be run by DBT.After the unit test models are run,dutwill validate that the DBT model produced bymodel.sqlis identical to the dataset inexpect.csv.Insidemodel.sqlTheexample_test/model.sqllooks like this:{{config(materialized='view')}}withbaseas(selectgrp,sum(metric)asmetric,max(batch)asbatchfrom{{ref('example_test_input')}}agroupbygrp)select*frombaseAny model/seed in the test can be referenced with{{ ref('<test_name>_<file_name>') }}. So this test uses this rubric with{{ ref('example_test_input') }}.Incremental ModelsIncremental models are often some of the trickiest to get right. This is because they rely on the state created by the previous runs to form their current run output.duthas a test pattern for this important use-case.Theinput.csvfile can have a column calledbatch, like this:batch,grp,metric1,0,101,0,101,1,101,1,102,1,102,2,102,2,103,3,10dut runwill automatically create and additional model called<test_name>_batchthat will be available withinmodel.sql.dut runruns the test models N (default 2) times configurable with--batches.On the first dbt run in ourexample_test, the model referenced by{{ ref('example_test_batch') }}will only have this data:batch,grp,metric1,0,101,0,101,1,101,1,10It only contains rows wherebatch <= 1. Subsequent dbt runs will filter less of the data wherebatch <= n,nbeing the ordinal number of the dbt run (the batch). On the final dbt run (the 2nd by default),{{ ref('example_test_batch') }}will always be equivalent to{{ ref('example_test_input') }}.Tutorial: Convertexample_testinto an incremental model.If we change the model configuration to:{{config(materialized='incremental',unique_key='grp')}}withbaseas(selectgrp,sum(metric)asmetric,max(batch)asbatchfrom{{ref('example_test_batch')}}agroupbygrp)select*frombaseOur test passes, but the model is not really incremental because the entire table is replaced on the last dbt run of thedut run. We want our model to consume only the new data on each run, as well as the previous state of the model.Let's change our model to this:{{config(materialized='incremental',unique_key='grp')}}withbaseas(selectgrp,sum(metric)asmetric,max(batch)asbatchfrom{{ref('example_test_batch')}}a{%ifis_incremental()%}wherebatch>(-- only pull new data on incremental runsselectbatchfrom{{this}}orderbybatchdesclimit1){%endif%}groupbygrp){%ifis_incremental()%}selectgrp,sum(metric)asmetric,-- sum of sumsmax(batch)asbatch-- update the most recent batch of the grpfrom(-- find any grp sums that should be updated by data in the most recent batchselectgrp,metric,batchfrom{{this}}wheregrpin(selectdistinctgrpfrombase)unionall-- add on the new batch dataselectgrp,metric,batchfrombase)asagroupbygrp{%else%}-- on a full refresh, just run the old queryselect*frombase{%endif%}Great! Now we have an incremental model that only requires the newest raw data, and the test passes.Wait, what are we testing?We have just proven that this incremental model operates the way we expect, but we need to be able to reuse this code. Let's package this as a macro:macros/incremental_sum.sql{%macroincremental_sum(reference,metric_field,group_field,load_time_field)%}withbaseas(select{{group_field}},sum({{metric_field}})as{{metric_field}},max({{load_time_field}})as{{load_time_field}}from{{reference}}a{%ifis_incremental()%}where{{load_time_field}}>(-- only pull new data on incremental runsselect{{load_time_field}}from{{this}}orderby{{load_time_field}}desclimit1){%endif%}groupby{{group_field}}){%ifis_incremental()%}select{{group_field}},sum({{metric_field}})as{{metric_field}},-- sum of sumsmax({{load_time_field}})as{{load_time_field}}-- update the most recent {{ load_time_field }} of the {{ group_field }}from(-- find any {{ group_field }} sums that should be updated by data in the most recent {{ load_time_field }}select{{group_field}},{{metric_field}},{{load_time_field}}from{{this}}where{{group_field}}in(selectdistinct{{group_field}}frombase)unionall-- add on the new {{ load_time_field }} dataselect{{group_field}},{{metric_field}},{{load_time_field}}frombase)asagroupby{{group_field}}{%else%}-- on a full refresh, just run the old queryselect*frombase{%endif%}{%endmacro%}The generic version of this code is not so easy on the eyes, but what is important is that the interface is nice and clean:example_test/model.sql{{config(materialized='incremental',unique_key='grp')}}{{incremental_sum(reference=ref('example_test_batch'),metric_field='metric',group_field='grp',load_time_field='batch')}}Now we can write many tests for this reusable code, and use it with confidence because we have tested it thoroughly!