Package: dbt-dry-run
dbt-dry-rundbtis a tool that helps manage data transformations using templated SQL queries. These SQL queries are
executed against a target data warehouse. It doesn't check the validity of SQL queries before it executes your project.
This dry runner uses BigQuery'sdry runcapability to allow you to check that SQL queries are valid before
trying to execute them.See theblog postfor more information on how the dry runner works.QuickstartInstallationThe dry runner can be installed via pip:pip install dbt-dry-runRunningThe dry runner has a single command calleddbt-dry-runin order for it to run you must first compile a dbt manifest
usingdbt compile.How much of the project should I compile?It is best practice to compile the entire dbt project when supplying a manifest for dry run. The 
      dry run loops through your project in the DAG order (staging -> intermediate -> mart) based on `ref` and predicts the 
      schema of each model as it progresses. If you dry run `marts` but have not compiled `staging` then it cannot 
      determine if `marts` will run as it does not know the predicted schema of the upstream models and you will see 
      `NotCompiledException` in the dry run output.Then on the same machine (So that the dry runner has access to your dbt project source and themanifest.yml) you can run the dry-runner in the same directory as ourdbt_project.yml:dbt-dry-runLike dbt it will search forprofiles.ymlin~/.dbt/and use the default target specified. Just like in the dbt CLI
you can override these defaults:dbt-dry-run default --project-dir /my_org_dbt/ --profiles-dir /my_org_dbt/profiles/ --target localThe full CLI help is shown below, anything prefixed with [dbt] can be used in the same way as a normal dbt parameter:❯ dbt-dry-run --help
   Usage: dbt-dry-run [OPTIONS]
   
   Options:
     --profiles-dir TEXT             [dbt] Where to search for `profiles.yml`
                                     [default: /Users/connor.charles/.dbt]
     --project-dir TEXT              [dbt] Where to search for `dbt_project.yml`
                                     [default: /Users/connor.charles/Code/dbt-
                                     dry-run]
     --vars TEXT                     [dbt] CLI Variables to pass to dbt
                                     [default: {}]
     --target TEXT                   [dbt] Target profile
     --target-path TEXT              [dbt] Target path
     --verbose / --no-verbose        Output verbose error messages  [default: no-
                                     verbose]
     --report-path TEXT              Json path to dump report to
     --skip-not-compiled             Whether or not the dry run should ignore
                                     models that are not compiled. This has
                                     several caveats that make this not a
                                     recommended option. The dbt manifest should
                                     generally be compiled with `--select *` to
                                     ensure good  coverage
     --full-refresh                  [dbt] Full refresh
     --extra-check-columns-metadata-key TEXT
                                     An extra metadata key that can be used in
                                     place of `dry_run.check_columns` for
                                     verifying column metadata has been specified
                                     correctly. `dry_run.check_columns` will
                                     always take precedence. The metadata key
                                     should be of boolean type or it will be cast
                                     to a boolean to be 'True/Falsey`
     --version
     --install-completion [bash|zsh|fish|powershell|pwsh]
                                     Install completion for the specified shell.
     --show-completion [bash|zsh|fish|powershell|pwsh]
                                     Show completion for the specified shell, to
                                     copy it or customize the installation.
     --help                          Show this message and exit.Reporting Results & FailuresIf the result is successful it will output the number of models that were tested like so:Dry running 3 models

DRY RUN SUCCESS!The process will also return exit code 0If there are failures it will print a summary table of the nodes that failed:Dry running 3 models
Node model.test_models_with_invalid_sql.second_layer failed with exception:
400 POST https://bigquery.googleapis.com/...: Column d in USING clause not found on left side of join at [6:88]

(job ID: 5e336f32-273d-480a-b8bb-cdf4fca66a98)

Total 1 failures:
1       :       model.test_models_with_invalid_sql.second_layer :       BadRequest      :       ERROR
DRY RUN FAILURE!`The process will also return exit code 1Column and Metadata LintingThe dry runner can also be configured to inspect your metadata YAML and assert that the predicted schema of your dbt
projects data warehouse matches what is documented in the metadata. To enable this for your models specify the keydry_run.check_columns: true. The dry runner will then fail if the model's documentation does not match. You can also
specify a custom extra key to enablecheck_columnsby setting the CLI argument--extra-check-columns-metadata-key.
For example the full metadata for this model:models:-name:badly_documented_modeldescription:This model is missing some columns in its docsmeta:dry_run.check_columns:truecolumns:-name:adescription:This is in the model-name:bdescription:This is in the model#      - name: c#        description: Forgot to document c-name:ddescription:This shouldn't be hereThis model is badly documented as the predicted schema is 3 columnsa,b,cthe dry runner will therefore output the
following error and fail your CI/CD checks:Dry running X models
Node model.test_column_linting.badly_documented_model failed linting with rule violations:
        UNDOCUMENTED_COLUMNS : Column not documented in metadata: 'c'
        EXTRA_DOCUMENTED_COLUMNS : Extra column in metadata: 'd'

Total 1 failures:
1       :       model.test_column_linting.badly_documented_model        :       LINTING :       ERROR
DRY RUN FAILURE!Currently, these rules can cause linting failures:UNDOCUMENTED_COLUMNS: The predicted schema of the model will have extra columns that have not been documented in the
YAMLEXTRA_DOCUMENTED_COLUMNS: The predicted schema of the model does not have this column that was specified in the
metadataUsage with dbt-external-tablesThe dbt packagedbt-external-tablesgives dbt support for staging and managingexternal tables. These sources do not produce any compiled sql in the manifest, so it is not
possible for the dry runner to predict their schema. Therefore, you must specify the resulting schema manually in the
metadata of the source.However, if thecolumnsschema is already defined under thenamein the yaml config, you do not need to specifydry_run_columnsunderexternal. The dry runner will use thecolumnsschema ifdry_run_columnsis not specified. This avoids duplicated schema definitions.For example if you were import data from a gcs bucket:version:2sources:-name:source_datasettables:-name:eventdescription:"Someeventsbucket.Ifexternalispopulatedthenthedryrunnerwillassumeitisusing`dbt-external-tables`"external:location:'gs://bucket/path/*'format:csvdry_run_columns:-name:string_fielddata_type:STRINGdescription:"Specifyeachcolumnintheyamlforexternalsources"-name:record_array_field[]data_type:RECORD[]description:"Forstruct/recordfieldsspecifythe`data_type`as`RECORD`"-name:record_array_field.foodata_type:NUMERICdescription:"Forrecordattributesusethedotnotation"-name:integer_arraydata_type:NUMERIC[]description:"Forrepeatedfieldssuffixdata_typewith[]"The dry runner cannot predict the schema, therefore, it is up to you to accurately describe the schema in the YAML otherwise
you may get false positive/negative results from the dry run.Report ArtefactIf you specify---report-patha JSON file will be outputted regardless of dry run success/failure with detailed
information of each node's predicted schema or error message if it has failed:{"success":false,"node_count":3,"failure_count":1,"failed_node_ids":["model.test_models_with_invalid_sql.second_layer"],"nodes":[{"unique_id":"seed.test_models_with_invalid_sql.my_seed","success":true,"status":"SUCCESS","error_message":null,"table":{"fields":[...]}},{"unique_id":"model.test_models_with_invalid_sql.first_layer","success":true,"status":"SUCCESS","error_message":null,"table":{"fields":[...]}},{"unique_id":"model.test_models_with_invalid_sql.second_layer","success":false,"status":"FAILURE","error_message":"BadRequest","table":null}]}Capabilities and LimitationsThings this can catchThe dry run can catch anything the BigQuery planner can identify before the query has run. Which includes:Typos in SQL keywords:selecinstead ofselectTypos in columns names:orders.produtsinstead oforders.productsProblems with incompatible data types: Trying to execute "4" + 4Incompatible schema changes to models: Removing a column from a view that is referenced by a downstream model
explicitlyIncompatible schema changes to sources: Third party modifies schema of source tables without your knowledgePermission errors: The dry runner should run under the same service account your production job runs under. This
allows you to catch problems with table/project permissions as dry run queries need table read permissions just like
the real queryIncorrect configuration of snapshots: For example a typo in theunique_keyconfig. Orcheck_colswhich do not
exist in the snapshotThings this can't catchThere are certain cases where a syntactically valid query can fail due to the data in the tables:Queries that run but do not return intended/correct result. This is checked usingtestsNULLvalues inARRAY_AGG(SeeIGNORE_NULLS bullet point)Bad query performance that makes it too complex/expensive to runThings still to do...The implementation of seeds is incomplete as we don't use them very much in our own dbt projects. The dry runner
will just use the datatypes thatagateinfers from the CSV files. It will ignore any type overrides you add in the YAML.If you see anything else that you think it should catch don't hesitate to raise an issue!LicenseCopyright 2022 Auto Trader LimitedLicensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the
License. You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "
AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific
language governing permissions and limitations under the License.