Package: dagrules
dagrulesdagrules is a tool that allows you to write your own dbt dag rules and check
that your dbt project is conforming to those rules.OverviewWhile the dbt community has established some excellent guidelines forhow to structure dbt
projects,
those conventions are not automatically enforced.  Those
conventions are simply guidelines, and each team may decide on a
slightly different set of conventions that work best for their
particular set up.  dagrules was developed to allow you to write your
own conventions in a simpleyamldocument, and have those
conventions enforced via your CI system.To use dagrules, all you need is a dbt project and adagrules.ymlfile located in the root of the dbt project (e.g.,dbt/dagrules.yml).  The yaml file should look like (for a more
complete example, seetests/dagrules.yml:---version:'1'rules:-name:The name of my rulesubject:...# How to select nodes to check that they satisfy the rulesmust:...# Define the rules that must be followed-name:Another one of my rules...Installation and running rulesdagrules can be installed using pip:pipinstalldagrulesAnd then rundagruleswith the--checkargument from your dbt project root:dagrules--checkdagrules assumes that it is being executed from the dbt project root and that there is
atarget/manifest.jsonfile already present (so the dbt project must be compiled
any time the dag is changed before dagrules can be run).  These defaults can
be overridden by setting theDBT_ROOTandDAGRULES_YAMLenvironment variable to
point to other locations.SubjectsFor every rule, a subject should be declared that defines how to
select nodes of the dbt dag to use for rule validation.  Omitting the
subject means that the rule will be applied to every dbt model.
dagrules currently supports two ways to select subjects: 1) by node
type (source, snapshot, model) and 2) by tags.  For example, the
follow subject includes all models that are tagged "staging":rules:-name:All staging models must ...subject:type:modeltags:stagingmust:...Tag selectionTag selection applies both tosubjectandmustsection of the
dagrules yaml spec.  Tags can be defined several ways.Single string- Selecting with a single tag can be expressed as a simple stringtags:stagingList of tags: match any- A list of tags can also be specified, and
dagrules will match nodes withanyof the tags in the list.  The
example below will match nodes having eitherbaseorintermediatetags.tags:-base-intermediateInclude: match all with exclusions- When you need to select nodes
that matchalltags in a list, and possibly exclude nodes with
some tags as well, you can use include/exclude.  The example below
will select any nodes that have both "staging" and "finance" tags, but
that don't also have thebasetag.tags:include:-staging-financeexclude:-baseThe arguments toincludeandexcludecan either be a list or single strings.Combine any/all- We can also combineanyandallsyntaxes
at once.  The following will select all nodes that are either
"non-base staging", "core", or "mart" models.:tags:-include:stagingexclude:base-core-martMusts"Musts" define the rules that must be adhered to by the subjects defined in thesubjectsection.  Multiple "musts" may be included in a rule definition, and all must be
satisfied for the rule to pass.Match name- Thematch-namerule requires that each subject adhere to a
particular naming pattern.  dagrules currently only supports regular expression matching.
For example, the following rule enforces that all snapshot models must be named with
asnap_prefix:rules:-name:Snapshot must be prefixed with snap_subject:type:snapshotmust:match-name:/snap_.*/Have tags- Thehave-tags-anyrule requires that all selected models must have
one of any of the listed tags.  The following example specifies that all nodes in the dag
must have at least one of the tags listed:rules:-name:All models must be tagged either snapshot, base, staging, intermediate, core, mart# Omit subject to include all nodesmust:have-tags-any:-snaphost-base-staging-intermediate-core-martHave parent or child relationship- Thehave-child-relationshipandhave-parent-relationshiprules require that the subjects have a
certain kind of relationship to either theirimmediatechildren or
parents.  The types of relationship can involve:cardinality- The cardinality of the relationship between a subject and its child/parent
can either beone_to_oneorone_to_many(default).  Ifone_to_oneis selected,
that a subject may only have one child/parent.required- Indicates whether a child/parent relationship is required or not.  The default
isTrue, meaning that if a relationship is defined, all subject must have at least
one child or parent node.  IfFalse, then a subject may have 0 children/parents.require-tags-any- Contains a list of tags that the parent/child
must have (with syntax defined in the "Tag selection" section
above).require-node-type- Indicates the node type (source, snapshot, model) that the child/parent
must be in order to pass.select-tags-any- Contains a list of tags that restricts the selection of parents/children
involved in the rule.select-node-type- Indicates that only the parents/children with the specified node
type are to be considered when checking the rule.For example,rules:-name:Snapshots must have 0 or 1 children, which must all be base modelssubject:type:snapshotmust:have-child-relationship:cardinality:one_to_onerequired:falserequire-tags-any:-base-name:Intermediate models may only depend on non-base staging, core, mart, or other intermediate modelssubject:tags:include:intermediatemust:have-parent-relationship:require-tags-any:-include:stagingexclude:base-core-mart-intermediateContributingWe welcome contributors!  Please submit any suggests or pull requests in Github.Developer setupCreate an appropriate python environment.  I likeminiconda,
but use whatever you like:conda create --name dagrules python=3.9
conda activate dagrulesThen install pip packagespip install pip-tools
pip install --ignore-installed -r requirements.txtrun tests viainv testand the linter viainv lint