Package: dbt-exposures-crawler
dbt exposures crawlerAutomates the creation of dbt exposures from different sources. Currently, only Tableau workbooks using Snowflake SQL
are supported as a source.This project is in an ALPHA stage. Internal and external APIs might change between minor versions.Please reach out if you try this at your own organization. Feedback is very appreciated, and we
would love to hear if you had any issues setting this up at your own.Version compatibilityThis project requires Python 3.8+. The latest published version of this package has been tested internally (at Voi) with
dbt 1.x, Tableau Server 2023.1 and Snowflake SQL dialect.Here is a summary of which versions from dbt and Tableau have been tested internally for recent versions of this
package, but it doesn't mean that other dbt and Tableau versions will not work.dbt-exposures-crawlerdbt versionTableau versionSQL dialect0.1.4 - 0.1.51.1 - 1.4Tableau Server 2022.1 - 2023.1SnowflakeInstallationYou can install the latest version of this package from PyPI by running the command below.$pipinstalldbt-exposures-crawlerUsageInternally, we use this automation at Voi as part of our dbt docs release pipeline. We have a GitHub Action that does
the following:Clone our dbt repository;Install dbt and our dependencies;Rundbt docs generate(it should be run against the production environment);Run this project (using themanifest.jsongenerated from the previous command as input);Publish the generated documentation portal;To run this project, we use:$python3-mexposurescrawler.crawlers.tableau\--manifest-path=~path/to/dbt/target/manifest.json\--dbt-package-name="your_dbt_package_name"\--tableau-ignore-projectsArchive\--verboseMake sure you check the.env.examplefile to see which environment variables must be defined.Project motivationdbtis an open-source tool to manage data transformations in SQL. It automatically generates
a documentation portal from your project which includes a dependency lineage graph. It is possible to add external
downstream dependencies to this graph (such as a dashboard on a Business Intelligence tool) through a dbt feature calledexposures, which are normally defined throughyamlfiles.This project automates the creation of exposures by implementing crawlers to parse the metadata of downstream tools.
Currently, only Tableau dashboards are supported.A few use cases on how having exposures can help:analysts working on a model can use the exposures to perform impact analysis and see which reports might be impacted
by their changes;report consumers can find their report on dbt and see which models are used and read their documentation;report consumers can find which other reports are using the same models as their favorite reports.How it worksThis Python package will, in summary:Retrieve dbt models and sources frommanifest.json;Extract metadata (custom SQL and table references) from Tableau workbooks using their GraphQL API;Try to find occurrences of the dbt models and sources in the Tableau SQL;Use the Tableau REST API to retrieve additional information about the workbooks (author, project, etc);Create the dbt exposures (in-memory) and write them back to themanifest.json.More in-depth explanation:First, you must provide the path to a dbt projectmanifest.
The metadata and fully qualified names (database, schema and object name) are extracted from all dbt models and sources
represented in the manifest. The combination of dbt models and dbt sources will from now on be referred as dbt nodes.Next, workbook metadata is extracted from Tableau using
theirMetadata API, including workbooks that use
custom SQL queries and workbooks that don't (which are referred in this project as "native SQL" workbooks). Note that
this API is included in all Tableau licenses (i.e. it does not require the Data Management Add-on), but must
bemanually enabledif you host your own Tableau Server.The SQL from the custom SQL workbooks and the table names from the native SQL workbooks are normalized through simple
heuristics, such as removing quotes and converting the custom SQL to lowercase. With the normalized SQL and
normalized table names from Tableau, and the fully qualified names for the dbt nodes available, this package tries
to find the occurrences of the latter in the former.The result is a mapping of workbooks and which dbt nodes they depend on. For every workbook (with mapped
dependencies available), extra metadata that was not available in the Metadata API is then retrieved from Tableau
using theirREST API, including when the
workbook was created, when it was last updated, to which folder it belongs to on Tableau and information about its
author.As a final step, the information above is written back in the providedmanifest.jsonin the form of exposures. Note
that instead of generating.yamlfiles on disk for each exposure, the exposures are written directly on
themanifest.json, which can then be used to publish the dbt documentation portal.ExampleTo better understand how this package works, let's take as an example
thejaffle_shopdbt sample project. It has, among other models,
acustomersand anordersmodel.Now suppose that you company has 4 workbooks on Tableau:Customers workbook: accesses thecustomersdbt model through custom SQL;Company KPIs workbook: accesses both models through custom SQL;Orders workbook: accesses theordersmodel without custom SQL;Unrelated workbook: a workbook that does not use the dbt project but instead has a static data source.When running this project, you would get the following console output:Themanifest.jsonthat you provided would have 3 new exposures added to it, such as:Those exposures can then be visualized through your dbt documentation portal, either by finding which exposures are
downstream dependencies of a certain model:Or by doing the inverse. Starting from an exposure, find which models are used on it:This example has been taken from the integration tests available in this project. You can read more in theTestingsection below.Features, assumptions and limitationsA folder name can be provided to ignore workbooks that belong to it. For example, if you have a folder calledArchiveon Tableau, you can pass--tableau-ignore-projects Archiveto ignore all workbooks that belong to it;For now, only Tableau workbooks (and not published data sources) are supported. Also, only Snowflake SQL is currently
supported. Custom SQL must use Snowflake fully qualified names (i.e.database.schema.object);Workbooks that are created under
Tableau'sPersonal spacesare ignored (since they usually not governed nor production-ready).Related projectsIf you use Metabase, thedbt-metabaseopen-source project
offers similar functionality to this project, but extracting exposures from Metabase dashboards instead of Tableau.DevelopmentClone the repository and install it in editable mode:$pipinstall-e.Before opening a pull request, make sure you run:make lint: runsmypy,blackandflake8;make test: runs all testsArchitectureThe entry point for the crawlers should be on thecrawlersmodule. For now, only Tableau is supported.Thetableaumodule contains all API clients (REST and GraphQL) and models.Thedbtmodule contains a model for representing a dbt exposure and utilities for parsing, interacting and saving dbt
manifests.Finally, theutilsmodule has functions for logging and string parsing.TestingFor the integration tests, we use a samplemanifest.jsonas a fixture. It was manually generated from
thejaffle_shop, an official dbt sample project. The following
steps can be used to recreate it or to generate a new manifest on a more recent dbt version.First, the dbt sample project should be cloned and dbt needs to be installed:$gitclonehttps://github.com/fishtown-analytics/jaffle_shop
$cdjaffle_shop
$pipenvshell
$pipinstalldbt-snowflake==1.4.4Then, a newprofiles.ymlshould be created in the same folder with the following configuration:jaffle_shop:target:devoutputs:dev:type:snowflakedatabase:sample_dbaccount:...user:...private_key_path:...private_key_passphrase:...warehouse:...role:...schema:...Afterward, the project can be compiled by running:$dbtcompile--targetdevThe generatedmanifest.jsonshould then be prettified and copied to thetests/_fixturesfolder in this repository.
You can also manually remove themacrosentries from the file since they are not needed and they make the file
much larger and harder to be manually inspected$cattarget/manifest.json|jq>$PROJECT_ROOT/tests/_fixtures/manifest.jsonFuture ideasAllow filters to be passed. E.g. only include Tableau workbooks with certain tags;Add support to Tableau published data sources;ContributingWe are open and would love to have contributions, both in Pull Requests but also in ideas and feedback. Don't hesitate
to create an Issue on this repository if you are trying this project in your organization or have anything to share.ReleaseThere is a GitHub Action that will trigger a release of this package on PyPI based on releases created on GitHub.
Steps:Loosely followsemantic versioningRemember to pretend the tag name withvUse the tag name as the release title on GitHubUse the auto-generated release notes from GitHubAppend a link at the end of the release notes to the released version on PyPILicenseThis project is licensed under the Apache License, Version 2.0:http://www.apache.org/licenses/LICENSE-2.0.