Package: dbt-superset-lineage
dbt-superset-lineageMakedbtdocs andApache Supersettalk to one anotherWhy do I need something like this?Odds are rather high that you use dbt together with a visualisation tool. If so, these questions might have popped
into your head time to time:"Could I get rid of this model? Does it get used for some dashboards? And in which ones, if yes?""It would be so handy to see all these well-maintained model and column descriptions when exploring and creating charts."In case your visualisation tool of choice is Supserset, you are in luck!Usingdbt-superset-lineage, you can:Add dependencies of Superset dashboards to your dbt sources and modelsSync model and column descriptions from dbt docs to SupersetThis will help you:Avoid broken dashboards because of deprecated or changed modelsChoosing the right attributes without navigating back and forth between chart and documentationDemoThe package was presented duringCoalesce, the annual dbt conference, as a part of the talkFrom 100 spreadsheets to 100 data analysts: the story of dbt at Slido.
Watch a demo in the video below.Installationpip install dbt-superset-lineageUsagedbt-superset-lineagecomes with two basic commands:pull-dashboardsandpush-descriptions.
The documentation for the individual commands can be shown by using the--helpoption.It includes a wrapper forSuperset API, one only needs to provideSUPERSET_ACCESS_TOKEN/SUPERSET_REFRESH_TOKEN(obtained via/security/login)
as environment variable or through--superset-access-token/superset-refresh-tokenoption.N.B.Make sure to rundbt compile(ordbt run) against the production profile, not your development profileIn case more databases are used within dbt and/or Superset and there are duplicate names (schema + table) across
them, specify the database through--dbt-db-nameand/or--superset-db-idoptionsCurrently,PUTrequests are only supported if CSRF tokens are disabled in Superset (WTF_CSRF_ENABLED=False).Tested on dbt v1.4.5 and Apache Superset v2.0.1. Other versions might face errors due to different underlying code and API.Pull dashboardsPull dashboards from Superset and add them asexposuresto dbt docs with
references to dbt sources and models, making them visible both separately and as dependencies.N.B.Only published dashboards are extracted.$cdjaffle_shop$dbtcompile# Compile project to create manifest.json$exportSUPERSET_ACCESS_TOKEN=<TOKEN>$dbt-superset-lineagepull-dashboardshttps://mysuperset.mycompany.com# Pull dashboards from Superset to /models/exposures/superset_dashboards.yml$dbtdocsgenerate# Generate dbt docs$dbtdocsserve# Serve dbt docsPush descriptionsPush model and column descriptions from your dbt docs to Superset as plain text so that they could be viewed
in Superset when creating charts.N.B.:Run carefully as this rewrites your datasets using merged metadata from Superset and dbt docs.Running with--superset-refresh-columnsoverridescolumns.filterableandcolumns.groupbytotrue,
because ofthis issue.Descriptions are rendered as plain text, hence no markdown syntax, incl. links, will be displayed.Avoid special characters and strings in your dbt docs, e.g.→or<null>.$cdjaffle_shop$dbtcompile# Compile project to create manifest.json$exportSUPERSET_ACCESS_TOKEN=<TOKEN>$dbt-superset-lineagepush-descriptionshttps://mysuperset.mycompany.com# Push descrptions from dbt docs to SupersetLicenseLicensed under the MIT license (seeLICENSE.mdfile for more details).