Package: dbt-starburst
dbt-starburstIntroductiondbtis a data transformation workflow tool that lets teams quickly and collaboratively deploy analytics code, following software engineering best practices like modularity, CI/CD, testing, and documentation. It enables anyone who knows SQL to build production-grade data pipelines.One frequently asked question in the context of usingdbttool is:Can I connect my dbt project to two databases?(see the answeredquestionon the dbt website).TL;DRdbtstands for transformation as inTwithinELTpipelines, it doesn't move data from source to a warehouse.dbt-starburstadapter usesTrinoas a underlying query engine to perform query federation across disperse data sources. Trino connects to multiple and diverse data sources (available connectors) via one dbt connection and process SQL queries at scale. Transformations defined in dbt are passed to Trino which handles these SQL transformation queries and translates them to queries specific to the systems it connects to create tables or views and manipulate data.This repository represents a fork of thedbt-prestowith adaptations to make it work with Trino.CompatibilityThis dbt plugin has been tested againstTrinoversion405,Starburst Enterpriseversion402-e.0andStarburst Galaxy.InstallationThis dbt adapter can be installed via pip:$pipinstalldbt-starburstConfiguring your profileA dbt profile can be configured to run against Trino using the following configuration:OptionDescriptionRequired?ExamplemethodThe Trino authentication method to useOptional (default isnone, supported methods areldap,kerberos,jwt,oauthorcertificate)noneorkerberosuserUsername for authenticationOptional (required ifmethodisnone,ldaporkerberos)commanderpasswordPassword for authenticationOptional (required ifmethodisldap)noneorabc123impersonation_userUsername override, used for impersonationOptional (applicable ifldap)impersonated_tomrolesCatalog rolesOptionalsystem: analystkeytabPath to keytab for kerberos authenticationOptional (may be required ifmethodiskerberos)/tmp/trino.keytabkrb5_configPath to config for kerberos authenticationOptional (may be required ifmethodiskerberos)/tmp/krb5.confprincipalPrincipal for kerberos authenticationOptional (may be required ifmethodiskerberos)trino@EXAMPLE.COMservice_nameService name for kerberos authenticationOptional (default istrino)abc123mutual_authenticationBoolean flag for mutual authenticationOptional (may be required ifmethodiskerberos)falseforce_preemptiveBoolean flag for preemptively initiate the Kerberos GSS exchangeOptional (may be required ifmethodiskerberos)falsehostname_overrideKerberos hostname for a host whose DNS name doesn't matchOptional (may be required ifmethodiskerberos)EXAMPLE.COMsanitize_mutual_error_responseBoolean flag to strip content and headers from error responsesOptional (may be required ifmethodiskerberos)truedelegateBoolean flag for credential delgation (GSS_C_DELEG_FLAG)Optional (may be required ifmethodiskerberos)falsejwt_tokenJWT token for authenticationOptional (required ifmethodisjwt)noneorabc123client_certificatePath to client certificate to be used for certificate based authenticationOptional (required ifmethodiscertificate)/tmp/tls.crtclient_private_keyPath to client private key to be used for certificate based authenticationOptional (required ifmethodiscertificate)/tmp/tls.keyhttp_headersHTTP Headers to send alongside requests to Trino, specified as a yaml dictionary of (header, value) pairs.OptionalX-Trino-Client-Info: dbt-starbursthttp_schemeThe HTTP scheme to use for requests to TrinoOptional (default ishttp, orhttpsformethod: kerberos,ldaporjwt)httpsorhttpcertThe full path to a certificate file for authentication with trinoOptionalsession_propertiesSets Trino session properties used in the connectionOptionalquery_max_run_time: 4hdatabaseSpecify the database to build models intoRequiredanalyticsschemaSpecify the schema to build models into. Note: it is not recommended to use upper or mixed case schema namesRequiredpublichostThe hostname to connect toRequired127.0.0.1portThe port to connect to the host onRequired8080threadsHow many threads dbt should useOptional (default is1)8prepared_statements_enabledEnable usage of Trino prepared statements (used indbt seedcommands)Optional (default istrue)trueorfalseretriesConfigure how many times a database operation is retried when connection issues ariseOptional (default is3)10timezoneThe time zone for the Trino sessionOptional (defaults to the client side local timezone)Europe/BrusselsExample profiles.yml entry:my-trino-db:target:devoutputs:dev:type:trinouser:commanderhost:127.0.0.1port:8080database:analyticsschema:publicthreads:8http_scheme:httpsession_properties:query_max_run_time:4hexchange_compression:Truetimezone:UTCExample profiles.yml entry for kerberos authentication:my-trino-db:target:devoutputs:dev:type:trinomethod:kerberosuser:commanderkeytab:/tmp/trino.keytabkrb5_config:/tmp/krb5.confprincipal:trino@EXAMPLE.COMhost:trino.example.comport:443database:analyticsschema:publicFor reference on which session properties can be set on the the dbt profile do executeSHOWSESSION;on your Trino instance.Usage NotesSupported authentication typesnone - No authenticationldap- Specify username inuserand password inpasswordkerberos- Specify username inuserjwt- Specify JWT token injwt_tokencertificate- Specify a client certificate inclient_certificateand private key inclient_private_keyoauth- It is recommended to install keyring to cache the OAuth2 token over multiple dbt invocations by runningpip install 'trino[external-authentication-token-cache]', keyring is not installed by default.See also:https://trino.io/docs/current/security/authentication-types.htmlSession properties per modelIn some specific cases, there may be needed tuning through the Trino session properties only
for a specific dbt model.
In such cases, using thedbt hooksmay come to the rescue:{{
  config(
    pre_hook="set session query_max_run_time='10m'"
  )
}}MaterializationsTabledbt-starburstsupports two modes intablematerializationrenameanddropconfigured usingon_table_exists.rename- creates intermediate table, then renames the target to backup one and renames intermediate to target one.drop- drops and recreates a table. It overcomes table rename limitation in AWS Glue.By defaulttablematerialization useson_table_exists = 'rename', see an examples below how to change it.In model add:{{
  config(
    materialized = 'table',
    on_table_exists = 'drop`
  )
}}or indbt_project.yaml:models:path:materialized:table+on_table_exists:dropUsingtablematerialization andon_table_exists = 'rename'with AWS Glue may result in below error:TrinoUserError(type=USER_ERROR, name=NOT_SUPPORTED, message="Table rename is not yet supported by Glue service")ViewAdapter supports two security modes inviewmaterializationDEFINERandINVOKERconfigured usingview_security.SeeTrino docsfor more details about security modes in views.By defaultviewmaterialization usesview_security = 'definer', see an examples below how to change it.In model add:{{
  config(
    materialized = 'view',
    view_security = 'invoker'
  )
}}or indbt_project.yaml:models:path:materialized:view+view_security:invokerIncrementalUsing an incremental model limits the amount of data that needs to be transformed, vastly reducing the runtime of your transformations. This improves performance and reduces compute costs.{{
    config(
      materialized = 'incremental', 
      unique_key='<optional>',
      incremental_strategy='<optional>',)
}}
select * from {{ ref('events') }}
{% if is_incremental() %}
  where event_ts > (select max(event_ts) from {{ this }})
{% endif %}Use the+on_schema_changeproperty to define how dbt-starburst should handle column changes. Seedbt docs.Set the+views_enabledtofalseif your connector doesn't support views.append(default)The default incremental strategy isappend.appendonly adds the new records based on the condition specified in theis_incremental()conditional block.{{
    config(
      materialized = 'incremental')
}}
select * from {{ ref('events') }}
{% if is_incremental() %}
  where event_ts > (select max(event_ts) from {{ this }})
{% endif %}delete+insertThrough thedelete+insertincremental strategy, you can instruct dbt to use a two-step incremental approach. It will first delete the records detected through the configuredis_incremental()block and re-insert them.{{
    config(
      materialized = 'incremental',
      unique_key='user_id',
      incremental_strategy='delete+insert',
      )
}}
select * from {{ ref('users') }}
{% if is_incremental() %}
  where updated_ts > (select max(updated_ts) from {{ this }})
{% endif %}mergeThrough themergeincremental strategy, dbt-starburst constructs aMERGEstatementwhichINSERTs new andUPDATEs existing records based on the unique key (specified byunique_key).Ifunique_keyis not uniquedelete+insertstrategy can be used.
Note that some connectors in Trino have limited or no support forMERGE.{{
    config(
      materialized = 'incremental',
      unique_key='user_id',
      incremental_strategy='merge',
      )
}}
select * from {{ ref('users') }}
{% if is_incremental() %}
  where updated_ts > (select max(updated_ts) from {{ this }})
{% endif %}Incremental overwrite on hive modelsIn case that the target incremental model is being accessed withhiveTrino connector, aninsert overwritefunctionality can be achieved when using:<hive-catalog-name>.insert-existing-partitions-behavior=OVERWRITEsetting on the Trino hive connector configuration.Below is a sample hive profile entry to deal withOVERWRITEfunctionality for the hive connector calledminio:trino-incremental-hive:target:devoutputs:dev:type:trinomethod:noneuser:adminpassword:catalog:minioschema:tinyhost:localhostport:8080http_scheme:httpsession_properties:minio.insert_existing_partitions_behavior:OVERWRITEthreads:1Existing partitions in the target model that match the staged data will be overwritten.
The rest of the partitions will be simply appended to the target model.NOTE that this functionality works on incremental models that use partitioning:{{
    config(
        materialized = 'incremental',
        properties={
          "format": "'PARQUET'",
          "partitioned_by": "ARRAY['day']",
        }
    )
}}Materialized viewThe adapter also supportsmaterialized views.
At every subsequentdbt runcommand, the materialized view isrefreshed.You can also define custom properties for the materialized view through thepropertiesconfig.This materialization supports thefull_refreshconfig and flag.
Whenever you want to rebuild your materialized view, e.g. when changing underlying SQL query, rundbt run --full-refresh.In model add:{{
  config(
    materialized = 'materialized_view',
    properties = {
      'format': "'PARQUET'"
    },
  )
}}or indbt_project.yaml:models:path:materialized:materialized_viewproperties:format:"'PARQUET'"SnapshotsCommonly, analysts need to "look back in time" at some previous state of data in their mutable tables. While some source data systems are built in a way that makes accessing historical data possible, this is often not the case. dbt provides a mechanism, snapshots, which records changes to a mutable table over time.Snapshots implement type-2 Slowly Changing Dimensions over mutable source tables. These Slowly Changing Dimensions (or SCDs) identify how a row in a table changes over time. Imagine you have an orders table where the status field can be overwritten as the order is processed.See also the dbt docs about snapshots.An example is given below.{% snapshot orders_snapshot %}
{{
    config(
        target_database='analytics',
        target_schema='snapshots',
        unique_key='id',
        strategy='timestamp',
        updated_at='updated_at',
    )
}}
select * from {{ source('jaffle_shop', 'orders') }}
{% endsnapshot %}Note that the Snapshot feature depends on thecurrent_timestampmacro. In some connectors the standard precision (TIMESTAMP(3) WITH TIME ZONE) is not supported by the connector eg. Iceberg.If necessary, you can override the standard precision by providing your own version of thetrino__current_timestamp()macro as in following example:{% macro trino__current_timestamp() %}
    current_timestamp(6)
{% endmacro %}Use table properties to configure connector specificsTrino connectors use table properties to configure connector specifics.Check the Trino connector documentation for more information.{{
  config(
    materialized='table',
    properties={
      "format": "'PARQUET'",
      "partitioning": "ARRAY['bucket(id, 2)']",
    }
  )
}}SeedsSeeds are CSV files in your dbt project (typically in your data directory), that dbt can load into your data warehouse using the dbt seed command.For dbt-starburst batch_size is defined in macrotrino__get_batch_size()and default value is1000.
In order to override default value define within your project a macro like the following:{% macro default__get_batch_size() %}
  {{ return(10000) }}
{% endmacro %}Persist docsPersist docs optionally persist resource descriptions as column and relation comments in the database. By default, documentation persistence is disabled, but it can be enabled for specific resources or groups of resources as needed.Detailed documentation can be foundhere.Generating lineage flow in docsIn order to generate lineage flow in docs usereffunction in the place of table names in the query. It builts dependencies between models and allows to create DAG with data flow. Refer to exampleshere.dbtdocsgenerate# generate docsdbtdocsserve--port8081# starts local server (by default docs server runs on 8080 port, it may cause conflict with Trino in case of local development)Using Custom schemasBy default, all dbt models are built in the schema specified in your target. But sometimes you wish to build some of the models in a custom schema. In order to do so, use theschemaconfiguration key to specify a custom schema for a model. Seeherefor the documentation. It is important to note that by default, dbt will generate the schema name for a model by concatenating the custom schema to the target schema, as in:<target_schema>_<custom_schema>.Prepared statementsThedbt seedfeature usesTrino's prepared statements.Python's http client has a hardcoded limit of 65536 bytes for a header line.When executing a prepared statement with a large number of parameters, you might encounter following error:requests.exceptions.ConnectionError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line')).The prepared statements can be disabled by settingprepared_statements_enabledtotruein your dbt profile (reverting back to the legacy behavior using Python string interpolation). This flag may be removed in later releases.GrantsPlease note that grants are only supported inStarburst EnterpriseandStarburst Galaxyand Hive (sql-standard).You can manage access to the datasets you're producing with dbt by using grants. To implement these permissions, define grants as resource configs on each model, seed, or snapshot. Define the default grants that apply to the entire project in your dbt_project.yml, and define model-specific grants within each model's SQL or YAML file.models:-name:specific_modelconfig:grants:select:['reporter','bi']Read everything about grants in thedbt docs.ContributingWant to report a bug or request a feature? Let us know onSlackin the#db-presto-trinochannel, or openan issueWant to help us build dbt-starburst? Check out theContributing GuideRelease processBefore doing a release, it is required to bump the dbt-starburst version by triggering release workflowversion-bump.yml. The major and minor part of the dbt version are used to associate dbt-starburst's version with the dbt version.Next step is to merge the bump PR and making sure that test suite pass.Finally, to releasedbt-starburstto PyPi and GitHub trigger release workflowrelease.yml.Backport processSometimes it is necessary to backport some changes to some older versions. In that case, create branch fromx.x.latestbranch. There is ax.x.latestfor each minor version, e.g.1.3.latest. Make a fix and open PR back tox.x.latest. Create changelog bychangie newas ususal, as separate changlog for each minor version is kept on everyx.x.latestbranch.
After merging, to make a release of that version, just follow instructions fromRelease processsection, but run every workflow onx.x.latestbranch.Code of ConductEveryone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected
to follow thePyPA Code of Conduct.