Package: dbt-starrocks
dbt-starrocksThis project isunder development.Thedbt-starrockspackage contains all the code to enabledbtto work withStarRocks.This is an experimental plugin:We have not tested it extensivelyRequires StarRocks version 2.5.0 or higherversion 3.1.x is recommendedStarRocks versions 2.4 and below are no longer supportedInstallationThis plugin can be installed via pip:$pipinstalldbt-starrocksSupported featuresStarrocks <= 2.5Starrocks 2.5 ~ 3.1Starrocks >= 3.1Feature✅✅✅Table materialization✅✅✅View materialization❌❌✅Materialized View materialization❌✅✅Incremental materialization❌✅✅Primary Key Model✅✅✅Sources✅✅✅Custom data tests✅✅✅Docs generate❌❌✅Expression Partition❌❌❌KafkaNoticeWhen StarRocks Version < 2.5,Create table ascan only set engine='OLAP' and table_type='DUPLICATE'When StarRocks Version >= 2.5,Create table assupports table_type='PRIMARY'When StarRocks Version < 3.1 distributed_by is requiredProfile ConfigurationExample entry for profiles.yml:starrocks:
  target: dev
  outputs:
    dev:
      type: starrocks
      host: localhost
      port: 9030
      schema: analytics
      username: your_starrocks_username
      password: your_starrocks_passwordOptionDescriptionRequired?ExampletypeThe specific adapter to useRequiredstarrockshostThe hostname to connect toRequired192.168.100.28portThe port to useRequired9030schemaSpecify the schema (database) to build models intoRequiredanalyticsusernameThe username to use to connect to the serverRequireddbt_adminpasswordThe password to use for authenticating to the serverRequiredcorrect-horse-battery-stapleversionLet Plugin try to go to a compatible starrocks versionOptional3.1.0Exampledbt seed properties(yml):Complete configuration:models:
  materialized: table                   // table or view or materialized_view
  engine: 'OLAP'
  keys: ['id', 'name', 'some_date']
  table_type: 'PRIMARY'                 // PRIMARY or DUPLICATE or UNIQUE
  distributed_by: ['id']
  buckets: 3                            // default 10
  indexs=[{ 'columns': 'idx_column' }]  
  partition_by: ['some_date']
  partition_by_init: ["PARTITION p1 VALUES [('1971-01-01 00:00:00'), ('1991-01-01 00:00:00')),PARTITION p1972 VALUES [('1991-01-01 00:00:00'), ('1999-01-01 00:00:00'))"]
  // RANGE, LIST, or Expr partition types should be used in conjunction with partition_by configuration
  // Expr partition type requires an expression (e.g., date_trunc) specified in partition_by
  order_by: ['some_column']             // only for PRIMARY table_type
  partition_type: 'RANGE'               // RANGE or LIST or Expr Need to be used in combination with partition_by configuration
  properties: [{"replication_num":"1", "in_memory": "true"}]
  refresh_method: 'async'               // only for materialized view default manualdbt run config:Example configuration:{{ config(materialized='view') }}
{{ config(materialized='table', engine='OLAP', buckets=32, distributed_by=['id']) }}
{{ config(materialized='table', indexs=[{ 'columns': 'idx_column' }]) }}
{{ config(materialized='table', partition_by=['date_trunc("day", first_order)'], partition_type='Expr') }}
{{ config(materialized='table', table_type='PRIMARY', keys=['customer_id'], order_by=['first_name', 'last_name'] }}
{{ config(materialized='incremental', table_type='PRIMARY', engine='OLAP', buckets=32, distributed_by=['id']) }}
{{ config(materialized='materialized_view') }}
{{ config(materialized='materialized_view', properties={"storage_medium":"SSD"}) }}
{{ config(materialized='materialized_view', refresh_method="ASYNC START('2022-09-01 10:00:00') EVERY (interval 1 day)") }}For materialized view only support partition_by、buckets、distributed_by、properties、refresh_method configuration.Read From CatalogFirst you need to add this catalog to starrocks. The following is an example of hive.CREATEEXTERNALCATALOG`hive_catalog`PROPERTIES("hive.metastore.uris"="thrift://127.0.0.1:8087","type"="hive");How to add other types of catalogs can be found in the documentation.https://docs.starrocks.io/en-us/latest/data_source/catalog/catalog_overviewThen write the sources.yaml file.sources:-name:external_exampleschema:hive_catalog.hive_dbtables:-name:hive_table_nameFinally, you might use below marco quote{{ source('external_example', 'hive_table_name') }}Test Adapterconsultthe projectContributingWe welcome you to contribute to dbt-starrocks. Please see theContributing Guidefor more information.