Package: dbt-bytehouse
Introductiondbt(Data Building Tool) is an open source tool that enables data analysts and engineers to transform
data in their warehouses simply by writing select statements.dbtperforms the T (Transform) of ETL and
allows companies to write transformations as queries and orchestrate them in a more efficient way.
ByteHouse dbt connector is a plugin enabling users to build their data warehouse ecosystem with dbt
and ByteHouse.Table of ContentsIntroductionRequirementsCreating ByteHouse AccountInstallationdbt Project Setupdbt_project.ymlprofiles.ymlConnection & Authentication ConfigurationsByteHouse RegionsRegion & Password ConfigurationRegion & API Key ConfigurationHost Address & Password ConfigurationHost Address & API Key ConfigurationProject InitializationTest Warehouse ConnectionDataset Ingestiondbt Modelsschema.ymlMaterialization types of ModelsView MaterializationsTable MaterializationsIncremental MaterializationsHow it worksProject DocumentationLocal DevelopmentOriginal AuthorLicenseRequirementsMake sure you havedbt&pythoninstalled on your machine. If not, then you can follow this guidehttps://docs.getdbt.com/docs/get-started/installationdbt v1.3.0 or greaterpython v3.7 or greaterCreating ByteHouse AccountYou need to create ByteHouse account in order to use bytehouse-dbt connector. You can simply create a free account with
the process mentioned in our official website documentation:https://docs.bytehouse.cloud/en/docs/quick-startYou can also create ByteHouse account through Volcano Engine by ByteDance:https://www.volcengine.com/product/bytehouse-cloudInstallationCreate a new repository where we will instantiate aPythonvirtual environment.mkdir dbt_bytehouse_demo
cd dbt_bytehouse_demo

python -m venv venv
source venv/bin/activateLatest release version can be installed from here:pip install dbt-bytehouseCurrent development version can be installed from here:pip install git+https://github.com/bytehouse-cloud/bytehouse-dbt@master#egg=bytehouse-driverCheck whether installation is successful by verifying bytehouse is available under Plugins.dbt --versiondbt Project Setupdbt_project.ymlEverydbtproject needs adbt_project.ymlfile — this is howdbtknows a directory is adbtproject.dbt_project.ymlfile holds the context of your project and tellsdbthow to build your data sets. Some common configurations
fordbt_project.ymlare:YAML keyValuenameYour project’s name in snake caseversionVersion of your projectprofileThe profile dbt uses to connect to ByteHousemodel-pathsDirectories to where your model and source files liveseed-pathsDirectories to where your seed files livetest-pathsDirectories to where your test files livesnapshot-pathsDirectories to where your snapshots livedocs-pathsDirectories to where your docs blocks liveprofiles.ymlWhen you invokedbtfrom the command line,dbtparses yourdbt_project.ymland obtains the profile name.dbtthen
checks yourprofiles.ymlfile for a profile with the same name. A profile contains all the details/credentials
required to connect to ByteHouse.dbtwill search the current working directory for theprofiles.ymlfile and will
default to the~/.dbt/ directoryif not found.<profile-name>:target:<target-name>outputs:<target-name>:type:bytehouseschema:<database-name>user:<username>password:<password>driver:native#optional fieldshost:<hostname>port:<port>region:<region-name>account:<account-name>warehouse:<warehouse-name>retries:1secure:Trueconnect_timeout:10send_receive_timeout:300custom_settings:<empty>YAML keyValue<profile-name>Name of the profile. Has to be the same name as the profile indicated in your dbt_project.yml filetargetDefault target your dbt project will use. It must be one of the targets you define in your profiletypeMust be set to bytehouseschemaDatabase nameuserUsername with adequate permissions to access the specified schema. For API Key authentication, user must be set to bytehousepasswordThe password associated with the specified userdriverMust be set to nativehost[Optional] The host name of the connectionport[Optional] The port number of the host serverregion[Optional] Alias for host & portaccount[Optional] ByteHouse account numberwarehouse[Optional] The name of the virtual warehouse that you want to use for this sessionretries[Optional] Number of times to retry the initial connection attempt if the error appears to be recoverablesecure[Optional] Whether the connection is secured by TLS. Suggested to set it to Trueconnect_timeout[Optional] Connection timeout in seconds. Default is 10 secondssend_receive_timeout[Optional] Timeout for receiving data from or sending data to ByteHouse. Default is 5 minutes (300 seconds)custom_settings[Optional] A mapping of ByteHouse specific user settings to use with the connectionConnection & Authentication ConfigurationsByteHouse RegionsCurrently, the driver supports the following region names across different cloud providers. Alternatively, if you know
the host address of ByteHouse server, you can directly use host address & omit region name.Region NameTarget ServerAP-SOUTHEAST-1gateway.aws-ap-southeast-1.bytehouse.cloud:19000VOLCANO-CN-NORTH-1bytehouse-cn-beijing.volces.com:19000Region & Password ConfigurationRequired parameters:accountuserpasswordregionbytehouse_profile:target:devoutputs:dev:type:bytehousedriver:native# databaseschema:$DATABASE_NAME# target server addressregion:$REGION_NAME# account credentialsaccount:$ACCOUNT_NAMEuser:$USER_NAMEpassword:$PASSWORD# additional settingssecure:TrueRegion & API Key ConfigurationRequired parameters:regionuserpasswordbytehouse_profile:target:devoutputs:dev:type:bytehousedriver:native# databaseschema:$DATABASE_NAME# target server addressregion:$REGION_NAME# account credentialsuser:bytehousepassword:$API_KEY# additional settingssecure:TrueHost Address & Password ConfigurationRequired parameters:hostportaccountuserpasswordbytehouse_profile:target:devoutputs:dev:type:bytehousedriver:native# databaseschema:$DATABASE_NAME# target server addresshost:$HOST_ADDRESSport:$PORT_NUMBER# account credentialsaccount:$ACCOUNT_NAMEuser:$USER_NAMEpassword:$PASSWORD# additional settingssecure:TrueHost Address & API Key ConfigurationRequired parameters:hostportuserpasswordbytehouse_profile:target:devoutputs:dev:type:bytehousedriver:native# databaseschema:$DATABASE_NAME# target server addresshost:$HOST_ADDRESSport:$PORT_NUMBER# account credentialsuser:bytehousepassword:$API_KEY# additional settingssecure:TrueProject Initializationdbt initcommand will prompt for project name & database adapters, where you have to select bytehouse. This will create
a new folder with your project name, sample files &dbt_project.ymlconfig file, enough to get you started with dbt.dbt initUpdate your profiles.yml with required authentication & target server credentials.nano ~/.dbt/profiles.ymlAsdbthas created a new folder with the same name as your project name, change your current
directory to the project folder.cd dbt_bytehouse_demoUpdate yourdbt_project.ymlfile to have the same profile_name as~/.dbt/profiles.yml.nano dbt_project.ymlTest Warehouse ConnectionUsedbt debugcommand to verify required dependencies & warehouse connection. In case of success, it will show you
"All checks passed!"dbt debugDataset IngestionTo showcase differentdbtfunctionalities, we will ingest a small imdb movie dataset, with the following schema.
The DDL & insertion queries can be found herehttps://github.com/bytehouse-cloud/bytehouse-dbt/examples/data_loading.sql.
You can use ByteHouse SQL worksheet to create the schema & insert the dataset.To verify that dataset preparation was successful, we will execute this following query to summarize each actor along
with their total number of movie appearances.SELECT id,
  any(actor_name) as name,
  uniqExact(movie_id)    as num_movies,
  avg(rating)                as avg_rating,
  max(created_at) as updated_at
FROM (
  SELECT actors.id as id,
       concat(actors.first_name, ' ', actors.last_name) as actor_name,
       movies.id as movie_id,
       movies.rating as rating,
       created_at
  FROM  imdb.actors
        JOIN imdb.roles ON roles.actor_id = actors.id
        LEFT OUTER JOIN imdb.movies ON movies.id = roles.movie_id
)
GROUP BY idThe resultset should be like this:dbt ModelsIn adbtproject, a model is a sql file located insidemodels/directory which will contain aSELECTstatement
referring to a transformation. The name of the model file will refer to the name of future table/view afterdbtexecution. When we executedbt runcommand,dbtwill build this model directly into ByteHouse by wrapping
it in a create table / view materialization. Materialization type of your model will determine the actual SQL
thatdbtwill use to create model in the warehouse.schema.ymlTheschema.ymlfile will define our tables & columns by referring through alias name. This schemas can later be
used in different models/macros viasource()function.
Createschema.ymlfile undermodels/directorytouch models/schema.ymlDefine our model schema like thisversion:2sources:-name:imdbtables:-name:actors-name:roles-name:moviesMaterialization types of ModelsView MaterializationsIn case of view materialization, a model is transformed to a view on each single run byCREATE VIEWAS statement in
ByteHouse. View materializations won't store the actual data, so it would be slower to query than table materializations.
Let's createmodels/actor_insight.sqlas view materialization.touch models/actors_insight.sql{{ config(materialized='view') }}

SELECT id,
  any(actor_name) as name,
  uniqExact(movie_id)    as num_movies,
  avg(rating)                as avg_rating,
  max(created_at) as updated_at
FROM (
  SELECT actors.id as id,
       concat(actors.first_name, ' ', actors.last_name) as actor_name,
       movies.id as movie_id,
       movies.rating as rating,
       created_at
  FROM  {{ source('imdb', 'actors') }}
        JOIN {{ source('imdb', 'roles') }} ON roles.actor_id = actors.id
        LEFT OUTER JOIN {{ source('imdb', 'movies') }} ON movies.id = roles.movie_id
)
GROUP BY idLet's executedbt runcommand to build this model in ByteHouse.Querying this view, we can replicate the results of our earlier query with a simpler syntax:SELECT * FROM imdb.actors_insight ORDER BY num_movies DESC;Table MaterializationsIn case of table materialization, your model would be rebuilt as a table on each singledbt runvia aCREATE TABLEAS statement.touch models/actors_insight_table.sqlWe can use our previous view materialization sql with config change for table materialization sql.{{ config(order_by='(updated_at, id, name)', materialized='table') }}We can verify that both view & table materializations generate the same response.SELECT * FROM imdb.actors_insight_table ORDER BY num_movies DESC;Incremental MaterializationsFor our previous table materialization,dbtwill construct a table every time to materialize the model. For larger
or complex transformations, this would be redundant and costly in terms of computing power. Incremental
materializations solve this problem.The first time a model is run, the table is built by transforming all rows of source data. On subsequent runs,dbttransforms only the rows in your source data that you telldbtto filter for, inserting them into the target
table which is the table that has already been built.To telldbtwhich rows it should transform on an incremental run, wrap valid SQL that filters for these rows
in theis_incremental()macro. Youris_incremental()code will check for rows created or modified since the
last timedbtran this model.touch models/actors_insight_incremental.sql{{ config(order_by='(updated_at, id, name)', materialized='incremental') }}

SELECT id,
  any(actor_name) as name,
  uniqExact(movie_id)    as num_movies,
  avg(rating)                as avg_rating,
  max(created_at) as updated_at
FROM (
  SELECT actors.id as id,
       concat(actors.first_name, ' ', actors.last_name) as actor_name,
       movies.id as movie_id,
       movies.rating as rating,
       created_at
  FROM  {{ source('imdb', 'actors') }}
        JOIN {{ source('imdb', 'roles') }} ON roles.actor_id = actors.id
        LEFT OUTER JOIN {{ source('imdb', 'movies') }} ON movies.id = roles.movie_id
)
GROUP BY id

{% if is_incremental() %}

-- this filter will only be applied on an incremental run
where id > (select max(id) from {{ this }}) or updated_at > (select max(updated_at) from {{this}})

{% endif %}We can verify that view, table & incremental materializations, all generate the same response.SELECT * FROM imdb.actors_insight_table ORDER BY num_movies DESC;dbt run -m models/actors_insight_incremental.sqlLet's insert a few more rows to demonstrate the power of incremental materializations. Now the most
appeared actor should be 'Chris Pratt'.INSERT INTO imdb.movies VALUES (9, 'Passengers', 2016, 7);
INSERT INTO imdb.movies VALUES (10, 'The Tomorrow War', 2021, 6.5);

INSERT INTO imdb.roles (actor_id, movie_id, role_name) VALUES(4, 9, 'Jim Preston');
INSERT INTO imdb.roles (actor_id, movie_id, role_name) VALUES(4, 10, 'Dan Forester');dbt run -m models/actors_insight_incremental.sqlSELECT * FROM imdb.actors_insight_incremental ORDER BY num_movies DESC;How it worksdbtwill first create a temporary table namedactors_insight_incremental_tmp& insert all those rows which
pass ouris_incremental()filter.A new tableactors_insight_incremental_newwill be created & rows from the old tableactors_insight_incrementalwill be ingested here.dbtwill make sure thatunique_key(if any declared in config) constraint is maintained,
by not allowing those rows which have the sameunique_keyas the previous temporary table.The rows from the temporary table would be ingested into the new table.Our previous table (actors_insight_incremental) & new table (actors_insight_new) will be exchanged.Project Documentationdbtprovides a way to generate documentation for your dbt project and render it as a website.
Createmodels/actors_insight_incremental.ymlto generate documentation for our models.version:2models:-name:actors_insight_incrementaldescription:Actor appearance summary based on rolescolumns:-name:iddescription:The id number of actor-name:namedescription:The name of actor-name:num_moviesdescription:The number of movies actor has appeared-name:avg_ratingdescription:Average rating-name:updated_atdescription:Latest update timeUsedbt docs generateto generate the documentation for your models &dbt docs serveto serve the documentation in
your local browser on port 8000.Local DevelopmentUpdatetests/integration/confest.pyfile to include your connection credentials. For running tests locally, follow
these steps:pip install -r dev_requirements.txt
python -m pytestOriginal AuthorByteHouse wants to thank ClickHouse for original contribution to this connector.LicenseThis project is distributed under the terms of the Apache License (Version 2.0).