Package: dbt-flink-adapter
dbt Flink AdapterThis is an MVP of dbt Flink Adapter. It allows materializing of dbt models as Flink cluster streaming pipelines and batch jobs.Check out ourblogpost about dbt-flink-adapter with tutorialPrerequisitesFlink 1.16+ with Flink SQL GatewayPython 3.8+ with pip(Optionally) venvSetupThis adapter is connecting to Flink SQL Gateway which is not started in Flink by default.
Please refer toflink-doc/starting-the-sql-gatewayon how to start SQL gateway in your cluster.For testing and developing purposes you can useenvs/flink-1.16/docker-compose.ymlto start one node Flink cluster with SQL Gateway.$cdenvs/flink-1.16
$dockercomposeupInstalldbt-flink-adapterCreate virtual environment and installdbt-flink-adapterfromPyPI/dbt-flink-adapterwithpip$python3-mvenv~/.virtualenvs/dbt-example1
$source~/.virtualenvs/dbt-example1/bin/activate
$pip3installdbt-flink-adapter
$dbt--version
...
Plugins:-flink:x.y.zCreate and initialize dbt projectNavigate to directory in which you want to create your project. If you are using Flink with SQL Gateway started
from docker-compose.yml file in this repo you can leave all values as defaults.$dbtinit
Enteranameforyourproject(letters,digits,underscore):example1
Whichdatabasewouldyouliketouse?[1]flink

Enteranumber:1host(FlinkSQLGatewayhost)[localhost]:
port[8083]:
session_name[test_session]:
database(Flinkcatalog)[default_catalog]:
schema(Flinkdatabase)[default_database]:
threads(1ormore)[1]:

$cdexample1Creating and running dbt modelOn how to create and run dbt model please refer todbt-docs.
This README will focus on things that are specific for this adapter.dbtrunSourceIn typical use-case dbt connects and runs its ETL processes on database engine that already has connection with underlying persistence layer.
In case of Flink however it's only a processing engine, and we need to define connectivity with external persistence.
To do so we have to define sources in our dbt model.Connector propertiesdbt-flink-adapter will readconfig/connector_propertieskey and use it as connector properties.TypeFlink supports sources in batch and streaming mode, usetypeto select what execution environment will be used during source creation.column typecurrent has these values, refer toflink-doc/create-tablephysical (default)metadatacomputedWatermarkTo provide watermark passcolumnandstrategyreference underwatermarkkey in config.Please refer to Flink documentation about possible watermark strategies:flink-doc/watermarkExamplesources:-name:my_sourcetables:-name:clickstreamconfig:type:streamingconnector_properties:connector:'kafka'properties.bootstrap.servers:'kafka:29092'topic:'clickstream'watermark:column:event_timestampstrategy:event_timestampcolumns:-name:iddata_type:BIGINT-name:id2column_type:computedexpression:id + 1-name:event_timestampdata_type:TIMESTAMP(3)-name:ts2column_type:metadatadata_type:TIMESTAMP(3)expression:timestampSQL passed to Flink will look like:CREATETABLEIFNOTEXISTSmy_source(`id`BIGINT,`id2`ASid+1,`event_timestamp`TIMESTAMP(3),`ts2`TIMESTAMP(3)METADATAFROM'timestamp',WATERMARKFORevent_timestampASevent_timestamp)WITH('connector'='kafka','properties.bootstrap.servers'='kafka:29092','topic'='clickstream')ModelThis adapter currently supports two types of materializationtableandview. Because in Flink table has to be
associated with a connectortypeandconnector_propertieshave to be provided similar like in case of defining sources.Examplemodels.ymlmodels:-name:my_modelconfig:type:streamingconnector_properties:connector:'kafka'properties.bootstrap.servers:'kafka:29092'topic:'some-output'my_model.sqlselect*from{{source('my_source','clickstream')}}whereevent='some-event'Seeddbt-flink-adapter can use Flink to insert seed data in any Flink supported connector.
Similar like in case of sources and models you have to provide connector configuration.Exampleseeds.ymlseeds:-name:clickstreamconfig:connector_properties:connector:'kafka'properties.bootstrap.servers:'kafka:29092'topic:'clickstream'TestsDbt also allows executing assertions in a form of tests to validate input data or model output if it does not contain abnormal values.
All generic tests are a select statement which is considered as passed when it did not found any rows.The problem is how to define such thing in streaming pipeline? It is not possible to tell that in entire stream there are no such entries
as stream by definition is infinite. What we can do however is to have run the test for some specific time and if in that time there are no
abnormal values, test will be considered as passed.To facilitate it dbt-flink-adapter when writing a sql query supports fetch_timeout_ms and mode directive.select/** fetch_timeout_ms(5000) mode('streaming') */*from{{ref('my_model')}}whereevent<>'some-event'In this example we are telling dbt-flink-adapter to fetch for 5 seconds in streaming mode.dbt_project.ymlYou can extract common configurations of your model and sources intodbt_project.ymldbt-docs/general-configuration.
If you define the same kay indbt_project.ymland in your model or source dbt will always override entire key value.
In case you wish to extract some keys from underconnector_propertiesyou can specify configuration underdefault_connector_propertieswhich will get merged withconnection_properies.Exampledbt_project.ymlmodels:example1:+materialized:table+type:streaming+default_connector_properties:connector:'kafka'properties.bootstrap.servers:'kafka:29092'sources:example1:+type:streaming+default_connector_properties:connector:'kafka'properties.bootstrap.servers:'kafka:29092'seeds:example1:+default_connector_properties:connector:'kafka'properties.bootstrap.servers:'kafka:29092'models.ymlmodels:-name:my_modelconfig:connector_properties:topic:'some-output'sources.ymlsources:-name:my_sourcetables:-name:clickstreamconfig:connector_properties:topic:'clickstream'watermark:column:event_timestampstrategy:event_timestampcolumns:-name:event_timestampdata_type:TIMESTAMP(3)seeds.ymlseeds:-name:clickstreamconfig:connector_properties:topic:'clickstream'SessionsOur interaction with Flink cluster is done in sessions any table and view created in one session will not be visible in another session.
Session by default is only valid for 10 minutes. Because of that if you will run dbt test after more than 10 minutes from dbt run
it will fail and in Flink logs you will find that it cannot find your tables. Currently, the only way to run this would be to rerun entire model.Session handler is stored in~/.dbt/flink-session.ymlfile, if you want to force new session you can simply delete that file.ReleasingTo release new version first executeprepare-releaseaction.
Please keep in mind that major and minor version have to be exactly the same as major and minor version of dbt-core.This action will create a release branch with bumped version and changelog prepared for release. It will also open a Pull Request to main branch if everything is ok with it - merge it.Next executepublishon branch that was just created by prepare-release action.