Package: ascend-io-dbt-utils
Utility to convert a dbt project with compiled SQL models into Ascend transformsOverviewThis utility allows you to convert a dbt project with compiled SQL models into Ascend transforms. It will use the SQL files generated withdbt compilein conjunction withmanifest.jsongenerated bydbt compileto create the transforms.UsageRun--helpto see the usage:Usage: ascend_dbt_utils [OPTIONS] COMMAND [ARGS]...

  Collection of utilities to help convert dbt projects into Ascend dataflows.

Options:
  --help  Show this message and exit.

Commands:
  dbt-model   Operations to perform on an individual dbt model.
  dbt-models  Operations to perform on a collection of dbt models together.
  dbt-tests   Operations to perform on dbt tests.PrerequisitesSuper-useful to run using venv:python3 -m venv .venv
source venv/bin/activateClone the Github repo, you know how to do it. Install the required modules.pip install -r requirements.txtMay require installing additional modules that are not mentioned in therequirements.txtfile, depending on your environment. The following modules are optional if you want to use Ascend CLI or do dbt compilation locally:pip install ascend-io-cli
pip install dbt-core
pip install dbt-snowflakePlace the dbt project in a subdirectory. Compile the project withdbt compile. Ensure thetarget/compileddirectory is present. Themanifest.jsonfile should be present in thetarget/compileddirectory, as well as the SQL files for each model..envfile supportTo save the amount of command line params typing, the utility supports the.envfile to specify the parameters instead of specifying them on the command line. The.envfile should be placed in the same directory where theascend_dbt_utilsutility is run from. The following environment variables are supported:ASCEND_HOSTNAME=myhost.ascend.io
ASCEND_DATA_SERVICE=My_Snowflake_Service
ASCEMD_DATAFLOW=dbt_model_flow
DBT_MANIFEST_FILE=target/manifest.json
ASCEND_LANGUAGE=bigquery-sql # Or, "snowflake-sql" by default when omitted.Workflowascend_dbt_utilssupports .env file instead of specifying the parameters on the command lineExample of the.envfile:ASCEND_HOSTNAME=myhost.ascend.io
ASCEND_DATA_SERVICE=My_Snowflake_Service
ASCEND_DATAFLOW=dbt_model_flow
DBT_MANIFEST_FILE=target/manifest.jsonUse thedbt-models showcommand first to see the dependencies, seeds and modelsThedbt-models showcommand will display the list of seeds and models defined in the dbt project. Seeds will have to be pre-created in Ascend dataflow to use them as transformer inputs. Models will be converted into transforms. The following output is an example of thedbt-models showcommand:Node: seed.acme.subscription_periods
  -> Depends on: 
Node: model.acme.util_months
  -> Depends on: 
Node: model.acme.customer_revenue_by_month
  -> Depends on: seed.acme.subscription_periods
  -> Depends on: model.acme.util_months
Node: model.acme.customer_churn_month
  -> Depends on: model.acme.customer_revenue_by_month
Node: model.acme.mrr
  -> Depends on: model.acme.customer_revenue_by_month
  -> Depends on: model.acme.customer_churn_monthThe output above requires us to have an Ascend dataflow with a connector or a data share connection calledsubscription_periodsto allow the transforms connect to it. The transforms will be created in the dataflow with the following names:util_months
customer_revenue_by_month
customer_churn_month
mrrIt is useful to check the following before creating the transforms:The transforms will be created in the dataflow with the same names as the models. If there are any name conflicts, the transforms will not be created.The utility doesn't interfere with the schema, if the transforms SQL relies on certain fields, they should be present in the seed connector and other transforms. Some manual cleanup may be required after the transforms are created to clean up errors.dbt allows for models without inputs, but Ascend doesn't. Ensure that all the generated SQL files are referencing other models or seeds. If needed, add a dummy SQL statement connecting the model to a seed or another model.The utility will determine the right order of model transforms creation based onmanifest.json. If there are any dependencies missing, the transforms will not be created.Use thedbt-models validatecommand to ensure all the required seeds are present in the flowThedbt-models validatecommand will validate that all the seeds required by the transforms are present in the dataflow. The following output is an example of thedbt-models validatecommand:Nodes present in the dataflow:
  raw_customers
  raw_orders
  raw_payments
Nodes absent in the dataflow:Use thedbt-models mergecommand to create the transforms or merge the transforms into an existing flowThedbt-models mergecommand will merge the transforms into the dataflow. The following output is an example of thedbt-models mergecommand:I1021 12:52:07.756062 75551 applier.py:350] Apply Dataflow: (ds=Mikes_Snowflake_demos df=dbt_test)
I1021 12:52:08.019690 75551 applier.py:360] Update Dataflow: (ds=Mikes_Snowflake_demos df=dbt_test)
I1021 12:52:09.084092 75551 applier.py:756] Apply Component: (ds=Mikes_Snowflake_demos df=dbt_test util_months)
I1021 12:52:09.356551 75551 applier.py:697] Create Transform: (ds=Mikes_Snowflake_demos df=dbt_test util_months)
I1021 12:52:10.335402 75551 applier.py:756] Apply Component: (ds=Mikes_Snowflake_demos df=dbt_test customer_revenue_by_month)
I1021 12:52:10.617655 75551 applier.py:697] Create Transform: (ds=Mikes_Snowflake_demos df=dbt_test customer_revenue_by_month)
I1021 12:52:11.593106 75551 applier.py:756] Apply Component: (ds=Mikes_Snowflake_demos df=dbt_test customer_churn_month)
I1021 12:52:11.856432 75551 applier.py:697] Create Transform: (ds=Mikes_Snowflake_demos df=dbt_test customer_churn_month)
I1021 12:52:12.824820 75551 applier.py:756] Apply Component: (ds=Mikes_Snowflake_demos df=dbt_test mrr)
I1021 12:52:13.091333 75551 applier.py:697] Create Transform: (ds=Mikes_Snowflake_demos df=dbt_test mrr)Thedbt-models mergecommand requires the pointers to themanifest.jsonfile that contains the required information. The transforms will be created in the dataflow with the same names as the models. If there are any name conflicts, the transforms will not be created. The--default-seedoption allows you to specify the default seed to use for the transforms that are not connected to anything, but this requires their SQL files to be edited to reference the default seed asXXX.YYY.subscription_periods. If the transforms SQL relies on certain fields, they should be present in the seed connector and other transforms. Some manual cleanup may be required after the transforms are created to clean up errors.Use thedbt-models update-sqlcommand to only update the SQL statements of the existing transformsThedbt-models update-sqlcommand will only update the SQL statements of the existing transforms and leave the rest of the configuration as-is. This is useful to update in place the SQL of the existing transforms that were created from the dbt models.If needed, use thedbt-models deletecommandThedbt-models deletecommand will delete the transforms created from dbt models from the dataflow. Make sure to delete the deployed tests first withdbt-tests delete, otherwise the deletion of transforms will fail. The following output is an example of thedbt-models deletecommand:Deleting node mrr
Deleting node customer_churn_month
Deleting node customer_revenue_by_month
Deleting node util_monthsTestingTo run dbt tests in Ascend you will need to first deploy the tests. Once deployed, the tests will appear as grouped transform components in the data flow. Once deployed, the tests will automatically run and will remain running continuously in the flow. Hence, the dbt tests can not only be used to validate the deployment but can also serve as continuously running data quality tests to validate the correct operation of your data flow.
Thedbt-tests deploy-testscommand will deploy the tests to the dataflow. Thedbt-tests delete-testscommand will remove the deployed tests. Thedbt-tests check-test-resultscommand will check the test results. The following output is an example of thedbt-tests check-test-resultscommand:Checking test results based on the number of records returned by tests...
Test unique_customers_customer_id produced 0 records. Test result is 'Ok'.
Test not_null_customers_customer_id produced 0 records. Test result is 'Ok'.
Test unique_orders_order_id produced 0 records. Test result is 'Ok'.
Test not_null_orders_order_id produced 0 records. Test result is 'Ok'.
...

Checking test results based on data quality check results...
Test unique_customers_customer_id produced 'passed'
Test not_null_customers_customer_id produced 'passed'
Test unique_orders_order_id produced 'passed'
Test not_null_orders_order_id produced 'passed'
...Creating individual transformsThedbt-model create-componentcommand will create a single transform from the model. This command can also create the associated tests and group them separately per model.Updating the SQL of individually created transformsThedbt-model update-sqlcommand will update the SQL of the transform created from the model.