Package: dbloy
DBloyA Databricks deployment CLI tool to enable Continuous Delivery of PySpark Notebooks based jobs.Installation$pipinstalldbloyUsageAuthenticate with Databricks using authentication token:$dbloyconfigureUpdate Databricks Job$dbloyapply--deploy-ymldeploy.yml--configmap-ymlconfigmap.yml--version<my_version_number>wheredeploy.ymlandconfigmap.ymlcontain the Job specification. The Job version is specified in<my_version_number>Workflowsource:https://databricks.com/blog/2017/10/30/continuous-integration-continuous-delivery-databricks.htmlExample UsageSeeexample/gitlab_my-etl-jobfor a example ETL repository using Gitlab's CI/CD.A Deployment requires the following:Deployment manifestConfiguration manifestA main Databricks Notebook source file available locally.(Optional) Attached python library containing the core logic. This allows easier unit testing ofCreating a Deploymentdeploy.ymlkind:Deploymentmetadata:name:my-etl-jobworkspace:Sharedtemplate:job:name:My ETL Jobnotifications:email:no_alert_for_skipped_runs:trueon_failure:-my_email@my_org.combase_notebook:mainnotebooks:-EPHEMERAL_NOTEBOOK_1:notebook_name1-EPHEMERAL_NOTEBOOK_2:notebook_name2libraries:-egg_main:dbfs:/python35/my_python_lib/my_python_lib-VERSION-py3.5.egg-egg:dbfs:/python35/static_python_lib.egg-pypi:package:scikit-learn==0.20.3-pypi:package:statsmodels==0.10.1-pypi:package:prometheus-client==0.7.1-jar:dbfs:/FileStore/jars/e9b87e4c_c754_4707_a62a_44ef47535b39-azure_cosmosdb_spark_2_4_0_2_11_1_3_4_uber-38021.jarrun:max_concurrent_runs:1max_retries:1min_retry_interval_millis:600000retry_on_timeout:truetimeout_seconds:10800configmap.ymlkind:ConfigMapmetadata:namespace:productionparams:DB_URL:production_db_url_1DB_PASSWORD:production_password123job:id:289schedule:quartz_cron_expression:"000**?"timezone_id:"Europe/Berlin"max_retries:"1"cluster:spark_version:"5.3.x-scala2.11"node_type_id:"Standard_DS3_v2"driver_node_type_id:"Standard_DS3_v2"autoscale:min_workers:1max_workers:2spark_env_vars:PYSPARK_PYTHON:"/databricks/python3/bin/python3"In this example:Job id289on Databricks, indicated by the.job.idfield inconfigmap.yml, will be updated with the nameMy ETL Job, indicated by the.template.job.namefield indeploy.yml.A cluster will be created on demand which is specified by the field.clusterinconfigmap.yml. Seehttps://docs.databricks.com/api/latest/clusters.html#request-structurefor a complete list of cluster settings.Note: Setting.cluster.existing_cluster_idwill use an existing cluster.Libraries specified by the field.template.librariesin.deploy.ymlwill be installed on the cluster. Seehttps://docs.databricks.com/api/latest/libraries.html#library.Note: The field.template.libraries.egg_mainis reserved for python.eggfile that is versioned with the ETL job.
For example when the main logic of the ETL job is put into a library. The.eggversion number is expected to be the same as the ETL version number.The main task notebook that will be executed by the job is defined by the field.template.base_notebookindeploy.yml. Task parameters are specified by the field.paramsinconfigmap.ymlwhich will be accessible in the Notebooks viadbutils.The notebookmain, indicated by the field.template.base_notebookis the Task notebook. This notebook should be found in the workspace/Shared/my-etl-job/<my_version_number>/mainspecified by the fields.metadataand.template.base_notebookindeploy.yml. The version number<my_version_number>will be specified in the CLI command.Two ephemeral notebooks are available under/Shared/my-etl-job/<my_version_number>/notebook_name1and/Shared/my-etl-job/<my_version_number>/notebook_name2. This allows the main task to execute nested Notebooks, e.g.notebook_path_1 = dbutils.widgets.get("EPHEMERAL_NOTEBOOK_1")
dbutils.notebook.run(notebook_path_1)Create the Deployment by running the following command:$dbloyapply--deploy-ymldeploy.yml--configmap-ymlconfigmap.yml--version<my_version_number>