Package: databricks-pybuilder-plugin
databricks-pybuilder-pluginThe plugin is considered to be used to deploy assets to a Databricks environment.The plugin is activated with the following command in your build.by:use_plugin('pypi:databricks_pybuilder_plugin')It provides a set of tasks for uploading resources, workspaces, deploying jobs,
or installing locally built whl dependencies into a databricks cluster.Deployment to DatabricksAutomated deployment is implemented on the PyBuilder tasks basis.The list of available tasks:export_workspace- exporting a workspace.
A workspace is considered to be a folder in Databricks holding a notebook, or a set of notebooks.The task uploads thesrc/main/scripts/content into a Databricks workspace.
It overrides files of the same names and leaves other files as is.By default, a git branch name is used as a nested folder of a workspace
for uploading the content into the Databricks workspace, if it's available.
Thedefaultfolder is used otherwise.
Use thebranchparameter to import a workspace in your own folder:pyb export_workspace -P branch={custom_directory}The final output path would be/team_folder/application_name/{custom_directory}in this way.Executing the command from a master branchpyb export_workspacewould upload the workspace files into/team_folder/application_name/master.Here is the list of related deployment settingsPropertyValueDescriptionproject_workspace_pathsrc/main/scripts/The path to a folder in the project tree holding notebooks.remote_workspace_path/team_folder/application_name/The Databricks folder the notebooks would be uploaded into from project_workspace_path.All of the properties could be overridden with a -P parameter.Usage example:pyb export_workspace [-P env={env}] [-P branch={branch}]Environment specific properties
Disabled by default.export_resources- exporting resources into dbfs.
Uploads resource files into dbfs if any. Existing files are to be overridden.Here is the list of related deployment settingsPropertyValueDescriptionproject_resources_pathsrc/main/resources/The path to the project resources.All of the properties could be overridden with a -P parameter.Usage example:pyb export_resources [-P env={env}] [-P branch={branch}]install_library- deploying an whl-archive to a Databricks cluster.
Uploads an whl archive to dbfs, and re-attaches the library to a cluster by name.
Re-installing a new library version triggers the cluster starting
to uninstall old libraries versions and to install a new one.
Repetitive installations of a library of the same version don't start the cluster.
The library is just re-attached to a cluster in this way.
Other installed libraries are not affected.Here is the list of related deployment settingsPropertyValueDescriptionremote_cluster_nameTest_cluster_nameThe name of a remote Databricks cluster the library to be installed to.dbfs_library_pathdbfs:/FileStore/jarsThe dbfs path to a folder holding the whl archives.All of the properties could be overridden with a -P parameter.Usage example:pyb install_librarydeploy_to_cluster- a full deployment to a cluster.
Runs theexport_resources,export_workspace,install_libraryin a row.Usage example:pyb deploy_to_clusterdeploy_job- deploying a job to the Databricks by name.
Please, make sure that the job is created on the Databricks side.Executesexport_resourcesandexport_workspacetasks preliminarily.
Updates the existing job using a job definition file.
The definition file supports the jinja2 template syntax.
Please, check documentation for details:https://jinja.palletsprojects.com/en/2.11.x/Here is the list of related deployment settingsPropertyValueDescriptionjob_definition_pathsrc/main/databricks/databricks_job_settings.jsonThe project path to a job definition file.All of the properties could be overridden with a -P parameter.Usage example:pyb deploy_job [-P env={env}] [-P branch={branch}]To Run a notebook with a custom dependencyBuild the whl-archive with thepybcommand.Deploy all the assets using the commandpyb deploy_to_cluster.Get to the target folder in the Databricks workspace.Attach the notebook to a cluster and run the script.All properties listPropertyDefault ValueDescriptiondatabricks_credentials{'dev': {'host': '', 'token': ''}'qa': {'host': '', 'token': ''}'prod': {'host': '', 'token': ''}}Please specify credentials in the dictionary format: host and token.default_environmentdevThere are 3 supported environments:dev,qaandprod.project_workspace_pathsrc/main/scripts/The directory content is going to be uploaded into a databricks workspace.These are considered to be notebook scripts.remote_workspace_pathThe databricks workspace that files inproject_workspace_pathare copied to.include_git_branch_into_output_workspace_pathTrueThe flag enables adding an extra directory with thebranchname to theremote_workspace_path. Requires git to be installed.enable_env_sensitive_workspace_propertiesFalseThe flag enables environment properties chosen by theenv_config_workspace_path.env_config_workspace_pathenvironment-settings/{env}.pyThe path to a property file to be chosen as a env properties. By defaultenvincluded into a file name is used to pick properties.env_config_nameenvThe expected environment properties file name. Theenv_config_workspace_pathwill be copied to databricks workspace with name.with_dbfs_resourcesFalseThe flage enables uploading resource files from theproject_resources_pathdirectory to databricks hdfsdbfs_resources_path.project_resources_pathsrc/main/resources/The local directory path holding resource files to be copied (txt, csv etc).dbfs_resources_pathThe output hdfs directory on databricks environment holding resources.dbfs_library_pathdbfs:/FileStore/jarsThe output hdfs directory on databricks environment holding a built dependency (whl-archive).attachable_lib_envs['dev']The list of environments that requires a dependency attached to a databricks cluster. The dependency is preliminary must be uploaded to thedbfs_library_path.cluster_init_timeout5 * 60The timeout of waiting a databricks cluster while it changes its state (initiating, restarting etc).remote_cluster_nameThe name of a databricks cluster that dependency is attached to.job_definition_path'src/main/databricks/job_settings.json'The path to a dataricks job configuration in a json format -https://docs.databricks.com/dev-tools/api/2.0/jobs.html. It supports Jinja template syntax in order to setup env sensitive properties. It also supports multiple jobs definitions - use a json array for that. the list of properties available by default: env, branch, remote_workspace_path, remote_workspace_pathdeploy_single_jobThe name of a job to be deployed. If your databricks job config contains multiple definitions, you can deploy just one of these jobs specifying a name of the particular job.extra_rendering_argsCustom properties to be populated in the job definition file. Use a dicionary as an argument. For example:{'app_name': name}.