Package: prefect-databricks
Integrate Databricks jobs into your dataflow with prefect-databricksVisit the full docshereto see additional examples and the API reference.The prefect-databricks collection makes it easy to coordiante Databricks jobs with other tools in your data stack using Prefect. Check out the examples below to get started!Getting StartedIntegrate with Prefect flowsUsing Prefect with Databricks allows you to define and orchestrate complex data workflows that take advantage of the scalability and performance of Databricks.This can be especially useful for data-intensive tasks such as ETL (extract, transform, load) pipelines, machine learning training and inference, and real-time data processing.Below is an example of how you can incorporate Databricks notebooks within your Prefect flows.Be sure to installprefect-databricksandsave a credentials blockto run the examples below!If you don't have an existing notebook ready on Databricks, you can copy the following, and name itexample.ipynb. This notebook, accepts a name parameter from the flow and simply prints a message.name=dbutils.widgets.get("name")message=f"Don't worry{name}, I got your request! Welcome to prefect-databricks!"print(message)Here, the flow launches a new cluster to runexample.ipynband waits for the completion of the notebook run. Replace the placeholders and run.fromprefectimportflowfromprefect_databricksimportDatabricksCredentialsfromprefect_databricks.flowsimportjobs_runs_submit_and_wait_for_completionfromprefect_databricks.models.jobsimport(AutoScale,AwsAttributes,JobTaskSettings,NotebookTask,NewCluster,)@flowdefjobs_runs_submit_flow(block_name:str,notebook_path:str,**base_parameters):databricks_credentials=DatabricksCredentials.load(block_name)# specify new cluster settingsaws_attributes=AwsAttributes(availability="SPOT",zone_id="us-west-2a",ebs_volume_type="GENERAL_PURPOSE_SSD",ebs_volume_count=3,ebs_volume_size=100,)auto_scale=AutoScale(min_workers=1,max_workers=2)new_cluster=NewCluster(aws_attributes=aws_attributes,autoscale=auto_scale,node_type_id="m4.large",spark_version="10.4.x-scala2.12",spark_conf={"spark.speculation":True},)# specify notebook to use and parameters to passnotebook_task=NotebookTask(notebook_path=notebook_path,base_parameters=base_parameters,)# compile job task settingsjob_task_settings=JobTaskSettings(new_cluster=new_cluster,notebook_task=notebook_task,task_key="prefect-task")run=jobs_runs_submit_and_wait_for_completion(databricks_credentials=databricks_credentials,run_name="prefect-job",tasks=[job_task_settings])returnrunjobs_runs_submit_flow(block_name="BLOCK-NAME-PLACEHOLDER"notebook_path="/Users/<EMAIL_ADDRESS_PLACEHOLDER>/example.ipynb",name="Marvin")Upon execution, the notebook run should output:Don't worry Marvin, I got your request! Welcome to prefect-databricks!!!! info "Input dictionaries in the place of models"Instead of using the built-in models, you may also input a valid dictionary.

For example, the following are equivalent:

```python
auto_scale=AutoScale(min_workers=1, max_workers=2)
```

```python
auto_scale={"min_workers": 1, "max_workers": 2}
```If you have an existing Databricks job, you can run it usingjobs_runs_submit_by_id_and_wait_for_completion:fromprefectimportflowfromprefect_databricksimportDatabricksCredentialsfromprefect_databricks.flowsimport(jobs_runs_submit_by_id_and_wait_for_completion,)@flowdefexisting_job_submit(databricks_credentials_block_name:str,job_id):databricks_credentials=DatabricksCredentials.load(name=block_name)run=jobs_runs_submit_by_id_and_wait_for_completion(databricks_credentials=databricks_credentials,job_id=job_id)returnrunexisting_job_submit(databricks_credentials_block_name="db-creds",job_id="YOUR-JOB-NAME")ResourcesFor more tips on how to use tasks and flows in a Collection, check outUsing Collections!Note, the tasks within this collection were created by a code generator using the service's OpenAPI spec.The service's REST API documentation can be foundhere.InstallationInstallprefect-databrickswithpip:pipinstallprefect-databricksRequires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.These tasks are designed to work with Prefect 2. For more information about how to use Prefect, please refer to thePrefect documentation.Saving Credentials to BlockTo use theloadmethod on Blocks, you must already have a block documentsaved through codeorsaved through the UI.Below is a walkthrough on saving block documents through code; simply create a short script, replacing the placeholders.Head over toDatabricks.Login to your Databricks account and select a workspace.On the top right side of the nav bar, click on your account name -> User Settings.Click Access tokens -> Generate new token -> Generate and copy the token.Note down your Databricks instance from the browser URL, formatted likehttps://<DATABRICKS-INSTANCE>.cloud.databricks.com/Create a short script, replacing the placeholders.fromprefect_databricksimportDatabricksCredentialscredentials=DatabricksCredentials(databricks_instance="DATABRICKS-INSTANCE-PLACEHOLDER"token="TOKEN-PLACEHOLDER")connector.save("BLOCK_NAME-PLACEHOLDER")Congrats! You can now easily load the saved block, which holds your credentials:fromprefect_databricksimportDatabricksCredentialsDatabricksCredentials.load("BLOCK_NAME-PLACEHOLDER")!!! info "Registering blocks"Register blocks in this module to
[view and edit them](https://orion-docs.prefect.io/ui/blocks/)
on Prefect Cloud:

```bash
prefect block register -m prefect_databricks
```FeedbackIf you encounter any bugs while usingprefect-databricks, feel free to open an issue in theprefect-databricksrepository.If you have any questions or issues while usingprefect-databricks, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to star or watchprefect-databricksfor updates too!ContributingIf you'd like to help contribute to fix an issue or add a feature toprefect-databricks, pleasepropose changes through a pull request from a fork of the repository.Here are the steps:Fork the repositoryClone the forked repositoryInstall the repository and its dependencies:pip install -e ".[dev]"Make desired changesAdd testsInsert an entry toCHANGELOG.mdInstallpre-committo perform quality checks prior to commit:pre-commit installgit commit,git push, and create a pull request