Package: astro-provider-databricks
Databricks Workflows in AirflowThe Astro Databricks Provider is an Apache Airflow provider to writeDatabricks Workflowsusing Airflow as the authoring interface. Running your Databricks notebooks as Databricks Workflows can result in a75% cost reduction($0.40/DBU for all-purpose compute, $0.07/DBU for Jobs compute).While this is maintained by Astronomer, it's available to anyone using Airflow - you don't need to be an Astronomer customer to use it.There are a few advantages to defining your Databricks Workflows in Airflow:via Databricksvia AirflowAuthoring interfaceWeb-based via Databricks UICode via Airflow DAGWorkflow compute pricing✅✅Notebook code in source control✅✅Workflow structure in source control✅Retry from beginning✅✅Retry single task✅Task groups within Workflows✅Trigger workflows from other DAGs✅Workflow-level parameters✅ExampleThe following Airflow DAG illustrates how to use theDatabricksTaskGroupandDatabricksNotebookOperatorto define a Databricks Workflow in Airflow:frompendulumimportdatetimefromairflow.decoratorsimportdag,task_groupfromairflow.operators.trigger_dagrunimportTriggerDagRunOperatorfromastro_databricksimportDatabricksNotebookOperator,DatabricksWorkflowTaskGroup# define your cluster spec - can have from 1 to many clustersjob_cluster_spec=[{"job_cluster_key":"astro_databricks","new_cluster":{"cluster_name":"",# ...},}]@dag(start_date=datetime(2023,1,1),schedule_interval="@daily",catchup=False)defdatabricks_workflow_example():# the task group is a context manager that will create a Databricks WorkflowwithDatabricksWorkflowTaskGroup(group_id="example_databricks_workflow",databricks_conn_id="databricks_default",job_clusters=job_cluster_spec,# you can specify common fields here that get shared to all notebooksnotebook_packages=[{"pypi":{"package":"pandas"}},],# notebook_params supports templatingnotebook_params={"start_time":"{{ ds }}",})asworkflow:notebook_1=DatabricksNotebookOperator(task_id="notebook_1",databricks_conn_id="databricks_default",notebook_path="/Shared/notebook_1",source="WORKSPACE",# job_cluster_key corresponds to the job_cluster_key in the job_cluster_specjob_cluster_key="astro_databricks",# you can add to packages & params at the task levelnotebook_packages=[{"pypi":{"package":"scikit-learn"}},],notebook_params={"end_time":"{{ macros.ds_add(ds, 1) }}",})# you can embed task groups for easier dependency management@task_group(group_id="inner_task_group")definner_task_group():notebook_2=DatabricksNotebookOperator(task_id="notebook_2",databricks_conn_id="databricks_default",notebook_path="/Shared/notebook_2",source="WORKSPACE",job_cluster_key="astro_databricks",)notebook_3=DatabricksNotebookOperator(task_id="notebook_3",databricks_conn_id="databricks_default",notebook_path="/Shared/notebook_3",source="WORKSPACE",job_cluster_key="astro_databricks",)notebook_4=DatabricksNotebookOperator(task_id="notebook_4",databricks_conn_id="databricks_default",notebook_path="/Shared/notebook_4",source="WORKSPACE",job_cluster_key="astro_databricks",)notebook_1>>inner_task_group()>>notebook_4trigger_workflow_2=TriggerDagRunOperator(task_id="trigger_workflow_2",trigger_dag_id="workflow_2",execution_date="{{ next_execution_date }}",)workflow>>trigger_workflow_2databricks_workflow_example_dag=databricks_workflow_example()Airflow UIDatabricks UIQuickstartCheck out the following quickstart guides:With the Astro CLIWithout the Astro CLIDocumentationThe documentation is a work in progress--we aim to follow theDiátaxissystem:Reference GuideChangelogAstro Databricks followssemantic versioningfor releases. Readchangelogto understand more about the changes introduced to each version.Contribution guidelinesAll contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.Read theContribution Guidelinesfor a detailed overview on how to contribute.Contributors and maintainers should abide by theContributor Code of Conduct.LicenseApache Licence 2.0