Package: brickflows
BrickflowBrickFlow is specifically designed to enable the development of Databricks workflows using Python, streamlining the 
process through a command-line interface (CLI) tool.ContributorsThanks to all thecontributorswho have helped ideate, develop and bring Brickflow to its current state.ContributingWe're delighted that you're interested in contributing to our project! To get started,
please carefully read and follow the guidelines provided in ourcontributingdocument.DocumentationBrickflow documentation can be foundhere.Getting StartedPrerequisitesInstall brickflowspipinstallbrickflowsInstallDatabricks CLIcurl-fsSLhttps://raw.githubusercontent.com/databricks/setup-cli/main/install.sh|sudoshConfigure Databricks cli with workspace token. This configures your~/.databrickscfgfile.databricksconfigure--tokenHello World workflowCreate your first workflow using brickflowmkdirhello-world-brickflowcdhello-world-brickflow
brickflowprojectsaddProvide the following inputsProjectname:hello-world-brickflow
Pathfromreporoottoprojectroot(optional)[.]:.
Pathfromprojectroottoworkflowsdir:workflows
Githttpsurl:https://github.com/Nike-Inc/brickflow.git
Brickflowversion[auto]:<hitenter>
Sparkexpectationsversion[0.5.0]:0.8.0
Skipentrypoint[y/N]:NNote: You can provide your own github repo url.Create a new file hello_world_wf.py in the workflows directorytouchworkflows/hello_world_wf.pyCopy the following code in hello_world_wf.py filefrombrickflowimport(ctx,Cluster,Workflow,NotebookTask,)fromairflow.operators.bashimportBashOperatorcluster=Cluster(name="job_cluster",node_type_id="m6gd.xlarge",spark_version="13.3.x-scala2.12",min_workers=1,max_workers=2,)wf=Workflow("hello_world_workflow",default_cluster=cluster,tags={"product_id":"brickflow_demo",},common_task_parameters={"catalog":"<uc-catalog-name>","database":"<uc-schema-name>",},)@wf.task# this task does nothing but explains the use of context objectdefstart():print(f"Environment:{ctx.env}")@wf.notebook_task# this task runs a databricks notebookdefexample_notebook():returnNotebookTask(notebook_path="notebooks/example_notebook.py",base_parameters={"some_parameter":"some_value",# in the notebook access these via dbutils.widgets.get("some_parameter")},)@wf.task(depends_on=[start,example_notebook])# this task runs a bash commanddeflist_lending_club_data_files():returnBashOperator(task_id=list_lending_club_data_files.__name__,bash_command="ls -lrt /dbfs/databricks-datasets/samples/lending_club/parquet/",)@wf.task(depends_on=list_lending_club_data_files)# this task runs the pyspark codedeflending_data_ingest():ctx.spark.sql(f"""CREATE TABLE IF NOT EXISTS{ctx.dbutils_widget_get_or_else(key="catalog",debug="development")}.\{ctx.dbutils_widget_get_or_else(key="database",debug="dummy_database")}.\{ctx.dbutils_widget_get_or_else(key="brickflow_env",debug="local")}_lending_data_ingestUSING DELTA -- this is default just for explicit purposeSELECT * FROM parquet.`dbfs:/databricks-datasets/samples/lending_club/parquet/`""")Note: Modify the values of catalog/database for common_task_parameters.Create a new file example_notebook.py in the notebooks directorymkdirnotebooks
touchnotebooks/example_notebook.pyCopy the following code in the example_notebook.py file# Databricks notebook sourceprint("hello world")Deploy the workflow to databricksbrickflowprojectsdeploy--projecthello-world-brickflow-elocalRun the demo workflowLogin to databricks workspaceGo to the workflows and select the workflow4. click on the run buttonExamplesRefer to theexamplesfor more examples.