Package: pydbr
pydbrDatabricks client SDK for Python with command line interface for Databricks REST APIs.{:toc}IntroductionPydbr (short of Python-Databricks) package provides python SDK for Databricks REST API:dbfsworkspacejobsrunsThe package also comes with a useful CLI which might be very helpful in automation.Installation$pipinstallpydbrDatabricks CLIDatabricks command line client provides convenient way to interact with Databricks cluster at the command line. A very popular use of such approach in in automation tasks, like DevOps pipelines or third party workflow managers.You can call the Databricks CLI using convenient shell commandpydbr:$pydbr--helpor using python module:$python-mpydbr.cli--helpTo connect to the Databricks cluster, you can supply arguments at the command line:--bearer-token--url--cluster-idAlternatively, you can define environment variables. Command line arguments take precedence.exportDATABRICKS_URL='https://westeurope.azuredatabricks.net/'exportDATABRICKS_BEARER_TOKEN='dapixyz89u9ufsdfd0'exportDATABRICKS_CLUSTER_ID='1234-456778-abc234'exportDATABRICKS_ORG_ID='87287878293983984'DBFSList DBFS items# List items on DBFSpydbrdbfsls--json-indent3FileStore/movielens[{"path":"/FileStore/movielens/ml-latest-small","is_dir":true,"file_size":0,"is_file":false,"human_size":"0 B"}]Download file from DBFS# Download a file and print to STDOUTpydbrdbfsgetml-latest-small/movies.csvDownload directory from DBFS# Download recursively entire directory and store locallypydbrdbfsget-oml-localml-latest-smallWorkspaceDatabricks workspace contains notebooks and other items.List workspace##################### List workspace# Default path is root - '/'$pydbrworkspacels# auto-add leading '/'$pydbrworkspacels'Users'# Space-indentend json output with number of spaces$pydbrworkspace--json-indent4ls# Custom indent string$pydbrworkspacels--json-indent='>'Export items from Databricks workspace###################### Export workspace items# Export everything in source format using defaults: format=SOURCE, path=/pydbrworkspaceexport-o./.dev/export# Export everything in DBC formatpydbrworkspaceexport-fDBC-o./.dev/export.# When path is folder, export is recursivepydbrworkspaceexport-o./.dev/export-utils'Utils'# Export single ITEMpydbrworkspaceexport-o./.dev/GetML'Utils/Download MovieLens.py'RunsThis command group implements thejobs/runsDatabricks REST API.Submit a notebookImplements:https://docs.databricks.com/dev-tools/api/latest/jobs.html#runs-submit$pydbrrunssubmit"Utils/Download MovieLens"{"run_id": 4}You can retrieve the job information usingruns get:$pydbrrunsget4-i3If you need to pass parameters, use the--parametersor-poption and specify JSON text.$pydbrrunssubmit-p'{"run_tag":"20250103"}'"Utils/Download MovieLens"You can refer also to parameters in JSON file:$pydbrrunssubmit-p'@params.json'"Utils/Download MovieLens"You can use the parameters in the notebook and will also be able to see them in the run metadata:pydbrrunsget-output-i38{"notebook_output":{"result":"Downloaded files (tag: 20250103): README.txt, links.csv, movies.csv, ratings.csv, tags.csv","truncated":false},"error":null,"metadata":{"job_id":8,"run_id":8,"creator_user_name":"your.name@gmail.com","number_in_job":1,"original_attempt_run_id":null,"state":{"life_cycle_state":"TERMINATED","result_state":"SUCCESS","state_message":""},"schedule":null,"task":{"notebook_task":{"notebook_path":"/Utils/Download MovieLens","base_parameters":{"run_tag":"20250103"}}},"cluster_spec":{"existing_cluster_id":"xxxx-yyyyyy-zzzzzz"},"cluster_instance":{"cluster_id":"xxxx-yyyyyy-zzzzzzzz","spark_context_id":"8734983498349834"},"overriding_parameters":null,"start_time":1592067357734,"setup_duration":0,"execution_duration":11000,"cleanup_duration":0,"trigger":null,"run_name":"pydbr-1592067355","run_page_url":"https://westeurope.azuredatabricks.net/?o=89349849834#job/8/run/1","run_type":"SUBMIT_RUN"}}Get run metadataImplements:Databricks REST runs/get$pydbrrunsget-i36{"job_id":6,"run_id":6,"creator_user_name":"your.name@gmail.com","number_in_job":1,"original_attempt_run_id":null,"state":{"life_cycle_state":"TERMINATED","result_state":"SUCCESS","state_message":""},"schedule":null,"task":{"notebook_task":{"notebook_path":"/Utils/Download MovieLens"}},"cluster_spec":{"existing_cluster_id":"xxxx-yyyyy-zzzzzz"},"cluster_instance":{"cluster_id":"xxxx-yyyyy-zzzzzz","spark_context_id":"783487348734873873"},"overriding_parameters":null,"start_time":1592062497162,"setup_duration":0,"execution_duration":11000,"cleanup_duration":0,"trigger":null,"run_name":"pydbr-1592062494","run_page_url":"https://westeurope.azuredatabricks.net/?o=398348734873487#job/6/run/1","run_type":"SUBMIT_RUN"}List RunsImplements:Databricks REST runs/list$pydbrrunslsTo get only the runs for a particular job:# Get job with job-id=4$pydbrrunsls4-i3{"runs":[{"job_id":4,"run_id":4,"creator_user_name":"your.name@gmail.com","number_in_job":1,"original_attempt_run_id":null,"state":{"life_cycle_state":"PENDING","state_message":""},"schedule":null,"task":{"notebook_task":{"notebook_path":"/Utils/Download MovieLens"}},"cluster_spec":{"existing_cluster_id":"xxxxx-yyyy-zzzzzzz"},"cluster_instance":{"cluster_id":"xxxxx-yyyy-zzzzzzz"},"overriding_parameters":null,"start_time":1592058826123,"setup_duration":0,"execution_duration":0,"cleanup_duration":0,"trigger":null,"run_name":"pydbr-1592058823","run_page_url":"https://westeurope.azuredatabricks.net/?o=abcdefghasdf#job/4/run/1","run_type":"SUBMIT_RUN"}],"has_more":false}Export runImplements:Databricks REST runs/export$pydbrrunsexport--content-only4>.dev/run-view.htmlGet run outputImplements:Databricks REST runs/get-output$pydbrrunsget-output-i36{"notebook_output":{"result":"Downloaded files: README.txt, links.csv, movies.csv, ratings.csv, tags.csv","truncated":false},"error":null,"metadata":{"job_id":5,"run_id":5,"creator_user_name":"your.name@gmail.com","number_in_job":1,"original_attempt_run_id":null,"state":{"life_cycle_state":"TERMINATED","result_state":"SUCCESS","state_message":""},"schedule":null,"task":{"notebook_task":{"notebook_path":"/Utils/Download MovieLens"}},"cluster_spec":{"existing_cluster_id":"xxxx-yyyyy-zzzzzzz"},"cluster_instance":{"cluster_id":"xxxx-yyyyy-zzzzzzz","spark_context_id":"8973498743973498"},"overriding_parameters":null,"start_time":1592062147101,"setup_duration":1000,"execution_duration":11000,"cleanup_duration":0,"trigger":null,"run_name":"pydbr-1592062135","run_page_url":"https://westeurope.azuredatabricks.net/?o=89798374987987#job/5/run/1","run_type":"SUBMIT_RUN"}}To get only the exit output:$pydbrrunsget-output-r6Downloaded files: README.txt, links.csv, movies.csv, ratings.csv, tags.csvPython Client SDK for Databricks REST APIsTo implement your own Databricks REST API client, you can use the Python Client SDK for Databricks REST APIs.Create Databricks connection# Get Databricks workspace connectiondbc=pydbr.connect(bearer_token='dapixyzabcd09rasdf',url='https://westeurope.azuredatabricks.net')DBFS# Get list of items at path /FileStoredbc.dbfs.ls('/FileStore')# Check if file or directory existsdbc.dbfs.exists('/path/to/heaven')# Make a directory and it's parentsdbc.dbfs.mkdirs('/path/to/heaven')# Delete a directory recusivelydbc.dbfs.rm('/path',recursive=True)# Download file block starting 1024 with size 2048dbc.dbfs.read('/data/movies.csv',1024,2048)# Download entire filedbc.dbfs.read_all('/data/movies.csv')Databricks workspace# List root workspace directorydbc.workspace.ls('/')# Check if workspace item existsdbc.workspace.exists('/explore')# Check if workspace item is a directorydbc.workspace.is_directory('/')# Export notebook in default (SOURCE) formatdbc.workspace.export('/my_notebook')# Export notebook in HTML formatdbc.workspace.export('/my_notebook','HTML')Build and publishpipinstallwheeltwine
pythonsetup.pysdistbdist_wheel
python-mtwineuploaddist/*