Package: pyspark-me
pyspark-meDatabricks client SDK for Python with command line interface for Databricks REST APIs.[TOC]IntroductionPysparkme package provides python SDK for Databricks REST API:dbfsworkspacejobsrunsThe package also comes with a useful CLI which might be very helpful in automation.Python Client SDK for Databricks REST APIsCreate Databricks connection# Get Databricks workspace connectiondbc=pysparkme.databricks.connect(bearer_token='dapixyzabcd09rasdf',url='https://westeurope.azuredatabricks.net')DBFS# Get list of items at path /FileStoredbc.dbfs.ls('/FileStore')# Check if file or directory existsdbc.dbfs.exists('/path/to/heaven')# Make a directory and it's parentsdbc.dbfs.mkdirs('/path/to/heaven')# Delete a directory recusivelydbc.dbfs.rm('/path',recursive=True)# Download file block starting 1024 with size 2048dbc.dbfs.read('/data/movies.csv',1024,2048)# Download entire filedbc.dbfs.read_all('/data/movies.csv')Databricks workspace# List root workspace directorydbc.workspace.ls('/')# Check if workspace item existsdbc.workspace.exists('/explore')# Check if workspace item is a directorydbc.workspace.is_directory('/')# Export notebook in default (SOURCE) formatdbc.workspace.export('/my_notebook')# Export notebook in HTML formatdbc.workspace.export('/my_notebook','HTML')Databricks CLIdbr-meYou can call the Databricks CLI using convenient shell commanddbr-me:$dbr-me--helpor using python module:$python-mpysparkme.databricks.cli--helpTo connect to the Databricks cluster, you can supply arguments at the command line:--bearer-token--url--cluster-idAlternatively, you can define environment variables. Command line arguments take precedence.exportDATABRICKS_URL='https://westeurope.azuredatabricks.net/'exportDATABRICKS_BEARER_TOKEN='dapixyz89u9ufsdfd0'exportDATABRICKS_CLUSTER_ID='1234-456778-abc234'exportDATABRICKS_ORG_ID='87287878293983984'Workspace##################### List workspace# Default path is root - '/'dbr-meworkspacels# auto-add leading '/'dbr-meworkspacels'Users'# Space-indentend json output with number of spacesdbr-meworkspace--json-indent4ls# Custom indent stringdbr-meworkspacels--json-indent='>'###################### Export workspace items# Export everything in source format using defaults: format=SOURCE, path=/dbr-meworkspaceexport-o./.dev/export# Export everything in DBC formatdbr-meworkspaceexport-fDBC-o./.dev/export.# When path is folder, export is recursivedbr-meworkspaceexport-o./.dev/export-utils'Utils'# Export single ITEMdbr-meworkspaceexport-o./.dev/GetML'Utils/Download MovieLens.py'DBFSList DBFS items# List items on DBFSdbr-medbfsls--json-indent3FileStore/movielens[{"path":"/FileStore/movielens/ml-latest-small","is_dir":true,"file_size":0,"is_file":false,"human_size":"0 B"}]# Download a file and print to STDOUTdbr-medbfsgetml-latest-small/movies.csv# Download recursively entire directory and store locallydbr-medbfsget-oml-localml-latest-smallRunsSubmit a notebookImplements:https://docs.databricks.com/dev-tools/api/latest/jobs.html#runs-submit$dbr-merunssubmit"Utils/Download MovieLens"{"run_id": 4}You can retrieve the job information usingruns get:$dbr-merunsget4-i3Get run metadataImplements:Databricks REST runs/get$dbr-merunsget-i36{"job_id":6,"run_id":6,"creator_user_name":"your.name@gmail.com","number_in_job":1,"original_attempt_run_id":null,"state":{"life_cycle_state":"TERMINATED","result_state":"SUCCESS","state_message":""},"schedule":null,"task":{"notebook_task":{"notebook_path":"/Utils/Download MovieLens"}},"cluster_spec":{"existing_cluster_id":"xxxx-yyyyy-zzzzzz"},"cluster_instance":{"cluster_id":"xxxx-yyyyy-zzzzzz","spark_context_id":"783487348734873873"},"overriding_parameters":null,"start_time":1592062497162,"setup_duration":0,"execution_duration":11000,"cleanup_duration":0,"trigger":null,"run_name":"pyspark-me-1592062494","run_page_url":"https://westeurope.azuredatabricks.net/?o=398348734873487#job/6/run/1","run_type":"SUBMIT_RUN"}List RunsImplements:Databricks REST runs/list$dbr-merunslsTo get only the runs for a particular job:# Get job with job-id=4$dbr-merunsls4-i3{"runs":[{"job_id":4,"run_id":4,"creator_user_name":"your.name@gmail.com","number_in_job":1,"original_attempt_run_id":null,"state":{"life_cycle_state":"PENDING","state_message":""},"schedule":null,"task":{"notebook_task":{"notebook_path":"/Utils/Download MovieLens"}},"cluster_spec":{"existing_cluster_id":"xxxxx-yyyy-zzzzzzz"},"cluster_instance":{"cluster_id":"xxxxx-yyyy-zzzzzzz"},"overriding_parameters":null,"start_time":1592058826123,"setup_duration":0,"execution_duration":0,"cleanup_duration":0,"trigger":null,"run_name":"pyspark-me-1592058823","run_page_url":"https://westeurope.azuredatabricks.net/?o=abcdefghasdf#job/4/run/1","run_type":"SUBMIT_RUN"}],"has_more":false}Export runImplements:Databricks REST runs/export$dbr-merunsexport--content-only4>.dev/run-view.htmlGet run outputImplements:Databricks REST runs/get-output$dbr-merunsget-output-i36{"notebook_output":{"result":"Downloaded files: README.txt, links.csv, movies.csv, ratings.csv, tags.csv","truncated":false},"error":null,"metadata":{"job_id":5,"run_id":5,"creator_user_name":"your.name@gmail.com","number_in_job":1,"original_attempt_run_id":null,"state":{"life_cycle_state":"TERMINATED","result_state":"SUCCESS","state_message":""},"schedule":null,"task":{"notebook_task":{"notebook_path":"/Utils/Download MovieLens"}},"cluster_spec":{"existing_cluster_id":"xxxx-yyyyy-zzzzzzz"},"cluster_instance":{"cluster_id":"xxxx-yyyyy-zzzzzzz","spark_context_id":"8973498743973498"},"overriding_parameters":null,"start_time":1592062147101,"setup_duration":1000,"execution_duration":11000,"cleanup_duration":0,"trigger":null,"run_name":"pyspark-me-1592062135","run_page_url":"https://westeurope.azuredatabricks.net/?o=89798374987987#job/5/run/1","run_type":"SUBMIT_RUN"}}To get only the exit output:$dbr-merunsget-output-r6Downloaded files: README.txt, links.csv, movies.csv, ratings.csv, tags.csvBuild and publishpythonsetup.pysdistbdist_wheel
python-mtwineuploaddist/*