Package: databricks-sdk-secure
Databricks SDK for Python (Beta)Beta: This SDK is supported for production use cases,
but we do expect future releases to have some interface changes; seeInterface stability.
We are keen to hear feedback from you on these SDKs. Pleasefile issues, and we will address them.
| See also theSDK for Java| See also theSDK for Go| See also theTerraform Provider| See also cloud-specific docs (AWS,Azure,GCP)
| See also theAPI reference on readthedocsThe Databricks SDK for Python includes functionality to accelerate development withPythonfor the Databricks Lakehouse.
It covers all publicDatabricks REST APIoperations.
The SDK's internal HTTP client is robust and handles failures on different levels by performing intelligent retries.ContentsGetting startedCode examplesAuthenticationLong-running operationsPaginated responsesSingle-sign-on with OAuthLoggingIntegration withdbutilsInterface stabilityGetting startedPlease install Databricks SDK for Python viapip install databricks-sdkand instantiateWorkspaceClient:fromdatabricks.sdkimportWorkspaceClientw=WorkspaceClient()forcinw.clusters.list():print(c.cluster_name)Databricks SDK for Python is compatible with Python 3.7(untilJune 2023), 3.8, 3.9, 3.10, and 3.11.Note:Databricks Runtime starting from version 13.1 includes a bundled version of the Python SDK.It is highly recommended to upgrade to the latest version which you can do by running the following in a notebook cell:%pipinstall--upgradedatabricks-sdkfollowed bydbutils.library.restartPython()Code examplesThe Databricks SDK for Python comes with a number of examples demonstrating how to use the library for various common use-cases, includingUsing the SDK with OAuth from a webserverUsing long-running operationsAuthenticating a client app using OAuthThese examples and more are located in theexamples/directory of the Github repository.Some other examples of using the SDK include:Unity Catalog Automated Migrationheavily relies on Python SDK for working with Databricks APIs.ip-access-list-analyzerchecks & prunes invalid entries from IP Access Lists.AuthenticationIf you use Databricksconfiguration profilesor Databricks-specificenvironment variablesforDatabricks authentication, the only code required to start
working with a Databricks workspace is the following code snippet, which instructs the Databricks SDK for Python to use
itsdefault authentication flow:fromdatabricks.sdkimportWorkspaceClientw=WorkspaceClient()w.# press <TAB> for autocompletionThe conventional name for the variable that holds the workspace-level client of the Databricks SDK for Python isw, which is shorthand forworkspace.In this sectionDefault authentication flowDatabricks native authenticationAzure native authenticationOverriding .databrickscfgAdditional authentication configuration optionsDefault authentication flowIf you run theDatabricks Terraform Provider,
theDatabricks SDK for Go, theDatabricks CLI,
or applications that target the Databricks SDKs for other languages, most likely they will all interoperate nicely together.
By default, the Databricks SDK for Python tries the followingauthenticationmethods,
in the following order, until it succeeds:Databricks native authenticationAzure native authenticationIf the SDK is unsuccessful at this point, it returns an authentication error and stops running.You can instruct the Databricks SDK for Python to use a specific authentication method by setting theauth_typeargument
as described in the following sections.For each authentication method, the SDK searches for compatible authentication credentials in the following locations,
in the following order. Once the SDK finds a compatible set of credentials that it can use, it stops searching:Credentials that are hard-coded into configuration arguments.:warning:Caution: Databricks does not recommend hard-coding credentials into arguments, as they can be exposed in plain text in version control systems. Use environment variables or configuration profiles instead.Credentials in Databricks-specificenvironment variables.For Databricks native authentication, credentials in the.databrickscfgfile'sDEFAULTconfiguration profilefrom its default file location (~for Linux or macOS, and%USERPROFILE%for Windows).For Azure native authentication, the SDK searches for credentials through the Azure CLI as needed.Depending on the Databricks authentication method, the SDK uses the following information. Presented are theWorkspaceClientandAccountClientarguments (which have corresponding.databrickscfgfile fields), their descriptions, and any corresponding environment variables.Databricks native authenticationBy default, the Databricks SDK for Python initially triesDatabricks token authentication(auth_type='pat'argument). If the SDK is unsuccessful, it then tries Databricks basic (username/password) authentication (auth_type="basic"argument).For Databricks token authentication, you must providehostandtoken; or their environment variable or.databrickscfgfile field equivalents.For Databricks basic authentication, you must providehost,username, andpassword(for AWS workspace-level operations); orhost,account_id,username, andpassword(for AWS, Azure, or GCP account-level operations); or their environment variable or.databrickscfgfile field equivalents.ArgumentDescriptionEnvironment variablehost(String)The Databricks host URL for either the Databricks workspace endpoint or the Databricks accounts endpoint.DATABRICKS_HOSTaccount_id(String)The Databricks account ID for the Databricks accounts endpoint. Only has effect whenHostis eitherhttps://accounts.cloud.databricks.com/(AWS),https://accounts.azuredatabricks.net/(Azure), orhttps://accounts.gcp.databricks.com/(GCP).DATABRICKS_ACCOUNT_IDtoken(String)The Databricks personal access token (PAT)(AWS, Azure, and GCP)or Azure Active Directory (Azure AD) token(Azure).DATABRICKS_TOKENusername(String)The Databricks username part of basic authentication. Only possible whenHostis*.cloud.databricks.com(AWS).DATABRICKS_USERNAMEpassword(String)The Databricks password part of basic authentication. Only possible whenHostis*.cloud.databricks.com(AWS).DATABRICKS_PASSWORDFor example, to use Databricks token authentication:fromdatabricks.sdkimportWorkspaceClientw=WorkspaceClient(host=input('Databricks Workspace URL: '),token=input('Token: '))Azure native authenticationBy default, the Databricks SDK for Python first tries Azure client secret authentication (auth_type='azure-client-secret'argument). If the SDK is unsuccessful, it then tries Azure CLI authentication (auth_type='azure-cli'argument). SeeManage service principals.The Databricks SDK for Python picks up an Azure CLI token, if you've previously authenticated as an Azure user by runningaz loginon your machine. SeeGet Azure AD tokens for users by using the Azure CLI.To authenticate as an Azure Active Directory (Azure AD) service principal, you must provide one of the following. See alsoAdd a service principal to your Azure Databricks account:azure_workspace_resource_id,azure_client_secret,azure_client_id, andazure_tenant_id; or their environment variable or.databrickscfgfile field equivalents.azure_workspace_resource_idandazure_use_msi; or their environment variable or.databrickscfgfile field equivalents.ArgumentDescriptionEnvironment variableazure_workspace_resource_id(String)The Azure Resource Manager ID for the Azure Databricks workspace, which is exchanged for a Databricks host URL.DATABRICKS_AZURE_RESOURCE_IDazure_use_msi(Boolean)trueto use Azure Managed Service Identity passwordless authentication flow for service principals.This feature is not yet implemented in the Databricks SDK for Python.ARM_USE_MSIazure_client_secret(String)The Azure AD service principal's client secret.ARM_CLIENT_SECRETazure_client_id(String)The Azure AD service principal's application ID.ARM_CLIENT_IDazure_tenant_id(String)The Azure AD service principal's tenant ID.ARM_TENANT_IDazure_environment(String)The Azure environment type (such as Public, UsGov, China, and Germany) for a specific set of API endpoints. Defaults toPUBLIC.ARM_ENVIRONMENTFor example, to use Azure client secret authentication:fromdatabricks.sdkimportWorkspaceClientw=WorkspaceClient(host=input('Databricks Workspace URL: '),azure_workspace_resource_id=input('Azure Resource ID: '),azure_tenant_id=input('AAD Tenant ID: '),azure_client_id=input('AAD Client ID: '),azure_client_secret=input('AAD Client Secret: '))Please see more examples inthis document.Google Cloud Platform native authenticationBy default, the Databricks SDK for Python first tries GCP credentials authentication (auth_type='google-credentials', argument). If the SDK is unsuccessful, it then tries Google Cloud Platform (GCP) ID authentication (auth_type='google-id', argument).The Databricks SDK for Python picks up an OAuth token in the scope of the Google Default Application Credentials (DAC) flow. This means that if you have rungcloud auth application-default loginon your development machine, or launch the application on the compute, that is allowed to impersonate the Google Cloud service account specified ingoogle_service_account. Authentication should then work out of the box. SeeCreating and managing service accounts.To authenticate as a Google Cloud service account, you must provide one of the following:hostandgoogle_credentials; or their environment variable or.databrickscfgfile field equivalents.hostandgoogle_service_account; or their environment variable or.databrickscfgfile field equivalents.ArgumentDescriptionEnvironment variablegoogle_credentials(String)GCP Service Account Credentials JSON or the location of these credentials on the local filesystem.GOOGLE_CREDENTIALSgoogle_service_account(String)The Google Cloud Platform (GCP) service account e-mail used for impersonation in the Default Application Credentials Flow that does not require a password.DATABRICKS_GOOGLE_SERVICE_ACCOUNTFor example, to use Google ID authentication:fromdatabricks.sdkimportWorkspaceClientw=WorkspaceClient(host=input('Databricks Workspace URL: '),google_service_account=input('Google Service Account: '))Overriding.databrickscfgForDatabricks native authentication, you can override the default behavior for using.databrickscfgas follows:ArgumentDescriptionEnvironment variableprofile(String)A connection profile specified within.databrickscfgto use instead ofDEFAULT.DATABRICKS_CONFIG_PROFILEconfig_file(String)A non-default location of the Databricks CLI credentials file.DATABRICKS_CONFIG_FILEFor example, to use a profile namedMYPROFILEinstead ofDEFAULT:fromdatabricks.sdkimportWorkspaceClientw=WorkspaceClient(profile='MYPROFILE')# Now call the Databricks workspace APIs as desired...Additional authentication configuration optionsFor all authentication methods, you can override the default behavior in client arguments as follows:ArgumentDescriptionEnvironment variableauth_type(String)When multiple auth attributes are available in the environment, use the auth type specified by this argument. This argument also holds the currently selected auth.DATABRICKS_AUTH_TYPEhttp_timeout_seconds(Integer)Number of seconds for HTTP timeout. Default is60.(None)retry_timeout_seconds(Integer)Number of seconds to keep retrying HTTP requests. Default is300 (5 minutes).(None)debug_truncate_bytes(Integer)Truncate JSON fields in debug logs above this limit. Default is 96.DATABRICKS_DEBUG_TRUNCATE_BYTESdebug_headers(Boolean)trueto debug HTTP headers of requests made by the application. Default isfalse, as headers contain sensitive data, such as access tokens.DATABRICKS_DEBUG_HEADERSrate_limit(Integer)Maximum number of requests per second made to Databricks REST API.DATABRICKS_RATE_LIMITFor example, to turn on debug HTTP headers:fromdatabricks.sdkimportWorkspaceClientw=WorkspaceClient(debug_headers=True)# Now call the Databricks workspace APIs as desired...Long-running operationsWhen you invoke a long-running operation, the SDK provides a high-level API totriggerthese operations andwaitfor the related entities
to reach the correct state or return the error message in case of failure. All long-running operations return genericWaitinstance withresult()method to get a result of long-running operation, once it's finished. Databricks SDK for Python picks the most reasonable default timeouts for
every method, but sometimes you may find yourself in a situation, where you'd want to providedatetime.timedelta()as the value oftimeoutargument toresult()method.There are a number of long-running operations in Databricks APIs such as managing:Clusters,Command executionJobsLibrariesDelta Live Tables pipelinesDatabricks SQL warehouses.For example, in the Clusters API, once you create a cluster, you receive a cluster ID, and the cluster is in thePENDINGstate Meanwhile
Databricks takes care of provisioning virtual machines from the cloud provider in the background. The cluster is
only usable in theRUNNINGstate and so you have to wait for that state to be reached.Another example is the API for running a job or repairing the run: right after
the run starts, the run is in thePENDINGstate. The job is only considered to be finished when it is in either
theTERMINATEDorSKIPPEDstate. Also you would likely need the error message if the long-running
operation times out and fails with an error code. Other times you may want to configure a custom timeout other than
the default of 20 minutes.In the following example,w.clusters.createreturnsClusterInfoonly once the cluster is in theRUNNINGstate,
otherwise it will timeout in 10 minutes:importdatetimeimportloggingfromdatabricks.sdkimportWorkspaceClientw=WorkspaceClient()info=w.clusters.create_and_wait(cluster_name='Created cluster',spark_version='12.0.x-scala2.12',node_type_id='m5d.large',autotermination_minutes=10,num_workers=1,timeout=datetime.timedelta(minutes=10))logging.info(f'Created:{info}')Please look at theexamples/starting_job_and_waiting.pyfor a more advanced usage:importdatetimeimportloggingimporttimefromdatabricks.sdkimportWorkspaceClientimportdatabricks.sdk.service.jobsasjw=WorkspaceClient()# create a dummy file on DBFS that just sleeps for 10 secondspy_on_dbfs=f'/home/{w.current_user.me().user_name}/sample.py'withw.dbfs.open(py_on_dbfs,write=True,overwrite=True)asf:f.write(b'import time; time.sleep(10); print("Hello, World!")')# trigger one-time-run job and get waiter objectwaiter=w.jobs.submit(run_name=f'py-sdk-run-{time.time()}',tasks=[j.RunSubmitTaskSettings(task_key='hello_world',new_cluster=j.BaseClusterInfo(spark_version=w.clusters.select_spark_version(long_term_support=True),node_type_id=w.clusters.select_node_type(local_disk=True),num_workers=1),spark_python_task=j.SparkPythonTask(python_file=f'dbfs:{py_on_dbfs}'),)])logging.info(f'starting to poll:{waiter.run_id}')# callback, that receives a polled entity between state updatesdefprint_status(run:j.Run):statuses=[f'{t.task_key}:{t.state.life_cycle_state}'fortinrun.tasks]logging.info(f'workflow intermediate status:{", ".join(statuses)}')# If you want to perform polling in a separate thread, process, or service,# you can use w.jobs.wait_get_run_job_terminated_or_skipped(#   run_id=waiter.run_id,#   timeout=datetime.timedelta(minutes=15),#   callback=print_status) to achieve the same results.## Waiter interface allows for `w.jobs.submit(..).result()` simplicity in# the scenarios, where you need to block the calling thread for the job to finish.run=waiter.result(timeout=datetime.timedelta(minutes=15),callback=print_status)logging.info(f'job finished:{run.run_page_url}')Paginated responsesOn the platform side the Databricks APIs have different wait to deal with pagination:Some APIs follow the offset-plus-limit paginationSome start their offsets from 0 and some from 1Some use the cursor-based iterationOthers just return all results in a single responseThe Databricks SDK for Python hides this  complexity
underIterator[T]abstraction, where multi-page resultsyielditems. Python typing helps to auto-complete
the individual item fields.importloggingfromdatabricks.sdkimportWorkspaceClientw=WorkspaceClient()forrepoinw.repos.list():logging.info(f'Found repo:{repo.path}')Please look at theexamples/last_job_runs.pyfor a more advanced usage:importloggingfromcollectionsimportdefaultdictfromdatetimeimportdatetime,timezonefromdatabricks.sdkimportWorkspaceClientlatest_state={}all_jobs={}durations=defaultdict(list)w=WorkspaceClient()forjobinw.jobs.list():all_jobs[job.job_id]=jobforruninw.jobs.list_runs(job_id=job.job_id,expand_tasks=False):durations[job.job_id].append(run.run_duration)ifjob.job_idnotinlatest_state:latest_state[job.job_id]=runcontinueifrun.end_time<latest_state[job.job_id].end_time:continuelatest_state[job.job_id]=runsummary=[]forjob_id,runinlatest_state.items():summary.append({'job_name':all_jobs[job_id].settings.name,'last_status':run.state.result_state,'last_finished':datetime.fromtimestamp(run.end_time/1000,timezone.utc),'average_duration':sum(durations[job_id])/len(durations[job_id])})forlineinsorted(summary,key=lambdas:s['last_finished'],reverse=True):logging.info(f'Latest:{line}')Single-Sign-On (SSO) with OAuthAuthorization Code flow with PKCEFor a regular web app running on a server, it's recommended to use the Authorization Code Flow to obtain an Access Token
and a Refresh Token. This method is considered safe because the Access Token is transmitted directly to the server
hosting the app, without passing through the user's web browser and risking exposure.To enhance the security of the Authorization Code Flow, the PKCE (Proof Key for Code Exchange) mechanism can be
employed. With PKCE, the calling application generates a secret called the Code Verifier, which is verified by
the authorization server. The app also creates a transform value of the Code Verifier, called the Code Challenge,
and sends it over HTTPS to obtain an Authorization Code. By intercepting the Authorization Code, a malicious attacker
cannot exchange it for a token without possessing the Code Verifier.Thepresented sampleis a Python3 script that uses the Flask web framework along with Databricks SDK for Python to demonstrate how to
implement the OAuth Authorization Code flow with PKCE security. It can be used to build an app where each user uses
their identity to access Databricks resources. The script can be executed with or without client and secret credentials
for a custom OAuth app.Databricks SDK for Python exposes theoauth_client.initiate_consent()helper to acquire user redirect URL and initiate
PKCE state verification. Application developers are expected to persistRefreshableCredentialsin the webapp session
and restore it viaRefreshableCredentials.from_dict(oauth_client, session['creds'])helpers.Works for both AWS and Azure. Not supported for GCP at the moment.fromdatabricks.sdk.oauthimportOAuthClientoauth_client=OAuthClient(host='<workspace-url>',client_id='<oauth client ID>',redirect_url=f'http://host.domain/callback',scopes=['clusters'])importsecretsfromflaskimportFlask,render_template_string,request,redirect,url_for,sessionAPP_NAME='flask-demo'app=Flask(APP_NAME)app.secret_key=secrets.token_urlsafe(32)@app.route('/callback')defcallback():fromdatabricks.sdk.oauthimportConsentconsent=Consent.from_dict(oauth_client,session['consent'])session['creds']=consent.exchange_callback_parameters(request.args).as_dict()returnredirect(url_for('index'))@app.route('/')defindex():if'creds'notinsession:consent=oauth_client.initiate_consent()session['consent']=consent.as_dict()returnredirect(consent.auth_url)fromdatabricks.sdkimportWorkspaceClientfromdatabricks.sdk.oauthimportSessionCredentialscredentials_provider=SessionCredentials.from_dict(oauth_client,session['creds'])workspace_client=WorkspaceClient(host=oauth_client.host,product=APP_NAME,credentials_provider=credentials_provider)returnrender_template_string('...',w=workspace_client)SSO for local scripts on development machinesFor applications, that do run on developer workstations, Databricks SDK for Python providesauth_type='external-browser'utility, that opens up a browser for a user to go through SSO flow. Azure support is still in the early experimental
stage.fromdatabricks.sdkimportWorkspaceClienthost=input('Enter Databricks host: ')w=WorkspaceClient(host=host,auth_type='external-browser')clusters=w.clusters.list()forclinclusters:print(f' -{cl.cluster_name}is{cl.state}')Creating custom OAuth applicationsIn order to use OAuth with Databricks SDK for Python, you should useaccount_client.custom_app_integration.createAPI.importlogging,getpassfromdatabricks.sdkimportAccountClientaccount_client=AccountClient(host='https://accounts.cloud.databricks.com',account_id=input('Databricks Account ID: '),username=input('Username: '),password=getpass.getpass('Password: '))logging.info('Enrolling all published apps...')account_client.o_auth_enrollment.create(enable_all_published_apps=True)status=account_client.o_auth_enrollment.get()logging.info(f'Enrolled all published apps:{status}')custom_app=account_client.custom_app_integration.create(name='awesome-app',redirect_urls=[f'https://host.domain/path/to/callback'],confidential=True)logging.info(f'Created new custom app: 'f'--client_id{custom_app.client_id}'f'--client_secret{custom_app.client_secret}')LoggingThe Databricks SDK for Python seamlessly integrates with the standardLogging facility for Python.
This allows developers to easily enable and customize logging for their Databricks Python projects.
To enable debug logging in your Databricks Python project, you can follow the example below:importlogging,syslogging.basicConfig(stream=sys.stderr,level=logging.INFO,format='%(asctime)s[%(name)s][%(levelname)s]%(message)s')logging.getLogger('databricks.sdk').setLevel(logging.DEBUG)fromdatabricks.sdkimportWorkspaceClientw=WorkspaceClient(debug_truncate_bytes=1024,debug_headers=False)forclusterinw.clusters.list():logging.info(f'Found cluster:{cluster.cluster_name}')In the above code snippet, the logging module is imported and thebasicConfig()method is used to set the logging level toDEBUG.
This will enable logging at the debug level and above. Developers can adjust the logging level as needed to control the verbosity of the logging output.
The SDK will log all requests and responses to standard error output, using the format>for requests and<for responses.
In some cases, requests or responses may be truncated due to size considerations. If this occurs, the log message will include
the text... (XXX additional elements)to indicate that the request or response has been truncated. To increase the truncation limits,
developers can set thedebug_truncate_bytesconfiguration property or theDATABRICKS_DEBUG_TRUNCATE_BYTESenvironment variable.
To protect sensitive data, such as authentication tokens, passwords, or any HTTP headers, the SDK will automatically replace these
values with**REDACTED**in the log output. Developers can disable this redaction by setting thedebug_headersconfiguration property toTrue.2023-03-22 21:19:21,702 [databricks.sdk][DEBUG] GET /api/2.0/clusters/list
< 200 OK
< {
<   "clusters": [
<     {
<       "autotermination_minutes": 60,
<       "cluster_id": "1109-115255-s1w13zjj",
<       "cluster_name": "DEFAULT Test Cluster",
<       ... truncated for brevity
<     },
<     "... (47 additional elements)"
<   ]
< }Overall, the logging capabilities provided by the Databricks SDK for Python can be a powerful tool for monitoring and troubleshooting your
Databricks Python projects. Developers can use the various logging methods and configuration options provided by the SDK to customize
the logging output to their specific needs.Interaction withdbutilsYou can use the client-side implementation ofdbutilsby accessingdbutilsproperty on theWorkspaceClient.
Most of thedbutils.fsoperations anddbutils.secretsare implemented natively in Python within Databricks SDK. Non-SDK implementations still require a Databricks cluster,
that you have to specify through thecluster_idconfiguration attribute orDATABRICKS_CLUSTER_IDenvironment variable. Don't worry if cluster is not running: internally,
Databricks SDK for Python callsw.clusters.ensure_cluster_is_running().fromdatabricks.sdkimportWorkspaceClientw=WorkspaceClient()dbutils=w.dbutilsfiles_in_root=dbutils.fs.ls('/')print(f'number of files in root:{len(files_in_root)}')Alternatively, you can importdbutilsfromdatabricks.sdk.runtimemodule, but you have to make sure that all configuration is alreadypresent in the environment variables:fromdatabricks.sdk.runtimeimportdbutilsforsecret_scopeindbutils.secrets.listScopes():forsecret_metadataindbutils.secrets.list(secret_scope.name):print(f'found{secret_metadata.key}secret in{secret_scope.name}scope')Interface stabilityDatabricks is actively working on stabilizing the Databricks SDK for Python's interfaces.
API clients for all services are generated from specification files that are synchronized from the main platform.
You are highly encouraged to pin the exact dependency version and read thechangelogwhere Databricks documents the changes. Databricks may have minordocumentedbackward-incompatible changes, such as renaming some type names to bring more consistency.