Saravanan666/pypi-org huggingface dataset:
Package: databricks-aws-utils
Databricks AWS UtilsDatabricks AWS Utils is a library to abstract Databricks integration with AWS ServicesFeaturesConvert Delta Table to be consumed by AWS Athena with Schema evolutionRun queries against AWS RDS using AWS Secrets Manager to retrieve the connection properties and returns as Spark DataFrameInstallpip install databricks-aws-utilsDelta Table to AWS AthenaMotivationCurrently, delta tables are only compatible with AWS Athena engine v3, however, even with the compatibility, there are some limitations regarding the schema evolution, where the schema is not fully or correctly synchronized with the AWS Glue catalog, causing problems when querying the table.To solve this problem, we created this library to convert the delta table columns to be compatible with the AWS Glue catalog and update the table metadata, allowing the table to be queried correctly by AWS Athena.Usagefromdatabricks_aws_utils.delta_tableimportDeltaTableUtils...DeltaTableUtils(spark,'my_schema.my_table_name').to_athena_v3()Theto_athena_v3function uses the spark session to capture the current delta schema and update the glue table.NOTE: This feature is only compatible with AWS Athena engine v3, and the Databricks cluster must have access to the AWS Glue catalog.NOTE: This feature is not supported by Databricks Unity Catalog, since it does not allow queries from AWS Athena.Custom IAM RoleIf you need to use a custom IAM Role to update the AWS Glue table, you can pass the role name as a parameter to theDeltaTableUtilsclass.fromdatabricks_aws_utils.delta_tableimportDeltaTableUtils...DeltaTableUtils(spark,'my_schema.my_table_name',iam_role='my_custom_iam_role').to_athena_v3()NOTE: The Databricks cluster must have permission to assume the custom IAM Role.Athena Engine v2AWS Athena engine v2 doesn't support delta tables, so, to query a delta table using AWS Athena engine v2, it's necessary to generate Hive Symlink from the delta table and point to a different table.fromdatabricks_aws_utils.delta_tableimportDeltaTableUtils...DeltaTableUtils(spark,'my_schema.my_table_name').to_athena('my_schema','my_symlink_table_name')NOTE: The schema name provided in theto_athenadoesn't need to be the same as the delta table schema.ContributingSee ourContributing GuideChange LogSee ourChange Log
Package: databricks-azure-ad-sync-provider
databricks-azure-ad-sync-providerdatabricks-azure-ad-sync-provideris a package that allows to sync users, groups and service principals from Microsoft Entra ID to Databricks.Requirementsdatabricks-azure-ad-sync-provideris built on top of:python 3.9+AuthenticationAuthenticate to you Microsoft Entra ID and Azure Databricks to use the package.
There are multiple ways available for that, we suggest the following:1. With Azure CLIAuthenticate to your Azure account on your machine withaz login.Specify necessary variablesWith environment variables: Setup forDATABRICKS_HOSTandDATABRICKS_ACCOUNT_IDWith a.databrickscfgfile: Create the.databrickscfgfile (~ for Linux or macOS, and %USERPROFILE% for Windows) containing the
following info:[DEFAULT]
host = https://accounts.azuredatabricks.net/
account_id = <Databricks account id>2. With Microsoft Entra ID service principal:Create a service principal in Microsoft Entra ID and add it to Azure Databricks and grant it target permissions (seereference documentation).Specify necessary variablesWith environment variablesFor Azure: AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_CLIENT_SECRET (seeMicrosoft Entra ID authentication)For Databricks: DATABRICKS_HOST, DATABRICKS_ACCOUNT_ID, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET (seeDatabricks authentication)With a.databrickscfgfile: Create a .databrickscfg file (~ for Linux or macOS, and %USERPROFILE% for Windows) containing the following info:[DEFAULT]
host = https://accounts.azuredatabricks.net/
account_id = <Databricks account id>
azure_tenant_id = <Azure tenant id>
azure_client_id = <Azure service principal application ID>
azure_client_secret = <Azure service principal secret>Installationpip install databricks-azure-ad-sync-providerUsageSync execution<fct_name> -f/--file <path to your yaml file> -d/--delete-f/--file :  it's required to provide the object ID(s) of Azure groups and (optionally) exclude object ID(s) in a yaml file (ex. see test.yaml).-d/--delete: If provided, this enablew to delete identities in Databricks.
Package: databricks-cicd
Databricks CI/CDThis is a tool for building CI/CD pipelines for Databricks. It is a python package that
works in conjunction with a custom GIT repository (or a simple file structure) to validate
and deploy content to databricks. Currently, it can handle the following content:Workspace- a collection of notebooks written in Scala, Python, R or SQLJobs- list of Databricks jobsClustersInstance PoolsDBFS- an arbitrary collection of files that may be deployed on a Databricks workspaceInstallationpip install databricks-cicdRequirementsTo use this tool, you need a source directory structure (preferably as a private GIT repository)
that has the following structure:any_local_folder_or_git_repo/
├── workspace/
│   ├── some_notebooks_subdir
│   │   └── Notebook 1.py
│   ├── Notebook 2.sql
│   ├── Notebook 3.r
│   └── Notebook 4.scala
├── jobs/
│   ├── My first job.json
│   └── Side gig.json
├── clusters/
│   ├── orion.json
│   └── Another cluster.json
├── instance_pools/
│   ├── Pool 1.json
│   └── Pool 2.json
└── dbfs/
    ├── strawbery_jam.jar
    ├── subdir
    │   └── some_other.jar
    ├── some_python.egg
    └── Ice cream.jpegNote:All folder names represent the default and can be configured. This is just a sample.UsageFor the latest options and commands run:cicd -hA sample command could be:cicddeploy\-wsample_12432.7.azuredatabricks.net\-ujohn.smith@domain.com\-tdapi_sample_token_0d5-2\-lp'~/git/my-private-repo'\-tp/blabla\-cDEV.ini\--verboseNote:Paths for windows need to be in double quotesThe default configuration is defined indefault.iniand can be overridden with a
custom ini file using the -c option, usually one config file per target environment. (sample)Create contentNotebooks:Add a notebook to sourceOn the databricks UI go to your notebook.Click onFile -> Export -> Source file.Add that file to theworkspacefolder of this repowithout changing the file name.Jobs:Add a job to sourceGet the source of the job and write it to a file. You need to have theDatabricks CLIandJQinstalled.
For Windows, it is easier to rename thejq-win64.exetojq.exeand place it
inc:\Windows\System32folder. Then on Windows/Linux/MAC:databricks jobs get --job-id 74 | jq .settings > Job_Name.jsonThis downloads the source JSON of the job from the databricks server and pulls only the settings from it,
then writes it in to a file.Note:The file name should be the same as the job name within the json file. Please, avoid spaces
in names.Add that file to thejobsfolderClusters:Add a cluster to sourceGet the source of the cluster and write it to a file.databricks clusters get --cluster-name orion > orion.jsonNote:The file name should be the same as the cluster name within the json file. Please, avoid spaces
in names.Add that file to theclustersfolderInstance pools:Add an instance pool to sourceSimilar to clusters, just useinstance-poolsinstead ofclustersDBFS:Add a file to dbfsJust add a file to the thedbfsfolder.
Package: databricks-clay
Documentationhttps://databricks-clay.readthedocs.io/en/latest/Development SetupCreate virual environment and install dependencies for local development:python3.7 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install -r dev_requirements.txtExporting variables doesn't make for a great development experience so I recommend using the enviroment manager tools of your editor and for testing create a ./pytest.ini that looks like this:[pytest]
env =
    VARIABLE=value
    VARIABLE2=value2REMINDER: do NOT commit any files that contain security tokensGit ignore already contains an exclusion for pytest.iniBuildBuild python wheel:python setup.py sdist bdist_wheelThere is a CI build configured for this repo that builds on main origin on a private Azure DevOps service.TestDependencies for testing:pip install --editable .Run tests:pytestTest Coverage:pytest --cov=databricks-clay --cov-report=htmlView the report in a browser:./htmlcov/index.html
Package: databricks-cli
NoteDatabricks recommends that you use newer Databricks CLI versions 0.200 and above instead of legacy Databricks CLI versions 0.17 and below (located in this repository). You can find the new CLI at https://github.com/databricks/cli.If you have been using the code in this repository as a Python SDK for interacting with the Databricks APIs, we recommend you use the new dedicated SDK package that can be found at https://github.com/databricks/databricks-sdk-py.databricks-cliThe Databricks Command Line Interface (CLI) is an open source tool which provides an easy to use interface to
the Databricks platform. The CLI is built on top of the Databricks REST APIs.Note: This CLI is no longer under active development and has been released as an experimental client.Please leave bug reports as issues on ourGitHub project.RequirementsPython Version >= 3.7InstallationTo install simply runpip install--upgradedatabricks-cliThen set up authentication using username/password orauthentication token. Credentials are stored at~/.databrickscfg.databricks configure(enter hostname/username/password at prompt)databricks configure--token(enter hostname/auth-token at prompt)Multiple connection profiles are also supported withdatabricks configure--profile<profile>[--token].
The connection profile can be used as such:databricks workspace ls--profile<profile>.To test that your authentication information is working, try a quick test likedatabricks workspace ls.Known IssuesAttributeError: 'module' object has no attribute 'PROTOCOL_TLSv1_2'The Databricks web service requires clients speak TLSV1.2. The built in
version of Python for MacOS does not have this version of TLS built in.To use the Databricks CLI you must install a version of Python that hasssl.PROTOCOL_TLSv1_2.
For MacOS, the easiest way may be to install Python withHomebrew.Using Docker# build image
docker build -t databricks-cli .

# run container
docker run -it databricks-cli

# run command in docker
docker run -it databricks-cli fs --helpDocumentationFor the latest CLI documentation, seeDatabricksAzure Databricks
Package: databricks-cli-uc
The Databricks Command Line Interface (CLI) is an open source tool which provides an easy to use interface to
the Databricks platform. The CLI is built on top of the Databricks REST APIs.This-ucvariant ofdatabricks-cliincludes commands for interacting with the Unity Catalog API using theunity-catalogcommand. This UC functionality is in the process of being merged to the mainlinedatabricks-climodule and is available here for early access.Note: This CLI is under active development and is released as an experimental client. This means that interfaces are still subject to change.If you’re interested in contributing to the project please reach out.
In addition, please leave bug reports as issues on ourGitHub project.RequirementsPython Version > 2.7.9 or > 3.6InstallationTo install simply runpip install--upgradedatabricks-cliThen set up authentication using username/password orauthentication token. Credentials are stored at~/.databrickscfg.databricks configure(enter hostname/username/password at prompt)databricks configure--token(enter hostname/auth-token at prompt)Multiple connection profiles are also supported withdatabricks configure--profile<profile>[--token].
The connection profile can be used as such:databricks workspace ls--profile<profile>.To test that your authentication information is working, try a quick test likedatabricks workspace ls.Known IssuesAttributeError: 'module' object has no attribute 'PROTOCOL_TLSv1_2'The Databricks web service requires clients speak TLSV1.2. The built in
version of Python for MacOS does not have this version of TLS built in.To use the Databricks CLI you must install a version of Python that hasssl.PROTOCOL_TLSv1_2.
For MacOS, the easiest way may be to install Python withHomebrew.Using Docker# build image
docker build -t databricks-cli .

# run container
docker run -it databricks-cli

# run command in docker
docker run -it databricks-cli fs --helpDocumentationFor the latest CLI documentation, seeDatabricksAzure Databricks
Package: databricks-connect
Databricks ConnectDatabricks Connect allows you to connect popular IDEs such as Visual Studio Code, PyCharm, IntelliJ IDEA,
notebook servers, and other custom applications to Databricks clusters.Databricks Connect is a client library for the Databricks Runtime. It allows you to write code using Spark APIs and
run them remotely on a Databricks cluster instead of in the local Spark session.Refer to the official docs athttps://docs.databricks.com/en/dev-tools/databricks-connect/
Package: databricks-converter
DatabricksConverterVersion0.0.9DescriptionThis cli tool helps you to work in your favourite python IDE and then converts the files in databricks notebook.
You can also do the reverse conversion. Starting from a databricks notebook, you can convert it into a python file.WhyI struggled a lot working in the databricks interface because you don't have a lot of IDE feature like:autocompletecode analysiscode lintingtestingHow it worksYou can install the package with pip:pip install Databricks2PyThen you have two options:if you need to convert a databricks notebook to python, simply rundatabricks-converter to-py file/folder --destination your-destination-pathif you need to convert a python module to a databricks notebook, simply rundatabricks-converter to-databricks file/folder --destination your-destination-pathThe converted files have a specific suffix_to_py.pyand_to_databricks.py. So, if you do not want to put these files under version control, you must add these rules to you.gitignorefile:# in .gitignore
*_to_py.py
*_to_databricks.pyTODOImprove conversion of complex import path
Package: databricks-dash
databricks-dashdatabricks-dash is a part ofDash Enterprise,
Plotly's end-to-end Dash application platform. This public copy of the package
is a stub published on pypi.org designed to shield Dash Enterprise users fromdependency confusion,
a remote code execution attack publicly disclosed in 2021 by Alex Birsan.If you are a Dash Enterprise user, you will only see this public package if your
app is not properly configured to find the local package repository built into
Dash Enterprise. Visit the docs for your Dash Enterprise installation for details.If you never see this public package, it's still doing its job of protecting you
from the dependency confusion attack!
Package: databricks-feature-engineering
The Databricks Feature Engineering client is used to:Create, read, and write feature tablesTrain models on feature dataPublish feature tables to online stores for real-time servingUpgrade workspace feature table metadata to Unity CatalogDocumentationDocumentation can be found per-cloud at:AWSAzureGCPFor release notes, seeAWSAzureGCPLimitationsThis library can only run on Databricks runtimes. It can be installed locally or in CI/CD environments for unit testing.DB licenseCopyright (2022) Databricks, Inc.This library (the "Software") may not be used except in connection with the Licensee's use of the Databricks Platform Services
pursuant to an Agreement (defined below) between Licensee (defined below) and Databricks, Inc. ("Databricks"). This Software
shall be deemed part of the Downloadable Services under the Agreement, or if the Agreement does not define Downloadable Services,
Subscription Services, or if neither are defined then the term in such Agreement that refers to the applicable Databricks Platform
Services (as defined below) shall be substituted herein for "Downloadable Services". Licensee's use of the Software must comply at
all times with any restrictions applicable to the Downlodable Services and Subscription Services, generally, and must be used in
accordance with any applicable documentation.Additionally, and notwithstanding anything in the Agreement to the contrary:THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.If you have not agreed to an Agreement or otherwise do not agree to these terms, you may not use the Software.This license terminates automatically upon the termination of the Agreement or Licensee's breach of these terms.Agreement: the agreement between Databricks and Licensee governing the use of the Databricks Platform Services, which shall be, with
respect to Databricks, the Databricks Terms of Service located atwww.databricks.com/termsofservice, and with respect to Databricks
Community Edition, the Community Edition Terms of Service located atwww.databricks.com/ce-termsofuse, in each case unless Licensee
has entered into a separate written agreement with Databricks governing the use of the applicable Databricks Platform Services.Databricks Platform Services: the Databricks services or the Databricks Community Edition services, according to where the Software is used.Licensee: the user of the Software, or, if the Software is being used on behalf of a company, the company.
Package: databricks-feature-lookup
The Databricks feature lookup client retrieves features from online stores prior to model inference.DocumentationDatabricks Feature Store onAWS|Azure|GCPFor release notes, seeAWSAzureGCPKnown issuesKnown conflicts when installed on Databricks ML RuntimesThis library is for online model serving systems and not intended to work on Databricks ML runtimes. Installing this on Databricks Runtime for ML will conflict with pre-installed Feature Store client modules.LicenseDB licenseCopyright (2021) Databricks, Inc.This library (the "Software") may not be used except in connection with the Licensee's use of the Databricks Platform Services
pursuant to an Agreement (defined below) between Licensee (defined below) and Databricks, Inc. ("Databricks"). This Software
shall be deemed part of the Downloadable Services under the Agreement, or if the Agreement does not define Downloadable Services,
Subscription Services, or if neither are defined then the term in such Agreement that refers to the applicable Databricks Platform
Services (as defined below) shall be substituted herein for "Downloadable Services". Licensee's use of the Software must comply at
all times with any restrictions applicable to the Downlodable Services and Subscription Services, generally, and must be used in
accordance with any applicable documentation.Additionally, and notwithstanding anything in the Agreement to the contrary:THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.If you have not agreed to an Agreement or otherwise do not agree to these terms, you may not use the Software.This license terminates automatically upon the termination of the Agreement or Licensee's breach of these terms.Agreement: the agreement between Databricks and Licensee governing the use of the Databricks Platform Services, which shall be, with
respect to Databricks, the Databricks Terms of Service located atwww.databricks.com/termsofservice, and with respect to Databricks
Community Edition, the Community Edition Terms of Service located atwww.databricks.com/ce-termsofuse, in each case unless Licensee
has entered into a separate written agreement with Databricks governing the use of the applicable Databricks Platform Services.Databricks Platform Services: the Databricks services or the Databricks Community Edition services, according to where the Software is used.Licensee: the user of the Software, or, if the Software is being used on behalf of a company, the company.
Package: databricks-feature-store
Starting with v0.17.0, thedatabricks-feature-storepackage isdeprecated. Usedatabricks-feature-engineeringinstead.Migrate to databricks-feature-engineeringUsepip install databricks-feature-engineeringto replacepip install databricks-feature-store.All existing modules indatabricks-feature-storehave been moved todatabricks-feature-engineering, so you do not have to change any code. Import statements such asfrom databricks.feature_store import FeatureStoreClientwill continue working after installingdatabricks-feature-engineering.LicenseDB licenseCopyright (2022) Databricks, Inc.This library (the "Software") may not be used except in connection with the Licensee's use of the Databricks Platform Services
pursuant to an Agreement (defined below) between Licensee (defined below) and Databricks, Inc. ("Databricks"). This Software
shall be deemed part of the Downloadable Services under the Agreement, or if the Agreement does not define Downloadable Services,
Subscription Services, or if neither are defined then the term in such Agreement that refers to the applicable Databricks Platform
Services (as defined below) shall be substituted herein for "Downloadable Services". Licensee's use of the Software must comply at
all times with any restrictions applicable to the Downlodable Services and Subscription Services, generally, and must be used in
accordance with any applicable documentation.Additionally, and notwithstanding anything in the Agreement to the contrary:THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.If you have not agreed to an Agreement or otherwise do not agree to these terms, you may not use the Software.This license terminates automatically upon the termination of the Agreement or Licensee's breach of these terms.Agreement: the agreement between Databricks and Licensee governing the use of the Databricks Platform Services, which shall be, with
respect to Databricks, the Databricks Terms of Service located atwww.databricks.com/termsofservice, and with respect to Databricks
Community Edition, the Community Edition Terms of Service located atwww.databricks.com/ce-termsofuse, in each case unless Licensee
has entered into a separate written agreement with Databricks governing the use of the applicable Databricks Platform Services.Databricks Platform Services: the Databricks services or the Databricks Community Edition services, according to where the Software is used.Licensee: the user of the Software, or, if the Software is being used on behalf of a company, the company.