Saravanan666/pypi-org huggingface dataset:
Package: databricks-filestore-uploader
Databricks FileStore UploaderA convenience tool for uploading local directories to the DBFS FileStoreUpload Local FilesThis tool uploads specified source directories to specified paths in theDBFSFileStore.Create a file adjacent tomain.pycalledinput.yml, and copy the following:host : <workspace-url>
token: <user-access-token>

# THIS TOOL CREATES OR OVERWRITES THE TARGET FOLDER!
# Folder paths don't start with './'
# Folder paths don't end with '/'
payloads:
  some-local-folder : some-filestore-folder
  some-other-local-folder : some-other/filestore-folder
  # ...Replace<user-access-token>with your token in databricks, and replace the example text underpayloads:one or more "source : destination" pairs.These local source directory pathsare relative tomain.py.You can then upload all files contained in the local source folders to their specified destinations with:./main.pyThough make sure you havePyYAMLinstalled.Verify UploadsIn a databricks notebook, you can verify uploads with:display(dbutils.fs.ls("/FileStore/some-filestore-folder"))Please Note❗This tool CREATES or OVERWRITES it's target dbfs:/FileStore path. Only target folders that are meant to be overwritten, or don't exist.
Package: databricks-google-sheets
databricks-google-sheetsInteract with Google Sheets from Databricks.Please follow these steps. Be sure to replace all {{PLACEHOLDERS}} with appropriate values:Create a spreadsheet in Google Sheets, and note the spreadsheet_id (You can get this from the URL)Go to console.cloud.google.com and create a new project (OR select an existing project)Within this project, create a service account [IAM & Admin --> Service Accounts --> Create], and call itdatabricks-google-sheets. You can skip the optional stepsNote down the email address associated with your new service account. Eg.databricks-google-sheets@{{MY-AWESOME-PROJECT}}.iam.gserviceaccount.comSwitch to the Keys tab and create a new key. Choose JSON as the Key type. This will generate a JSON file and download it locallyOn your local machine, use the Databricks CLI to create a new secret scope. Eg.databricks secrets create-scope --scope databricks-google-sheets --profile DEFAULTCreate a new secret within the newly created secret scope. Eg.databricks secrets put --scope databricks-google-sheets --key "databricks-google-sheets@{{MY-AWESOME-PROJECT}}.iam.gserviceaccount.com" --binary-file {{PATH-TO-JSON-FILE-FROM-STEP-5}} --profile DEFAULTShare your Google Sheet with the userdatabricks-google-sheets@{{MY-AWESOME-PROJECT}}.iam.gserviceaccount.com, and give Editor privilegesUsage within Databricks Python notebookfromdatabricks_google_sheetsimportDatabricksGoogleSheetsdgs=DatabricksGoogleSheets(dbutils)# Pass in dbutils from the global scope of your Databricks runtime# OPTION 1 - Create a Pandas dataframe from a Spark SQL query in Pythondf=spark.sql("SELECT 'Hello from databricks-google-sheets' greeting;").toPandas()# OPTION 2 - Create a dataframe directly in Pandasimportpandasaspddf=pd.DataFrame({'greeting':['Hello from databricks-google-sheets']})# Write this dataframe to Google Sheetsdgs.df_to_sheets(df,spreadsheet_id='{{SPREADSHEET-ID-FROM-STEP-1}}',sheet_name='test',dbsecret_scope='databricks-google-sheets',dbsecret_key='databricks-google-sheets@{{MY-AWESOME-PROJECT}}.iam.gserviceaccount.com')# Read from Google Sheets into a new dataframenewdf=dgs.sheets_to_df(spreadsheet_id='{{SPREADSHEET-ID-FROM-STEP-1}}',sheet_name='test',dbsecret_scope='databricks-google-sheets',dbsecret_key='databricks-google-sheets@{{MY-AWESOME-PROJECT}}.iam.gserviceaccount.com')# Compare the new dataframe to the original to ensure they are the sameassertdf.equals(newdf)==True
Package: databricks-kakapo
Contributing to KakapoOverviewWe happily welcome contributions to Kakapo as long as CLA has been accepted.
We use GitHub Issues to track community reported issues and GitHub Pull Requests for accepting changes.Repository structureThe repository is structured as follows:.githubCICD for both kakapo and for solution examples.python/kakapo/Kakapo python module and tests. This code is used to build databricks-kakapo pip dependency.docs/Source code for documentation. Documentation is built via sphinx.Test & build KakapoPythonThe python bindings can be tested usingunittest.Move to thepython/directory and install the project and its dependencies:pip install .Run the tests usingunittest:python -m unittestThe project wheel file can be built withbuild.Install the build requirements:pip install build wheel.Build the wheel usingpython -m build.Collect the .whl file frompython/dist/DocumentationThe documentation has been produced usingSphinx.To build the docs:Install the pandoc library (follow the instructions for your platformhere).Install the python requirements fromdocs/docs-requirements.txt.Build the HTML documentation by runningmake htmlfromdocs/.You can locally host the docs by running thereload.pyscript in thedocs/source/directory.StyleTools we use for code formatting and checking:scalafmtandscalastylein the main scala project.blackandisortfor the python bindings.
Package: databrickslabs-jupyterlab
Local JupyterLab connecting to Databricks via SSH (V2)This package allows to connect to a remote Databricks cluster from a locally running JupyterLab.For more details see:Prerequisites, installation and usageSwitching kernels and restart after cluster auto-terminationCreating a mirror of a remote Databricks clusterDetailed databrickslabs_jupyterlab command overviewHow it worksTroubleshooting
Package: databrickslabs-jupyterlab-status
databrickslabs_jupyterlab_statusA JupyterLab extension to show starting status of Databricks clusters.RequirementsJupyterLab >= 3.0Installpipinstalldatabrickslabs_jupyterlab_statusContributingDevelopment installNote: You will need NodeJS to build the extension package.Thejlpmcommand is JupyterLab's pinned version ofyarnthat is installed with JupyterLab. You may useyarnornpmin lieu ofjlpmbelow.# Clone the repo to your local environment# Change directory to the databrickslabs_jupyterlab_status directory# Install package in development modepipinstall-e.# Link your development version of the extension with JupyterLabjupyterlabextensiondevelop.--overwrite# If using a server extension, it must be manually installed in develop modejupyterserverextensionenable<extension_name># Rebuild extension Typescript source after making changesjlpmrunbuildYou can watch the source directory and run JupyterLab at the same time in different terminals to watch for changes in the extension's source and automatically rebuild the extension.# Watch the source directory in one terminal, automatically rebuilding when neededjlpmrunwatch# Run JupyterLab in another terminaljupyterlabWith the watch command running, every saved change will immediately be built locally and available in your running JupyterLab. Refresh JupyterLab to load the change in your browser (you may need to wait several seconds for the extension to be rebuilt).By default, thejlpm run buildcommand generates the source maps for this extension to make it easier to debug using the browser dev tools. To also generate source maps for the JupyterLab core extensions, you can run the following command:jupyterlabbuild--minimize=FalseUninstallpipuninstalldatabrickslabs_jupyterlab_status
Package: databricks-mosaic-gdal
OverviewCurrent version is 3.4.3 (to match GDAL).This is a filetree (vs apt based) drop-in packaging of GDAL with Java Bindings fromUbuntuGIS PPAfor Ubuntu 20.04 (Focal Fossa) which is used byDatabricks Runtime(DBR) 11+.Note: This is a very specific packaging for GDAL + dependencies which removes any libraries that are already provided by DBR, so not useful outside Databricks.gdal-3.4.3-filetree.tar.xzis ~50MB - it is extracted withtar -xf gdal-3.4.3-filetree.tar.xz -C /gdal-3.4.3.-symlinks.tar.xzis ~19MB - it is extracted withtar -xhf gdal-3.4.3-symlinks.tar.xz -C /An exampleinit scriptis provided at./resources/scripts/mosaic-gdal-3.4.3-filetree-init.shfor managing install of the tarballs across a DBR cluster.
Package: databricks-registry-webhooks
Databricks Model Registry Webhooksenable you to automate and integrate your machine learning pipelines with a variety of CI/CD tools and workflows. Databricks Model Registry Webhooks integrate with theDatabricks MLflow Model Registryto provide event-based triggers for Model Registry actions, such as the creation of a new model or the transition of a model version into production.Use cases for Databricks Model Registry Webhooks include triggering CI builds when a new model version is created, notifying team members each time a model transition to production is requested, and much more.This package provides a Python client for managing Databricks Model Registry Webhooks. For more information about the feature and Python client usage, explore the documentation athttps://docs.databricks.com/applications/mlflow/model-registry-webhooks.html.
Package: databricks-runtime
Databricks LTS Python RuntimesThis repository contains a simple python packageruntimeto facilitate
referencing the supported python libraries in thelong-term support
databricks runtimes.This should facilitate adding the LTS dependencies in external python projects,
specially when building python wheels.InstallationInstall via pip:pip install databricks-runtime==7.3.dev4DevelopmentClone this repo and install locally with:$ pip install -e .Install the development reqs:$ pip install -r requirements-development.txtTools & common scripts:Style check:bash check-style.shType check:bash check-types.shProfiles:bash profiler.shSupported LTS Versions7.3 LTSReference to official documentationhere.Python version:3.7.5Release date:Sep 24, 2020
Package: databricks-session
Databricks Session UtilA simple utility for spark and mlflow session objectsSetupQuick Installpython-mpipinstalldatabricks_sessionBuild from sourceClone the repositorygitclonehttps://github.com/Broomva/databricks_session.gitInstall the packagecddatabricks_session&&makeinstallBuild manuallyAfter cloning, create a virtual environmentcondacreate-ndatabricks_sessionpython=3.10
condaactivatedatabricks_sessionInstall the requirementspipinstall-rrequirements.txtRun the python installationpythonsetup.pyinstallUsageThe deployment requires a .env file created under local folder:touch.envIt should have a schema like this:databricks_experiment_name=''databricks_experiment_id=''databricks_host=''databricks_token=''databricks_username=''databricks_password=''databricks_cluster_id=''databricks_sql_http_path=''importdatabricks_session# Create a Spark sessionspark=DatabricksSparkSession().get_session()# Connect to MLFLow Artifact Servermlflow_session=DatabricksMLFlowSession().get_session()
Package: databricks-sql-cli
IntroductionDatabricks SQL CLI is a command line interface (CLI) forDatabricks SQLthat can do auto-completion and syntax highlighting, and is a proud member of the dbcli community.Quick StartInstall viapipYou'll need Python 3.7 or newer.python3-mpipinstalldatabricks-sql-cliYou can now rundbsqlclifrom your terminal.AuthenticationTo connect with SQL Endpointsdbsqlclineeds the host name and http path from theconnection detailsscreen in Databricks SQL and apersonal access token. You can provide these todbsqlclias command line arguments, by setting environment variables, or by writing them into the[credentials]section of thedbsqlclircfile (see below).ConfigA config file is automatically created at~/.dbsqlcli/dbsqlclircat first launch (rundbsqlcli). See the file itself for a description of all available options.Run a query$dbsqlcli-e'select id, name from minifigs LIMIT 10'Run a .sql file$dbsqlcli-equery.sqlRun a .sql file and save to CSV$dbsqlcli-equery.sql>output.csvREPL$cd<directorycontainingdbsqlclibinary>
$./dbslqcli[<database_name>]Run thehelp;command to see a list of shortcutsFeaturesAuto-completes as you type for SQL keywords as well as tables and columns in the database.Syntax highlighting.Smart-completion will suggest context-sensitive completion.SELECT * FROM <tab>will only show table names.SELECT * FROM users WHERE <tab>will only show column names.Pretty prints tabular data and various table formats.Some special commands. e.g. Favorite queries.Alias support. Column completions will work even when table names are aliased.Usages$dbsqlcli--help
Usage:dbsqlcli[OPTIONS][DATABASE]ADBSQLterminalqueryingclientwithauto-completionandsyntaxhighlighting.Examples:-dbsqlcli-dbsqlclimy_database

Options:-e,--executeTEXTExecuteacommand(orafile)andquit.--hostnameTEXTHostname[envvar:DBSQLCLI_HOST_NAME]--http-pathTEXTHTTPPath[envvar:DBSQLCLI_HTTP_PATH]--access-tokenTEXTAccessToken[envvar:DBSQLCLI_ACCESS_TOKEN]--clircFILELocationofclircfile.--table-formatTEXTTableformatusedwith-eoption.--helpShowthismessageandexit.ContributionsWe usePoetryfor development. Follow the instructions to install Poetry on your system.Clone this repositorypoetry installwill install its dependenciespoetry shellwill activate the local virtual environmentpython app.pywill rundbsqlcliincorporating any of your local changesCreditsHuge thanks to the maintainers ofhttps://github.com/dbcli/athenacliupon which this project is built.Similar projectsTheDBCLIorganization on Github maintains CLIs for numerous database platforms including MySQL, Postgres, and MSSQL.https://github.com/dbcli/myclihttps://github.com/dbcli/pgclihttps://github.com/dbcli/mssql-cli
Package: databricks-sql-connector
Databricks SQL Connector for PythonThe Databricks SQL Connector for Python allows you to develop Python applications that connect to Databricks clusters and SQL warehouses. It is a Thrift-based client with no dependencies on ODBC or JDBC. It conforms to thePython DB API 2.0 specificationand exposes aSQLAlchemydialect for use with tools likepandasandalembicwhich use SQLAlchemy to execute DDL. Usepip install databricks-sql-connector[sqlalchemy]to install with SQLAlchemy's dependencies.pip install databricks-sql-connector[alembic]will install alembic's dependencies.This connector uses Arrow as the data-exchange format, and supports APIs to directly fetch Arrow tables. Arrow tables are wrapped in theArrowQueueclass to provide a natural API to get several rows at a time.You are welcome to file an issue here for general use cases. You can also contact Databricks Supporthere.RequirementsPython 3.8 or above is required.DocumentationFor the latest documentation, seeDatabricksAzure DatabricksQuickstartInstall the library withpip install databricks-sql-connectorNote: Don't hard-code authentication secrets into your Python. Use environment variablesexportDATABRICKS_HOST=********.databricks.comexportDATABRICKS_HTTP_PATH=/sql/1.0/endpoints/****************exportDATABRICKS_TOKEN=dapi********************************Example usage:importosfromdatabricksimportsqlhost=os.getenv("DATABRICKS_HOST")http_path=os.getenv("DATABRICKS_HTTP_PATH")access_token=os.getenv("DATABRICKS_TOKEN")connection=sql.connect(server_hostname=host,http_path=http_path,access_token=access_token)cursor=connection.cursor()cursor.execute('SELECT :param `p`, * FROM RANGE(10)',{"param":"foo"})result=cursor.fetchall()forrowinresult:print(row)cursor.close()connection.close()In the above example:server-hostnameis the Databricks instance host name.http-pathis the HTTP Path either to a Databricks SQL endpoint (e.g. /sql/1.0/endpoints/1234567890abcdef),
or to a Databricks Runtime interactive cluster (e.g. /sql/protocolv1/o/1234567890123456/1234-123456-slid123)personal-access-tokenis the Databricks Personal Access Token for the account that will execute commands and queriesContributingSeeCONTRIBUTING.mdLicenseApache License 2.0
Package: databricks-tooling
IntroductionTODO: Give a short introduction of your project. Let this section explain the objectives or the motivation behind this project.Getting StartedTODO: Guide users through getting your code up and running on their own system. In this section you can talk about:Installation processSoftware dependenciesLatest releasesAPI referencesBuild and TestTODO: Describe and show how to build your code and run the tests.ContributeTODO: Explain how other users and developers can contribute to make your code better.If you want to learn more about creating good readme files then refer the followingguidelines. You can also seek inspiration from the below readme files:ASP.NET CoreVisual Studio CodeChakra Core
Package: databricks-utils
databricks-utilsdatabricks-utilsis a python package that provide several utility classes/func
that improve ease-of-use in databricks notebook.Installationpipinstalldatabricks-utilsFeaturesS3Bucketclass to easily interact with aS3 bucketviadbfsand databricks spark.vega_embedto render charts fromVegaandVega-Litespecifications.DocumentationAPI documentation can be found athttps://e2fyi.github.io/databricks-utils/.Quick startS3Bucketimportjsonfromdatabricks_utils.awsimportS3Bucket# need to attach notebook's dbutils# before S3Bucket can be usedS3Bucket.attach_dbutils(dbutils)# create an instance of the s3 bucketbucket=(S3Bucket("somebucketname","SOMEACCESSKEY","SOMESECRETKEY").allow_spark(sc)# local spark context.mount("somebucketname"))# mount location name (resolves as `/mnt/somebucketname`)# show list of files/folders in the bucket "resource" folderbucket.ls("resource/")# read in a json file from the bucketdata=json.load(open(bucket.local("resource/somefile.json","r")))# read from parquet via sparkdataframe=spark.read.parquet(bucket.s3("resource/somedf.parquet"))# umountbucket.umount()VegaVegaandVega-Liteare high-level grammars of interactive graphics. They provide concise JSON
syntax for rapidly generating visualizations to support analysis.fromdatabricks_utils.vegaimportvega_embed# vega-lite spec for a bar chartspec={"data":{"values":[{"a":"A","b":28},{"a":"B","b":55},{"a":"C","b":43},{"a":"D","b":91},{"a":"E","b":81},{"a":"F","b":53},{"a":"G","b":19},{"a":"H","b":87},{"a":"I","b":52}]},"mark":"bar","encoding":{"x":{"field":"a","type":"ordinal"},"y":{"field":"b","type":"quantitative"}}}# plot out the vega chart in databricks notebookdisplayHTML(vega_embed(spec=spec))Developer# add a version to git tag and publish to pypi.add_tag.sh<VERSION>
Package: databricks-workspace-cleaner
databricks-workspace-cleanerdwc is a tool to clear run cells from notebooks, for example where there might be concern about data held in run cells, or as preparation for commit to source control.You can also use it to import/export multiple notebooks with this capability, in use cases where dbc export may not be possible due to volume limits.CommandsCommandSub-CommandParametersDescriptionlistnotebooksList all notebooks in workspace.listlibrariesList all libraries in workspace.exportnotebookspath: location to output zip of notebooksExports all notebooks from a workspace as base64 source code. The process will remove annotations for run cellsimportnotebookspath: location of notebooks.zipimport_prefix: folder to import into (default: IMPORT)Import notebooks into workspace.cleanfoldersDelete all empty folders in workspace.cleannotebooksRemove annotations for run cells from all notebooks in workspace.InstallationIn a python 3.7 environment install this repository, e.g:pip install git+https://github.com/frogrammer/fire-commands.gitThe tool can be installed to an azure cloud shell.Databricks Workspace LoginThe dwc CLI is built using the databricks CLI sdkhttps://github.com/databricks/databricks-cli, and uses its authentication mechanism to login to a workspace.To login to an azure databricks workspace using a user token:echoMY_TOKEN>>token.txt
databricksconfigure--hostMY_HOST-ftoken.txt
rmtoken.txt
Package: databricks-workspace-tool
databricks-workspace-tooldwt is a tool to clear run cells from notebooks, for example where there might be concern about data held in run cells, or as preparation for commit to source control.You can also use it to import/export multiple notebooks with this capability, in use cases where dbc export may not be possible due to volume limits.CommandsCommandSub-CommandParametersDescriptionlistnotebooksList all notebooks in workspace.listlibrariesList all libraries in workspace.exportnotebookspath: location to output zip of notebooksExports all notebooks from a workspace as base64 source code. The process will remove annotations for run cellsimportnotebookspath: location of notebooks.zipimport_prefix: folder to import into (default: IMPORT)Import notebooks into workspace.cleanfoldersDelete all empty folders in workspace.cleannotebooksRemove annotations for run cells from all notebooks in workspace.InstallationIn a python 3.7 environment install this repository, e.g:pip install git+https://github.com/frogrammer/fire-commands.gitThe tool can be installed to an azure cloud shell.Databricks Workspace LoginThe dwt CLI is built using the databricks CLI sdkhttps://github.com/databricks/databricks-cli, and uses its authentication mechanism to login to a workspace.To login to an azure databricks workspace using a user token:echoMY_TOKEN>>token.txt
databricksconfigure--hostMY_HOST-ftoken.txt
rmtoken.txt
Package: databrowse
DatabrowseDatabrowse:  An Extensible Data Management PlatformCopyright (C) 2012-2016 Iowa State University Research Foundation, Inc.DocumentationPlease see the documentation indoc/Manual/Manual.pdffor more
information and installation instructions.LicenseDatabrowse is released under the BSD 3-Clause License.  Please see the fileCOPYINGfor more information.AcknowledgmentsThis material is based on work supported by the Air Force Research Laboratory
under Contract #FA8650-10-D-5210, Task Order #023 and performed at Iowa State
University.DISTRIBUTION A.  Approved for public release:  distribution unlimited; 19 Aug
2016; 88ABW-2016-4051.This material is based on work supported by NASA under Contract
NNX16CL31C and performed by Iowa State University as a subcontractor
to TRI Austin.Approved for public release by TRI Austin: distribution unlimited;
02 August 2018; by Carl W. Magnuson (NDE Division Director).CEFDatabrowse Install InstructionsDependencies are platform specific and can be found inrequirements.From PIP:`> pip install databrowse`From source:`> cd databrowse_source_dir`
`> python setup.py install`Changelogv0.8.3Added find function (ctrl+f) in CEFDatabrowseImproved error reporting in CEFDatabrowseSwitched from x3dom to x_ite (x_ite supports built in x3d scripts)Added v1 of no provenance mode to db_generic_XML_file (removes all lip:process tags)Added v1 of data fusion custom view to db_directory (Simple search function and render for any tag in .xlp or .xlg files in current directory)Added v1 of db_git plugin (displays relavant details of a git repository such as commit history, directory structure, modified files/untracked files, and current branch) Requires GITPython module.Added prompt during install to allow user to decide if they want to use pip to automatically install dependencies or notCEFDatabrowse now uses user directory based configuration files instead of system wide config files.Many bug fixesv0.8Approval for Public release following work with TRI AustinCEFDatabrowse, a server-less client, ImplementedSDT File Viewer Plugin Implemented3D Model Viewer Plugin ImplementedSerial log viewer Plugin for LIMATIX thermoconductivity demoMagic bug fixesVarious other bug fixesv0.7Approval for Public ReleasePackaging CleanupLicense Change to BSD-3v0.6Databrowse Library ImplementedRestructuring of Files for Packaging CompletedStarting of DocumentationRestructuring of Directory Plugin OutputComplete Reworking of Speicmen Database PluginsNumerous Updates to Dataguzzler and DatacollectV2 PluginsAll Use of etree.register_namespace Global Removed and Basic Namespace
Handling ImplementedNumerous Bugfixesv0.5Directory Restructuring For DistributionStyle Updates for Several PluginsTabular Views in DatacollectV2 PluginMinor Bugfixesv0.4Major Updates to DatacollectV2, Checklist, Specimen/Xducer, and DataguzzlerNumerous BugfixesBranch Mergesv0.3Plugins Restructures into Python ModulesNumerous Plugin Updates and BugfixesMenus and Location Bars AddedNumerous Style Updatesv0.2Numerous Plugin UpdatesNew Stylesheet using latest version of ISU Templatev0.1Initial Commit of Baseline Plugins and Basic Infrastructure