Saravanan666/pypi-org huggingface dataset:
Package: dask-databricks
dask-databricksCluster tools for running Dask on Databricks multi-node clusters.QuickstartTo launch a Dask cluster on Databricks you need to create aninit scriptwith the following contents and configure your multi-node cluster to use it.#!/bin/bash# Install Dask + Dask Databricks/databricks/python/bin/pipinstall--upgradedask[complete]dask-databricks# Start Dask cluster componentsdaskdatabricksrunThen from your Databricks Notebook you can quickly connect a DaskClientto the scheduler running on the Spark Driver Node.importdask_databricksclient=dask_databricks.get_client()Now you can submit work from your notebook to the multi-node Dask cluster.definc(x):returnx+1x=client.submit(inc,10)x.result()DashboardYou can access theDask dashboardvia the Databricks driver-node proxy. The link can be found inClientorDatabricksClusterrepr or viaclient.dashboard_link.>>>print(client.dashboard_link)https://dbc-dp-xxxx.cloud.databricks.com/driver-proxy/o/xxxx/xx-xxx-xxxx/8087/statusReleasingReleases of this project are automated usingGitHub Actions and thepypa/gh-action-pypi-publishaction.To create a new release push a tag to the upstream repo in the formatx.x.x. The package will be built and pushed to PyPI automatically and then later picked up by conda-forge.# Make sure you have an upstream remotegitremoteaddupstreamgit@github.com:dask-contrib/dask-databricks.git# Create a tag and push it upstreamgittagx.x.x&&gitpushupstreammain--tags
Package: dask-deltalake
Dask DeltalakeReads and write to deltalake from Dask leveraging delta-rsDask Deltalake ReaderReads data from Deltalake with DaskTo Try out the package:pip install dask_deltalakeFeatures:Reads the parquet files based on delta logs parallely using dask engineSupports all three filesystem like s3, azurefs, gcsfsSupports some delta features likeTime TravelSchema evolutionparquet filtersrow filterpartition filterQuery Delta commit info - Historyvacuum the old/ unused parquet filesload different versions of data using datetime.Usage:import dask_deltalake as ddl

# read delta table
ddl.read_delta("delta_path")

# read delta table for specific version
ddl.read_delta("delta_path",version=3)

# read delta table for specific datetime
ddl.read_delta("delta_path",datetime="2018-12-19T16:39:57-08:00")


# read delta complete history
ddl.read_delta_history("delta_path")

# read delta history upto given limit
ddl.read_delta_history("delta_path",limit=5)

# read delta history to delete the files
ddl.vacuum("delta_path",dry_run=False)

# Can read from S3,azure,gcfs etc.
ddl.read_delta("s3://bucket_name/delta_path",version=3)
# please ensure the credentials are properly configured as environment variable or
# configured as in ~/.aws/credential

# can connect with AWS Glue catalog and read the complete delta table (currently only AWS catalog available)
# will take expilicit AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from environment
# variables if available otherwise fallback to ~/.aws/credential
ddl.read_delta(catalog=glue,database_name="science",table_name="physics")
Package: dask-deltatable
Dask-DeltaTableReading and writing to Delta Lake using Dask engine.InstallationTo install the package:pip install dask-deltatableFeatures:Read the parquet files from Delta Lake and parallelize with DaskWrite Dask dataframes to Delta Lake (limited support)Supports multiple filesystems (s3, azurefs, gcsfs)Subset of Delta Lake features:Time TravelSchema evolutionParquet filtersrow filterpartition filterNot supportedWriting to Delta Lake is still in development.optimizeAPI to run a bin-packing operation on a Delta Table.Reading from Delta Lakeimportdask_deltatableasddt# read delta tableddt.read_deltalake("delta_path")# with specific versionddt.read_deltalake("delta_path",version=3)# with specific datetimeddt.read_deltalake("delta_path",datetime="2018-12-19T16:39:57-08:00")Accessing remote file systemsTo be able to read from S3, azure, gcsfs, and other remote filesystems,
you ensure the credentials are properly configured in environment variables
or config files. For AWS, you may need~/.aws/credential; for gcsfs,GOOGLE_APPLICATION_CREDENTIALS. Refer to your cloud provider documentation
to configure these.ddt.read_deltalake("s3://bucket_name/delta_path",version=3)Accessing AWS Glue catalogdask-deltatablecan connect to AWS Glue catalog to read the delta table.
The method will look forAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYenvironment variables, and if those are not available, fall back to~/.aws/credentials.Example:ddt.read_deltalake(catalog="glue",database_name="science",table_name="physics")Writing to Delta LakeTo write a Dask dataframe to Delta Lake, useto_deltalakemethod.importdask.dataframeasddimportdask_deltatableasddtdf=dd.read_csv("s3://bucket_name/data.csv")# do some processing on the dataframe...ddt.to_deltalake(df,"s3://bucket_name/delta_path")Writing to Delta Lake is still in development, so be aware that some features
may not work.
Package: dask-dirac
dask-diracdask-dirac is a library for launching adaskcluster with (DIRAC)[https://dirac.readthedocs.io/en/latest/].Installationdask-dirac can be installedfrom PyPIusing pip.pipinstalldask-diracRequirementsOpen port on localhost (default is 8786). Required for schedulerDIRAC grid certificateAcknowledgementsLuke KreczkoNullBotsdependabot[bot]
Package: dask-drmaa
Deploy aDask.distributedcluster on top of a cluster running aDRMAA-compliant job scheduler.ExampleLaunch from Pythonfromdask_drmaaimportDRMAAClustercluster=DRMAACluster()fromdask.distributedimportClientclient=Client(cluster)cluster.start_workers(2)>>>future=client.submit(lambdax:x+1,10)>>>future.result()11Or launch from the command line:$ dask-drmaa 10  # starts local scheduler and ten remote workersInstallPython packages are available from PyPI and can be installed withpip:pip install dask-drmaaAlsocondapackages are available from conda-forge:conda install -c conda-forge dask-drmaaAdditionally the package can be installed from GitHub with the latest changes:pip install git+https://github.com/dask/dask-drmaa.git --upgradeor:git clone git@github.com:dask/dask-drmaa.git
cd dask-drmaa
pip install .You must have the DRMAA system library installed and be able to submit jobs
from your local machine. Please make sure to set the environment variableDRMAA_LIBRARY_PATHto point to the location oflibdrmaa.sofor your
system.TestingThis repository contains a Docker-compose testing harness for a Son of Grid
Engine cluster with a master and two slaves.  You can initialize this system
as follows:docker-composebuild./start-sge.shIf you have done this previously and need to refresh your solution you can do
the followingdocker-composestopdocker-composebuild--no-cache./start-sge.shAnd run tests with py.test in the master docker containerdockerexec-itsge_master/bin/bash-c"cd /dask-drmaa; python setup.py develop"dockerexec-itsge_master/bin/bash-c"cd /dask-drmaa; py.test dask_drmaa --verbose"Adaptive LoadDask-drmaa can adapt to scheduler load, deploying more workers on the grid when
it has more work, and cleaning up these workers when they are no longer
necessary.  This can simplify setup (you can just leave a cluster running) and
it can reduce load on the cluster, making IT happy.To enable this, call theadaptmethod of aDRMAACluster.  You can
submit computations to the cluster without ever explicitly creating workers.fromdask_drmaaimportDRMAAClusterfromdask.distributedimportClientcluster=DRMAACluster()cluster.adapt()client=Client(cluster)futures=client.map(func,seq)# workers will be created as necessaryExtensibleThe DRMAA interface is the lowest common denominator among many different job
schedulers like SGE, SLURM, LSF, Torque, and others.  However, sometimes users
need to specify parameters particular to their cluster, such as resource
queues, wall times, memory constraints, etc..DRMAA allows users to pass native specifications either when constructing the
cluster or when starting new workers:cluster=DRMAACluster(template={'nativeSpecification':'-l h_rt=01:00:00'})# orcluster.start_workers(10,nativeSpecification='-l h_rt=01:00:00')Related WorkDRMAA: The Distributed Resource Management Application API, a high level
API for general use on traditional job schedulersdrmaa-python: The Python bindings for DRMAADaskSGE: An earlier dask-drmaa implementationSon of Grid Engine: The default implementation used in testingDask.distributed: The actual distributed computing library this launches
Package: dask-elasticsearch
==================Dask-Elasticsearch==================.. image:: https://img.shields.io/pypi/v/dask-elasticsearch.svg        :target: https://pypi.python.org/pypi/dask-elasticsearch.. image:: https://img.shields.io/pypi/pyversions/dask-elasticsearch.svg        :target: https://pypi.python.org/pypi/dask-elasticsearch.. image:: https://readthedocs.org/projects/dask-elasticsearch/badge/?version=latest        :target: https://readthedocs.org/projects/dask-elasticsearch/?badge=latest        :alt: Documentation Status.. image:: https://img.shields.io/travis/rolando/dask-elasticsearch.svg        :target: https://travis-ci.org/rolando/dask-elasticsearch.. image:: https://codecov.io/github/rolando/dask-elasticsearch/coverage.svg?branch=master    :alt: Coverage Status    :target: https://codecov.io/github/rolando/dask-elasticsearch.. image:: https://landscape.io/github/rolando/dask-elasticsearch/master/landscape.svg?style=flat    :target: https://landscape.io/github/rolando/dask-elasticsearch/master    :alt: Code Quality Status.. image:: https://requires.io/github/rolando/dask-elasticsearch/requirements.svg?branch=master    :alt: Requirements Status    :target: https://requires.io/github/rolando/dask-elasticsearch/requirements/?branch=masterElasticsearch reader for Dask.* Free software: MIT license* Documentation: https://dask-elasticsearch.readthedocs.org.* Python versions: 2.7, 3.4+Features--------FIXMEQuickstart----------FIXMECredits-------This package was created with Cookiecutter_ and the `rolando/cookiecutter-pypackage`_ project template... _Cookiecutter: https://github.com/audreyr/cookiecutter.. _`rolando/cookiecutter-pypackage`: https://github.com/rolando/cookiecutter-pypackage=======History=======0.1.0 (2017-02-02)------------------* First release on PyPI.
Package: dask-elk
dask-elkUse dask to fetch data from Elasticsearch in parallel by sending the request to each shard separatelly.Table of ContentsIntroductionUsageIntroductionThe library tries to imitate the functionality of the ES Hadoop plugin for spark.dask-elkperforms a parallel read across all the target indices shards.
In order to achieve that it uses Elasticsearch scrolling mechanism.UsageTo use the library and read from an index:fromdask_elk.clientimportDaskElasticClient# First create a clientclient=DaskElasticClient()# localhost Elasticsearchindex='my-index'df=client.read(index=index,doc_type='_doc')You can even pass a query to push down to elasticsearch, so that any filtering can be done on the Elasticsearch side. Becausedask-elkuses scroll mechanism aggregations are not supportedfromdask_elk.clientimportDaskElasticClient# First create a clientclient=DaskElasticClient()# localhost Elasticsearchquery={"query":{"term":{"user":"kimchy"}}}index='my-index'df=client.read(query=query,index=index,doc_type='_doc')Read documentationhere
Package: dasker
daskermonadic daskexample:
dasker '{}' python -c "import dasker; print(dasker.get_client().submit(lambda: 'hello, world!').result())"equivalent example, because it's a monad:
dasker '{}' dasker '{}' python -c "import dasker; print(dasker.get_client().submit(lambda: 'hello, world!').result())"if dasker wasn't used to start dask, a default client is used. example:
python -c "import dasker; print(dasker.get_client().submit(lambda: 'hello, world!').result())"if you want to override the contextual dask, use --force-new. example:
dasker '{"processes": false}' dasker --force-new '{"processes": true, "nthreads": 1, "n_workers": 1}' command ...
Package: dask-expr
Dask ExpressionsDask DataFrames with query optimization.This is a proof-of-concept rewrite of Dask DataFrame that includes query
optimization and generally improved organization.More in our blog posts:Dask Expressions overviewTPC-H benchmark results vs. Dask DataFrameExampleimportdask_exprasdxdf=dx.datasets.timeseries()df.head()df.groupby("name").x.mean().compute()Query RepresentationDask-expr encodes user code in an expression tree:>>>df.x.mean().pprint()Mean:Projection:columns='x'Timeseries:seed=1896674884This expression tree will be optimized and modified before execution:>>>df.x.mean().optimize().pprint()Div:Sum:Fused(375f9):|Projection:columns='x'|Timeseries:dtypes={'x':<class'float'>} seed=1896674884Count:Fused(375f9):|Projection:columns='x'|Timeseries:dtypes={'x':<class'float'>} seed=1896674884StabilityThis project is a work in progress and will be changed without notice or
deprecation warning.  Please provide feedback, but it's best to avoid use in
production settings.API CoverageDask-Expr covers almost everything of the Dask DataFrame API. The only missing features are:meltnamed GroupBy Aggregations
Package: dask-felleskomponenter
DASK FelleskomponenterThis is a repo where we make available apache beam components that is utilized throughout the organization. DASK
felleskomponenter is still in an early stage of the development process.StructureWe structure the repo in one main components that contains the following packages:common classescustom apache beam sourcescustom apache beam transformscustom apache beam sinksDependenciesYou need to install Python3.7 and higher, and to install the dependencies of this project, please execute the following
commandpipinstall-rrequirements.txtBulding and publishing of packageStepsRemove old dist-folder, from last time you publishedUpdate version insetup.py, for instance0.0.7->0.0.8Add change info to CHANGES.txt(Runpip install -r requirements.txtif you haven't done that earlier)Runpython3 -m build(and wait some minutes...)Verity that dist contains a package with the new version in the package name.Runpython3 -m twine upload dist/*to upload to pypiTo upload to PyPi testReplace the last command withpython3 -m twine upload --repository testpypi dist/*
Package: dask-funk
dask-funk provides an extension todaskthat creates keyword named functions
(hence the funk!) fromdasksor function graphs represented as dictionaries.
It is inspired by Plumatic’s wonderfulGraphlibrary for clojure.Example>>>fromoperatorimportadd>>>importdaskfunkasdsf>>>inc=lambdax:x+1>>>double=lambdan:n*2>>>d={'n':inc,'doubled':double}>>>f=dsf.compile(d)>>>result=f(x=2)>>>result=={'n':3,'doubled':6}TrueInstallationcondainstalldask-funkorpipinstalldask-funkPlease fix or point out any errors, inaccuracies or typos you notice.
Package: dask-geopandas
Parallel GeoPandas with DaskDask-GeoPandas is a project merging the geospatial capabilities of GeoPandas
and scalability of Dask. GeoPandas is an open source project designed to make working with geospatial data in Python easier. GeoPandas extends the datatypes used by pandas to allow spatial operations on geometric types.
Dask provides advanced parallelism and distributed out-of-core computation with a dask.dataframe module designed to scale
pandas. Since GeoPandas is an extension to the pandas DataFrame, the same way Dask scales pandas can also be applied to GeoPandas.This project is a bridge between Dask and GeoPandas and offers geospatial capabilities of GeoPandas backed by Dask.DocumentationSee the documentation onhttps://dask-geopandas.readthedocs.io/en/latest/InstallationThis package depends on GeoPandas, Dask and PyGEOS.One way to install all required dependencies is to use thecondapackage manager to
create a new environment:conda create -n geo_env
conda activate geo_env
conda config --env --add channels conda-forge
conda config --env --set channel_priority strict
conda install dask-geopandasExampleGiven a GeoPandas dataframeimportgeopandasdf=geopandas.read_file('...')We can repartition it into a Dask-GeoPandas dataframe:importdask_geopandasddf=dask_geopandas.from_geopandas(df,npartitions=4)The familiar spatial attributes and methods of GeoPandas are also available
and will be computed in parallel:ddf.geometry.area.compute()ddf.within(polygon)
Package: dask-hpcconfig
dask-hpcconfigTo install, usepython-mpipinstallgit+https://github.com/umr-lops/dask-hpcconfig.git#egg=dask-hpcconfigor clone the source:gitclonehttps://github.com/umr-lops/dask-hpcconfig.gitcddask-hpcconfigand then install from there:python-mpipinstall.or as "editable":python-mpipinstall-e.Usageimportdask_hpcconfigTo list the available cluster definitions:dask_hpcconfig.print_clusters()or, as a mapping of name to type:clusters=dask_hpcconfig.available_clusters()To create a cluster, use:cluster=dask_hpcconfig.cluster(name)wherenameis the name of one of the available clusters.To override any particular setting:
For example on 'datarmor-local' to use only 7 workers for increasing memory size of each worker:overrides={"cluster.n_workers":7}cluster=dask_hpcconfig.cluster("datarmor-local",**overrides)For example on 'datarmor' to use only 7 workers for increasing memory size of each worker, and use 49 workers (i.e. 7 mpi_1 nodes) :overrides={"cluster.cores":7}cluster=dask_hpcconfig.cluster("datarmor",**overrides)cluster.scale(49)clustercan then be used to create aClient:fromdistributedimportClientclient=Client(cluster)
Package: dask-igzip
Dask indexed gzip##################|pypi-version| |travis| |coveralls|An implementation compatible with `dask read_text`_ interface,than can chunk a gzipped text file into several partitions,thanks to an index, provided by `indexed_gzip`_This is useful when your data resides in a big gzipped file,yet you want to leverage dask parallelism capabilities.Sample session---------------::>>> import os>>> import dask_igzip.. initalization>>> data_path = os.path.join(os.path.dirname(dask_igzip.__file__), "..", "test", "data")::>>> source = os.path.join(data_path, "sample.txt.gz")>>> # 3 lines per chunk (obviously this is for demoing)>>> bag = dask_igzip.read_text(source, chunk_size=3, encoding="utf-8")>>> lines = bag.take(4, npartitions=2)>>> print("".join(lines).strip())a first sentencea second sentencea third sentencea fourth sentence>>> bag.str.upper().str.strip().compute()[8]'LINE 9'Why ?-----Dask `read_text` creates a unique partition if you provide it with a gzip file.This limitations comes from the fact thatthere is no way to split the gzip file in a predictable yet coherent way.This project provides an implementation where the gzip is indexed,then lines positions are also indexed,so that reading the text can be done by chunk (thus enabling parallelism).On first run, indexes are saved on disk, so that subsequent runs are fast... _`indexed_gzip`: https://githuib.com/pauldmccarthy/indexed_gzip.. _`dask read_text`: https://dask.pydata.org/en/latest/bag-creation.html#db-read-text.. |pypi-version| image:: https://img.shields.io/pypi/v/dask-igzip.svg:target: https://pypi.python.org/pypi/dask-igzip:alt: Latest PyPI version.. |travis| image:: http://img.shields.io/travis/jurismarches/dask_igzip/master.svg?style=flat:target: https://travis-ci.org/jurismarches/dask_igzip.. |coveralls| image:: http://img.shields.io/coveralls/jurismarches/dask_igzip/master.svg?style=flat:target: https://coveralls.io/r/jurismarches/dask_igzipChangelog#########The format is based on `Keep a Changelog`_and this project tries to adhere to `Semantic Versioning`_... _`Keep a Changelog`: http://keepachangelog.com/en/1.0.0/.. _`Semantic Versioning`: http://semver.org/spec/v2.0.0.html0.2.0 - 2018-06-20==================New---- read_text now accept a limit parameter to limit the global amount of lines to readChanged-------- incompatible format for lines index0.1.0 - 2018-06-19==================New---- initial release- 100% code coverage
Package: dask-kafka
Dask-KafkaDask-Kafka readerFree software: MIT licenseDocumentation:https://dask-kafka.readthedocs.org.Python versions: 2.7, 3.4+FeaturesFIXMEQuickstartFIXMECreditsThis package was created withCookiecutterand therolando/cookiecutter-pypackageproject template.History0.1.0-dev (unreleased)First release on PyPI.
Package: dask-lightgbm
Dask-LightGBM - DEPRECATEDTHIS REPOSITORY IS DEPRECATEDThis repository is deprecated and it is no longer maintained. The code was migrated into LightGBM package -https://github.com/microsoft/LightGBM.Distributed training with LightGBM and Dask.distributedThis repository enables you to perform distributed training with LightGBM on
Dask.Array and Dask.DataFrame collections. It is based on dask-xgboost package.UsageLoad your data into distributed data-structure, which can be either Dask.Array or Dask.DataFrame.
Connect to a Dask cluster using Dask.distributed.Client.
Let dask-lightgbm train a model or make predictions for you.
See system tests for a sample code:https://github.com/dask/dask-lightgbm/blob/main/system_tests/test_fit_predict.pyHow this worksDask is used mainly for accessing the cluster and managing data.
The library assures that both features and a label for each sample are located on the same worker.
It also lets each worker to know addresses and available ports of all other workers.
The distributed training is performed by LightGBM library itself using sockets.
See more details on distributed training in LightGBM here:https://github.com/microsoft/LightGBM/blob/main/docs/Parallel-Learning-Guide.rst
Package: dask-lxplus
dask-lxplusBuilds on top of Dask-Jobqueue to enable jobs to run on the CERN HTCondor cluster via LXPLUS.SummaryfromdistributedimportClientfromdask_lxplusimportCernClusterimportsocketcluster=CernCluster(cores=1,memory='2000MB',disk='10GB',death_timeout='60',lcg=True,nanny=False,container_runtime='none',log_directory='/eos/user/b/ben/condor/log',scheduler_options={'port':8786,'host':socket.gethostname(),},job_extra={'MY.JobFlavour':'"longlunch"',},)CERN extrasThere are a few changes in the wrapper to address some of the particular features of the CERN
HTCondor cluster, but there are also a few changes to detail here.Optionslcg: If set toTruethis will validate and use the LCG python environment per the managedLCGreleases. It will send the environment of the submitting scheduler to the batch worker node. DASK
normally requires that both the scheduler and the worker is the same python versions and libraries.
At CERN this would mean that you should, assuming say the default ofCentOS7worker nodes, that
the scheduler is run on something likelxplus.cern.chalso running CentOS7`.
An example use would be to do the following before running dask:$./cvmfs/sft.cern.ch/lcg/views/LCG_102/x86_64-centos7-gcc11-opt/setup.shcontainer_runtime: Can be set to"singularity"ordockeror"none". If a runtime is needed
for the worker, this selects which will be used for theHTCondorjob the worker runs. In principle
it should not be necessary when usinglcgand should therefore be set to"none". Default though
is"singularity".worker_image: The image that will be used ifcontainer_runtimeis defined to use one. The default
is defined injobqueue-cern.yaml.batch_name: Optionally set a string that will identify the jobs inHTCondor. The default is"dask-worker"