Saravanan666/pypi-org huggingface dataset:
Package: dask_memusage
dask-memusageIf you're using Dask with tasks that use a lot of memory, RAM is your bottleneck for parallelism.
That means you want to know how much memory each task uses:So you can set the highest parallelism level (process or threads) for each machine, given available to RAM.In order to know where to focus memory optimization efforts.dask-memusageis an MIT-licensed statistical memory profiler for Dask's Distributed scheduler that can help you with both these problems.dask-memusagepolls your processes for memory usage and records the minimum and maximum usage in a CSV:task_key,min_memory_mb,max_memory_mb
"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 0)",44.84765625,96.98046875
"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 1)",47.015625,97.015625
"('sum-part-e15703211a549e75b11c63e0054b53e5', 0)",0,0
"('sum-part-e15703211a549e75b11c63e0054b53e5', 1)",0,0
sum-aggregate-apply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,0,0
apply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,47.265625,47.265625
task_key,min_memory_mb,max_memory_mb
"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 0)",44.84765625,96.98046875
"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 1)",47.015625,97.015625
"('sum-part-e15703211a549e75b11c63e0054b53e5', 0)",0,0
"('sum-part-e15703211a549e75b11c63e0054b53e5', 1)",0,0
sum-aggregate-apply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,0,0
apply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,47.265625,47.265625UsageImportant:Make sure your workers only have a single thread! Otherwise the results will be wrong.InstallationOn the machine where you are running the Distributed scheduler, run:$pipinstalldask_memusageOr if you're using Conda:$condainstall-cconda-forgedask-memusageAPI usage# Add to your Scheduler object, which is e.g. your LocalCluster's scheduler# attribute:fromdask_memoryusageimportinstallinstall(scheduler,"/tmp/memusage.csv")CLI usage$dask-scheduler--preloaddask_memusage--memusage.csv/tmp/memusage.csvLimitationsAgain, make sure you only have one thread per worker process.This is statistical profiling, running every 10ms.
Tasks that take less than that won't have accurate information.HelpNeed help? File a ticket athttps://github.com/itamarst/dask-memusage/issues/new
Package: dask.mesos
[![Build Status](http://52.0.47.203:8000/api/badges/lensacom/dask.mesos/status.svg)](http://52.0.47.203:8000/lensacom/dask.mesos)[![Join the chat at https://gitter.im/lensacom/dask.mesos](https://badges.gitter.im/lensacom/dask.mesos.svg)](https://gitter.im/lensacom/dask.mesos?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)[Apache Mesos](http://mesos.apache.org/) backend for [Dask](https://github.com/dask/dask) scheduling library.Run distributed python dask workflows on your Mesos cluster.## Notable Features- distributively run tasks in docker container- specify resource requirements per task- bin packing for optimized resource utilization## Installation**Prerequisits:** [satyr](https://github.com/lensacom/satyr), [dask](https://github.com/dask/dask.git), [toolz](https://pypi.python.org/pypi/toolz). All of them should be installed w/ the following commands:`pip install dask.mesos` or use [lensa/dask.mesos](https://hub.docker.com/r/lensa/dask.mesos/) Docker imageConfiguration:- `MESOS_MASTER=zk://127.0.0.1:2181/mesos`- `ZOOKEEPER_HOST=127.0.0.1:2181`## Example```pythonfrom __future__ import absolute_import, division, print_functionfrom dask_mesos import mesos, MesosExecutor@mesos(cpus=0.1, mem=64)def add(x, y):"""Run add on mesos with specified resources"""return x + y@mesos(cpus=0.3, mem=128, image='lensa/dask.mesos')def mul(x, y):"""Run mul on mesos in specified docker image"""return x * ywith MesosExecutor(name='dask') as executor:"""This context handles Mesos scheduler's lifecycle"""a, b = 23, 89alot = add(a, b)gigalot = mul(alot, add(10, 2))gigalot.compute(get=executor.get)  # (a + b) * (10 + 2)executor.compute([alot, gigalot])  # list of futures```## Configuring dask.mesos TasksYou can configure your mesos tasks in your decorator, currently the following options are available:* **cpus**: The amount of cpus to use for the task.* **mem**: Memory in MB to reserver for the task.* **disk**: The amount of disk to use for the task.* **image**: A docker image name. If not set, mesos containerizer will be used.Both mem and cpus are defaults to some low values set in _satyr_ so you're encouraged to override them here. More options like constraints, other resources are on the way.
Package: dask-mongo
Dask-MongoRead and write data to MongoDB with DaskInstallationdask-mongocan be installed withpip:pip install dask-mongoor withconda:conda install -c conda-forge dask-mongoExampleimportdask.bagasdbimportdask_mongo# Create Dask Bagrecords=[{"name":"Alice","fruit":"apricots"},{"name":"Bob","fruit":["apricots","cherries"]},{"name":"John","age":17,"sports":"cycling"},]b=db.from_sequence(records)# Write to a Mongo databasedask_mongo.to_mongo(b,database="your_database",collection="your_collection",connection_kwargs={"host":"localhost","port":27017},)# Read Dask Bag from Mongo databaseb=dask_mongo.read_mongo(database="your_database",collection="your_collection",connection_kwargs={"host":"localhost","port":27017},chunksize=2,)# Perform normal operations with Dasknames=b.pluck("name").compute()assertnames==["Alice","Bob","John"]LicenseBSD-3
Package: dask-optuna
Dask-OptunaDask-Optuna helps improve integration betweenOptunaandDaskby leveraging Optuna's existing distributed optimization capabilities to run
optimization trials in parallel on a Dask cluster. It does this by providing a
Dask-compatibledask_optuna.DaskStoragestorage class which wraps an
Optuna storage class (e.g. Optuna's in-memory or sqlite storage) and can be used
directly by Optuna. For example:importoptunaimportjoblibimportdask.distributedimportdask_optunadefobjective(trial):x=trial.suggest_uniform("x",-10,10)return(x-2)**2withdask.distributed.Client()asclient:# Create a study using Dask-compatible storagestorage=dask_optuna.DaskStorage()study=optuna.create_study(storage=storage)# Optimize in parallel on your Dask clusterwithjoblib.parallel_backend("dask"):study.optimize(objective,n_trials=100,n_jobs=-1)print(f"best_params ={study.best_params}")DocumentationSee theDask-Optuna documentationfor more information.LicenseMIT License
Package: daskperiment
Overviewdaskperimentis a tool to perform reproducible machine learning experiment.
It allows users to define and manage the history of trials
(given parameters, results and execution environment).The package is built onDask, a package for parallel computing with task
scheduling. Each experiment trial is internally expressed asDaskcomputation
graph, and can be executed in parallel.BenefitsCompatibility with standard Python/Jupyter environment (and optionally with standard KVS).No need to set up server applicationsNo need to registrate on any cloud servicesRun on standard / customized Python shellsIntuitive user interfaceFew modifications on existing codes are neededTrial histories are logged automatically (no need to write additional codes for logging)Daskcompatible APIEasily accessible experiments history (withpandasbasic operations)Less managiment works on Git (no need to make branch per trials)(Experimental) Web dashboard to manage trial historyTraceability of experiment related informationTrial result and its (hyper) parameters.Code contextsEnvironment informationDevice informationOS informationPython versionInstalled Python packages and its versionGit informationReproducibilityCheck function purity (each step should return the same output for the same inputs)Automatic random seedingAuto saving and loading of previous experiment historyParallel execution of experiment stepsExperiment sharingRedis backendMongoDB backendFuture ScopeMore efficient execution.Omit execution if depending parameters are the sameDistributed execution
Package: dask-pgen
dask-pgenLazy loading PLINK genotype data (.pgen or .bed) file with dask-array.Installationpipinstall-Ucythonnumpy
gitclonehttps://github.com/KangchengHou/dask-pgen.gitcddask-pgen;pipinstall-e.Or,pipinstallgit+https://github.com/KangchengHou/dask-pgen.gitExample# if dapgen is not found, set the path of dapgen yourself# chmod +x dask-pgen/bin/dapgen# dapgen=dask-pgen/bin/dapgen# replace "dapgen" with "$dapgen"dapgenscore\--plink<plink>\--weights<weights_path>\--weight-col-prefix<weight_column_prefix>\--chrom-colCHR--pos-colPOS--alt-colA1--ref-colA2\--out<out_path>\--centerTrue# center the genotype or not, default is False \--threads4\--memory20000
Package: dask-quik
dask-quikUtilities for transforming data using dask and dask_cudf. Most can take either object,
and will still process.This utility currently has:cartesian: The ability to create a sparse matrix of the entire universe of data (creating a cartesian product of your data), and "indexize" your data for future
encoding usescombine: a shortcut for a left merge, and the ability to prune rows based on what
should be grouped by, max, min, and/or avgdummy: The ability to run just dask functions, using a dask_cudf dummy classsplit: Splitting your train, validation, and testing datasets with test being the
most recent value (for recommendation systems)transform: The ability to transform back and forth from pandas, dask, and dask_cudf, and also a workaround to sort by index in dask_cudfutils: random utils to setup the dask_cudf cluster, shrink data types when defaulted to
64-bit, and a quick check if there are available GPUs (using nvidia-smi).
Package: dask-rasterio
dask-rasteriodask-rasterioprovides some methods for reading and writing rasters in
parallel usingRasterioandDaskarrays.UsageRead a multiband raster>>>fromdask_rasterioimportread_raster>>>array=read_raster('tests/data/RGB.byte.tif')>>>arraydask.array<stack,shape=(3,718,791),dtype=uint8,chunksize=(1,3,791)>>>>array.mean()dask.array<mean_agg-aggregate,shape=(),dtype=float64,chunksize=()>>>>array.mean().compute()40.858976977533935Read a single band from a raster>>>fromdask_rasterioimportread_raster>>>array=read_raster('tests/data/RGB.byte.tif',band=3)>>>arraydask.array<raster,shape=(718,791),dtype=uint8,chunksize=(3,791)>Write a singleband or multiband raster>>>fromdask_rasterioimportread_raster,write_raster>>>array=read_raster('tests/data/RGB.byte.tif')>>>new_array=array&(array>100)>>>new_arraydask.array<and_,shape=(3,718,791),dtype=uint8,chunksize=(1,3,791)>>>>prof=...# reuse profile from tests/data/RGB.byte.tif...>>>write_raster('processed_image.tif',new_array,**prof)Chunk sizeBothread_rasterandwrite_rasteraccept ablock_sizeargument that
acts as a multiplier to the block size of rasters. The default value is 1,
which means the dask array chunk size will be the same as the block size of
the raster file. You will have to adjust this value depending on the
specification of your machine (how much memory do you have, and the block
size of the raster).InstallInstall with pip:pip install dask-rasterioDevelopmentThis project is managed byPoetry.  If
you do not have it installed, please refer toPoetry instructions.Now, clone the repository and runpoetry install.  This will create a virtual
environment and install all required packages there.Runpoetry run pytestto run all tests.Runpoetry buildto build package ondist/.Issue trackerPlease report any bugs and enhancement ideas using the GitHub issue tracker:https://github.com/dymaxionlabs/dask-rasterio/issuesFeel free to also ask questions on ourGitter channel, or by email.Help wantedAny help in testing, development, documentation and other tasks is highly
appreciated and useful to the project.For more details, see the fileCONTRIBUTING.md.LicenseSource code is released under a BSD-2 license.  Please refer toLICENSE.mdfor more information.
Package: dask-regionprops
dask-regionpropsAboutThis is a small library that uses dask to compute regionprops in parallel.In addition to parallelization, it adds a few features/specializations on top of
thescikit-imageregionprops implementation.dask_regionpropswill return a dask dataframe containing the region properties as columns.Arrays can be numpy or dask arrays as well as xarray DataArrays backed by either array libary.ND arrays get processed as a sequence of 2D arrays. Typically we assume that the last two
dimenions contain the images and the leading dimensions will be looped over.In the ND case, the result dataframe will have columns that map each labelIntended Use CaseI wrote this library to help analyze microscopy datasets. After segmentation I typically have a 4D xarray DataArray
where the dimensions are (Position, Time, Y, X). Importantly, I reuse label values between positions but not times
so for all of the time-points in positionS, the region labelledrshould refer to the same cell. Hopefully this
motivated the decision to return the leading dimensions in the dataframe. For instance if you want to get the properties
of a cell 5 in position 2 you could do something like:from dask_regionprops import regionprops

# Assume data is a numpy/dask array that has dims corresponding to (S,T,Y,X)
props = regionprops(data)
single_cell_props = props.loc[(props["dim-0"]==2)&(props["label"]==5)]If you are a more advanced pandas user, and you want to do this sort of analysis for many cells,
you might consider using the leading dimensions and region labels as amultiindexto more efficiently
access the data in this way.Finally, a useful downstream application is to use the region properties as features for a classifer
or maybe even a clustering algorithm. I have personally input labelled regions and the corresponding
fluorescence images to identify progression through the cell cycle.ContributionsPlease feel free to open an issue or pull-request if you have questions or improvements for this library.
Package: dask-sas-reader
Dask SAS ReaderRead SAS files into a Python Dask DataFrame.UsageInstall from PyPIpip install dask_sas_readerImport and use module.import dask_sas_reader

import dask
import dask.dataframe as dd
from dask.delayed import delayed


data_dir = Path('../data')
file_path = data_dir / 'bond.sas7bdat'

dd_df = sas.dask_sas_reader(data_dir, blocksize=800000)
dd_df.head()TestTest simply using:pytest --collect-only
pytest
Package: dask-searchcv
Tools for performing hyperparameter search withScikit-LearnandDask.HighlightsDrop-in replacement for Scikit-Learn’sGridSearchCVandRandomizedSearchCV.Hyperparameter optimization can be done in parallel using threads, processes,
or distributed across a cluster.Works well with Dask collections. Dask arrays, dataframes, and delayed can be
passed tofit.Candidate estimators with identical parameters and inputs will only be fit
once. For composite-estimators such asPipelinethis can be significantly
more efficient as it can avoid expensive repeated computations.For more information, check out thedocumentation.InstallDask-searchcv is available viacondaorpip:# Install with conda
$ conda install dask-searchcv -c conda-forge

# Install with pip
$ pip install dask-searchcvExamplefromsklearn.datasetsimportload_digitsfromsklearn.svmimportSVCimportdask_searchcvasdcvimportnumpyasnpdigits=load_digits()param_space={'C':np.logspace(-4,4,9),'gamma':np.logspace(-4,4,9),'class_weight':[None,'balanced']}model=SVC(kernel='rbf')search=dcv.GridSearchCV(model,param_space,cv=3)search.fit(digits.data,digits.target)
Package: dask-snowflake
Dask-SnowflakeThis connector is in an early experimental/testing phase.Reach out to usif you are interested in trying
it out!Installationdask-snowflakecan be installed withpip:pipinstalldask-snowflakeor withconda:condainstall-cconda-forgedask-snowflakeUsagedask-snowflakeprovidesread_snowflakeandto_snowflakemethods
for parallel IO from Snowflake with Dask.>>>fromdask_snowflakeimportread_snowflake>>>example_query='''...    SELECT *...    FROM SNOWFLAKE_SAMPLE_DATA.TPCH_SF1.CUSTOMER;... '''>>>ddf=read_snowflake(...query=example_query,...connection_kwargs={..."user":"...",..."password":"...",..."account":"...",...},...)>>>fromdask_snowflakeimportto_snowflake>>>to_snowflake(...ddf,...name="my_table",...connection_kwargs={..."user":"...",..."password":"...",..."account":"...",...},...)See their docstrings for further API information.TestsRunning tests requires a Snowflake account and access to a database.
The test suite will automatically look for specificSNOWFLAKE_*environment variables (listed below) that must be set.It's recommended (though not required) to store these environment variables
in a local.envfile in the root of thedask-snowflakerepository.
This file will be automatically ignored bygit, reducing the risk of accidentally
commiting it.Here's what an example.envfile looks like:SNOWFLAKE_USER="<test user name>"
SNOWFLAKE_PASSWORD="<test_user_password>"
SNOWFLAKE_ACCOUNT="<account>.<region>.aws"
SNOWFLAKE_WAREHOUSE="<test warehouse>"
SNOWFLAKE_ROLE="<test role>"
SNOWFLAKE_DATABASE="<test database>"
SNOWFLAKE_SCHEMA="<test schema>"You may thensource .envor installpytest-dotenvto automatically set these environment variables.Note:If you run the tests and get anMemoryErrormentioning
"write+execute memory for ffi.callback()", you probably have stale
build ofcffifrom conda-forge. Remove it and install the version
usingpip:condaremovecffi--force
pipinstallcffiLicenseBSD-3
Package: dask-spark
Launch Dask from Spark and Spark from Dask.  This project is not mature.Examplespip install dask-sparkCreate Spark cluster from a Dask cluster>>>fromdask.distributedimportClient>>>client=Client('scheduler-address:8786')>>>client<Client:scheduler='tcp://scheduler-address:8786'processes=8cores=64>>>>fromdask_sparkimportdask_to_spark>>>sc=dask_to_spark(client)>>>sc<pyspark.context.SparkContextat0x7f62fa4bb550>Create Dask cluster from a Spark cluster>>>importpyspark>>>sc=pyspark.SparkContext('local[4]')<pyspark.context.SparkContextat0x7f8b908b0128>>>>fromdask_sparkimportspark_to_dask>>>client=spark_to_dask(sc)>>>client<Client:scheduler="'tcp://127.0.0.1:8786'">Requirements and How this WorksThis depends on a relatively recent version of Dask.distributed.For starting Spark from Dask this assumes that you have Spark installed and
that thestart-master.shandstart-slave.shSpark scripts are available
on the PATH of the workers.  This starts a long-running Spark master process on
the Dask Scheduler and starts long running Spark slaves on Dask workers.  There
will only be one slave per worker.  We set the number of cores and the amount
of memory to match the Dask workers and available memory.When starting Dask from Spark this will block the Spark cluster.  We start a
scheduler on the local machine and then run a long-running function that starts
up a Dask worker usingRDD.mapPartitions.TODO[ ] This almost certainly fails in non-trivial situations[ ] Enable user specification of Java flags for memory and core use[ ] Support multiple spark clusters per Dask cluster
Package: dask-sphinx-theme
This is the official Sphinx theme for Dask documentation.  It extends thesphinx-book-themeproject, but adds custom styling and a navigation bar to
additional Dask subprojects.When creating a Dask subproject you can include this theme by changing this
line in your conf.py filehtml_theme='dask_sphinx_theme'And by includingdask_sphinx_themeas a requirement in your documentation
installation.ReleasingThis projectuses GitHub Actionsto automatically push a new release to PyPI whenever
a git tag is pushed. For example, to release a newx.y.zversion ofdask-sphinx-theme, checkout the commit you would like to release,
add a git tag, and push the tag to themainbranch of thedask/dask-sphinx-themerepository:$ git checkout main
$ git tag -a x.y.z -m 'Version x.y.z'
$ git push upstream main --tagsAfter a new release is published on PyPI, a pull request to thedask-sphinx-themeconda-forge feedstockfor the newx.y.zrelease will automatically be opened by conda-forge bots.
Package: dask-tensorflow
Start TensorFlow clusters from DaskExampleGiven a Dask clusterfromdask.distributedimportClientclient=Client('scheduler-address:8786')Get a TensorFlow cluster, specifying groups by namefromdask_tensorflowimportstart_tensorflowtf_spec,dask_spec=start_tensorflow(client,ps=2,worker=4)>>>tf_spec{'worker':['192.168.1.100:2222','192.168.1.101:2222','192.168.1.102:2222','192.168.1.103:2222'],'ps':['192.168.1.104:2222','192.168.1.105:2222']}This creates atensorflow.train.Serveron each Dask worker and sets up a
Queue for data transfer on each worker.  These are accessible directly astensorflow_serverandtensorflow_queueattributes on the workers.More Complex WorkflowTypically then we set up long running Dask tasks that get these servers and
participate in general TensorFlow compuations.fromdask.distributedimportworker_clientdefps_function(self):withworker_client()asc:tf_server=c.worker.tensorflow_servertf_server.join()ps_tasks=[client.submit(ps_function,workers=worker,pure=False)forworkerindask_spec['ps']]defworker_function(self):withworker_client()asc:tf_server=c.worker.tensorflow_server# ... use tensorflow as desired ...worker_tasks=[client.submit(worker_function,workers=worker,pure=False)forworkerindask_spec['worker']]One simple and flexible approach is to have these functions block on queues and
feed them data from dask arrays, dataframes, etc.defworker_function(self):withworker_client()asc:tf_server=c.worker.tensorflow_serverqueue=c.worker.tensorflow_queuewhilenotstopping_condition():batch=queue.get()# train with batchAnd then dump blocks of numpy and pandas dataframes to these queuesfromdistributed.worker_clientimportget_workerdefdump_batch(batch):worker=get_worker()worker.tensorflow_queue.put(batch)importdask.dataframeasdddf=dd.read_csv('hdfs:///path/to/*.csv')# clean up dataframe as necessarypartitions=df.to_delayed()# delayed pandas dataframesclient.map(dump_batch,partitions)
Package: dask-xgboost
WarningDask-XGBoost has been deprecated and is no longer maintained. The functionality
of this project has been included directly in XGBoost. To use Dask and XGBoost
together, please usexgboost.daskinsteadhttps://xgboost.readthedocs.io/en/latest/tutorials/dask.html.Distributed training with XGBoost and Dask.distributedThis repository offers a legacy option to perform distributed training
with XGBoost on Dask.array and Dask.dataframe collections.pip install dask-xgboostPlease note that XGBoost now includes a Dask API as part of its official Python package.
That API is independent ofdask-xgboostand is now the recommended way to use Dask
adn XGBoost together. Seethe xgb.dask documentation here https://xgboost.readthedocs.io/en/latest/tutorials/dask.htmlfor more details on the new API.Examplefromdask.distributedimportClientclient=Client('scheduler-address:8786')# connect to clusterimportdask.dataframeasdddf=dd.read_csv('...')# use dask.dataframe to load anddf_train=...# preprocess datalabels_train=...importdask_xgboostasdxgbparams={'objective':'binary:logistic',...}# use normal xgboost paramsbst=dxgb.train(client,params,df_train,labels_train)>>>bst# Get back normal XGBoost result<xgboost.core.Boosterat...>predictions=dxgb.predict(client,bst,data_test)How this worksFor more information on using Dask.dataframe for preprocessing see theDask.dataframe documentation.Once you have created suitable data and labels we are ready for distributed
training with XGBoost.  Every Dask worker sets up an XGBoost slave and gives
them enough information to find each other.  Then Dask workers hand their
in-memory Pandas dataframes to XGBoost (one Dask dataframe is just many Pandas
dataframes spread around the memory of many machines).  XGBoost handles
distributed training on its own without Dask interference.  XGBoost then hands
back a singlexgboost.Boosterresult object.Larger ExampleFor a more serious example seeThis blogpostThis notebookThis screencastHistoryConversation during development happened atdmlc/xgboost #2032