Saravanan666/pypi-org huggingface dataset:
Package: distex
IntroductionDistex offers a distributed process pool to utilize multiple CPUs or machines.
It usesasyncioto efficiently manage the worker processes.Features:Scales from 1 to 1000’s of processors;Can handle in the order of 50.000 small tasks per second;Easy to use with SSH (secure shell) hosts;Full async support;Maps over unbounded iterables;Compatible withconcurrent.futures.ProcessPool(orPEP3148).Installationpip3 install -U distexWhen using remote hosts then distex must be installed on those too.
Make sure that thedistex_procscript can be found in the path.For SSH hosts: Authentication should be done with SSH keys since there is
no support for passwords. The remote installation  can be tested with:ssh <host> distex_procDependencies:Pythonversion 3.6 or higher;On Unix theuvlooppackage is recommended:pip3 install uvloopSSH client and server (optional).ExamplesA process pool can have local and remote workers.
Here is a pool that uses 4 local workers:fromdisteximportPooldeff(x):returnx*xpool=Pool(4)foryinpool.map(f,range(100)):print(y)To create a pool that also uses 8 workers on hostmaxi, using ssh:pool=Pool(4,'ssh://maxi/8')To use a pool in combination witheventkit:fromdisteximportPoolimporteventkitasevimportbz2pool=Pool()# await pool  # un-comment in Jupyterdata=[b'A'*1000000]*1000pipe=ev.Sequence(data).poolmap(pool,bz2.compress).map(len).mean().last()print(pipe.run())# in Jupyter: print(await pipe)pool.shutdown()There is full support for every asynchronous construct imaginable:importasynciofromdisteximportPooldefinit():# pool initializer: set the start time for every workerimporttimeimportbuiltinsbuiltins.t0=time.time()asyncdeftimer(i=0):# async code running in the poolimporttimeimportasyncioawaitasyncio.sleep(1)returntime.time()-t0asyncdefait():# async iterator running on the user sideforiinrange(20):awaitasyncio.sleep(0.1)yieldiasyncdefmain():asyncwithPool(4,initializer=init,qsize=1)aspool:asyncfortinpool.map_async(timer,ait()):print(t)print(awaitpool.run_on_all_async(timer))asyncio.run(main())High level architectureDistex does not use remote ‘task servers’.
Instead it is done the other way around: A local
server is started first; Then the local and remote workers are started
and each of them will connect on its own back to the server. When all
workers have connected then the pool is ready for duty.Each worker consists of a single-threaded process that is running
an asyncio event loop. This loop is used both for communication and for
running asynchronous tasks. Synchronous tasks are run in a blocking fashion.When using ssh, a remote (or ‘reverse’) tunnel is created from a remote Unix socket
to the local Unix socket that the local server is listening on.
Multiple workers on a remote machine will use the same Unix socket and
share the same ssh tunnel.The plainsshexecutable is used instead of much nicer solutions such
asAsyncSSH. This is to keep the
CPU usage of encrypting/decrypting outside of the event loop and offload
it to thesshprocess(es).DocumentationDistex documentationauthor:Ewald de Wit <ewald.de.wit@gmail.com>
Package: dist-exe
Distributed ExecutorA cross-platform reference implementation for processing shell commands over
a distributed, asynchronous queue.dist_exeis a single producer
(server) many consumer (client) system. It functions similar toGNU Parallelbut scales out to many servers and even across operating systems.This tool uses only standard library methods so there are no dependencies.
It has been tested on Linux, macOS, and Windows 10 in Python 3.7 environments.
The server and clients don't even need to be using the same platform.Installationpip install dist_exe [--upgrade] [--user]UsageTheserverrole reads from standard input and publishes to a queue. If you don't
have something actively creating work a good approach is to create aMANIFESTfile
and usetail -fto wait for work to arrive.user@host-01: ~ $ touch MANIFEST
user@host-01: ~ $ tail -f MANIFEST | dist_exe server -H 0.0.0.0 -p 5050 -k SECRETTheclientrole connects to the remote queue and pulls items off, executing them using the local shell.
You can specify a template for the command line invocation, otherwise the items are take as being the entire
invocation.user@host-02: ~ $ dist_exe client -H host-01 -p 5050 -k SECRET -t 'echo {}'
Package: distext
Some useful distutils/setuptools/distribute extensions to deal with our needs @
Silveregg.The main features are:support for arbitrary and customizable installation of data files, ala
autoconf. For example, you can set a file to be installed in $sysconfdir, which
will expand to /etc by default, but can be customized from the command
line. Only sysconfdir, bin and libdir are supported for now.optionally generate a python module which contains the aforementioned
paths so that they can be retrieved at runtimeoption to install init scripts, optionally from templates using install
variables (think init scripts which need to know where other bits are
installed). The templating format is Jinja2.basic code to write hg revision in a version file
Package: dist-gb-akhlaqi
dist_gbGuassian and Binomial distribution package📝 Table of ContentsAboutGetting StartedDeploymentUsageBuilt UsingTODOContributingAuthorsAcknowledgments🧐 AboutThis initial package is a simple package for doing statistical operation on a dataset.
Currently, this package support two distribution including Guassian and Binomial.
The future contibutions is encouraged to add more functionality in this pakcage.🏁 Getting StartedThese instructions will get you a copy of the project up and running on your local machine for development and testing purposes. Seedeploymentfor notes on how to deploy the project on a live system.PrerequisitesTo draw the plot you need to install matplotlib.
There is no more prerequisit for this package.pip install matplotlibInstallingThis package is poblicly available in pypi repository (https://pypi.org/project/dist-gb-akhlaqi/). To install from pypi repository run:pip install dist-gb-akhlaqiTo easy install and use this package follow this commands:Clone the package source code into your local systemGit clone https://github.com/myakhlaqi/distributions_package.gitGo to dist_gb directory, run:pip install pyproject.tomlThe package will be installed under the name "dist_gb:x.x.x" to make sure run:pip list | grep dist🔧 Running the testsRun testTo run the test cases on this package there is a simple test.py file in the
main direcotry. You can add more test case or just run the existing one to
make sure that the code run error free.To run the test type:pytest test.py🎈 UsageHow to use examples:import imp
from dist_gb.src import Gaussian

g1= Gaussian()
g1.read_data_file('data.csv')
g1.plot_histogram()
print(g1.calculate_mean())
print(g1.calculate_stdev)🚀 DeploymentAdd additional notes about how to deploy this on a live system.⛏️ Built Using✍️ Authors@myakhlaqi🎉 AcknowledgementsI inspired by the Udacity AWS ML engineering nano-degreee program to write this package.Thanks from AWS and Udacity for giving this opportinityhttps://www.udacity.com/scholarships/aws-machine-learning-scholarship-program
Package: distGB-probability
#distGB prabality packageThis is a package to calculate the Gausian and Bianomial Probability#FilesGeneraldistribution.pyGaussiandistribution.pyBinomialdistribution.py#installationto install use the below codepip install distGB-probabilityAfter the installation import the functions as shown below and use the packagefrom Gaussiandistribution import Gaussian
from Binomialdistribution import Binomial
Package: distgen
Distribution oriented templating systemThe problem this project tries to mitigate is "portable" scripting for variety
of operating systems (currently Linux distributions only) in the wild.  While
writing an "universal" script, one needs to take into account small or bigger
differences among operating systems (like package installation tools, versions
of utilities, expected directories for binaries, libraries, etc.).Thedistgenproject is thus something like database of OS differences together
with convenience tool-set that allows you to instantiate valid script for
particular distribution.  The concept is to havetemplatefile (mostly raw
jinja2 template) together with "declarative"specfile (YAML file) that
fulfils the needs of particular template.You can find distgen documentation athttp://distgen.readthedocs.io.Download/InstallationStable releases of distgen are available as RPMs in Fedora and EPEL
repositories, and inpypi:$ sudo dnf install distgen

$ pip install distgenDevelopment (git snapshot) RPMs are automatically built inFedora
Copr.You can also run development version directly fromgit, simply use thedgshell wrapper available in this git root directory.Example with DockerfileTypical example is the need to instantiate workingFOOpackage orientedDockerfiles for all supportedFedora/RHELversions.To achieve that goal with distgen, you need to write something likedocker.tpltemplate andFOO.yamlspec file.  If the system-defaultdocker.tpltemplate is good enough, its enough to write proper spec file:createFOO.yamlspec:$ cat FOO.yaml
maintainer: John Doe <jdoe@example.com>
parts:
  pkginstall:
    data:
      - type: pkg
        action: install
        packages:
          - vim
  footer:
    cmd: ["vim"]Rundgtool to generateFedora 22Dockerfile:$ dg --template docker.tpl \
     --spec FOO.yaml \
     --distro fedora-22-x86_64.yaml \
> Dockerfile
$ cat Dockerfile
FROM index.docker.io/fedora:22
MAINTAINER John Doe <jdoe@example.com>

ENV container="docker"

RUN dnf -y --setopt=tsflags=nodocs install vim \
    && dnf -y --setopt=tsflags=nodocs clean all --enablerepo='*'

CMD ["vim"]Rundgtool again to generateRHEL 7dockerfile:$ dg --template docker.tpl \
     --spec FOO.yaml \
     --distro rhel-7-x86_64.yaml \
> Dockerfile
$ cat Dockerfile
FROM registry.access.redhat.com/rhel7
MAINTAINER John Doe <jdoe@example.com>

ENV container="docker"

RUN yum -y --setopt=tsflags=nodocs install vim \
    && yum -y --setopt=tsflags=nodocs clean all --enablerepo='*'

CMD ["vim"]
Package: distgfs
distgfsDistributed computing framework for theGlobal Function Search(GFS) hyperparameter optimizer from theDliblibrary.
Based ongfsopt.Provides the following features:Parallel optimization: Run distributed hyperparameter searches viampi4py.Save and restore progress: Save/restore settings, parameters and optimization progress to/from HDF5 file.Average over multiple runs: Run a stochastic objective function using the same
parameters multiple times and report the average to Dlib's Global Function
Search. Useful in highly stochastic domains to avoid biasing the search towards
lucky runs.For theoretical background of GFS, see'A Global Optimization Algorithm Worth Using'andMalherbe & Vayatis 2017: Global optimization of Lipschitz functionsExample usageA basic example where we maximize Levi's function with as many parallel processes as there are logical cores, and save progress to file.importmath,distgfsdeflevi(x,y):"""Levi's function (see https://en.wikipedia.org/wiki/Test_functions_for_optimization).Has a global _minimum_ of 0 at x=1, y=1."""a=math.sin(3.*math.pi*x)**2b=(x-1)**2*(1+math.sin(3.*math.pi*y)**2)c=(y-1)**2*(1+math.sin(2.*math.pi*y)**2)returna+b+cdefobj_fun(pp,pid):""" Objective function to be _maximized_ by GFS. """x=pp['x']y=pp['y']res=levi(0.4*x,y)print(f"Iter:{pid}\tx:{x}, y:{y}, result:{res}")# Since Dlib maximizes, but we want to find the minimum,# we negate the result before passing it to the Dlib optimizer.return-res# For this example, we pretend that we want to keep 'y' fixed at 1.0# while optimizing 'x' in the range -4.5 to 4.5space={'x':[-4.5,4.5]}problem_parameters={'y':1.}# Create an optimizer parameter setdistgfs_params={'opt_id':'distgfs_levi','obj_fun_name':'obj_fun','obj_fun_module':'example_distgfs_levi_file','problem_parameters':problem_parameters,'space':space,'n_iter':10,'file_path':'distgfs.levi.h5','save':True,}distgfs.run(distgfs_params,verbose=True)For additional examples, seeexamples.
Package: distify
distifyWrapper around Ray for easy distributed processing in Python.Disclaimer: I use it for some of my projects, but there is no guarantee that the code will work as expected or that future versions will be backward compatible.Installpip install distify --upgradeFeaturesMultiple backends: Ray, Multiprocessing, Multithreading, Sequential.Logging.Progress bar.Can run in local or in multiple nodes.Individual timeout for map applications.Resume from checkpointing.Hydra integration.Quickstart/usagePlease refer to this example:https://github.com/jordiae/distify/tree/main/examples/basicDocumentationTODOLICENSEMITAuthorJordi Armengol Estapé
Package: distillable-vision-transformer
PyTorch Vision Transformers with DistillationBased on the paper "Training data-efficient image transformers & distillation through attention".This repository will allow you to use distillation techniques with vision transformers in PyTorch. Most importantly, you can use pretrained models for the teacher, the student, or even both! My motivation was to use transfer learning to decrease the amount of resources it takes to train a vision transformer.QuickstartInstall withpip install distillable_vision_transformerand load a pretrained transformer with:fromdistillable_vision_transformerimportDistillableVisionTransformermodel=DistillableVisionTransformer.from_pretrained('ViT-B_16')InstallationInstall via pip:pip install distillable_vision_transformerOr install from source:git clone https://github.com/Graeme22/DistillableVisionTransformer.git
cd DistillableVisionTransformer
pip install -e .UsageLoad a model architecture:fromdistillable_vision_transformerimportDistillableVisionTransformermodel=DistillableVisionTransformer.from_name('ViT-B_16')Load a pretrained model:fromdistillable_vision_transformerimportDistillableVisionTransformermodel=DistillableVisionTransformer.from_pretrained('ViT-B_16')Default hyper parameters:Param\ModelViT-B_16ViT-B_32ViT-L_16ViT-L_32image_size384, 384384, 384384, 384384, 384patch_size16321632emb_dim76876810241024mlp_dim3072307240964096num_heads12121616num_layers12122424num_classes1000100010001000attn_dropout_rate0.00.00.00.0dropout_rate0.10.10.10.1If you need to modify these hyperparameters, just overwrite them:model=DistillableVisionTransformer.from_name('ViT-B_16',patch_size=64,emb_dim=2048,...)TrainingWrap the student (instance ofDistillableVisionTransformer) and the teacher (any network that you want to use to train the student) with aDistillationTrainer:fromdistillable_vision_transformerimportDistillableVisionTransformer,DistillationTrainerstudent=DistillableVisionTransformer.from_pretrained('ViT-B_16')trainer=DistillationTrainer(teacher=teacher,student=student)# where teacher is some pretrained network, e.g. an EfficientNetFor the loss function, it is recommended that you use theDistilledLossclass, which is a kind of hybrid between cross-entropy and KL-divergence loss.
It takes as argumentsteacher_logits,student_logits, anddistill_logits, which are obtained from the forward pass onDistillationTrainer, as well as the true labelslabels.fromdistillable_vision_transformerimportDistilledLossloss_fn=DistilledLoss(alpha=0.5,temperature=3.0)loss=loss_fn(teacher_logits,student_logits,distill_logits,labels)InferenceFor inference, we want to use theDistillableVisionTransformerinstance, not itsDistillationTrainerwrapper.importtorchfromdistillable_vision_transformerimportDistillableVisionTransformermodel=DistillableVisionTransformer.from_pretrained('ViT-B_16')model.eval()inputs=torch.rand(1,3,*model.image_size)# we can discard the distillation tokens, as they are only needed to calculate lossoutputs,_=model(inputs)ContributingIf you find a bug, create a GitHub issue, or even better, submit a pull request. Similarly, if you have questions, simply post them as GitHub issues.
Package: distillation
DistillationPerform a calculation for distillation of mixtures
of chemical compounds.Currently only supports configurations with total condensers and partial reboilers.
Equilibrium data currently only comes from DePriester charts
and is assumed to only depend on pressure and temperature.TutorialA tutorial can be foundhereDocumentationThe online documentation can be foundhere.ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Package: distill.py
distill.pyI love blogging. In fact, I love blogging too much. I used to use wordpress, mkdocs but I don’t satisfy with these platform. So I create my own platform FOR ME.Notes:It’s quick and very dirty project.I’m sure that I don’t known how to “finish it”However I guarantee it will task some years.And finally I will make fun (and many bugs)History0.0.1 (2017-08-03)Hello WorldRefactor feature_engineering code
Package: distilNLP
distilNLPdistilNLP is a low-dependency, GPU-free natural language processing toolkit designed for building natural language processing applications and preprocessing for deep learning.Installpip install -U distilNLPUsageText NormalizeText normalization processing. Correct common character errors in text from the internet.fromdistilnlpimporttext_normalizenorm=text_normalize('Proposed programme budget for the biennium 2006-2007： Section 29， Office of Internal Oversight Services。 ',lang='en')# got: 'Proposed programme budget for the biennium 2006-2007: Section 29, Office of Internal Oversight Services。'norm=text_normalize(' 2006-2007两年期拟议方案预算:😇第29款,内部监督事务厅.',lang='zh')# got: '2006-2007两年期拟议方案预算：第29款，内部监督事务厅。'norm=text_normalize('百度的网址是:  http：//baidu.com',lang='zh')# got: '百度的网址是： http://baidu.com'
Package: distimate
Distimate - Distributions visualizedDistimate approximates and plots common statistical functions from histograms.Distimate can aggregate empirical distributions of random variables.
The distributions are represented as histograms with user-defined bucket edges.
This is especially useful when working with large datasets
that can be aggregated to histograms at database level.importdistimateimportmatplotlib.pyplotaspltedges=[0,1,2,5,10,15,20,50]values=[291,10,143,190,155,60,90,34,27]dist=distimate.Distribution.from_histogram(edges,values)plt.title(f"x̃={dist.quantile(0.5):.2f}")plt.xlim(0,50)plt.ylim(0,1)plt.plot(dist.cdf.x,dist.cdf.y,label="CDF")plt.plot(dist.pdf.x,dist.pdf.y,label="PDF")plt.legend(loc="lower right")Features:Histogram creation and mergingProbability density function (PDF)Cumulative distribution function (CDF or ECDF)Quantile (percentile) functionPandas integration.Distimate is hosted atGitHuband
it can be installed fromPyPI.DocumentationDistimate documentation from thedocs/directory can be read
online atRead the Docs.LicenseCopyright 2020 Akamai Technologies, Inc

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.ContributingBy submitting a contribution (the “Contribution”) to this project,
and for good and valuable consideration, the receipt and sufficiency of which
are hereby acknowledged, you (the “Assignor”) irrevocably convey, transfer,
and assign the Contribution to the owner of the repository (the “Assignee”),
and the Assignee hereby accepts, all of your right, title, and interest in and
to the Contribution along with all associated copyrights, copyright
registrations, and/or applications for registration and all issuances,
extensions and renewals thereof (collectively, the “Assigned Copyrights”).
You also assign all of your rights of any kind whatsoever accruing under
the Assigned Copyrights provided by applicable law of any jurisdiction,
by international treaties and conventions and otherwise throughout the world.
Package: distinctid
DISTINCTIDWithdistinctid, you can generate distinct/unique IDs. Just think of it as something that can replace your UUID. It has the advantage of being a long integer and can be sorted in ASC or DESC order.PhilosophyThis code is pure Python and relies onthis articlewhich led tothis,thisand this present project you are reading now.100% Unittest CoverageSee the unittest to know how to use itpipinstallcoverage# unittestpython-munittestdiscover# OR# coverage with unittestcoveragerun-munittestdiscover\&&coveragehtml&&coveragereport# browse: htmlcov/index.htmlHow to Use# Browse a short video:https://twitter.com/zuoike/status/1544337418467397633
Package: distinctiveness
Distinctiveness CentralityA Python library to calculate Distinctiveness Centrality in social networks.Tutorials and code available on GitHub:https://github.com/iandreafc/distinctivenessPlease cite asFronzetti Colladon, A., & Naldi, M. (2020). Distinctiveness Centrality in Social Networks.PLoS ONE,15(5), e0233276.https://doi.org/10.1371/journal.pone.0233276
Package: distinfo
Distinfodistinfois a tool for extracting metadata, including full dependencies, from
Python source distributions.UsageCLIDump json metadata to stdout:$ distinfo /path/to/package/sourceLibraryPrint dependencies and metadata:>>>fromdistinfoimportDistribution,dump>>>>>>dist=Distribution("/path/to/package/source")>>>dump(dist.requires){"build":["setuptools-scm"],"dev":["pycmd",],"run":["click","requests",],"test":["pytest",]}>>>dump(dist.metadata){"author":"A N Other","author_email":"a@example.org","extensions":{"distinfo":{"imports":{"distinfo":["click","requests"],"tests":["pytest",]},"tests":["tests",],}},"license":"GPL-3.0-or-later","metadata_version":"2.1","name":"example","provides_extra":["build","dev","test"],"requires_dist":["click","pycmd; extra == 'dev'","pytest; extra == 'test'","requests""setuptools-scm; extra == 'build'",],"summary":"Example package","version":"0.0.0"}Specificationshttps://packaging.python.org/specifications/MetadataPEP 241 - Metadata for Python Software Packages 1.0PEP 314 - Metadata for Python Software Packages 1.1PEP 345 - Metadata for Python Software Packages 1.2PEP 426 - Metadata for Python Software Packages 2.0PEP 566 - Metadata for Python Software Packages 2.1PEP 459 -- Standard Metadata Extensions for Python Software PackagesDependenciesPEP 440 - Version Identification and Dependency SpecificationPEP 508 - Dependency specification for Python Software PackagesPEP 518 - Specifying Minimum Build System Requirements for Python Projects
Package: distinguishedname
distinguishednameParse RFC 2253 Distinguished Names from strings and create strings from DNs (which
are implemented here as lists of RDNs - which are lists of strings in "TAG=VALUE"
format).Examples>>>string_to_dn(r'CN=   James \"Bond\" +UID= 007, OU = 00 Division\, Special Services, O=MI5,C=UK')[['CN=James "Bond"','UID=007'],['OU=00 Division, Special Services'],['O=MI5'],['C=UK']]>>>dn_to_string([['CN="Pinkie" Stevens','UID=1'],['O= Name With Spaces ']])'CN=\\"Pinkie\\" Stevens+UID=1,O=\\20Name With Spaces\\20'RationaleI couldn't find any existing libraries for this.  In theory,python-ldaphas a DN class that
should be able to do this, but I've never been able topip installit successfully.This is essentially a single file, so you could also just drop this file into your project, ansible
module, CLI script or whatever.