Saravanan666/pypi-org huggingface dataset:
Package: parallel-hadd
This is a python package to allow usage of hadd (from ROOT CERN) in
parallel. It allows adding histogram and.rootfiles. The addition
will occur in blocks instead of one at all. It was designed to work with
limited memory (local development primilary). The addition will usemultiprocessingif avilable in your python enviroment.Requirements:The minimum supported python version is 3.6 and 3.7 and 3.8 are tested.
Not tested on python 3.9 yet.Installclone this repositorygit clone https://github.com/MohamedElashri/hadd-parallelbuild the package from source usingpippip3 install ./hadd-parallelManualThe command to use the package isphaddyou can use it as the following (with default arguments)phadd out.root *.rootwhereout.rootwill be the output file the contains all the*.rootfiles.more information about how to customize the settings are available on
program help pagephadd -hHelp pageusage: phadd [-h] [-t TMPDIR] [-j NUM_JOBS] [-n NUM_FILES] [-f] [-s] [-l {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [-v]
             output_file input_files [input_files ...]

Hadd ROOT histograms in parallel

positional arguments:
  output_file           the output file
  input_files           one or more input files

optional arguments:
  -h, --help            show this help message and exit
  -t TMPDIR, --tmpdir TMPDIR
                        the temporary directory to store intermediate files
  -j NUM_JOBS, --jobs NUM_JOBS
                        the number of jobs to run in parallel , [default cpu_count * 2 = {N_JOBS}
  -n NUM_FILES, --num_files NUM_FILES
                        the number of files to hadd at once
  -f, --force_overwrite
                        force overwrite of output file
  -s, --save-tmp        save the intermediate files, otherwise they will be deleted (which is the default)
  -l {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --log {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        the log level, [default Warning]
  -v, --version         show program's version number and exit
Package: parallelhtmlscraper
Parallel HTML ScraperHelps you to web scrape html file in parallel without async / await syntax.FeatureThis project helps you to web scrape html file in parallel without async / await syntax.Installationpip install parallelhtmlscraperUsageMinimum example:frombs4importBeautifulSoupfromparallelhtmlscraper.html_analyzerimportHtmlAnalyzerfromparallelhtmlscraper.parallel_html_scraperimportParallelHtmlScraperclassAnalyzerExample(HtmlAnalyzer):asyncdefexecute(self,soup:BeautifulSoup)->str:returnsoup.find('title').texthost_google='https://www.google.com'path_and_content=['',# Google Search'/imghp?hl=EN',# Google Images'/shopping?hl=en',# Google Shopping'/save',# Collection'https://www.google.com/maps?hl=en',# Google Maps'https://www.google.com/drive/apps.html',# Google drive'https://www.google.com/mail/help/intl/en/about.html?vm=r',# GMail]list_response=ParallelHtmlScraper.execute(host_google,path_and_content,AnalyzerExample())print(list_response)$pipenvrunpythontest.py['\n      Gmail - Email from Google\n    ', 'Google Images', '  Google Maps  ', 'Using Google Drive - New Features, Benefits & Advantages of Google Cloud Storage', 'Google Shopping', 'Google', 'Collections']APIParallelMediaDownloader.executeclassParallelHtmlScraper:"""API of parallel HTML scraping."""@staticmethoddefexecute(base_url:str,list_url:Iterable[str],analyzer:HtmlAnalyzer[_T],*,limit:int=5interval:int=1)->List[_T]:base_url: strCommon part of request URL.
This will be help to download URLs got from HTML.list_url: Iterable[str]List of URL. Method will download them in parallel.
Absolute URL having same base URL asbase_urlalso can be specified.analyzer: HtmlAnalyzer[_T]The instance extendsHtmlAnalyzerto analyze HTML by usingBeautifulSoup.
Following example will be help to understand its roll:classAnalyzerExample(HtmlAnalyzer):asyncdefexecute(self,soup:BeautifulSoup)->str:returnsoup.find('title').textlimit: int = 5Limit number of parallel processes.interval: int = 1Interval between each request(second).
Package: parallelic
ParallelicParallelic is a hyperparallel multi-node task execution engine with shared data and wokspace capabilities.Note of warningParallelic is not a containerization/sandboxing engine. It does not constitute a full task isolation, and provides no guarantee of such. That may change in the future, and feel free to contribute your code towards that goal, but in the mean time, keep this in consideration when giving access to a Parallelic system to third parties.InstallationFrom gitClone the git repo locally.Download python3(.7) and corresponding pipInstallPoetryRunpoetry installto create a virtualenv and install dependenciesAt this point, you can use parallelic throughpoetry run parallelicRunpoetry buildto build a wheelRunpip install dist/parallelic-[version]-py3-none-any.whlNow you can use parallelic without poetry:python -m parallelicFrom pipRunpip install parallelicUsageRunning a taskTo run an already defined task, you upload the task package (a zipped up task root directory) via the Parallelic WebUI, or Parallelic CLI client, to the Parallelic manager instance.You may need to provide access credentials before being allowed to upload the task package, as per your Parallelic system configuration.From there, the Parallelic manager instance will take care of everything else.Defining a taskThe task root contains atask.tomlfile, that contains metadata required for the manager to set up and prepare resources for the compute nodes in order to run the particullar task.If the task requires no additional files, the task definition can be only thetask.tomlfile.The directory tree doesn't follow a particullar convention, and can be different from task to task. The task definition file should contain a section where the entrypoint and working directory are defined. Both the entrypoint and the working directory have to be relative to the task root.CreditsPackage maintained by Trickster Animations
Package: parallelism
parallelismUnlock advanced task scheduling in Python withparallelism.
Seamlessly coordinate parallel and concurrent execution, optimizing performance while ensuring code integrity.
Embrace expert scheduling techniques to elevate your programming to new heights of efficiency and responsiveness.InstallationYou can install this package using pip:pipinstallparallelismBasic UsageExplore an effortless approach to task creation and management.
Follow these steps to get started:Import the necessary classes and functions in your Python code.# Built-in modulesfrommultiprocessingimportProcessfromthreadingimportThread# Third-party librariesfromparallelismimportscheduled_task,task_schedulerDefine your task functions. These user-defined functions will be executed in parallel and concurrently.deffunc(*args,**kwargs):ifnotargsandnotkwargs:raiseValueError('Missing *args or **kwargs')returnargs,kwargsCreate task instances using thescheduled_taskfunction, specifying the execution unit (Process or Thread), task name, function, and provide any required positional or keyword arguments.task1=scheduled_task(Process,'task1',func,args=(1,2,3),continual=True)task2=scheduled_task(Process,'task2',func,kwargs={'a':10,'b':20},continual=True)task3=scheduled_task(Thread,'task3',func,continual=True)Schedule tasks using thetask_schedulerfunction, Specify the tasks to be executed along with the desired number of processes and threads.result=task_scheduler(tasks=(task1,task2,task3),processes=2,threads=4)Access task execution details and results through the result object, providing insights into execution times, elapsed times, exceptions, and return values:>>>result.execution_time{'task1':datetime.datetime(%Y,%m,%d,%H,%M,%S,%f),'task2':datetime.datetime(%Y,%m,%d,%H,%M,%S,%f),'task3':datetime.datetime(%Y,%m,%d,%H,%M,%S,%f),}>>>result.elapsed_time{'task1':float(...),'task2':float(...),'task3':float(...),}>>>result.raise_exception{'task3':ValueError('Missing *args or **kwargs'),}>>>result.return_value{'task1':((1,2,3),{}),'task2':((),{'a':10,'b':20}),}For more comprehensive documentation and advanced usage, please refer to the fullAPI Documentation.
Package: parallelizer
parallelizerParallel execution of your tasks simplified! This package can be used as an alternative to Python'sconcurrent.futuresexecutors, or as an analog of Java'sExecutorService.Two simple wrappers (thread-based and process-based), that allow for easy split and parallel processing of your jobs.DescriptionThis package provides two classes:ThreadParallelizerto execute your jobs in new threadsProcessParallelizerto execute your jobs in new processesThis is a pure python implementation, with usage ofthreadingandmultiprocessingpackagesNote that the wrappers appear to be in sync (from the caller's perspective), they will wait until all inner tasks are completed.InstallationNormal installationpipinstallparallelizerDevelopment installationgitclonehttps://github.com/jpleorx/parallelizer.gitcdparallelizer
pipinstall--editable.ExampleimporttimefromparallelizerimportThreadParallelizer,ProcessParallelizer,repeatdefpower_function(base:int,power:int)->int:time.sleep(1)returnbase**powerinputs=[1,2,3,4,5,6,7,8,9,10]number_of_threads=5thread_parallelizer=ThreadParallelizer(number_of_threads)results=thread_parallelizer.execute(power_function,[inputs,repeat(2,len(inputs))])process_parallelizer=ProcessParallelizer(number_of_threads)results=process_parallelizer.execute(power_function,[inputs,repeat(2,len(inputs))])
Package: paralleller
paralleller 介绍用于进行并行处理的接口库。开源许可: MIT文档:https://llango.github.io/paralleller.特征TODO项目布局main.py    # 项目主文件，服务使用它启动。 uvicorn main:app --reload
.env       # 环境变量文件
requirements.txt    # 依赖文件
readme.md       # 简单说明文件
test.db         # 生成的数据库文件
mkdocs.yml      # mkdocs文档构建配置文件
LICENSE         # 开源许可文件
Procfile        # 容器用来执行用户设定的命令
Makefile        # 用来启动docker-compose 文件
.dockerignore   # docker打包上传到容器里过滤文件
.gitignore      # 版本控制过滤上传文件
docker-compose.yaml # docker-compose配置启动文件
Dockerfile      # Docker文件
core/
    blog.py     # 与数据库进行博客交互文件
    user.py     # 与数据库进行用户交互文件
    __init__.py # 声明为包所需文件
database/
    config.py   # 数据库连接配置文件
    __init__.py # 声明为包所需文件
docs/
    index.md    # 文档首页
    database_config.md      # 对应database/config.py 文件的讲解
models/
    models.py   # 用来建立数据库表映射的类文件
    __init__.py # 声明为包所需文件
route/
    auth.py     # 登录认证接口文件
    blog.py     # 博客接口文件
    user.py     # 用户接口文件
    __init__.py # 声明为包所需文件
schema/
    schemas.py  # 声明类型文件
    __init__.py # 声明为包所需文件
static/
    css/
        index.css   # 样式文件
    js/
        main.js     # 脚步
    images/         # 图片文件夹
templates/
    index.html      # 模板文件
test/               # 测试目录
utils/
    hash.py         # 密码处理文件
    oa2.py          # 认证处理文件
    token.py        # 指令处理文件
    __init__.py     # 声明为包所需文件项目启动1. 进行该项目中，建立虚拟环境
```
virtualenv venv
source  venv/bin/activate
```
2. 安装依赖
```
pip install -r requirements.txt
```
3. 直接使用如下命令启动:
    ``` 
    uvicorn main:app --realod 
    ```文档启动mkdocs serve --livereload -t shadocs -a localhost:8080制作该包使用Cookiecutter和llango/cookiecutter-mkdoc-shapackage项目模版创建。
Package: parallel-manager
parallel_managerA simple shell command manager. Used for running parallel simulations.How-toFirst install the package withpip:pipinstallparallel_managerTo run the manager, we will need to wrap the manager initialization and request creations all in a async function like this:fromparallel_manager.managerimportBaseShellManagerfromparallel_manager.workerGroupimportShellWorkerGroupimportloggingimportasyncioasyncdefMain():# Init the managersimpleShellWorkergroup=ShellWorkerGroup("simpleShellWorkergroup",logging.getLogger(),"./log",10)simpleShellManager=BaseShellManager("simpleShellManager")simpleShellManager.add_workergroup("shell",simpleShellWorkergroup)awaitsimpleShellManager.init()# Adding tasks# Here we just run 100 echoforiinrange(100):simpleShellManager.add_shell_request(f"echoing loop-{i}",f"echo{i}")# Wait for the manager to finishawaitsimpleShellManager.done()Then we simply useasyncio.run()to run the above function:asyncio.run(Main())The manager will run at most10processes at a given time and will terminate when all100requests are done.Full example is athereStructureWorkerReceive request from queue and process itWorkerGroupManage workers of same classCollect basic stats from workersManagerMight host various workergroups with different worker typeSupport various ways to add tasksControl workergroups/workersVarious middleware/plugin to the BaseManager to implement other features like http web monitoring and task submissionTODOsShort-term todosManager statistics pluginManager summary when all tasks are doneTimeout supportStatus monitor, progress monitorDump failed task/able to rerunDump unfinished tasks when script terminatesLong-term todosThread-mode support (completely non-blocking)Use a separate thread to run the workersAllow to submit task at any timeHttp monitorInteractive control for status monitoring and changing configuration dynamically, treat worker as a server that can submit task to, like a local slurm controlPythonFunctionWorkerclass (execute python function calls) and other worker class as wellStandardize plugin format for each classFor instance, http server monitor could be a plugin that can be added to any manager class.
Package: parallel-map
# A simple parallel map utility## InstallingFirst find a Python 3 executable.### From PyPI` $ pip install parallel_map `### Manual InstallThe following step creates build abuild/directory, which can be added
to yourPYTHONPATH.` $ python setup.py build `To build the software into the global installation of your environment (ideally,
but not necessarily, a virtual environment), use:` $ pip install . `## Example`python from parallel_map import parallel_map parallel_map(lambda x:x**2,range(10)) `## LicenseApache License 2.0, see LICENSE.txt.This repository contains a stand-alone module of a useful parallel mapping
feature, originally written by Taylor R. Campbell as part of the Venture
project.
Package: parallelmediadownloader
Parallel Media DownloaderHelps you to download media file in parallel without async / await syntax.FeatureThis project helps you to download media files in parallel without async / await syntax.Installationpip install parallelmediadownloaderUsageMinimum example:fromdatetimeimportdatetimefromparallelmediadownloader.media_download_coroutineimportDownloadOrderfromparallelmediadownloader.media_fileimportSaveOrderfromparallelmediadownloader.parallel_media_downloaderimportParallelMediaDownloaderpath_directory_download="path/directory/download"created_date_time=datetime.now()list_download_order=[DownloadOrder("https://example.com/test01.png",SaveOrder(path_directory_download,"test01.png",created_date_time,),),DownloadOrder("https://example.com/test02.png",SaveOrder(path_directory_download,"test02.png",created_date_time,),),DownloadOrder("https://example.com/test03.png",SaveOrder(path_directory_download,"test03.png",created_date_time,),),]list_media_download_result=ParallelMediaDownloader.execute(list_download_order)APIParallelMediaDownloader.executeclassParallelMediaDownloader:"""API of parallel media downloading."""@staticmethoddefexecute(list_download_order:Iterable[DownloadOrder],*,limit:int=5,media_filter:Optional[MediaFilter]=None,allow_http_status:List[int]=None)->List[MediaDownloadResult]:list_download_order: Iterable[DownloadOrder]List ofDownloadOrder. Method will download them in parallel.limit: int = 5Limit number of parallel processes.media_filter: Optional[MediaFilter] = NoneFilter extendsMediaFilterto remove downloaded media file depending on file or content of media.NotImageFilterwill be help to understand its roll:classNotImageFilter(MediaFilter):def_filter(self,media_file:MediaFile)->bool:returnnotstr(media_file.path_file).lower().endswith((".png",".jpg",".jpeg",".gif"))allow_http_status: List[int] = NoneBy default, ParallelMediaDownloader.execute will check HTTP status code byResponse.raise_for_statusand whole process will stop.
When HTTP status applies allow_http_status, process will continue.
Then,MediaDownloadResult.media_filewill beNone.
Package: parallel-numpy-rng
parallel-numpy-rngA multi-threaded random number generator backed by Numpy RNG, with parallelism provided by Numba.OverviewUses the "fast-forward" capability of the PCG-family of RNG, as exposed by the
new-style Numpy RNG API, to generate random numbers in a multi-threaded manner. The key
is that you get the same random numbers regardless of how many threads were used.Only a two types of random numbers are supported right now: uniform and normal. More
could be added if there is demand, although some kinds, like bounded random ints, are
hard to parallelize in the approach used here.Example + Performanceimportnumpyasnpimportparallel_numpy_rngseed=123parallel_rng=parallel_numpy_rng.default_rng(seed)numpy_rng=np.random.default_rng(seed)%timeitnumpy_rng.random(size=10**9,dtype=np.float32)# 2.85 s%timeitparallel_rng.random(size=10**9,dtype=np.float32,nthread=1)# 3.34 s%timeitparallel_rng.random(size=10**9,dtype=np.float32,nthread=128)# 67.8 ms%timeitnumpy_rng.standard_normal(size=10**8,dtype=np.float32)# 1.12 s%timeitparallel_rng.standard_normal(size=10**8,dtype=np.float32,nthread=1)# 1.85 s%timeitparallel_rng.standard_normal(size=10**8,dtype=np.float32,nthread=128)# 43.5 msNote that theparallel_rngis slower than Numpy when using a single thread, because the parallel implementation requires a slower algorithm in some cases (this is especially noticeable for float64 and normals)InstallationThe code works and isreasonably well tested, so it's probably ready for use.  It can be installed from PyPI:$pipinstallparallel-numpy-rng
Package: parallel-phpunit
Parallel Test Cases Runner for PHPUnitInstallationComposerTo install with Composer, execute the following command inside your project:composerrequire--devsocialpoint/parallel-phpunitAfter installation, the binary can be found atvendor/bin/parallel-phpunitPipparallel-phpunit can be installed via “pip”, the Python package manager, with:pip installparallel-phpunitAfter installation, you can use executingparallel-phpunitUsageUsage is as follows:parallel-phpunit [-h] [--max_concurrency MAX_CONCURRENCY]
                      [--configuration CONFIGURATION]
                      [--phpunit_bin PHPUNIT_BIN]
                      [--test_suffix TEST_SUFFIX]
                      [TestCases Dirs [TestCases Dirs ...]]

positional arguments:
  TestCases Dirs        Directories of test cases

optional arguments:
  -h, --help            show this help message and exit
  --max_concurrency MAX_CONCURRENCY
                        Max TestCase processing concurrency (8 by default)
  --configuration CONFIGURATION, -c CONFIGURATION
                        Read configuration from XML file
  --phpunit_bin PHPUNIT_BIN
                        phpunit bin path
  --test_suffix TEST_SUFFIX
                        phpunit test suffix (by default Test.php)
Package: parallel-pro
parallel_proparallel_pro is a simple package for parallel processing.Installationpipinstallparallel_proHow to usefromparallel_proimportrun_parallelfromdatetimeimportdatetimedeftest(data):ss=list()foriindata:ss.append(i*2)returnssdefmain():start=datetime.now()data=run_parallel(list_of_data=list(range(100)),process_fun=test,workers=4,verbose=False)end=datetime.now()# print(data)diff=end-startprint(diff.total_seconds()/60)
Package: parallel-processor
介绍我们提供了一个可快速处理大量数据的多进程数据处理模块此模块可以处理list、numpy、pandas等数据，并且可以以tuple的形式输入，以tuple的形式输出中间的处理过程可以让用户来决定，只要传入一个自定义函数即可可设置进程数样例引入：fromparallel_processorimportprocess_data# 注意在向进程池传入处理函数时，不能使用lambda的方式，否则会报错fromfunctoolsimportpartial样例1：分词defseg(x):result=[]foriintqdm_notebook(x):result.append(list(jieba.cut(i)))returnresultdata=process_data(data,seg,num_workers=16)样例2：text2idsdefconvert_example(x,f_token,max_seq_len,return_raw_data=False,mask_rate=0,x1=None,avg_split=False):input_ids_list=[]input_mask_list=[]segment_ids_list=[]input_ids_gt_list=[]contents=[]ifx1isnotNone:if"JPY_PARENT_PID"inos.environ:bar=tqdm_notebook(zip(x,x1))else:bar=tqdm(zip(x,x1))else:if"JPY_PARENT_PID"inos.environ:bar=tqdm_notebook(x)else:bar=tqdm(x)forlineinbar:ifx1isnotNone:line,line1=line[0],line[1]else:line1=''ifmask_rate<=0:input_ids,input_mask,segment_ids=convert_single_example(max_seq_len,f_token,line,line1,avg_split=avg_split)else:input_ids,input_mask,segment_ids,input_ids_gt=convert_single_example_with_mlm(max_seq_len,f_token,line,line1,avg_split=avg_split)input_ids=np.array(input_ids)input_mask=np.array(input_mask)segment_ids=np.array(segment_ids)input_ids_list.append(input_ids)input_mask_list.append(input_mask)segment_ids_list.append(segment_ids)ifmask_rate>0:input_ids_gt=np.array(input_ids_gt)input_ids_gt_list.append(input_ids_gt)ifreturn_raw_data:contents.append(list(line.replace('\n','')))ifmask_rate>0:ifreturn_raw_data:returnnp.array(input_ids_list),np.array(input_mask_list),np.array(segment_ids_list),\np.array(input_ids_gt_list),np.array(contents)returnnp.array(input_ids_list),np.array(input_mask_list),np.array(segment_ids_list),\np.array(input_ids_gt_list)else:ifreturn_raw_data:returnnp.array(input_ids_list),np.array(input_mask_list),np.array(segment_ids_list),np.array(contents)returnnp.array(input_ids_list),np.array(input_mask_list),np.array(segment_ids_list)def_process(text,**kwargs):f_token=kwargs.get('f_token')max_seq_len=kwargs.get('max_seq_len',128)return_raw_data=kwargs.get('return_raw_data',False)mask_rate=kwargs.get('mask_rate',0)avg_split=kwargs.get('avg_split',False)iflen(text.shape)>1:text_a,text_b=text[:,0],text[:,1]returnconvert_example(text_a,f_token,max_seq_len,return_raw_data,mask_rate,text_b,avg_split=avg_split)else:returnconvert_example(text,f_token,max_seq_len,return_raw_data,mask_rate,avg_split=avg_split)vocab_file=os.path.join(bert_model_dir,"vocab.txt")f_token=FullTokenizer(vocab_file)text=data[text_cols].valuesfunc_kwargs={'f_token':f_token,'max_seq_len':max_seq_len,'return_raw_data':return_raw_data,'mask_rate':mask_rate,'avg_split':avg_split}result=process_data(text,_process,num_workers=16,is_tuple_data=False,func_kwargs=func_kwargs)input_ids=result[0]input_mask=result[1]segment_ids=result[2]
Package: parallelprocs
parallelprocsRun commands via GNU Parallel or something like it.Synopsis$ cat test.py
#!/usr/bin/env python

from parallelprocs import run

try:
    run(['echo foo', 'echo bar', 'echo baz'], verbose=True)
except Exception as e:
    print(e)
$ ./test.py
Running job (# jobs = 3)
bar
baz
fooDescriptionThe module exposes a single functionrunthat takes the following arguments:Required Arguments:commands [str]: commandsKeyword Optionsparallel (str): path to "parallel", default "which('parallel')"num_procs (int): num of concurrent processes, default "0" to use all CPUsverbose (bool): print messages to sys.stderr, default "False"halt (int): number of failed jobs to trigger halt, default "0" for no haltAuthorKen Youens-Clarkkyclark@gmail.com