Saravanan666/pypi-org huggingface dataset:
Package: pipelib
PipelibPipelib is a library for creating pipelines. You can parse, compare, and order iterables. With Pipelib you can:Create a custom pipeline to parse and compare version stringsUse a collection of provided sorting functions for custom sorts.Assemble different processing blocks to pre-process inputs first.The initial ideas came fromSingularity Registry HPC (shpc)that had a need to parse and compare version strings from docker container tags.⭐️Documentation⭐️TODOadd tests for wrappers and pipelinesautomated detection / docs for pipelines tooshould be able to print pretty a pipeline / stepsToInteger doesn't work because we choose an int wrapper - we need to be able to detect output and apply a different wrapper type given int.pipeline steps will need a way to sort / compare / filterContributorsWe use theall-contributorstool to generate a contributors graphic below.Vanessasaurus💻LicenseThis code is licensed under the MPL 2.0LICENSE.
Package: pipeline
Provides a file-like interface to other system processes and shell-likepipelines.  In contrast with os.popen, the shell is not used to interpret thecommands, and the exit status of each componenet of the pipeline is availabe.An alternative to builtin packages such as subprocess, commands, os.popen,popen2, os.system, etc.Examples of how to feed the pipe with a file-like object are included in thedocstrings. Requires os.fork, but does not require any threadinglibraries.
Package: pipeline-anomaly-detection-gazprom
Модуль Python для проверки точности определения размеров дефектов газопроводаЭтот модуль Python реализует требования Газпрома к порогу обнаружения и точности определения размеров дефектов трубопровода при продольном и поперечном намагничивании
согласно "Р Газпром2-2.3-919-2015 ОСНОВНОЕ И ВСПОМОГАТЕЛЬНОЕ ОБОРУДОВАНИЕ ДЛЯ ВНУТРИТРУБНОГО ДИАГНОСТИРОВАНИЯ Технические требования".Установкаpipinstallpipeline-anomaly-detection-gazpromИспользованиефункцияis_detectableопределяет, должен ли выявляться дефект заданных размеров на заданной толщине стенки трубы
при указанном типе намагничивания.Дефект размером 10x10 мм глубиной 5 мм на стенке трубы толщиной 11 мм должен определяться снарядами с продольным намагничиванием.frompipeline_anomaly_detection_gazpromimportis_detectable,MagnetTypeassertis_detectable((10,10,5),11,magnet_type=MagnetType.MFL)функцияis_in_limitsпроверяет, укладываются ли в допустимую погрешность вычисленные размеры дефекта по сравнению с его реальными размерами
при заданных толщине стенки трубы и типе намагничивания.frompipeline_anomaly_detection_gazpromimportis_in_limits# реальные размеры дефекта: длина 90 мм, ширина 12 мм, глубина 4 ммreal=(90,12,4)# вычисленные размеры дефекта: длина 72 мм, ширина 11 мм, глубина 1 ммcalcked=(72,11,1)# проверяем, соответствуют ли вычисленные размеры дефекта допускам, если толщина стенки трубы равна 16.6 мм и тип намагничивания поперечныйlength_ok,width_ok,depth_ok=is_in_limits(calcked,real,16.6,magnet_type=MagnetType.TFI)# вычисленная длина укладывается в допустимую погрешностьassertlength_ok# вычисленная ширина укладывается в допустимую погрешностьassertwidth_ok# вычисленная глубина НЕ укладывается в допустимую погрешностьassertnotdepth_okРазработкаgitclonegit@github.com:vb64/pipeline.anomaly.detection.gazprom.gitcdpipeline.anomaly.detection.gazpromПри использовании Python 2makesetup2PYTHON_BIN=/path/to/python27/executable
maketests2При использовании Python 3makesetupPYTHON_BIN=/path/to/python37/executable
maketests
Package: pipeline-anomaly-dimension-class
Python class for pipeline anomaly dimension classificationThis Python module implements the definition of the "defect dimension class" of a pipeline according toSpecifications and requirements for intelligent pig inspection of pipelines.Installationpipinstallpipeline-anomaly-dimension-classUsagefrompipeline_anomaly_dimension_classimportsize_class,FeatureClass# a defect measuring 100x100 mm on a pipe wall 10 mm thick is a defect of the "GENE" classassertsize_class(100,100,10)==FeatureClass.GENE# defects with zero dimensions are not allowedsize_class(0,100,10)pipeline_anomaly_dimension_class.Error:WrongFeatureClassparams.l=0w=100t=10Developmentgitclonegit@github.com:vb64/pipeline.anomaly.dimension.class.gitcdpipeline.anomaly.dimension.classWith Python 2makesetup2PYTHON_BIN=/path/to/python27/executable
maketests2With Python 3makesetupPYTHON_BIN=/path/to/python3/executable
maketests
Package: pipeline-cli
Command-line interface to the Sepior Pipeline.RequirementsPython 3.5Install using pyenv or pyvenvInstallation$pipinstallpipeline_cliUsageThe tool will show a description of usage when given the–helpoption:$pipeline--helpusage:pipeline[-h][--version][--hostHOST_NAME]{start,enqueue}...Thepipelineutilitypositionalarguments:{start,enqueue}sub-commandsstartstarthelpenqueueenqueuedownstreamdependencieshelpoptionalarguments:-h,--helpshowthishelpmessageandexit--version,-vshowprogramsversionnumberandexit--hostHOST_NAMEspecifypipelineserverhost.Defaultishttps://pipeline.sepior.net```The tool is able to:
1. start a pipeline
2. enqueue downstream pipelinesStarting a pipelineWe assume you already have a pipeline file. If the pipeline file
is in the current directory then simply run$pipelinestart<version>Seepipeline start –helpfor more advanced usecases.Enqueue downstream pipelinesThis command is relevant if the pipeline you have is building inside
the pipeline or if you manually want to (re)start downstream pipelines.Seepipeline enqueue –helpfor parameters.Building the toolMake sure you have a Python 3.5 environment with the requirements.E.g. use pyenv:$pyenvvirtualenv3.5.0pipeline-cli-venv$pyenvactivatepipeline-cli-venvMake sure pip is up-to-date:$pipinstall--upgradepipInstall requirements:$pipinstall--upgrade-rrequirements.txtBuild the wheel:$makeDevelopmentUse the following command to install the package in the local
environment during development.$pipinstall-e.This allows you to change the code and test the pipeline cli
directly.ContributingWhen contributing changes remember to update theCHANGELOG.md.LicenseProprietary. Copyright 2016 Sepior ApS.ReleasingDo the following to release a new version:Commit changesPush changesMerge with masterUpdate local masterFind the next release version, e.g. 6.6.6Create new branch with name core/release-6.6.6Bump version in __about__.pyCommitMerge with masterUpdate local masterRun./release.sh 6.6.6Upload to PypiFirst, perform a test upload to verify everything is nice and dandy.
Then perform the real upload.Build the code$makeThen define the following environment variables:$exportPYPI_TEST_PASSWORD=""$exportPYPI_PASSWORD=""Test upload to pypiYou may need to register on the Pypi test server. This can be done here:$https://testpypi.python.org/pypiRegister:$twineregister-p${PYPI_TEST_PASSWORD}-rpypitestdist/pipeline_cli-6.6.6-py3-none-any.whlUpload$maketestpypi_uploadGoto:$https://testpypi.python.org/pypi/pipeline-cli/6.6.6A check that everything looks nice.You can check the HTML by running:$pythonsetup.py--long-description|rst2html.py--no-raw>output.htmlTest if it installs (do it in a different environment):$pipinstall-ihttps://testpypi.python.org/pypipipeline-cliReal upload to PypiUpload$makepypi_uploadGoto:$https://pypi.python.org/pypi/pipeline-cli/6.6.6And check that every things looks nice.
Package: pipeline-composer
CodePipeline ComposerGenerate your CodeCommit/CodeBuild/CodePipeline templates made easierFree software: BSD licenseDocumentation:https://pipeline-composer.readthedocs.io.FeaturesTODOCreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.0.1 (2020-04-05)First release on PyPI.
Package: pipeline-core
pipeline-coreThe PipelineCore repo.commandsauthpcore credentials|credential add|rm|ls|get|use gcp <credentials-name>create radish deploymentpcore clusters|cluster create|rm pcloud|gcp|aws|azure <cluster-name>radish cluster managementpcore clusters|cluster config <cluster-alias> [sub command]:scaling-strategy|caching-strategy|routing-strategy set|get|listgpu-pools add|rm|list|get
Package: pipelined
Overview…Usage examplesimportpipelineddefdetect_ddos(stream):access=pipelined.utils.limit_counter(10*1000)forloginstream:access[log.ip]+=1yield fromaccess.over(100)defblock_ips(stream):foripinstream:# block the ipyieldipddos_monitor=[detect_ddos,block_ips,]source_logs=pipelined.files.follow('/path/to/server/access.log')foripinpipelined.run(ddos_monitor,source_logs):print(ip,"has been blocked.")ResourcesSource:https://github.com/hanula/pipelinedDocumentation:https://github.com/hanula/pipelinedLicenseSeeLICENSE.
Package: pipeline-deploy
Pipeline Deploy is a CLI tool that makes it possible to quickly and easily deploy data pipelines across a variety of systems.## Requirements
* Python Version > 3.6## InstallationTo install, simply runpip install –upgrade pipeline-deploy.## Platform-Specific Requirements
Pipeline Deploy was created to work with a variety of systems, however, each of these systems have prerequisites that need to be met in order to ensure a fully functional experience.### Databricks
Once installation of the Databricks Deploy package is complete, follow the Databricks CLI [installation](https://github.com/databricks/databricks-cli#installation) and [configuration](https://docs.databricks.com/dev-tools/cli/index.html#set-up-authentication) instructions.NOTE:you MUST complete the configuration of your Databricks CLI in order for pipeline-deploy to work.### Other Platforms
Coming soon…## DocumentationLooking more information on how to get started withpipeline-deploy? Check out the project [wiki](https://github.com/comcast/pipeline-deploy/wiki) for this info and more.## Help and SupportFor bugs or other questions regarding this project, please create a [new Github Issue](https://github.com/comcast/pipeline-deploy/issues/new/choose).## License
Pipeline Deploy is licensed under the [Apache License, Version 2.0](LICENSE).
Package: pipeline-diacritizer
Multi-components system for automatic Arabic diacritics restorationAboutThis tool is a command-line application written in Python 3 that automatically add diacritics to
raw undiacritized Arabic text. To accomplish this task, it uses several techniques: Deep Learning, rule-based and
statistical corrections. The deep learning part was implemented using Tensorflow. It was released as a support for the
research paper:"Multi-components system for automatic Arabic diacritization" at ECIR2020.InstallationThis tool is available as a Python 3 packagepipeline-diacritizerinstallable throughpip. For installation
instructions check theDownload and installationwiki page.FunctionsThis tool has 4 main functions: preprocessing of the data, training on the processed data, testing, and
restoring the diacritics of an undiacritized text. In addition, it can calculates some statistics on a given dataset and
the ratio of Out-of-Vocabulary words in a testing set according to a train set.This is a quick introduction to the most important ones, without mentioning all the possible options for each one. For
additional options, consider calling any subcommand with the option--helpor-h(ex:pipeline_diacritizer train --help) orcheck the wikifor more details.PreprocessingBefore feeding the new data to this application for training or testing, it needs to be converted to the standard format
of this application: one sentence per line, where a sentence is delimited by a dot, a comma, or an end of line
character.$ pipeline_diacritizer process <source_file> <destination_file>If the data is not yet partitioned into training, validation and testing sets, the program can help in this task using
the following command:$ pipeline_diacritizer partition <dataset_file>TrainingTo run the training and validation on selected training/validation sets, use the next command:$ pipeline_diacritizer train --train-data <train_file> --val-data <val_file>TestingTo evaluate the performances of the application on a testing set, use this command:$ pipeline_diacritizer test <test_file>DiacritizationThe following command restores the diacritics of the Arabic words from the supplied text file and outputs a diacritized
copy:$ pipeline_diacritizer diacritize <text_file>StatisticsTo get some statistics about the dataset, such as the count of tokens, arabic words, numbers... use the following
command:$ pipeline_diacritizer stat <dataset_file>OoV CountingTo calculate the ratio of the Out-of-Vocabulary words between the train set and the validation/test set, use the next
command:$ pipeline_diacritizer oov <train_file> <test_file>LicensePipeline-diacritizer code is licensed underMIT License.
Package: pipeline-dp
PipelineDPPipelineDP is a framework for applying differentially private aggregations to large
datasets using batch processing systems such as Apache Spark, Apache Beam,
and more.To make differential privacy accessible to non-experts, PipelineDP:Provides a convenient API familiar to Spark or Beam developers.Encapsulates the complexities of differential privacy, such as:protecting outliers and rare categories,generating safe noise,privacy budget accounting.Supports many standard computations, such as count, sum, and average.Additional information can be found atpipelinedp.io.Notethat this project is still experimental and is subject to change.
At the moment we don't recommend its usage in production systems as it's not
thoroughly tested yet. You can learn more in theRoadmap section.The project is a collaboration between OpenMined and Google in an effort
to bring Differential Privacy to production.Getting startedHere are some examples of how to use PipelineDP:Apache Spark exampleApache Beam exampleFramework-free exampleExample with all frameworksPlease check out thecodelabfor a more detailed demonstration of the API functionality and usage.Code sample showing private processing on Spark:# Define the privacy budget available for our computation.budget_accountant=pipeline_dp.NaiveBudgetAccountant(total_epsilon=1,total_delta=1e-6)# Wrap Spark's RDD into its private version. You will use this private wrapper# for all further processing instead of the Spark's RDD. Using the wrapper ensures# that only private statistics can be released.private_movie_views=\make_private(movie_views,budget_accountant,lambdamv:mv.user_id)# Calculate the private sum of ratings per moviedp_result=private_movie_views.sum(SumParams(# The aggregation key: we're grouping data by moviespartition_extractor=lambdamv:mv.movie_id,# The value we're aggregating: we're summing up ratingsvalue_extractor=lambdamv:mv.rating,# Limits to how much one user can contribute:# .. at most two movies rated per user#    (if there's more, randomly choose two)max_partitions_contributed=2,# .. at most one ratings for each moviemax_contributions_per_partition=1,# .. with minimal rating of "1"#    (automatically clip the lesser values to "1")min_value=1,# .. and maximum rating of "5"#    (automatically clip the greater values to "5")max_value=5))budget_accountant.compute_budgets()# Save the resultsdp_result.saveAsTextFile(FLAGS.output_file)InstallationPipelineDP without any frameworks:pip install pipeline-dpIf you like to run PipelineDP on Apache Spark:pip install pipeline-dp pysparkon Apache Beam:pip install pipeline-dp apache-beam.Supported Python version >= 3.8.Note for Apple Silicon users:PipelineDP pip package is currently available only
for x86 architecture. The reason is thatPyDPdoes not
have pip pacakge. It might be possible to compile it from sources for Apple Silicon.DevelopmentTo setup a local environment and contribute with the development of PipelineDP, please see our guidelines inCONTRIBUTING.Support and Community on SlackIf you have questions about the PipelineDP, joinOpenMined's Slackand check the#differential-privacychannel.
Package: pipeline-dsl
Pipeline DSL forConcourseFeaturesDevelop complete Concourse pipelines in plain PythonTest Pipelines inside Concourse without pushing them to githubTest Pipelines locallyInstallationInstallation fromPyPIpip3installpipeline-dslInstallation from GitHubpip3install--upgradegit+https://github.com/sap/pipeline-dsl.git@mainExamplefrompipeline_dslimportPipeline,GitRepowithPipeline("c21s")aspipeline:pipeline.resource("shalm",GitRepo("https://github.com/wonderix/shalm"))withpipeline.job("create-cluster")asjob:shalm=job.get("shalm")cluster_name="test"@job.task()defcreate_shoot():print(f"Create cluster{cluster_name}")returncluster_name@job.task(secrets={"home":"HOME"})definstall_shalm(home=None):print("HOME="+home)print(f"Installing shalm{shalm.path}into{create_shoot()}")return"Hello"DocumentationUsers guideReference guideContribution guidelinesExamples
Package: pipeline-exec
pipeline-execAboutpipeline-exec is an execution Pipeline to apply to  an Object Collection. It's like a
group of functions to run on a list of objects.
The original goal behind this project was to create a framework to help create a funnel pipeline for Django models instance.
We're sure there can be many more uses.InstallationJust run:pip install -U pipeline-execUsageYou need to create a pipeline and add a few funnels to it:frompipeline_exec.pipelineimportPipe,Pipeline# creating our funnelsclassMyTestPipePlusOne(Pipe):defrun(self,l):return[i+1foriinl]classMyTestPipePlusTwo(Pipe):defrun(self,l):return[i+2foriinl]classMyTestPipeMinusOne(Pipe):defrun(self,l):return[i-1foriinl]# creating our pipelinepipeline=Pipeline([MyTestPipePlusOne(),MyTestPipePlusTwo(),MyTestPipeMinusOne()])# run the pipelinepipeline.run([1,1,1])# returns [3, 3, 3]ContributeFor now, it's very straight forward. Everyone is welcome to contribute. All pull request are against the develop branch.
Unit tests and the flake8 linter run at every push.
Thanks in advance for your contribution :)
Package: pipeline-feature
Feature Pipelinecd/path/to/pipeline-feature~/mlops-pipeline-v1/pipeline-feature$curl-s-omake_venv.sh\https://raw.githubusercontent.com/gao-hongnan/common-utils/main/scripts/devops/make_venv.sh&&\bashmake_venv.shvenv_pipeline_feature--pyproject--dev&&\sourcevenv_pipeline_feature/bin/activate&&\rmmake_venv.shpythonpipeline-feature/mlops_pipeline_feature_v1/pipeline.py
Package: pipeline-functions
PIPELINE-FUNCTIONSClass for manipulating a pipeline of functions, creating chains of functions processing the same context.
In this way it is possible that functions can send information between them through the execution chain.Example of how it works-> Function one receives a parameter, executes and returns data.
-> Function two receives as a parameter the result of the previous one, executes and returns data.
...
-> Final function takes data from all previous functions and processes.Install$pipinstallpipeline-functionsExamplefrompipeline_functionsimportPipelineFunctionsdefhello(parameter:dict={}):print("Calling hello...")parameter.update({"hello":"Hello!"})returnparameterdefmiddle(parameter:dict={}):print("Calling middle...")raiseException()returnparameterdefworld(parameter:dict={}):print("Calling world...")parameter.update({"world":"World!"})returnparameterdeffinish(parameter:dict={}):print("Calling finish...")print(parameter)returnparameterif__name__=="__main__":pipeline=PipelineFunctions(functions=[hello,middle,world,finish],debug_mode=True,block_mode=False)pipeline.execute(param={"universe":"universe!"})
Package: pipeline-material
Pipeline material class for static integrity calculationInstallationpipinstallpipeline-materialUsagefrompipeline_materialimportPipelineMaterialmaterial=PipeMaterial("Steel",20000)assertmaterial.smys==20000assert"smys 20000"instr(material)Development$ git clone git@github.com:vb64/pipeline.material.git
$ cd pipeline.materialWith Python 3:$ make setup PYTHON_BIN=/path/to/python3
$ make testsWith Python 2:$ make setup2 PYTHON_BIN=/path/to/python2
$ make tests2
Package: pipeline-materials-file
Pipeline materials file for static integrity calculationInstallationpipinstallpipeline-materials-fileUsagefrompipeline_materials_fileimportMaterialsfrompipeline_csv.csvfileimportFilefrompipeline_csv.csvfile.rowimportRowdeftable=File(1000)deftable.data=[Row.as_weld(1000),Row.as_thick(1010,105),Row.as_weld(12000),Row.as_thick(12010,110),Row.as_weld(24000),Row.as_thick(12010,110),Row.as_weld(36000),]warnings=[]data=Materials.from_deftable(deftable,warnings)assertlen(data.data)==2assertnotwarningsdata.to_csv('materials.csv')Development$ git clone git@github.com:vb64/pipeline.material.git
$ cd pipeline.material
$ make setup PYTHON_BIN=/path/to/python3
$ make tests
Package: pipeline-mermaid
Pipeline as mermaidThe useful tool to show Gitlab pipeline as mermaidrequired environmentsGITLAB_API_TOKEN- gitlab api tokenUsingShow current pipeline in merge request notes.Example merge requestin .gitlab-ci.ymlnotify merge request:
  image: registry.gitlab.com/nov1kov/pipeline_to_mermaid:0.5
  stage: deploy
  only:
    - merge_requests
  script:
    - pipeline-mermaid show_current_pipelineas python packageinstallpip install pipeline-mermaidshow current pipeline in current merge request as gantt diagrampipeline-mermaid show_pipelineshow specific pipeline in specific merge request as gantt diagrampipeline-mermaid show_pipeline 249125838 gantt 3as python libraryfrompipeline_mermaid.gitlab_helperimportGitlabHelpergl=GitlabHelper()gl.show_pipeline(249125838,"gantt",3)Gitlab pipeline as mermaidgraph LR

classDef failed fill:white,stroke:#db3b21,color:black;
classDef success fill:white,stroke:#1aaa55,color:black;
classDef warning fill:white,stroke:#fc9403,color:black;
classDef skipped fill:white,stroke:#999,color:black;

730991283(android)
730991285(s3)
730991286(firebase)
730991287(slack)

730991283 --> 730991285
730991283 --> 730991286
730991285 --> 730991287
730991286 --> 730991287

class 730991283 success
class 730991285 warning
class 730991286 success
class 730991287 failedgantt

dateFormat  YYYY-MM-DDTHH:mm:ss.SSSZ
axisFormat  %H:%M:%S

section test
unit tests : 730991283, 2020-09-12T12:26:05.370Z, 2020-09-12T12:26:41.665Z

section build
build : 730991284, 2020-09-12T12:26:42.182Z, 2020-09-12T12:27:38.126Z

section deploy
s3 : 730991285, 2020-09-12T12:27:43.757Z, 2020-09-12T12:28:35.406Z
firebase : 730991286, 2020-09-12T12:27:43.757Z, 2020-09-12T12:28:44.406Zjourney
section build
  android: -1: running
  ios: -1: running
section deploy
  slack-android: -1: created
  slack-ios: -1: createdtodo:gitlabtry depend on CI_JOB_TOKENmermaidinteractionshttps://mermaid-js.github.io/mermaid/diagrams-and-syntax-and-examples/gantt.html#interactionstatedocumentaionhttps://pdoc3.github.io/pdoc/https://pydoc-markdown.readthedocs.io/en/latest/docs/api-documentation/processors/ci toolshttps://gitlab.version.fz-juelich.de/vis/jusense-cicd/-/wikis/discussion-on-howto-include-badges-in-gitlab-...https://github.com/jongracecox/anybadgehttps://docs.gitlab.com/ee/api/wikis.htmlgitlab release versionspylint (show errors in MR)0.5support downstream pipelines0.4global command "pipeline-mermaid"custom message ability0.3support repeated jobs0.2support journey diagrammethods for using by one line0.1Show gitlab pipeline as gantt diagramShow gitlab pipeline as graph LR