Saravanan666/pypi-org huggingface dataset:
Package: pipelite
The pipelite ProjectEmpower your data workflows effortlessly withpipelite, a lightweight Python program designed for seamlessdata pipeline creation and execution. Using a simple JSON configuration, users can build complex pipelines without writing code. What sets pipelite apart is its total extensibility—anyone can easily create and integrate new connectors or transformations, enhancing the program's capabilities.It's also possible to add new way to manage the flow of the transformations if needed. With a MIT license fostering collaboration, this flexible tool is perfect for users of all levels. Craft, execute, and extend your data pipelines withpipelite, your go-to solution for adaptable and scalable data processing.Some characteristics:Simple JSON configurationLightweight and code-free (MIT license for flexibility)Python Code (leverage the basics libraries instead addind many heavy and complex libs)Effortless pipeline creation and high integrability thanks to the json configurationStreamlined execution processTotal extensibility (connectivity, transformation, pipeline management)Boost data processing efficiencyQuick learning curveEmpower your data workflows in a simple waySo in one word ... pipelite is your extensible solution for dynamic data pipelines.Currently this solution provides data access and load from these data sources :✅External file (csv)✅External Excel Spreadsheet (xls, xlsx, xlsm, xlsb, odf, ods and odt) (read only)✅External XES File (read only)✅ODBC Data Sources (checked with SQL Server, SQLite) by using an configurable SQL query (Read Only)✅SAP Read Table via SAP RFC (Read Only)✅ABBYY Timeline PI (write only in Repository)And provides those transformers✅Pass Through (Ex. just to change the Data Sources names IN-OUT)✅Dataset Profiling✅Concat 2 Data sources✅Join data sources✅Lookup✅SubString✅Rename Column Name✅Column Transformation👉Jump to the wiki from hereInstallationjust use pip by typingpip install pipelite... and just use it !See hereExternal modulesjinja2BSD-3-Clause licensejsonschemaMIT License (MIT)jsonpath-ngApache Software License (Apache 2.0)xmltodictMIT License (MIT)requestsApache Software License (Apache 2.0)pyrfcApache Software License (Apache 2.0)pyodbcMIT License (MIT)openpyxlMIT License (MIT)pandasBSD 3-Clause License
Package: pipelitools
DescriptionPipelines for data analysis and machine learning.pipelitools/pipelitools
	|-- __init__.py  
	|-- preprocessing  
		|-- __init__.py  
		|-- eda.py  
		|-- feature_selection.py  
		|-- outliers.py  
	|-- models  
		|-- __init__.py  
		|-- models.py  
		|-- metrics.py  
		|-- clustering.py 
	|-- utils.pyInstallationpip install pipelitoolsUsageThis package was used inUniversity of Oxford MSc dissertation project.History0.1.0 (2021-06-11)First release.1.1.0 (2021-07-01)Added pipeline to compute models1.1.1 (2021-07-08)Added tests for every moduleRenaming1.1.2 (2021-07-08)Bug fixes1.1.3 (2021-07-13)README.mdfixes
Package: pipeln
A Python open source package to create a custom pipeline💾 Installation & UpgradeInstallation:pip install pipelnUpgrade:pip install pipeln --upgrade📦 What is this?Pipeln is a Python package aimed at creating a pipeline or a sequence of custom methods execution. The goal is to enable the creation of different procedural sequences that can be encapsulated within a method, allowing for the selection of order, return values, etc. Although some functionalities are still pending implementation, for a better overview, please refer to the objectives calendar.📃​Last version 0.2.11-alpha out now!📃​💬 Contribution & QuestionsContribution & Questions TypePlatforms🐞​​Bug ReportsGitHub Issues📦​Feature Requests & IdeasGitHub Discussions🛠️​Usage Questions & DiscusionsGitHub Discussions​🔧​ Install & Use PipelnTo start using Pipeln use the next command:pipinstallpipelnTo use the library properly, we follow these steps:Import the library.Initialize an object of the Pipeln class, providing the names and parameters of our methods, and setting the execution order.Create the execution structure.Execute the pipeline.fromPipeln.pipelineimportPipelinefromfeatures.auxiliar_test_methodsimportadd,sub,capobj=Pipeline(methods=[add,sub,cap],params=[{'a':2,'b':4},('1',2),('spain',)],orders=[3,1,2],debug=True)obj.create()obj.run()Expected output:DEBUG:*****************************
DEBUG:>Startingpipeline
DEBUG:
DEBUG:EXEC:Methodaddexecutedsuccessfully.
ERROR:ERROR:Thereisanerrorinsubmethod.Inappropiateargumenttype.
DEBUG:EXEC:Methodcapexecutedsuccessfully.
DEBUG:
DEBUG:Pipelinefinishedin10.9
DEBUG:*****************************💳 LicensePipeln is licensed underMIT License.🗃️ Shields
Package: pipeml
ContentsWhat is PipeML?InstallationDocumentationBug reportsLicense informationWhat is PipeML?PipeML constitutes a set of tools to work with machine learning models. PipeML does not implement machine learning models nor does it serve to build model. It is simply used to interface more easily with machine learning modelsInstallationGet the latest version of PipeML fromPyPI. If you have an installation of Python with pip, simple install it with:$ pip install pipemlTo get the git version, run:$ git clone git://github.com/yoelcortes/pipemlDocumentationTODOBug reportsTo report bugs, please use the PipeML’s Bug Tracker at:https://github.com/yoelcortes/pipemlLicense informationSeeLICENSE.txtfor information on the terms & conditions for usage
of this software, and a DISCLAIMER OF ALL WARRANTIES.Although not required by the PipeML license, if it is convenient for you,
please cite PipeML if used in your work. Please also consider contributing
any changes you make back, and benefit the community.
Package: pipen-annotate
pipen-annotateUse docstring to annotatepipenprocessesInstallationpipinstall-Upipen-annotateUsagefrompprintimportpprintfrompipenimportProcfrompipen_annotateimportannotateclassProcess(Proc):"""Short descriptionLong descriptionInput:infile: An input fileinvar: An input variableOutput:outfile: The output fileEnvs:ncores: Number of cores"""input="infile:file, invar"output="outfile:file:output.txt"args={'ncores':1}annotated=annotate(Process)# prints:{'Envs':{'ncores':{'attrs':OrderedDiot([('default',1)]),'help':'Number of cores','terms':OrderedDiot([])}},'Input':{'infile':{'attrs':{'action':'extend','itype':'file','nargs':'+'},'help':'An input file','terms':OrderedDiot([])},'invar':{'attrs':{'action':'extend','itype':'var','nargs':'+'},'help':'An input variable','terms':OrderedDiot([])}},'Output':{'outfile':{'attrs':{'default':'output.txt','otype':'file'},'help':'The output file','terms':OrderedDiot([])}},'Summary':{'long':'Long description\n','short':'Short description'}}
Package: pipen-args
pipen-argsCommand line argument parser forpipenUsagefrompipenimportProc,PipenclassProcess(Proc):"""My processInput:a: Input data"""input='a'input_data=range(10)script='echo {{in.a}}'Pipen().set_start(Process).run()$pythonexample.py--help
Usage:test.py[-h|-h+][options]Undescribedprocess.
Use`@configfile`toloaddefaultvaluesfortheoptions.

PipelineOptions:--nameNAMEThenameforthepipeline,willaffectthedefaultworkdirandoutdir.[default:pipen-0]--profilePROFILEThedefaultprofilefromtheconfigurationtorunthepipeline.Thisprofilewillbeusedunlessaprofileisspecifiedintheprocessorinthe.runmethodofpipen.Youcanchecktheavailableprofilesbyrunning`pipenprofile`--outdirOUTDIRTheoutputdirectoryofthepipeline[default:./<name>-output]--forksFORKSHowmanyjobstorunsimultaneouslybythescheduler--schedulerSCHEDULERTheschedulertorunthejobsNamespace<in>:--in.aA[A...]Inputdata

OptionalArguments:-h,--help,-h+,--help+showhelpmessage(with+toshowmoreoptions)andexitSee more examples intests/pipelines/folder.Plugin optionsargs_hide: (process level) Hide the arguments in the help message. Default:Falseargs_group: (pipeline level) The group name for the arguments. Default:pipeline optionsargs_flatten: (pipeline level) Flatten the arguments in the help message when there is only one process in the pipeline. Default:auto(flatten if single process, otherwise not)Metadata for Proc envs itemsThe metadata in the docstring of env items determines how the arguments are defined.classProcess(Proc):"""My process# other docstring sectionsEnvs:a (<metadata>): ..."""The metadata could be key-value pairs separated by;. The separator:or=is used to
separate the key and value. The value is optional. If the value is not specified, it
will be set toTrue. The keys are valid arguments ofargx.ArgumentParser.add_argument, except thathiddenwill be interpreted asshow=Falseinargx.ArgumentParser.add_argument. If the value ofchoicesis not specified, the subkeys of the env item will be used as the choices.
Package: pipen-cli-ref
pipen-cli-refMake reference documentation forpipenpipeline/processesInstallationpipinstallpipen-cli-refUsagepipenref--helpUsage: pipen ref [-h] -p PIPELINE -d DESTDIR [-s] [-r REPLACE_TITLES]
                 [-i HIDE_SECTIONS]

Make API reference documentation for pipen pipeline/processes.

Required Arguments:
  -p PIPELINE, --pipeline PIPELINE
                        Specify the pipeline to generate the docs for, in the
                        format of `part1:part2` where `part1` could be either:
                        1. a package name that can be imported by
                        `importlib.import_module`, or
                        2. a submodule of a package, for example,
                        `package.module`; or
                        3. a path to a python file, for example,
                        `path/to/file.py`.
                        and `part2` is the name of the pipeline class.
                        When the module is loaded, the pipeline class will be
                        searched in the module.
  -d DESTDIR, --destdir DESTDIR
                        The destination directory to save the docs.

Options:
  -h, --help            show help message and exit
  -s, --show-hidden-items
                        Do not include hidden items.
  -r REPLACE_TITLES, --replace-titles REPLACE_TITLES
                        Replace the title of a section, in the format of
                        `old=new`. Multiple replacements can be specified.
  -i HIDE_SECTIONS, --hide-sections HIDE_SECTIONS
                        Hiden some sections in docstring
Package: pipen-cli-require
pipen-cli-requireChecking the requirements for processes of apipenpipelineInstallpipinstall-Upipen-cli-requireUsageDefining requirements of a process# example_pipeline.pyfrompipenimportPipen,ProcclassP1(Proc):"""Process 1Requires:pipen: Run `pip install -U pipen` to install- check: |{{proc.lang}} -c "import pipen"liquidpy: Run `pip install -U liquidpy` to install- check: |{{proc.lang}} -c "import liquid"nonexist: Run `pip install -U nonexist` to install- check: |{{proc.lang}} -c "import nonexist"conditional:- if: {{envs.require_conditional}}- check:{{proc.lang}} -c "import optional""""input="a"output="outfile:file:out.txt"envs={"require_conditional":False}lang="python"# Setup the pipeline# Must be outside __main__# Or define a function to return the pipelineclassPipeline(Pipen):starts=P1if__name__=='__main__':# Pipeline must run with __main__Pipeline().run()Parsing process requirements using APIfrompipen_cli_requireimportparse_proc_requirementsdefparse_proc_requirements(proc:Type[Proc])->Tuple[OrderedDiot,OrderedDiot]:"""Parse the requirements of a processArgs:proc: The process classReturns:A tuple of two OrderedDiot's.The first one is the annotated sections by pipen_annotateThe second one is the requirements. The key is the name of therequirement, the value is a dict with message, check and if_ keys."""Checking the requirements via the CLI>pipenrequire--verbose--ncores2example_pipeline.py:pipeline

Checkingrequirementsforpipeline:PIPEN-0
│
└──P1:Process1├──✅pipen├──✅liquidpy├──❎nonexist:Run`pipinstall-Unonexist`toinstall│└──Traceback(mostrecentcalllast):│File"<string>",line1,in<module>│ModuleNotFoundError:Nomodulenamed'nonexist'│└──⏩conditional(skippedbyif-statement)Checking requirements with runtime argumentsFor example, when I use a different python to run the pipeline:Add this to the head ofexample_pipeline.py:importpipen_argsSee alsotests/pipen_args_pipeline.pyThen specify the path of the python to use:pipenrequireexample_pipeline.py:pipeline--P1.lang/path/to/another/python
Package: pipen-diagram
pipen-diagramDraw pipeline diagrams forpipen.FeaturesDiagram themingHiding processes from diagramConfigurationsdiagram_theme: The name of the theme to use, or a dict of a custom theme.Seepipen_diagram/diagram.pyfor the a theme definitionSeehttps://graphviz.org/for theme itemsdiagram_savedot: Whhether to save the dot file (for debugging purpose)diagram_hide: Process-level item, whether to hide current process from the diagramInstallationpipinstall-Upipen-diagramEnabling/Disabling the pluginThe plugin is registered via entrypoints. It's by default enabled. To disable it:plugins=[..., "no:diagram"], or uninstall this plugin.Usageexample.pyfrompipenimportProc,Pipen,ProcGroupclassA(Proc):"""Process A"""input="a"classB(Proc):"""Process B"""requires=Ainput="b"plugin_opts={"diagram_hide":True}classPG(ProcGroup):"""Process Group"""@ProcGroup.add_procdefc(self):"""Process C"""classC(Proc):input="c"returnC@ProcGroup.add_procdefc1(self):"""Process C1"""classC1(Proc):requires=self.cinput="c1"plugin_opts={"diagram_hide":True}returnC1@ProcGroup.add_procdefd(self):"""Process D"""classD(Proc):input="d"requires=self.c1returnDpg=PG()pg.c.requires=BclassE(Proc):"""Process E"""input="e1,e2"requires=pg.d,AclassF(Proc):"""Process F"""input="f"requires=EPipen("MyPipeline").set_start(A).run()# Dark theme# Pipen("MyPipeline", plugin_opts={"diagram_theme": "dark"}).set_start(A).run()Runningpython example.pywill generateMyPipeline-output/diagram.svg:Default themeDark themeFancy themeFancy dark theme
Package: pipen-filters
pipen-filtersAdd a set of useful filters forpipentemplates.These filters can be used for both liquid and jinja2 templating in pipen.Installationpipinstall-Upipen-filtersEnabling/Disabling the pluginThe plugin is registered via entrypoints. It's by default enabled. To disable it:plugins=[..., "no:filters"], or uninstall this plugin.UsagefrompipenimportProcclassMyProc(Proc):input="infile:file"output="outfile:file:{{in.infile | stem}}.txt"...FiltersParse the symbolic linksrealpath:os.path.realpathreadlink:os.readlinkFind common prefix of given pathscommonprefix:>>>commonprefix("/a/b/abc.txt","/a/b/abc.png")>>># "abc.">>>commonprefix("/a/b/abc.txt","/a/b/abc.png",basename_only=False)>>># "/a/b/abc."Get parts of the pathdirname:path.dirnamebasename:path.basenameext: get the extension (/a/b/c.txt -> .txt)ext0: get the extension without dot (/a/b/c.txt -> txt)prefix: get the prefix of a path (/a/b/c.d.txt -> /a/b/c.d)prefix0: get the prefix of a path without dot in basename (/a/b/c.d.txt -> /a/b/c)filename,fn,stem: get the stem of a path (/a/b.c.txt -> b.c)filename0,fn0,stem0: get the stem of a path without dot (/a/b.c.txt -> b)joinpaths: join path parts (os.path.join)as_path: convert a string into apathlib.PathobjectPath statisdir:os.path.isdirisfile:os.path.isfileislink:os.path.islinkexists:os.path.existsgetsize:os.path.getsize, return -1 if the path doesn't existgetmtime:os.path.getmtime, return -1 if the path doesn't existgetctime:os.path.getctime, return -1 if the path doesn't existgetatime:os.path.getatime, return -1 if the path doesn't existisempty: check if a file is emptyQuote dataquote: put double quotes around data (1 -> "1")squote: put single quotes around data (1 -> '1')Configurationsjson:json.dumpsjson_dump: Load json froma  filejson_dumps: Alias ofjsonjson_loads:json.loadstoml:toml.dumpstoml_dump: Load toml from a filetoml_dumps: Alias oftomltoml_loads:toml.loadsconfig: Load configuration from an object, a string or a fileGlobsglob: Likeglob.glob, but allows passing multiple parts of a pathglob0: Likeglob, but only returns the first matched pathRead file contentsread: Read file content. You can also pass arguments toopenreadlines: Read file content as a list of lines. Additional arguments will be passed toopenOtherregex_replace: Replace a string using regex
Package: pipen-log2file
pipen-log2fileSave running logs to file forpipen.The log file is saved to<workdir>/<pipeline>/.logs/run-<date-time>.logby default.
A symlink<workdir>/<pipeline>/run-latest.logis created to the latest log file.The xqute logs are also saved to<workdir>/<pipeline>/<proc>/proc.xqute.logNote that the original handler of xqute logger is removed during pipeline running.Optionsplugin_opts.log2file_xqute: Whether to save xqute logs. Default:True.
if False, the xqute logger will be kept intact.plugin_opts.log2file_xqute_level: The log level for xqute logger. Default:INFO.plugin_opts.log2file_xqute_append: Whether to append to the log file. Default:False.Installationpip install -U pipen-log2fileEnabling/Disabling the pluginThe plugin is registered via entrypoints. It's by default enabled. To disable it:plugins=[..., "no:log2file"], or uninstall this plugin.
Package: pipenode
pipenodepipenode supported mode:threadprocesscoroutineexampleimport pipenode
import time
def work(**kwargs):
    print('my work', kwargs.get('name'))
    time.sleep(1)

server = pipenode.Server(run_type='thread|process|coroutine', max_workers=10)
_ = [server.add_worker(work, name='luo') for _ in range(10)]
server.run_executor()ExtensionsExtensions folder include modulers or other tools someone sharing. You can push your code without bugs.|logo||Build Status||Demo|DocumentationHEADGitHub.Roadmap..
Package: pipe-nonblock
pipe-nonblockNon-blocking multiprocessing pipeInstallingpipinstallpipe_nonblockSupported versionsPython 3.4+Usagefrompipe_nonblockimportPipec1,c2=Pipe(duplex=True,conn1_nonblock=True,conn2_nonblock=True)# create a new duplex non-blocking pipeassertlen(list(c2.recv(32)))==0# try receive data from a connection, it's empty at the momentc1.send('foo')# send python string objectbuf,=c2.recv(32)# recv return generator with received objectsassertbuf=='foo'c2.send('bar')buf,=c1.recv(32)assertbuf=='bar'Overview of Pipe functiondefPipe(duplex,conn1_nonblock,conn1_nonblock)->Tuple[connection1,connection2]:parametertypedescriptionduplexboolIf duplex is True (the default) then the pipe is bidirectional. If duplex is False then the pipe is unidirectional: conn1 can only be used for receiving messages and conn2 can only be used for sending messages.conn1_nonblockboolIf conn1_nonblock is True then the connection1 is  non-blockingconn2_nonblockboolIf conn2_nonblock is True then the connection2 is  non-blocking
Package: pipen-poplog
pipen-poplogPopulate logs from jobs to running log of the pipeline forpipen.Installationpipinstall-Upipen-poplogEnabling/Disabling the pluginThe plugin is registered via entrypoints. It's by default enabled. To disable it:plugins=[..., "no:log2file"], or uninstall this plugin.UsagefrompipenimportProc,PipenclassPoplog(Proc):input="var:var"input_data=[0,1,2]script="""echo -n "[PIPEN-POPLOG][INFO] Log message "sleep 1  # Simulate message not read in timeecho "by {{in.var}} 1"sleep 1echo "[PIPEN-POPLOG][ERROR] Log message by {{in.var}} 2"sleep 1echo "[PIPEN-POPLOG][INFO] Log message by {{in.var}} 3""""if__name__=="__main__":Pipen().run()01-12 11:23:52 I core    ╭═══════════════ PoplogDefault ═════════════════╮
01-12 11:23:52 I core    ║ A default poplog proc                         ║
01-12 11:23:52 I core    ╰═══════════════════════════════════════════════╯
01-12 11:23:52 I core    PoplogDefault: Workdir: '.pipen/Pipeline/PoplogDefault'
01-12 11:23:52 I core    PoplogDefault: <<< [START]
01-12 11:23:52 I core    PoplogDefault: >>> [END]
01-12 11:23:56 I poplog  PoplogDefault: [0/2] Log message by 0 1
01-12 11:23:59 E poplog  PoplogDefault: [0/2] Log message by 0 2
01-12 11:24:02 I poplog  PoplogDefault: [0/2] Log message by 0 3Configurationplugin_opts.poplog_loglevel: The log level for poplog. Default:info.plugin_opts.poplog_pattern: The pattern to match the log message. Default:r'\[PIPEN-POPLOG\]\[(?P<level>\w+)\] (?P<message>.*)'.plugin_opts.poplog_jobs: The job indices to be populated. Default:[0](the first job).plugin_opts.poplog_max: The total max number of the log message to be poplutated. Default:99.plugin_opts.poplog_source: The source of the log message. Default:stdout.
Package: pipen-report
Report generation system forpipen.Installation❯pipinstall-Upipen-report# update configurations if needed:❯pipenreportconfig# then update frontend dependencies:❯pipenreportupdateEnabling/Disabling the pluginThe plugin is registered via entrypoints. It's by default enabled. To disable it:plugins=[..., "no:report"], or uninstall this plugin.ExampleSee the pipeline inexample/, and the reports generated by the pipeline:https://pwwang.github.io/pipen-report/example/output/REPORTSDocumentationSee:https://pwwang.github.io/pipen-report
Package: pipen-runinfo
pipen-runinfoGenerate running information for jobs inpipenpipelines.Including session info (packages and versions), time, and device info.Installpipinstall-Upipen-runinfoEnable/Disable the pluginThe plugin is registered via entrypoints. It's by default enabled. To disable it: plugins=[..., "no:runinfo"], or uninstall this plugin.Plugin optionsruninfo_lang: The name of the language to run the job script for session info.
Default isNone, which means it will be inferred from theproc.langThis should be a process-level option, unless you only have one single
process in your pipeline.runinfo_path: Whether to include paths for the modules in the session information.
Default isTrue.
This option could be either specified in the process-level or the pipeline-level.
Only works forpython.runinfo_submod: Whether to include submodules in the session information.
Default isFalse.
This option could be either specified in the process-level or the pipeline-level.
Only works forpython.Supported languages for session infopython,R,bash, andfish.UsageThe plugin will generate 3 files in the job directory of the pipeline.job.runinfo.sessionThe session information of the job, including the interpreter, packages and their versions.PythonGenerates a TSV file with the following columns:Name: The name of the module, or python itself__version__: The version fetched bymodule.__version__ormodule.versionimportlib.metadata: The version fetched byimportlib.metadata.version(package)Path: The path of the module (only ifruninfo_pathisTrue)RGenerates a text filesessionInfo()output.BashGenerates a TSV file with the following columns:SHELL: The value of$SHELLBASH_VERSION: The value of$BASH_VERSIONBASH_ARGV0: The value of$BASH_ARGV0BASH_SOURCE: The value of$BASH_SOURCEproc-exe: The real path of the executable from/proc/<pid>/exeproc-exe-version: The version of the executable from/proc/<pid>/exe --versionFishGenerates a TSV file with the following columns:SHELL: The value of$SHELLFISH_VERSION: The value of$FISH_VERSIONproc-exe: The real path of the executable from/proc/<pid>/exeproc-exe-version: The version of the executable from/proc/<pid>/exe --versionjob.runinfo.timeThe time spent on the job, and more, generated bytime -vcommand.job.runinfo.deviceThe device (cpu and memory) information of the job, generated bylscpu/lsmemcommand.
Package: pip-ensure-version
Usage:importpip_ensure_version# pip_ensure_version.set_debug()pip_ensure_version.require_package("tqdm",# package name"4.47",# pin version (optional))pip_ensure_version.require_gitpackage("soraxas-toolbox",# package name"soraxas/python-soraxas_toolbox",# repo name (default to github)"aa00d757b58c43523d7c518cdac600d4a75c6015",# commit id (optional))pip_ensure_version.require_gitpackage("easy-visualiser",">0.1.2",)# at this point, the requested package will be up-to-dateimporttqdmimportsoraxas_toolboximporteasy_visualiser...
Package: pipenv2setup
DescriptionThis module can be used to generate ainstall_requires.pyfile from aPipfile. The array in this file can then be imported into yoursetup.pyfile and used as theinstall_requiresarguments in thesetupfunction.Install>>> pipenv install --dev pipenv2setupRun>>> pipenv run python -m pipenv2setupUseCreate asetup.pyfile that follows this format:from setuptools import setup
from install_requires import install_requires


setup(
   name='package_name',
   version='0.1.0',
   description='Library to do my cool thing',
   author='Victoria Snellgrove',
   packages=['package_name'],
   install_requires=install_requires
)