Package: autopipe
AutopipeA tool that allow one to create pipeline of automatic data processing.How to useTo create a pipeline, you must create a Coordinator. To do so, create a file and a class implementing the base Coordinator class.
For example, here is the code for a simple coordinator that download images corresponding to a querry from google image.fromtypingimportList,Union,CallablefromautopipeimportCoordinator,Pipe,APData,Outputfromautopipe.inputimportRssInputfromautopipe.pipeimportFileData,DownloaderPipeclassDownloadExample(Coordinator):def__init__(self,query:str="raccoon"):super().__init__()self.query=query@classmethoddefname(cls):return"DownloadExample"@propertydefinput(self):returnRssInput(f"http://www.obsrv.com/General/ImageFeed.aspx?{self.query}",lambdax:FileData(None,x["media_content"][0]["url"],False))@propertydefpipeline(self)->List[Union[Pipe,Callable[[APData],Union[APData,Pipe]]]]:return[Output(DownloaderPipe())]For this coordinator to be found by autopipe, you must use one of the three following wayPlace your coordinator file into theautopipe/coordinatorsfolders, import your coordinator in theautopipe/coordinators/__init__.pyfile and place your coordinator name in the__all__array of this file.Run autopipe with the coordinator argument set to the path of your file followed by ':' and the coordinator's class name. For example if your coordinator's file is namedcoordinator.py, is located in the current directory and your coordinator's name isDownloadExample, your coordinator argument would be../coordinator.py:DownloadExampleSend your coordinator file to the standard input of autopipe and use-as your coordinator name.SOONTo run this, you have three waysUse the autopipe file in the bin folder like so:./autopipe <coordinator> [coordinator_parameters]Use the module syntaxe like so:python -m autopipe <coordinator> [coordinator_parameters]Use the shebang#!/usr/bin/env autopipe -, set your coordinator file executable (chmod +x file.py) and execute it.SOONCoordinators optionsA pipeline always start with anInput. You specify the instance of thte input manager you will use in theget_input()method.An input will return one or multiples data that will be send to your pipeline one by one.The pipelineEach item is send to thePipeyou specify in thepipelineproperty of your coordinator. In this property, you can place instances of pipes or functions that take a singleAPDataas parameter and return anAPDataor aPipe.
A pipeline is finished for an item when anOutputpipe is reached. That can be by using one of the premade output or by wrapping aPipeor anAPDatawith anOutput()call.InterceptorsYou can add interceptors to your coordinator. An interceptor is a function that will be called between steps of your pipeline if the specified condition matches. This allow you to handle invalid cases of your data or specific cases that don't need a specific step in the pipeline. You can specify an interceptor using the@autopipe.interceptor(lambda data: condition)decorator.The default handlerAdefault_handlermethod can be specified in your coordinator. This special method will be called once the whole pipeline has been consumed but no Output has been returned. You can also use this method instead of thepipelineproperty by removing the property from your coordinator (or returning an empty list).Usageusage: autopipe [-h] [-V] [-v [lvl]] [-d] [-w dir] coordinator [coordinator ...]

Easily run advanced pipelines in a daemon or in one run sessions.

positional arguments:
  coordinator          The name of your pipeline coordinator.

optional arguments:
  -h, --help           show this help message and exit
  -V, --version        show program's version number and exit
  -v, --verbose [lvl]  Set the logging level. (default: warn ; available: trace, debug, info, warning, error)
  -d, --daemon         Enable the daemon mode (rerun input generators after a sleep cooldown)
  -w, --workdir dir    Change the workdir, default is the pwd.InstalationTo install autopipe, runsudo pip install autopipe. To use a developement version, you can clone this project and runpip install -e ..