Package: async-pipeline
async_pipelinecomplex async data pipelines with functional methodsFeaturesasync_pipelineaims to simplify the creation of asynchronous pipelines.Main features are:Simplicity.
Allows defining not only linear pipelines,
but also complex pipelines with cross structures.Fast.
Each pipeline process is implemented through a coroutine process,
which could avoid blocking by async functions.Easy-to-read.
Linear previous and after stages can be connected using pipeline operator,
making it more readable.Easy-to-use.
Provides functional like methods.Quick Startinstallpythonsetup.pyinstallSimple linear pipelineIn this example, the data is multiplied by 2, and then filtered out those greater than 5.
The functions in each stage can be synchronous or asynchronous.importasynciofromasync_pipeline.pipeimportPipeasyncdefdouble(x):awaitasyncio.sleep(0.01)returnx*2pipe=Pipe()# create a pipestage1=pipe.data(range(5))# define data stagestage2=pipe.map(func=double,src=stage1)# define map stage, async funcstage3=pipe.filter(func=lambdax:x>5,src=stage2)# define filter stage, sync funcstage4=pipe.result(src=stage3)# define result stage, store result in stage4.resultpipe.run()# run pipeprint(stage4.result)# [6, 8]Use pipe operator to abbreviate:pipe=Pipe()stage=range(5)|pipe.map(func=double)|pipe.filter(func=lambdax:x>5)|pipe.result()pipe.run()Complex non-liner pipelineThis pipeline can be described in the following diagram:graph LR;
      inp((data))--'>0'-->n1((21));
      inp((data))--'<0'-->n2((_2));
      inp((data))--'==0'-->n3((drop));
      n1((21))-->n4((31));
      n1((21))-->n5((32));
      n4((31))-->n6((4));
      n5((32))-->n6((4));
      n2((22))-->n7((5));
      n6((4))-->n7((5));
      n7((5))-->n8((end));fromasync_pipeline.pipeimportPipepipe=Pipe()stage1=[-1,0,1]|pipe.partition({'p':{'match':"elem > 0",'handle':pipe.map(func=lambdax:str(x)+'_21')|pipe.multiply(pipe.map(func=lambdax,y:str(x)+y,func_kw={'y':'_31'}),pipe.map(func=lambdax:str(x)+'_32'))},'n':{'match':"elem < 0",'handle':pipe.map(func=lambdax:str(x)+'_22')}})stage2=pipe.concat(pipe.concat(stage1['p'][0],stage1['p'][1])|pipe.map(func=lambdax:str(x)+'_4'),stage1['n'])\|pipe.map(func=lambdax:str(x)+'_5')\|pipe.result()pipe.run()print(sorted(stage2.result))# ['-1_22_5', '1_21_31_4_5', '1_21_32_4_5']Supported Python Versions3.73.83.93.10Unittestpipinstall-rrequirements/test.txt
python-munittestAPI ReferenceclassPipeAll methods are called through pipe instance.pipe = Pipe()Create apipe.Different pipes do not affect each other.pipe.run()Execute stages inpipe, let the data start flowing.Before this, all Stage functions are just record as coroutines.
Each pipe can be executed only once.await pipe.async_run()The asynchronous version ofrun.pipe.cancel()Cancel the coroutines in the pipeline that have not yet executed.classStageEach section of the pipeline, i.e. each process, is called astage.stages are always associated with a pipeline,
so the stage instance is created through thepipeinterfacepipe.data(src)Create aData Stage. Place it at the beginning.srcis data source, needs to be iterable or async iterable.This method is optional because the data source can be placed directly,
except that the code check will say "Class 'xx' does not define '_or_'"stage=pipe.data(range(10))pipe.result(src=None, output=None)Create aResult Stage. Place it at the end.srcis previous stage.
When using the pipeline operator, the left side of|is the previous Stage, so src is not specified at this time.outputlocate the pipeline output, can be a list or any object that acceptsappendmethod. Default is self.result.# 1previous=pipe.data(range(10))end=pipe.result(src=previous)# 2end=previous|pipe.result()# 3result=[]end=previous|pipe.result(output=result)pipe.map(func, src=None, func_kw=None)Create aMap Stage, it executesresult = func(elem, **func_kw)for each element passed in, and outputs theresult.funcis function object, either sync or async.func_kwis parameters offuncother than element, likekwargsdict.srcis previous stage,Nonewhen using|.# 1stage=pipe.map(func=lambdax:x+1,src=previous)# 2stage=previous|pipe.map(func=some_func)# 3stage=previous|pipe.map(func=lambdax,y:x+y,func_kw={'y':1})pipe.filter(func, src=None, func_kw=None)Create aFilter Stage, it computes flag = func(elem, **func_kw) for each element passed in, and outputs theelementwith flag True.funcis function object, either a sync or async.func_kwis parameters offuncother than element, likekwargsdictionary.srcis previous stage,Nonewhen using|.# 1stage=pipe.filter(func=lambdax:x>0,src=previous)# 2stage=previous|pipe.filter(func=some_bool_func)# 3stage=previous|pipe.filter(func=lambdax,y:x>y,func_kw={'y':0})pipe.flatten(src=None)Create aFlatten Stage, it flattens each element passed in and output.srcis previous stage which outputs a two-level list, or iterable or async iterative two-level list.Nonewhen using|.# 1stage=pipe.flatten(src=[[1],[2,3]])# 2stage=previous|pipe.flatten()pipe.concat(*src)Create aConcat Stage, it merges the list of stages.*srcis multi previous stages, or iterable or async iterative lists.# 1stage=pipe.concat([1,2],[3,4])# 2stage=pipe.concat(data|pipe.map(func=lambdax:x),data|pipe.filter(func=lambdax:x>0),)pipe.partition(cases, src=None)Create aPartition Stage, it outputs elements to branch by condition.casescan be a key-value dictionary. The key is the allies, and the value is the matching logic with single paramelem.Elements are calculated by dict order, and output to the first match. Those do not match any branch will drop.Next stage'ssrcis partition[allies]stage1=range(-5,6)|pipe.partition(cases={'p':'elem > 0','n':'elem < 0'})stage21=pipe.map(src=stage1['p'],func=lambdax:x+1)stage22=pipe.map(src=stage1['n'],func=lambdax:x-2)end1=pipe.result(src=stage21)# end1.result [2, 3, 4, 5, 6]end2=pipe.result(src=stage22)# end2.result [-7, -6, -5, -4, -3]casescan also be a complex dictionary define both matching logic and next processing.handle is the subsequent processing, it'ssrcuses the result by match.stage=range(-5,6)|pipe.partition({'p':{'match':'elem > 0','handle':pipe.map(func=lambdax:x+1)},'n':{'match':'elem < 0','handle':pipe.map(func=lambdax:x-2)}})end1=pipe.result(src=stage['p'])# end1.result [2, 3, 4, 5, 6]end2=pipe.result(src=stage['n'])# end2.result [-7, -6, -5, -4, -3]srcis previous stage,Nonewhen using|.pipe.multiply(*targets, src=None)Create aMultiply Stage, it multiply copies of the elements and output totargetsbranches.targetsare the subsequent pipelines.srcis previous stage,Nonewhen using|.# 1previous=pipe.data(range(10))stage=pipe.multiply(pipe.map(func=lambdax:x),pipe.map(func=lambdax:2*x),src=previous)# 2stage=pipe.data(range(10))pipe.map(func=lambdax:x),pipe.map(func=lambdax:2*x),)pipe.each(func, src=None, func_kw=None)Create aEach Stage.Similar tomap, but elements are dropped after computation.pipe.peek(func, src=None, func_kw=None)Create aPeek Stage.Similar tomap, but still outputs the elements after computation.Elements may changed if mutable.pipe.filter_not(func, src=None, func_kw=None)Create aFilterNot Stage.Similar tofilter, but the logic is reversed.pipe.distinct(func=hash, src=None, func_kw=None)Create aDistinct Stage.Similar tofilter, outputs the elements after de-duplication.funccalculate whether an element is duplicated or not, default is hash function.Notice: python hash(-1) == hash(-2)pipe.duplicate(func=hash, src=None, func_kw=None)Create aDuplicate Stage.Similar tofilter, outputs the duplicate elements.funccalculate whether an element is duplicated or not, default is hash function.Notice: python hash(-1) == hash(-2)pipe.limit(num, src=None)Create aLimit Stage.Similar tofilter, but only outputs the firstnumof elements.# 1previous=pipe.data(range(10))stage=pipe.limit(num=5,src=previous)# 2stage=pipe.data(range(10))|pipe.limit(num=5)