Package: iter-pipes
iter_pipes: Iterable PipesFunctional pythonic pipelines for iterables.pipinstalliter-pipesExamplesmap / filter:importmathfromiter_pipesimportPipelineFactorypipeline=(PipelineFactory[int]().map(math.exp).filter(lambdax:x>math.exp(2)).map(math.log).map(str))assertpipeline(range(5)).to_list()==["3.0","4.0"]Batch operationsdefget_user_names_from_db(user_ids:list[int])->list[str]:# typical batch operation:#   - duration is roughly constant for a batch#   - batch size has to be below a fixed thresholdprint("processing batch",user_ids)return[f"user_{user_id}"foruser_idinuser_ids]pipeline=(PipelineFactory[int]().batch(get_user_names_from_db,batch_size=3).for_each(lambdauser_name:print("Hello ",user_name)))pipeline(range(5)).to_list()# returns#   ["user_0", "user_1", "user_2", "user_3", "user_4"]# prints#   processing batch [0, 1, 2]#   Hello  user_0#   Hello  user_1#   Hello  user_2#   processing batch [3, 4]#   Hello  user_3#   Hello  user_4Storing stateClass with a__call__method provide a easy way to store a state during the processing.classCountUsers:def__init__(self):self._count=0def__call__(self,item:str)->str:self._count+=1returnf"{item}(position{self._count})"pipeline=PipelineFactory[int]().map(lambdax:f"user{x}").map(CountUsers())pipeline.process(range(5)).to_list()# return#    ['user 0 (position 1)', 'user 1 (position 2)', 'user 2 (position 3)', 'user 3 (position 4)', 'user 4 (position 5)']One could also use a closure:defcount_users():count=0defwrapper(item:str)->str:nonlocalcountcount+=1returnf"{item}(position{count})"returnwrapperpipeline=PipelineFactory[int]().map(lambdax:f"user{x}").map(count_users())pipeline.process(range(5)).to_list()# return#    ['user 0 (position 1)', 'user 1 (position 2)', 'user 2 (position 3)', 'user 3 (position 4)', 'user 4 (position 5)']Branchespipeline=(PipelineFactory[int]().branch(lambdax:x.filter(lambdax:x%2==0).map(lambdax:x**2),lambdax:x.map(lambdax:-x),).map(str))expected=["0","0","4","-1","-2","16","-3","-4","36","-5","-6","-7"]assertpipeline(range(8)).to_list()==expectedEach "branch" order will be preserved, but there is not guarantee in term of how the two are merged.There is alsobranch_offwhich discard the output of the branch:pipeline=(PipelineFactory[int]().branch_off(lambdax:x.filter(lambdax:x%2==0).map(lambdax:x**2),).map(str))expected=["0","0","4","16","36"]assertpipeline(range(8)).to_list()==expectedPipe operator overloadimportiter_pipes.functionalasitppipeline=(PipelineFactory[int]()|itp.map(math.exp)|itp.filter(lambdax:x>math.exp(2))# type checker might complain|itp.map(math.log)|itp.map(str))assertpipeline(range(6)).to_list()==["3.0","4.0","5.0"]note that typing of lambda function inside functional map is not as good as the one from thePipeline.XXXmethods. To work around this, one should either use the non functional style, either use fully typed function instead of lambda.Resumabilitypipeline=PipelineFactory[int]().branch(lambdax:x.filter(lambdax:x%3==0).map(str),lambdax:x,)print(pipeline.process(range(12)).to_list())# return#    ['0', 0, '3', 1, 2, 3, '6', 4, 5, 6, '9', 7, 8, 9, 10, 11]# note that between each yield from the first branch, the pipeline will yield everything# from the second branch so that we don't store too many messages in the inflight buffer.deffilter_out_everything(items:Iterable[int])->Iterable[int]:print("starting")foriteminitems:ifFalse:yielditempipeline=PipelineFactory[int]().branch(lambdax:x.pipe(filter_out_everything).map(str),lambdax:x,max_inflight=5,)print(pipeline.process(range(9)).to_list())# return#    [0, 1, 2, 3, 4, 5, 6, 7, 8]# print#    starting#    starting#    startingMotivationsGoal of the library is to provide a structure to work withcollection pipelines.Collection pipelines are a programming pattern where you organize some computation as a sequence of operations which compose by taking a collection as output of one operation and feeding it into the next.In this library, each "operation" is called a "step". We differentiate different subtype for steps:mapsteps will operate on each item of the collection, one by onefiltersteps will reduce the number of item in the collection, without changing their valuesfor_eachsteps will do some processing, but without impacting the following steps (they won't change the input)batchsteps will operate by batch of a fixed size - can be useful for example to batch database calls.In addition to that, we also define pipelinebranch, which allow to run several steps after a single one.Library goal:declarative, expressive syntax for the steps abovememory efficiency:pure python, so it's not optimal at allbut what we care about is ensuring that the memory used by the pipeline does not scale with the number of items in the collection.performant:pure python, so the code itself is not really performantbut the library allow for an optimal usage of the "slow" operations (network calls mainly) that are computed in the pipeline. This is what is meant by "performant"lightweight usage, as in existing function can be used as a step without the need for a wrapperprovide as good of a type experience as possibleDocumentationHave a look at thedocspart of the test suites for examples.ContributingPlease refer to thetestactions. 100% test coverage is a start.