Package: taskiq-pipelines
Pipelines for taskiqTaskiq pipelines is afire-and-forgetat its limit.Imagine you have a really tough functions and you want
to call them sequentially one after one, but you don't want to wait for them
to complete. taskiq-pipeline solves this for you.InstallationYou can install it from pypi:pip install taskiq-pipelinesAfter you installed it you need to add our super clever middleware
to your broker.This middleware actually decides what to do next, after current step
is completed.fromtaskiq_pipelines.middlewareimportPipelineMiddlewaremy_super_broker=...my_super_broker.add_middlewares([PipelineMiddleware(),])Also we have to admit that your broker MUST use result_backend that
can be read by all your workers. Pipelines work with inmemorybroker,
feel free to use it in local development.ExampleFor this example I'm going to use one single script file.importasynciofromtypingimportAny,Listfromtaskiq.brokers.inmemory_brokerimportInMemoryBrokerfromtaskiq_pipelinesimportPipelineMiddleware,Pipelinebroker=InMemoryBroker()broker.add_middlewares([PipelineMiddleware()])@broker.taskdefadd_one(value:int)->int:returnvalue+1@broker.taskdefrepeat(value:Any,reps:int)->List[Any]:return[value]*reps@broker.taskdefcheck(value:int)->bool:returnvalue>=0asyncdefmain():pipe=(Pipeline(broker,add_one,# First of all we call add_one function.)# 2.call_next(repeat,reps=4)#  Here we repeat our value 4 times# [2, 2, 2, 2].map(add_one)# Here we execute given function for each value.# [3, 3, 3, 3].filter(check)# Here we filter some values.# But sice our filter filters out all numbers less than zero,# our value won't change.# [3, 3, 3, 3])task=awaitpipe.kiq(1)result=awaittask.wait_result()print("Calculated value:",result.return_value)if__name__=="__main__":asyncio.run(main())If you run this example, it prints this:$pythonscript.py
Calculatedvalue:[3,3,3,3]Let's talk about this example.
Two notable things here:We must add PipelineMiddleware in the list of our middlewares.We can use only tasks as functions we wan to execute in pipeline.
If you want to execute ordinary python function - you must wrap it in task.Pipeline itself is just a convinient wrapper over list of steps.
Constructed pipeline has the same semantics as the ordinary task, and you can add steps
manually. But all steps of the pipeline must implementtaskiq_pipelines.abc.AbstractStepclass.Pipelines can be serialized to strings withdumpsmethod, and you can load them back withPipeline.loadsmethod. So you can share pipelines you want to execute as simple strings.Pipeline assigntask_idfor each task when you callkiq, and executes every step with pre-calculatedtask_id,
so you know all task ids after you call kiq method.How does it work?After you callkiqmethod of the pipeline it pre-calculates
all task_ids, serializes itself and adds serialized string to
the labels of the first task in the chain.All the magic happens in the middleware.
After task is executed and result is saved, you can easily deserialize pipeline
back and calculate pipeline's next move. And that's the trick.
You can get more information from the source code of each pipeline step.Available stepsWe have a few steps available for chaining calls:SequentialMapperFilterSequential stepsThis type of step is just an ordinary call of the function.
If you haven't specifiedparam_nameargument, then the result
of the previous step will be passed as the first argument of the function.
If you did specify theparam_nameargument, then the result of the previous
step can be found in key word arguments with the param name you specified.You can add sequential steps with.call_nextmethod of the pipeline.If you don't want to pass the result of the previous step to the next one,
you can use.call_aftermethod of the pipeline.Mapper stepThis step runs specified task for each item of the previous task's result spawning
multiple tasks.
But I have to admit, that the result of the previous task must be iterable.
Otherwise it will mark the pipeline as failed.After the execution you'll have mapped list.
You can add mappers by calling.mapmethod of the pipeline.Filter stepThis step runs specified task for each item of the previous task's result.
But I have to admit, that the result of the previous task must be iterable.
Otherwise it will mark the pipeline as failed.If called tasks returnedTruefor some element, this element will be added in the final list.After the execution you'll get a list with filtered results.
You can add filters by calling.filtermethod of the pipeline.