Package: aqute
AquteAsyncQUeueTaskEngineAqute is a minimalist yet potent Python library specifically designed
for hassle-free asynchronous task processing. Leveraging the power of async programming,
Aqute offers:Efficient Producer-Consumer Model: Utilizing the Producer-Consumer pattern with
multiple workers, Aqute ensures streamlined task distribution and swift concurrent processing.Worker count & Rate Limiting: Regulate the execution rate of tasks and configure
the number of workers for concurrent processing, ensuring optimal resource utilization.
You can use pre-programmed rate limiters or provide your own rate limiting implementation.Resilient Retry Mechanism: Tasks that encounter errors can automatically retry,
with options to specify which error types should trigger retries.
Exception in handler is returned as error-value.Versatile task adding: You can process the whole batch or add tasks on the fly,
depending on your needs.Lightweight and simple: Aqute operates efficiently without relying on any
external dependencies, ensuring seamless integration and minimal footprint in your projects.Aqute simplifies task management in asynchronous landscapes, allowing developers
to focus on the task logic rather than concurrency challenges.Table of ContentsInstallQuickstartHow to use it?Simple batch processingResult as async generator per completedRate limitingManual task adding, context manager and error retryManual flow management and custom result queueEven more manual management and internal worker queue sizeUse priroty queueTask timeout settingBarebone queue via ForemanSome caveatsStart load timeoutYou can't wait on not started AquteMiscInstance reuse after stop()Type checking and genericsInstallPython 3.9+ required:pipinstallaquteQuickstartApply your async function to each item of some iterable and get list of wrapped inAquteTaskresults, ordered the same way:importasynciofromaquteimportAquteasyncdefmain():asyncdefhandler(i:int)->int:awaitasyncio.sleep(i/20)returni*2aqute=Aqute(handle_coro=handler,workers_count=2)result=awaitaqute.apply_to_all(range(10))# Do not forget to extract result data from wrapper object with <result> propertyassert[t.resultfortinresult]==[i*2foriinrange(10)]asyncio.run(main())How to use it?While a deep dive is available through Aqute's method docstrings, it's not necessary.Aqute is easy to use for both simple and advanced workflows.Simple batch processingThe easiest way to use Aqute isapply_to_all()method:importasyncioimportloggingfromrandomimportrandint,randomfromaquteimportAqute,AquteErrorlogging.basicConfig(level=logging.INFO,format='%(asctime)s-%(levelname)s-%(name)s-%(message)s')logger=logging.getLogger("main")asyncdefhandler(i:int)->str:"""NOTE: This is a mock handler for demonstration purposes.Replace the logic below with your own specific task processing logic."""awaitasyncio.sleep(0.01+(0.09)*random())# Guaranties failures for some tasks of examplesifi>=19:raiseKeyError(f"The Key for{i}")# Here we have some randomness, so you can see retry after errors in playr=randint(1,101)ifr>=80:raiseValueError(f"Got big number for{i}")returnf"success{i}"asyncdefmain():# Getting started example, the most simpleinput_data=list(range(20))# This will apply handler to every item of iterable and return result as list# with task results ordered as input iterableaqute=Aqute(handle_coro=handler,workers_count=10,retry_count=2)result=awaitaqute.apply_to_all(input_data)# Each task result is wrapped in AquteTask instanceassert[t.datafortinresult]==input_data...asyncio.run(main())Result as async generator per completed# Like previous but the result is async generator and the tasks are yielded# in completition orderinput_data=list(range(20))aqute=Aqute(handle_coro=handler,workers_count=10)done,with_errors=[],[]# You can determine final task status with specific success fieldasyncfortaskinaqute.apply_to_each(input_data):iftask.success:done.append(task)else:with_errors.append(task)assertlen(done+with_errors)==len(input_data)Rate limitingYou can also add RateLimiter instance to Aqute for rate limiting:fromaqute.ratelimiterimportTokenBucketRateLimiter# Applied rate limiter with 5 handler calls per second.input_data=list(range(20))r_limit=TokenBucketRateLimiter(5,1)aqute=Aqute(handle_coro=handler,workers_count=10,rate_limiter=r_limit)result=[]asyncfortaskinaqute.apply_to_each(input_data):result.append(task)assertlen(result)==len(input_data)There are three avaliableRateLimiterimplementations:TokenBucketRateLimiter: steady rate by default, burstable withallow_burstoption;SlidingRateLimiter: next call will be avaliable after enough time from the oldest one;PerWorkerRateLimiter: enforces separate rate limits for each unique worker with separateTokenBucketRateLimiterinstances;RandomizedIntervalRateLimiter: introduces more random intervals between each call,
but enforcingmax_rateovertime_periodlimit.You can write your ownRateLimiterimplementation with specific algorithm if needed.Manual task adding, context manager and error retryThis can be most useful if not all of your tasks are avaliable at the start:# You can add tasks manually and also start/stop aqute with context# manager. And even add tasks on the fly.# Aqute is reliable for errors retry, you can specify your own# retry count (0 is used for no retries by default) and specify errors to retry or not# to keep retrying on all errorsaqute=Aqute(handle_coro=handler,workers_count=10,# We we will retry 5 more times after first failretry_count=5,# We retry only ValueError herespecific_errors_to_retry=(ValueError,),# Either you can set this option to not retry only on ValueError insteaderrors_to_not_retry=(ValueError,),)foriinrange(10):# You also can use your own task id for identificationawaitaqute.add_task(i,task_id=f"My task id:{i}")asyncwithaqute:awaitasyncio.sleep(0.1)foriinrange(10,15):awaitaqute.add_task(i,task_id=f"My task id:{i}")awaitasyncio.sleep(0.1)foriinrange(15,20):awaitaqute.add_task(i,task_id=f"My task id:{i}")# Set waiting for finalization when you have all tasks addedawaitaqute.wait_till_end()# You can simply extract all results from queue with this method if aqute has# finished, returns the list of AquteTaskfortrinaqute.extract_all_results():logger.info(f"{tr.success,tr.error,tr.result}")Manual flow management and custom result queue# You can manage the whole workflow manually if needed and use your own# result queue instance (with limited size for example)result_q=asyncio.Queue(5)aqute=Aqute(handle_coro=handler,workers_count=10,result_queue=result_q)foriinrange(10):# We can't await here cause we will hang without queue emptyingasyncio.create_task(aqute.add_task(i))awaitasyncio.sleep(0.1)# Starting the processingaqute.start()# Sleep enough for possibly all task to finishawaitasyncio.sleep(1)# We can see our result sizing worksassertresult_q.qsize()==5for_inrange(5):awaitresult_q.get()# Now wait till all finished via speicific method, this also notifies# aqute that we have added all tasksawaitaqute.wait_till_end()assertresult_q.qsize()==5# Stop the aquteawaitaqute.stop()Even more manual management and internal worker queue size# You can configure internal queue size for consumers if you want it to be limitedaqute=Aqute(handle_coro=handler,workers_count=10,input_task_queue_size=2)foriinrange(10):awaitaqute.add_task(i)# Should set it before awaiting bare start() if we wantaqute.set_all_tasks_added()aqute_run_aiotask=aqute.start()awaitaqute_run_aiotaskawaitaqute.stop()assertaqute.result_queue.qsize()==10Use priroty queueYou can prioretize tasks by settinguse_priority_queueflag:asyncdefhandler(i:int)->int:returni# Set flag for prioritezed queue, default task priority is 1_000_000aqute=Aqute(workers_count=1,handle_coro=handler,use_priority_queue=True,)awaitaqute.add_task(1_000_000)awaitaqute.add_task(10,task_priority=10)awaitaqute.add_task(5,task_priority=5)awaitaqute.add_task(10,task_priority=10)awaitaqute.add_task(1,task_priority=1)asyncwithaqute:awaitaqute.wait_till_end()results=aqute.extract_all_results()assert[t.datafortinresults]==[1,5,10,10,1_000_000]```Task timeout settingThetask_timeout_secondsoption is used to specify a time limit for each task.
If a task exceeds this duration, it is considered a timeout and is
handled according to the specified retry logic.Timeout with retriesBy default setting timeout will result in task timeout if task exceeds value, but
retry logic is applied ifretry_count > 0.aqute=Aqute(handle_coro=your_handler_coroutine,task_timeout_seconds=5,retry_count=2,# other parameters)Do not retry on timeoutTo disable this behavior you can seterrors_to_not_retrywithAquteTaskTimeoutError:aqute=Aqute(handle_coro=your_handler_coroutine,task_timeout_seconds=5,retry_count=2,errors_to_not_retry=AquteTaskTimeoutError,# other parameters)Barebone queue via ForemanIf you don't need retry flow and high-level helpers you can useForemanfor bare flow,
but still with rate limiting support:importasynciofromrandomimportrandomfromaqute.workerimportForemanfromaqute.ratelimiterimportTokenBucketRateLimiterasyncdefhandler(i:int)->str:awaitasyncio.sleep(0.01+(0.09)*random())returnf"Success{i}"asyncdefmain():# These are the supported options for Foremanforeman=Foreman(handle_coro=handler,workers_count=10,rate_limiter=TokenBucketRateLimiter(5,1),input_task_queue_size=100,)foriinrange(20):awaitforeman.add_task(AquteTask(i,f"{i}"))foreman.start()result=[]for_inrange(20):# Be aware that status and retries are not relevant here# But you can check the error field of outputr=awaitforeman.get_handeled_task()assertr.errorisNonelogger.info(r.result)result.append(r)# Do not finalize before result extractionawaitforeman.finalize()Some caveatsStart load timeoutIf no tasks will be provided, and you've set the timeout, Aqute will intentionally fail:try:asyncwithAqute(handle_coro=handler,workers_count=10,start_timeout_seconds=1,)asaqute:awaitasyncio.sleep(1.2)exceptAquteErrorasexc:logger.error(f"Aqute timeouted:{exc}")You can't wait on not started Aqute#aqute=Aqute(handle_coro=handler,workers_count=10)try:awaitaqute.wait_till_end()exceptAquteErrorasexc:logger.error(f"Aqute cannot be waited here:{exc}")MiscInstance reuse afterstop()# You can reuse same aqute instance after proper stop() callaqute=Aqute(handle_coro=handler,workers_count=5)asyncwithaqute:foriinrange(10):awaitaqute.add_task(i)awaitaqute.wait_till_end()asyncwithaqute:foriinrange(10,20):awaitaqute.add_task(i)awaitaqute.wait_till_end()assertaqute.result_queue.qsize()==20Type checking and genericsYou should get error during type check if you would try to use wrong type withAqutemethods (types are indered based on your provided handler):fromaquteimportAquteasyncdefhandler(i:int)->str:returnf"success{i}"asyncdefmain()->None:aqute=Aqute(handle_coro=handler,workers_count=10)# Mypy error: error: Argument 1 to "add_task" of "Aqute" has incompatible type "str"; expected "int"  [arg-type]awaitaqute.add_task("10")You can also provide the expected types of in/out via generics mechanism:fromaquteimportAquteasyncdefhandler(i:int)->str:returnf"success{i}"asyncdefmain()->None:# Mypy error: Argument "handle_coro" to "Aqute" has incompatible type "Callable[[int], Coroutine[Any, Any, str]]"; expected "Callable[[int], Coroutine[Any, Any, int]]"  [arg-type]aqute=Aqute[int,int](handle_coro=handler,workers_count=10)awaitaqute.add_task(123)