Package: atasker
ataskerPython library for modern thread / multiprocessing pooling and task processing
via asyncio.Warning:ataskeris not suitable for the lightweight tasks in high-load
environments. For such projects it's highly recommended to use lightweight
version:neotaskerNo matter how your code is written, atasker automatically detects blocking
functions and coroutines and launches them in a proper way, in a thread,
asynchronous loop or in multiprocessing pool.Tasks are grouped into pools. If there's no space in pool, task is being placed
into waiting queue according to their priority. Pool also has "reserve" for the
tasks with priorities "normal" and higher. Tasks with "critical" priority are
always executed instantly.This library is useful if you have a project with many similar tasks which
produce approximately equal CPU/memory load, e.g. API responses, scheduled
resource state updates etc.Installpip3installataskerSources:https://github.com/alttch/ataskerDocumentation:https://atasker.readthedocs.io/Whyasynchronous programming is a perfect way to make your code fast and reliablemultithreading programming is a perfect way to run blocking code in the
backgroundataskercombines advantages of both ways: atasker tasks run in separate
threads however task supervisor and workers are completely asynchronous. But
all their public methods are thread-safe.Why not standard Python thread pool?threads in a standard pool don't have prioritiesworkersWhy not standard asyncio loops?compatibility with blocking functionsasync workersWhy not concurrent.futures?concurrent.futuresis a great standard Python library which allows you to
execute specified tasks in a pool of workers.For thread-based tasks,ataskerextendsconcurrent.futures.ThreadPoolExecutorfunctionality.ataskermethodbackground_tasksolves the same problem but in slightly
different way, adding priorities to the tasks, whileataskerworkers do
absolutely different job:inconcurrent.futuresworker is a pool member which executes the single
specified task.inataskerworker is an object, which continuouslygeneratesnew tasks
with the specified interval or on external event, and executes them in thread
or multiprocessing pool.Code examplesStart/stopfromataskerimporttask_supervisor# set pool sizetask_supervisor.set_thread_pool(pool_size=20,reserve_normal=5,reserve_high=5)task_supervisor.start()# ...# start workers, other threads etc.# ...# optionally block current threadtask_supervisor.block()# stop from any threadtask_supervisor.stop()Background taskfromataskerimportbackground_task,TASK_LOW,TASK_HIGH,wait_completed# with annotation@background_taskdefmytask():print('I am working in the background!')return777task=mytask()# optionalresult=wait_completed(task)print(task.result)# 777print(result)# 777# with manual decorationdefmytask2():print('I am working in the background too!')task=background_task(mytask2,priority=TASK_HIGH)()Async tasks# new asyncio loop is automatically created in own threada1=task_supervisor.create_aloop('myaloop',default=True)asyncdefcalc(a):print(a)awaitasyncio.sleep(1)print(a*2)returna*3# call from sync code# put coroutinetask=background_task(calc)(1)wait_completed(task)# run coroutine and wait for resultresult=a1.run(calc(1))Worker examplesfromataskerimportbackground_worker,TASK_HIGH@background_workerdefworker1(**kwargs):print('I am a simple background worker')@background_workerasyncdefworker_async(**kwargs):print('I am async background worker')@background_worker(interval=1)defworker2(**kwargs):print('I run every second!')@background_worker(queue=True)defworker3(task,**kwargs):print('I run when there is a task in my queue')@background_worker(event=True,priority=TASK_HIGH)defworker4(**kwargs):print('I run when triggered with high priority')worker1.start()worker_async.start()worker2.start()worker3.start()worker4.start()worker3.put_threadsafe('todo1')worker4.trigger_threadsafe()fromataskerimportBackgroundIntervalWorkerclassMyWorker(BackgroundIntervalWorker):defrun(self,**kwargs):print('I am custom worker class')worker5=MyWorker(interval=0.1,name='worker5')worker5.start()