Package: aioprocessing
aioprocessingaioprocessingprovides asynchronous,asynciocompatible, coroutine
versions of many blocking instance methods on objects in themultiprocessinglibrary. To usedillfor universal pickling, install usingpip install aioprocessing[dill]. Here's an example demonstrating theaioprocessingversions ofEvent,Queue, andLock:importtimeimportasyncioimportaioprocessingdeffunc(queue,event,lock,items):""" Demo worker function.This worker function runs in its own process, and usesnormal blocking calls to aioprocessing objects, exactlythe way you would use oridinary multiprocessing objects."""withlock:event.set()foriteminitems:time.sleep(3)queue.put(item+5)queue.close()asyncdefexample(queue,event,lock):l=[1,2,3,4,5]p=aioprocessing.AioProcess(target=func,args=(queue,event,lock,l))p.start()whileTrue:result=awaitqueue.coro_get()ifresultisNone:breakprint("Got result{}".format(result))awaitp.coro_join()asyncdefexample2(queue,event,lock):awaitevent.coro_wait()asyncwithlock:awaitqueue.coro_put(78)awaitqueue.coro_put(None)# Shut down the workerif__name__=="__main__":loop=asyncio.get_event_loop()queue=aioprocessing.AioQueue()lock=aioprocessing.AioLock()event=aioprocessing.AioEvent()tasks=[asyncio.ensure_future(example(queue,event,lock)),asyncio.ensure_future(example2(queue,event,lock)),]loop.run_until_complete(asyncio.wait(tasks))loop.close()The aioprocessing objects can be used just like their multiprocessing
equivalents - as they are infuncabove - but they can also be
seamlessly used inside ofasynciocoroutines, without ever blocking
the event loop.What's newv2.0.1Fixed a bug that kept theAioBarrierandAioEventproxies returned fromAioManagerinstances from working. Thanks to Giorgos Apostolopoulos for the fix.v2.0.0Add support for universal pickling usingdill, installable withpip install aioprocessing[dill]. The library will now attempt to importmultiprocess, falling back to stdlibmultiprocessing. Force stdlib behaviour by setting a non-empty environment variableAIOPROCESSING_DILL_DISABLED=1. This can be used to avoiderrorswhen attempting to combineaioprocessing[dill]with stdlibmultiprocessingbased objects likeconcurrent.futures.ProcessPoolExecutor.How does it work?In most cases, this library makes blocking calls tomultiprocessingmethods
asynchronous by executing the call in aThreadPoolExecutor, usingasyncio.run_in_executor().
It doesnotre-implement multiprocessing using asynchronous I/O. This means
there is extra overhead added when you useaioprocessingobjects instead ofmultiprocessingobjects, because each one is generally introducing aThreadPoolExecutorcontaining at least onethreading.Thread. It also means
that all the normal risks you get when you mix threads with fork apply here, too
(Seehttp://bugs.python.org/issue6721for more info).The one exception to this isaioprocessing.AioPool, which makes use of the
existingcallbackanderror_callbackkeyword arguments in the variousPool.*_asyncmethods to run them asasynciocoroutines. Note thatmultiprocessing.Poolis actually using threads internally, so the thread/fork
mixing caveat still applies.Eachmultiprocessingclass is replaced by an equivalentaioprocessingclass,
distinguished by theAioprefix. So,PoolbecomesAioPool, etc. All methods
that could block on I/O also have a coroutine version that can be used withasyncio. For example,multiprocessing.Lock.acquire()can be replaced withaioprocessing.AioLock.coro_acquire(). You can pass anasyncioEventLoop object to anycoro_*method using theloopkeyword argument. For example,lock.coro_acquire(loop=my_loop).Note that you can also use theaioprocessingsynchronization primitives as replacements
for their equivalentthreadingprimitives, in single-process, multi-threaded programs
that useasyncio.What parts of multiprocessing are supported?Most of them! All methods that could do blocking I/O in the following objects
have equivalent versions inaioprocessingthat extend themultiprocessingversions by adding coroutine versions of all the blocking methods.PoolProcessPipeLockRLockSemaphoreBoundedSemaphoreEventConditionBarrierconnection.Connectionconnection.Listenerconnection.ClientQueueJoinableQueueSimpleQueueAllmanagers.SyncManagerProxyversions of the items above (SyncManager.Queue,SyncManager.Lock(), etc.).What versions of Python are compatible?aioprocessingwill work out of the box on Python 3.5+.GotchasKeep in mind that, while the API exposes coroutines for interacting withmultiprocessingAPIs, internally they are almost always being delegated
to aThreadPoolExecutor, this means the caveats that apply with usingThreadPoolExecutorwithasyncioapply: namely, you won't be able to
cancel any of the coroutines, because the work being done in the worker
thread can't be interrupted.