Package: asynccpu
Asynchronous CPUSupports async / await pattern for CPU-bound operations.AdvantageSupport async / await pattern for CPU-bound operationsFree from handling event loop1. Support async / await pattern for CPU-bound operationsThe async / await syntax makes asynchronous code as:SimpleReadableThis syntax is not only for I/O-bound but also CPU-bound operations.
This package supportsCoroutinefunction to run inProcessPoolExecutorand returnsAwaitableobject.2. Free from handling event loopasynciois focusing not CPU-bound but I/O-bound operations.
High-level APIs ofasynciodoesn't support CPU-bound operations
since it works based on notProcessPoolExecutorbutThreadPoolExecutor.
When we want to run CPU-bound operations concurrently withasyncio,
we need to use Low-level APIs which need finer control over the event loop behavior.Application developers should typically use the High-levelasynciofunctions, such as asyncio.run(),
and should rarely need to reference Low-level APIs, such as the Event Loop object or call its methods.See:Event Loop — Python 3 documentationQuickstart1. Installpip install asynccpu2. ImplementThis package providesProcessTaskPoolExecutorextendsProcessPoolExecutor,
And its instance has the method:create_process_task().Ex:importasynciofromasynccpuimportProcessTaskPoolExecutorasyncdefprocess_cpu_bound(task_id):"""CPU-bound operations will block the event loop:in general it is preferable to run them in a process pool."""returnf"task_id:{task_id}, result:{sum(i*iforiinrange(10**7))}"withProcessTaskPoolExecutor(max_workers=3,cancel_tasks_when_shutdown=True)asexecutor:awaitables={executor.create_process_task(process_cpu_bound,x)forxinrange(10)}results=awaitasyncio.gather(*awaitables)forresultinresults:print(result)NoteThe argument ofCoroutinerequires not "rawCoroutineobject" but "Coroutinefunction"
since rawCoroutineobject is not picklable.This specification is depend on the one of Pythonmultiprocessingpackage:multiprocessing — Process-based parallelismNote When an object is put on a queue, the object is pickled
and a background thread later flushes the pickled data to an underlying pipe.See:Answer: Python multiprocessing PicklingError: Can't pickle <type 'function'> - Stack OverflowHow do I...Capture log from subprocess?Ex:importasyncioimportmultiprocessingfromloggingimportDEBUG,Formatter,StreamHandler,getLogger,handlersfromasynccpuimportProcessTaskPoolExecutordeflistener_configurer():console_handler=StreamHandler()console_handler.setFormatter(Formatter("[%(levelname)s/%(processName)s]%(message)s"))# Key Point 4returnhandlers.QueueListener(queue,console_handler)defworker_configurer():root_logger=getLogger()root_logger.setLevel(DEBUG)withmultiprocessing.Manager()asmanager:# Key Point 2queue=manager.Queue()listener=listener_configurer()listener.start()withProcessTaskPoolExecutor(max_workers=3,cancel_tasks_when_shutdown=True,# Key Point 1queue=queue,# Key Point 3configurer=worker_configurer)asexecutor:futures={executor.create_process_task(process_cpu_bound,x)forxinrange(10)}returnawaitasyncio.gather(*futures)listener.stop()This implementation is based on following document:Logging to a single file from multiple processes | Logging Cookbook — Python 3 documentationKey PointsInjectspecialqueue.Queueobject into subprocessCreatespecialqueue.Queueobject viamultiprocessing.ManagerInject configurer to configure logger for WindowsConsider to uselogging.handlers.QueueListener1. Injectspecialqueue.Queueobject into subprocessWe can capture logs from subprocess viaqueue.Queueobject.logging.handlers.QueueHandleris often used for multi-threaded, multi-process code logging.See:Logging Cookbook — Python 3 documentationProcessTaskPoolExecutorautomatically setqueueargument into root logger aslogging.handlers.QueueHandlerifqueueargument is set.2. Createspecialqueue.Queueobject viamultiprocessing.ManagerWe have to createqueue.Queueobject viamultiprocessing.Managerdue to limitation ofProcessPoolExecutorrunning inside,
otherwise, following error raised when refer queue argument in child process:RuntimeError: Queue objects should only be shared between processes through inheritancemultiprocessing.Managerinstantiatesspecialqueue.Queueobject (Proxy Object).See:Using concurrent.futures.ProcessPoolExecutor | Logging Cookbook — Python 3 documentationProxy Objects | multiprocessing — Process-based parallelism — Python 3 documentation3. Inject configurer to configure logger for WindowsOn POSIX, subprocess will share loging configuration with parent process by process fork semantics.
On Windows you can't rely on fork semantics,
so each process requires to run the logging configuration code when it starts.ProcessTaskPoolExecutorwill automatically executeconfigurerargument
before startingCoroutinefunction.This design is based on following document:Logging to a single file from multiple processes | Logging Cookbook — Python 3 documentationFor instance, this allows us to set log level in subprocess on Windows.Note that configuring root logger in subprocess seems to effect parent process on POSIX.4. Consider to uselogging.handlers.QueueListenerWe don't have to create an implementation on the Listener process from scratch, we can use it right away withlogging.handlers.QueueListener.CreditsThis package was created withCookiecutterand theyukihiko-shinoda/cookiecutter-pypackageproject template.