Package: coroexecutor
WarningThis is alpha. Please don’t rely on this in a production
setting yet. I will remove this warning when it is ready.coroexecutorDemoDiscussionThrottling, usingmax_workersDealing with errors and cancellationExamplesUsingmapNestingProvides anExecutorinterface for running a group of coroutines
together in asyncio-native applications.DemoimportasynciofromcoroexecutorimportCoroutineExecutorasyncdeff(dt,msg=''):awaitasyncio.sleep(dt)print(f'completion message:{msg}')asyncdefmain():asyncwithCoroutineExecutor(max_workers=10)asexe:t1=awaitexe.submit(f,0.01,msg="task 1")t2=awaitexe.submit(f,0.05,msg="task 2")assertt1.done()assertt2.done()asyncio.run(main())max_workerscontrols how many submitted jobs can run concurrently.
These internal workers are lightweight of course, they’re justasyncio.Taskinstances. Millions of jobs can be pushed through
the executor. As is normal for asyncio, concurrency requires
that these jobs be IO-bound, and the upper bound for settingmax_workersis mainly going to depend on your CPU and RAM resources.DiscussionTheCoroutineExecutorcontext manager works very much like
theExecutorimplementations in theconcurrent.futurespackage in the standard library. This is the intention of
this package. The basic components of the interface are:The executor applies a context over the creation of jobsJobs are submitted to the executorAll jobs must be complete when the context manager for the executor exits.After creating a context manager usingCoroutineExecutor, the two
main features are thesubmit()method, and themap()method.It is impossible toexactlymatch theExecutorinterface in theconcurrent.futurespackage because some functions in this interface
need to beasyncfunctions. But we can get close; certainly close
enough that a user with experience using theThreadPoolExecutororProcessPoolExecutorshould be able to figure things out pretty quickly.There is a great deal of complexity that can arise. The “happy path” is
simple. You just submit jobs to the executor, and they will get
executed accordingly. But there are many corner cases:asyncio can concurrently execute thousands, or even tens of thousands
of (IO-bound) jobs concurrently. But how to handle more, say, millions
of jobs?If one job raises an exception, how to terminate all the other jobs?
In the CTRL-C case, this is desired, but what about other cases? Do
you always want a single task failure (with an unexpected exception)
to cancel the entire batch? And is there a difference between
a job raisingCancelledErrorversus raising some other kind of
exception?TheCoroutineExectutorprovides a context manager API: if
some code within the body of the context manager (that is not a task)
raises an exception, should all the submitted tasks also
be cancelled?Each of these will be discussed in more detail in the sections
that follow.Throttling, usingmax_workersEven though it is possible to concurrently execute a much larger number
of (IO bound) tasks with asyncio compared to threads or processes, there
will still be an upper limit the machine can handle based on either:memory limitations: many task object instancesCPU limitations: too many concurrent task objects and events for the event loop to process.Thus, we also have amax_workerssetting to limit concurrency. It might
not be obvious how that limitation is applied, say, in the scenario of
millions of jobs.TheCoroutineExecutor.submit()is anasync defmethod. This means
that you will have to await it, like so:importasynciofromcoroexecutorimportCoroutineExecutorasyncdeff():print('hi!')asyncdefmain():asyncwithCoroutineExecutor(max_workers=10)asexe:t1=awaitexe.submit(f)asyncio.run(main())If the total number of jobs already submitted is less thanmax_workers,
the call toawait exe.submit()will return immediately: the job will
begin executing, andsubmit()returns anasyncio.Taskinstance
for that job. However, if the total number of concurrently-running jobs
is greater than themax_workerssetting, this call will wait until
the number of currently-running jobs drops below the threshold before
adding the new job. This means thatsubmit()appliesback-pressure.Say you have a file containing ten million URLs that you want to fetch
using aiohttp. That program might look something like this:importasyncio,aiohttpfromcoroexecutorimportCoroutineExecutorasyncdeffetch(url:str):try:asyncwithaiohttp.ClientSession()assession:asyncwithsession.get(url)asresponse:print('body:',response.text())# or whateverexceptException:print('Problem with url:',url)asyncdefmain():asyncwithCoroutineExecutor(max_workers=10000)asexe:forlineinopen('urls.txt'):awaitexe.submit(fetch,line)asyncio.run(main())Assuming it takes 3 seconds to fetch a single url, this program
should take around 1e7 / 1e4 => 1000 seconds to fetch all of them.
About 17 minutes, since even though there are 10 million urls, we’re
doing 10k concurrently. (In practice, some of the endpoints will be
very slow to respond, if they respond at all. So for real code you’re
going to want to either use aiohttp facilities for timeouts on the.get(), or wrap the work inside anasyncio.wait_for()wrapper.)Note that we’re handling errors inside our job functionfetch().
By default, if jobs raise exceptions these will cancel all pending jobs
inside the executor, and shut it down. For long batch jobs, that may
not be what we want, and this is discussed next.Dealing with errors and cancellationGenerally, there are these kinds of error situations:A job is cancelled, and you want the executor to be shut downA job is cancelled, and the executor must NOT be shut downA job raises an exception (notCancelledError), and
you want the executor to shut downA job raises an exception (notCancelledError), and the
executor must NOT be shut downConsider the previous example using aiohttp to fetch URLs: inside
thefetch()function, we’re handlingException, which
includesasyncio.CancelledError. In general, this is the
correct thing to do because you can control what happens in
each of the scenarios presented above. But what happens
if your code is not supplying the jobs and you don’t control
how error handling inside them is being managed? By default,
if any job raises an exception (cancellation or otherwise)
that will initiate “shutdown” of the executor instance, and
all other pending jobs on that executor will be cancelled.If you have a situation where this is not desired, you can
askCoroutineExecutorto ignore all task errors for you:importasyncio,aiohttpfromcoroexecutorimportCoroutineExecutorasyncdefnaive_fetch(url:str):asyncwithaiohttp.ClientSession()assession:asyncwithsession.get(url)asresponse:print('body:',response.text())# or whateverasyncdefmain():asyncwithCoroutineExecutor(max_workers=10000,suppress_task_errors=True,)asexe:forlineinopen('urls.txt'):awaitexe.submit(naive_fetch,line)asyncio.run(main())In this modified example, the job functionnaive_fetchhas
no error handling. No matter, thesuppress_task_errorsparameter will allow the executor to absorb them all. Be careful
with this. I recommend against doing this wherever possible, and
handle exceptions andCancelledErrorexplicitly within
your job functions instead.ExamplesUsingmapTheconcurrent.futures.Executorinterface also definesmap()which
returns an iterator. However, it makes for sense for us to use anasynchronous generatorfor this purpose. Here’s an example from the tests:times=[0.01,0.02,0.03]asyncdeff(dt):awaitasyncio.sleep(dt)returndtasyncdefmain():asyncwithCoroutineExecutor()asexe:results=exe.map(f,times)assert[vasyncforvinresults]==timesasyncio.run(main())You can see howasync foris used to asynchronously loop over the
result from callingmap.If one of the function calls raises an error, all unfinished calls will
be cancelled, but you may still have received partial results. Here’s
another example from the tests:times=[0.01,0.02,0.1,0.2]results=[]asyncdeff(dt):awaitasyncio.sleep(dt)ifdt==0.1:raiseException('oh noes')returndtasyncdefmain():asyncwithCoroutineExecutor()asexe:asyncforrinexe.map(f,times):results.append(r)withpytest.raises(Exception):asyncio.run(main())assertresults==[0.01,0.02]The first two values of the batch finish quickly, and I saved these to theresultslist in the outer scope. Then, one of the jobs fails with
an exception. This results in the other pending jobs being cancelled (i.e.,
the “0.2” case in this example), theCoroutineExecutorinstance
re-raising the exception, and in this example, the exception raises all
the way out to the invocation of therun()function itself. However,
note that we still have the results from jobs that succeeded.NestingYou don’t always have to submit tasks to the executor in a single function.
The executor instance can be passed around and work can be added to it
from several different places.fromrandomimportrandomasyncdeff(dt):awaitasyncio.sleep(dt)asyncdefproducer1(executor:CoroutineExecutor):executor.submit(f,random())executor.submit(f,random())executor.submit(f,random())asyncdefproducer2(executor:CoroutineExecutor):executor.submit(f,random())executor.submit(f,random())executor.submit(f,random())asyncdefmain():asyncwithCoroutineExecutor(timeout=0.5)asexecutor:executor.submit(f,random())executor.submit(f,random())executor.submit(f,random())executor.submit(producer1,executor)executor.submit(producer2,executor)run(main())You can not only submit jobs within the executor context manager, but also
pass the instance around and collect jobs from other functions too. And the
timeout set when creating theCoroutineExecutorinstance will still
be applied.