Package: deadpool-executor
1DeadpoolDeadpoolis a process pool that is really hard to kill.Deadpoolis an implementation of theExecutorinterface
in theconcurrent.futuresstandard library.Deadpoolis
a process pool executor, quite similar to the stdlib’sProcessPoolExecutor.This document assumes that you are familiar with the stdlibProcessPoolExecutor. If you are not, it is important
to understand thatDeadpoolmakes very specific tradeoffs that
can result in quite different behaviour to the stdlib
implementation.2InstallationThe python package name isdeadpool-executor, so to install
you must type$ pip installdeadpool-executor. The import
name isdeadpool, so in your Python code you must typeimport deadpoolto use it.I try quite hard to keep dependencies to a minimum. CurrentlyDeadpoolhas no dependencies other thanpsutilwhich
is simply too useful to avoid for this library.3Why would I want to use this?I createdDeadpoolbecause I became frustrated with the
stdlibProcessPoolExecutor, and various other community
implementations of process pools. In particular, I had a use-case
that required a high server uptime, but also had variable and
unpredictable memory requirements such that certain tasks could
trigger theOOM killer, often resulting in a “broken” process
pool. I also needed task-specific timeouts that could kill a “hung”
task, which the stdlib executor doesn’t provide.You might wonder, isn’t it bad to just kill a task like that?
In my use-case, we had extensive logging and monitoring to alert
us if any tasks failed; but it was paramount that our services
continue to operate even when tasks got killed in OOM scenarios,
or specific tasks took too long. This is the primary trade-off
thatDeadpooloffers: the pool will not break, but tasks
can receive SIGKILL under certain conditions. This trade-off
is likely fine if you’ve seen many OOMs break your pools.I also tried using thePebblecommunity process pool. This is a cool project, featuring several
of the properties I’ve been looking for such as timeouts, and
more resilient operation. However, during testing I found several
occurrences of a mysteriousRuntimeErrorthat caused the Pebble
pool to become broken and no longer accept new tasks.My goal withDeadpoolis thatthe pool must never enter
a broken state. Any means by which that can happen will be
considered a bug.4What differs fromProcessPoolExecutor?Deadpoolis generally similar toProcessPoolExecutorsince it executes
tasks in subprocesses, and implements the standardExecutorabstract
interface. We can draw a few comparisons to the stdlib pool to guide
your decision process about whether this makes sense for your use-case:4.1SimilaritiesDeadpoolalso supports themax_tasks_per_childparameter (a new feature in
Python 3.11, although it was available inmultiprocessing.Poolsince Python 3.2).The “initializer” callback inDeadpoolworks the same.Deadpooldefaults to theforkservermultiprocessing
context, unlike the stdlib pool which defaults toforkon
Linux. It’s just a setting though, you can change it in the same way as
with the stdlib pool. Like the stdlib, I strongly advise you to avoid
usingforkbecause propagation threads and locks via fork is
going to ruin your day eventually. While this is a difference to the
default behaviour of the stdlib pool, it’s not a difference in
behaviour to the stdlib pool when you use theforkservercontext
which is the recommended context for multiprocessing.4.2Differences in existing behaviourDeadpooldiffers from the stdlib pool in the following ways:If aDeadpoolsubprocess in the pool is killed by some
external actor, for example, the OS runs out of memory and theOOM killerkills a pool subprocess that is using too much memory,Deadpooldoes not care and further operation is unaffected.Deadpoolwill not, and indeed cannot raiseBrokenProcessPoolorBrokenExecutor.Deadpoolprecreates all subprocesses up to the pool size on
creation.Deadpooltasks can have priorities. When the executor chooses
the next pending task to schedule to a subprocess, it chooses the
pending task with the highest priority. This gives you a way of
prioritizing certain kinds of tasks. For example, you might give
UI-sensitive tasks a higher priority to deliver a more snappy
user experience to your users. The priority can be specified in
thesubmitcall.The shutdown parameterswaitandcancel_futurescan behave
differently to how they work in theProcessPoolExecutor. This is
discussed in more detail later in this document.Deadpoolcurrently only works on Linux. There isn’t any specific
reason it can’t work on other platforms. The malloc trim feature also
requires a glibc system, so probably won’t work on Alpine.4.3New features in DeadpoolDeadpoolhas the following features that are not present in the
stdlib pool:WithDeadpoolyou can provider a “finalizer” callback that will
fire before a subprocess is shut down or killed. The finalizer callback
might be executed in a different thread than the main thread of the
subprocess, so don’t rely on the callback running in the main
subprocess thread. There are certain circumstances where the finalizer
will not run at all, such as when the subprocess is killed by the OS
due to an out-of-memory (OOM) condition. So don’t design your application
such that the finalizer is required to run for correct operation.Even thoughDeadpooltypically uses a hard kill to remove
subprocesses, it does still run any handlers registered withatexit.Deadpooltasks can have timeouts. When a task hits the timeout,
the underlying subprocess in the pool is killed withSIGKILL.
The entire process tree of that subprocess is killed. Your application
logic needs to handle this. Thefinalizerwill not run.Deadpoolalso allows afinalizer, with correspondingfinalargs, that will be called after a task is executed on
a subprocess, but before the subprocess terminates. It is
analogous to theinitializerandinitargsparameters.
Just like theinitializercallable, thefinalizercallable is executed inside the subprocess. It is not guaranteed that
the finalizer will always run. If a process is killed, e.g. due to a
timeout or any other reason, the finalizer will not run. The finalizer
could be used for things like flushing pending monitoring messages,
such as traces and so on.Deadpoolcan ask the system allocator (Linux only) to return
unused memory back to the OS based on exceeding a max threshold RSS.
For long-running pools and modern
kernels, the system memory allocator can hold onto unused memory
for a surprisingly long time, and coupled with bloat due to
memory fragmentation, this can result in carrying very large
RSS values in your pool. Themax_tasks_per_childhelps with
this because a subprocess is entirely erased when the max is
reached, but it does mean that periodically there will be a small
latency penalty from constructing the replacement subprocess. In
my opinion,max_tasks_per_childis appropriate for when you
know or suspect there’s a real memory leak somewhere in your code
(or a 3rd-party package!), and the easiest way to deal with that
right now is just to periodically remove a process.5Show me some code5.1Simple caseThe simple case works exactly the same as withProcessPoolExecutor:fromdeadpoolimportDeadpooldeff():return123withdeadpool.Deadpool()asexe:fut=exe.submit(f)result=fut.result()assertresult==123It is intended that all the basic behaviour should “just work” in the
same way, andDeadpoolshould be a drop-in replacement forProcessPoolExecutor; but there are some subtle differences so you
should read all of this document to see if any of those will affect you.5.2TimeoutsIf a timeout is reached on a task, the subprocess running that task will be
killed, as inSIGKILL.Deadpooldoesn’t mind, but your own
application should: if you use timeouts it is likely important that your tasks
beidempotent, especially if
your application will restart tasks, or restart them after application deployment,
and other similar scenarios.importtimeimportdeadpooldeff():time.sleep(10.0)withdeadpool.Deadpool()asexe:fut=exe.submit(f,deadpool_timeout=1.0)withpytest.raises(deadpool.TimeoutError)fut.result()The parameterdeadpool_timeoutis special and consumed byDeadpoolin the call. You can’t use a parameter with this name in your function
kwargs.5.3Handling OOM killed situationsimporttimeimportdeadpooldeff():x=list(range(10**100))withdeadpool.Deadpool()asexe:fut=exe.submit(f,timeout=1.0)try:result=fut.result()exceptdeadpool.ProcessError:print("Oh no someone killed my task!")As long as the OOM killer terminates merely a subprocess (and not the main
process), which is likely because it’ll be your subprocess that is using too
much memory, this will not hurt the pool, and it will be able to receive and
process more tasks. Note that this event will show up as aProcessErrorexception when accessing the future, so you have a way of at least tracking
these events.6Design Details6.1Typical Example - with timeoutsHere’s a typical example of how code using Deadpool might look. The
output of the code further below should be similar to the following:$pythonexamples/entrypoint.py...................xxxxxxxxxxx.xxxxxxx.x.xxxxxxx.x$Each.is a successfully completed task, and eachxis a task
that timed out. Below is the code for this example.importrandom,timeimportdeadpooldefwork():time.sleep(random.random()*4.0)print(".",end="",flush=True)return1defmain():withdeadpool.Deadpool()asexe:futs=(exe.submit(work,timeout=2.0)for_inrange(50))forfutindeadpool.as_completed(futs):try:assertfut.result()==1exceptdeadpool.TimeoutError:print("x",end="",flush=True)if__name__=="__main__":main()print()The work function will be busy for a random time period between 0 and
4 seconds.There is adeadpool_timeoutkwarg given to thesubmitmethod.
This kwarg is special and will be consumed by Deadpool. You cannot
use this kwarg name for your own task functions.When a task completes, it prints out.internally. But when a task
raises adeadpool.TimeoutError, axwill be printed out instead.When a task times out, keep in mind that the underlying process that
is executing that task is killed, literally with theSIGKILLsignal.6.2Deadpool tasks have priorityThe example below is similar to the previous one for timeouts. In fact
this example retains the timeouts to show how the different features
compose together. In this example we create tasks with different
priorities, and we change the printed character of each task to show
that higher priority items are executed first.The code example will print something similar to the following:$pythonexamples/priorities.py!!!!!xxxxxxxxxxx!x..!...x.xxxxxxxx.xxxx.x...xxxxxxYou can see how the!characters, used for indicating higher priority
tasks, appear towards the front indicating that they were executed sooner.
Below is the code.importrandom,timeimportdeadpooldefwork(symbol):time.sleep(random.random()*4.0)print(symbol,end="",flush=True)return1defmain():withdeadpool.Deadpool(max_backlog=100)asexe:futs=[]for_inrange(25):fut=exe.submit(work,".",deadpool_timeout=2.0,deadpool_priority=10)futs.append(fut)fut=exe.submit(work,"!",deadpool_timeout=2.0,deadpool_priority=0)futs.append(fut)forfutindeadpool.as_completed(futs):try:assertfut.result()==1exceptdeadpool.TimeoutError:print("x",end="",flush=True)if__name__=="__main__":main()print()When the tasks are submitted, they are given a priority. The default
value for thedeadpool_priorityparameter is 0, but here we’ll
write them out explicity.  Half of the tasks will have priority 10 and
half will have priority 0.A lower value for thedeadpool_priorityparameters means ahigherpriority. The highest priority allowed is indicated by 0. Negative
priority values are not allowed.I also specified themax_backlogparameter when creating the
Deadpool instance. This is discussed in more detail next, but quickly:
task priority can only be enforced on what is in the submitted backlog
of tasks, and themax_backlogparameter controls the depth of that
queue. Ifmax_backlogis too low, then the window of prioritization
will not include tasks submitted later which might have higher priorities
than earlier-submitted tasks. Thesubmitcall will in fact block
once themax_backlogdepth has been reached.6.3Controlling the backlog of submitted tasksBy default, themax_backlogparameter is set to 5. This parameter is
used to create the “submit queue” size. The submit queue is the place
where submitted tasks are held before they are executed in background
processes.If the submit queue is large (max_backlog), it will mean
that a large number of tasks can be added to the system with thesubmitmethod, even before any tasks have finished exiting. Conversely,
a lowmax_backlogparameter means that the submit queue will fill up
faster. If the submit queue is full, it means that the next call tosubmitwill block.This kind of blocking is fine, and typically desired. It means that
backpressure from blocking is controlling the amount of work in flight.
By using a smallermax_backlog, it means that you’ll also be
limiting the amount of memory in use during the execution of all the tasks.However, if you nevertheless still accumulate received futures as my
example code above is doing, that accumulation, i.e., the list of futures,
will contribute to memory growth. If you have a large amount of work, it
will be better to set acallbackfunction on each of the futures rather
than processing them by iterating overas_completed.The example below illustrates this technique for keeping memory
consumption down:importrandom,timeimportdeadpooldefwork():time.sleep(random.random()*4.0)print(".",end="",flush=True)return1defcb(fut):try:assertfut.result()==1exceptdeadpool.TimeoutError:print("x",end="",flush=True)defmain():withdeadpool.Deadpool()asexe:for_inrange(50):exe.submit(work,deadpool_timeout=2.0).add_done_callback(cb)if__name__=="__main__":main()print()With this callback-based design, we no longer have an accumulation of futures
in a list. We get the same kind of output as in the “typical example” from
earlier:$pythonexamples/callbacks.py.....xxx.xxxxxxxxx.........x..xxxxx.x....x.xxxxxxxSpeaking of callbacks, the customizedFutureclass used by Deadpool
lets you set a callback for when the task begins executing on a real
system process. That can be configured like so:withdeadpool.Deadpool()asexe:f=exe.submit(work)defcb(fut:deadpool.Future):print(f"My task is running on process{fut.pid}")f.add_pid_callback(cb)Obviously, both kinds of callbacks can be added:withdeadpool.Deadpool()asexe:f=exe.submit(work)f.add_pid_callback(lambdafut:f"Started on{fut.pid=}")f.add_done_callback(lambdafut:f"Completed{fut.pid=}")6.4More about shutdownIn the documentation forProcessPoolExecutor, the following function
signature is given for theshutdownmethod of the executor interface:shutdown(wait=True,*,cancel_futures=False)I want to honor this, but it presents some difficulties because the
semantics of thewaitandcancel_futuresparameters need to be
somewhat different for Deadpool.In Deadpool, this is what the combinations of those flags mean:Shutdown flagswaitcancel_futureseffectTrueTrueWait for already-running tasks to complete; theshutdown()call will unblock (return) when they’re done. Cancel
all pending tasks that are in the submit queue, but have not yet started
running. Thefut.cancelled()method will returnTruefor such
cancelled tasks.TrueFalseWait for already-running tasks to complete.
Pending tasks in the
submit queue that have not yet started running willnotbe cancelled, and
will all continue to execute. Theshutdown()call will return only
after all submitted tasks have completed.FalseTrueAlready-running taskswill be cancelledand this
means the underlying subprocesses executing these tasks will receive
SIGKILL. Pending tasks on the submit queue that have not yet started
running will also be cancelled.FalseFalseThis is a strange one. What to do if the caller
doesn’t want to wait, but also doesn’t want to cancel things? In this
case, already-running tasks will be allowed to complete, but pending
tasks on the submit queue will be cancelled. This is the same outcome as
aswait==Trueandcancel_futures==True. An alternative design
might have been to allow all tasks, both running and pending, to just
keep going in the background even after theshutdown()call
returns. Does anyone have a use-case for this?If you’re usingDeadpoolas a context manager, you might be wondering
how exactly to set these parameters in theshutdowncall, since that
call is made for you automatically when the context manager exits.For this, Deadpool provides additional parameters that can be provided
when creating the instance:# This is pseudocodeimportdeadpoolwithdeadpool.DeadPool(shutdown_wait=True,shutdown_cancel_futures=True):fut=exe.submit(...)7Developer Workflow7.1noxThis project usesnox. Follow the instructions for installing
nox at their page, and then come back here.While nox can be configured so that all the tools for each of
the tasks can be installed automatically when run, this takes
too much time and so I’ve decided that you should just have
the following tools in your environment, ready to go. They
do not need to be installed in the same venv or anything like
that. I’ve found a convenient way to do this is withpipx.
For example, to installblackusingpipxyou can do
the following:$pipxinstallblackYou must do the same forisortandruff. See the following
sections for actually usingnoxto perform dev actions.7.2testsTo run the tests:$nox-stestsTo run tests for a particular version, and say with coverage:$nox-stestcov-3.11To pass additional arguments to pytest, use the--separator:$nox-stestcov-3.11---ktest_deadpool-s<etc>This is nonstandard above, but I customized thenoxfile.pyto
allow this.7.3styleTo apply style fixes, and check for any remaining lints,$nox-tstyle7.4docsThe only docs currently are this README, which uses RST. Github
usesdocutilsto render RST.7.5releaseThis project uses flit to release the package to pypi. The whole
process isn’t as automated as I would like, but this is what
I currently do:Ensure thatmainbranch is fully up to date with all to
be released, and all the tests succeed.Change the__version__field indeadpool.py. Flit
uses this to stamp the version.Verify thatflit buildsucceeds. This will produce a
wheel in thedist/directory. You can inspect this
wheel to ensure it contains only what is necessary. This
wheel will be what is uploaded to PyPI.Commit the changed ``__version__``. Easy to forget this
step, resulting in multiple awkward releases to try to
get the state all correct again.Now create the git tag and push to github:$gittagYYYY.MM.patch$gitpush--tagsoriginmainNow deploy to PyPI:$flitpublish