Package: loky
Reusable Process Pool ExecutorGoalThe aim of this project is to provide a robust, cross-platform and
cross-version implementation of theProcessPoolExecutorclass ofconcurrent.futures. It notably features:Consistent and robust spawn behavior: All processes are started
using fork + exec on POSIX systems. This ensures safer interactions with
third party libraries. On the contrary,multiprocessing.Pooluses
fork without exec by default, causing third party runtimes to crash
(e.g. OpenMP, macOS Accelerate...).Reusable executor: strategy to avoid re-spawning a complete
executor every time. A singleton executor instance can be reused (and
dynamically resized if necessary) across consecutive calls to limit
spawning and shutdown overhead. The worker processes can be shutdown
automatically after a configurable idling timeout to free system
resources.Transparent cloudpickle integration: to call interactively
defined functions and lambda expressions in parallel. It is also
possible to register a custom pickler implementation to handle
inter-process communications.No need forif __name__ == "__main__":in scripts: thanks
to the use ofcloudpickleto call functions defined in the__main__module, it is not required to protect the code calling
parallel functions under Windows.Deadlock free implementation: one of the major concern in
standardmultiprocessingandconcurrent.futuresmodules is the
ability of thePool/Executorto handle crashes of worker
processes. This library intends to fix those possible deadlocks and
send back meaningful errors. Note that the implementation ofconcurrent.futures.ProcessPoolExecutorthat comes with Python 3.7+
is as robust as the executor from loky but the latter also works for
older versions of Python.InstallationThe recommended way to installlokyis withpip,pipinstalllokylokycan also be installed from sources usinggitclonehttps://github.com/joblib/lokycdloky
pythonsetup.pyinstallNote thatlokyhas an optional dependency onpsutilto allow early
memory leak detections.UsageThe basic usage oflokyrelies on theget_reusable_executor, which
internally manages a customProcessPoolExecutorobject, which is reused or
re-spawned depending on the context.importosfromtimeimportsleepfromlokyimportget_reusable_executordefsay_hello(k):pid=os.getpid()print(f"Hello from{pid}with arg{k}")sleep(.01)returnpid# Create an executor with 4 worker processes, that will# automatically shutdown after idling for 2sexecutor=get_reusable_executor(max_workers=4,timeout=2)res=executor.submit(say_hello,1)print("Got results:",res.result())results=executor.map(say_hello,range(50))n_workers=len(set(results))print("Number of used processes:",n_workers)assertn_workers==4For more advance usage, see ourdocumentationWorkflow to contributeTo contribute toloky, first create an account ongithub. Once this is done, fork theloky repositoryto have your own repository,
clone it using 'git clone' on the computers where you want to work. Make your
changes in your clone, push them to your github account, test them on several
computers, and when you are happy with them, send a pull request to the main
repository.Running the test suiteTo run the test suite, you need thepytest(version >= 3) andpsutilmodules. From the root of the project, run the test suite using:pipinstall-e.pytest.Why was the project namedloky?While developpingloky, we had some bad experiences trying to debug deadlocks
when usingmultiprocessing.Poolandconcurrent.futures.ProcessPoolExecutor,
especially when calling functions with non-picklable arguments or returned
values at the beginning of the project. When we had to chose a name, we had
dealt with so many deadlocks that we wanted some kind of invocation to repel
them! Henceloky: a mix of a god, locks and theythat make it somehow
cooler and nicer : (and also less likely to result in name conflict in google
results ^^).Fixes to avoid those deadlocks inconcurrent.futureswere also contributed
upstream in Python 3.7+, as a less mystical way to repel the deadlocks :DAcknowledgementThis work is supported by the Center for Data Science, funded by the IDEX
Paris-Saclay, ANR-11-IDEX-0003-02