Package: trio-parallel
Do you have CPU-bound work that just keeps slowing down yourTrioevent loop no
matter what you try? Do you need to get all those cores humming at once? This is the
library for you!The aim of trio-parallel is to use the lightest-weight, lowest-overhead, lowest-latency
method to achieve CPU parallelism of arbitrary Python code with a dead-simple API.ResourcesLicenseDocumentationChatForumIssuesRepositoryTestsCoverageStyleDistributionExampleimportfunctoolsimportmultiprocessingimporttrioimporttrio_paralleldefloop(n):# Arbitrary CPU-bound workfor_inrange(n):passprint("Loops completed:",n)asyncdefamain():t0=trio.current_time()asyncwithtrio.open_nursery()asnursery:# Do CPU-bound work in parallelforiin[6,7,8]*4:nursery.start_soon(trio_parallel.run_sync,loop,10**i)# Event loop remains responsivet1=trio.current_time()awaittrio.sleep(0)print("Scheduling latency:",trio.current_time()-t1)# This job could take far too long, make it cancellable!nursery.start_soon(functools.partial(trio_parallel.run_sync,loop,10**20,cancellable=True))awaittrio.sleep(2)# Only explicitly cancellable jobs are killed on cancelnursery.cancel_scope.cancel()print("Total runtime:",trio.current_time()-t0)if__name__=="__main__":multiprocessing.freeze_support()trio.run(amain)Additional examples and the full API are available in thedocumentation.FeaturesBypasses the GIL for CPU-bound workMinimal API complexitylooks and feels like TriothreadsMinimal internal complexityNo reliance onmultiprocessing.Pool,ProcessPoolExecutor, or any background threadsCross-platformprintjust worksSeamless interoperation withcoverage.pyviztracercloudpickleAutomatic LIFO caching of subprocessesCancel seriously misbehaving code via SIGKILL/TerminateProcessConvert segfaults and other scary things to catchable errorsFAQHow does trio-parallel run Python code in parallel?Currently, this project is based onmultiprocessingsubprocesses and
has all the usual multiprocessingcaveats(freeze_support, pickleable objects
only, executing the__main__module).
The case for basing these workers on multiprocessing is that it keeps a lot of
complexity outside of the project while offering a set of quirks that users are
likely already familiar with.The pickling limitations can be partially alleviated by installingcloudpickle.Can I have my workers talk to each other?This is currently possible through the use ofmultiprocessing.Manager,
but we don’t and will not officially support it.This package focuses on providing
a flat hierarchy of worker subprocesses to run synchronous, CPU-bound functions.
If you are looking to create a nested hierarchy of processes communicating
asynchronously with each other, while preserving the power, safety, and convenience of
structured concurrency, look intotractor.
Or, if you are looking for a more customized solution, try usingtrio.run_processto spawn additional Trio runs and have them talk to each other over sockets.Can I let my workers outlive the main Trio process?No. Trio’s structured concurrency strictly bounds job runs to within a giventrio.runcall, while cached idle workers are shutdown and killed if necessary
by ouratexithandler, so this use case is not supported.How should I map a function over a collection of arguments?This is fully possible but we leave the implementation of that up to you. Think
of us as alokyfor yourjoblib, but natively async and Trionic.
We take care of the worker handling so that you can focus on the best concurrency
for your application. That said, some example parallelism patterns can be found in
thedocumentation.Also, look intoaiometer?ContributingIf you notice any bugs, need any help, or want to contribute any code, GitHubissuesand pull requests are very welcome! Please read thecode of conduct.