Package: unidist
Unified Distributed ExecutionWhat is unidist?unidist is a framework that is intended to provide the unified API for distributed execution by supporting various performant execution backends. At the moment the following backends are supported under the hood:MPIDask DistributedRayPython Multiprocessingunidist is designed to work in atask-based parallelmodel.Also, the framework provides a Python Sequential backend (pyseq), that can be used for debugging.InstallationUsing pipunidist can be installed withpipon Linux, Windows and MacOS:pipinstallunidist# Install unidist with dependencies for Python Multiprocessing and Python Sequential backendsunidist can also be used with MPI, Dask or Ray execution backend.
If you don't have MPI, Dask or Ray installed, you will need to install unidist with one of the targets:pipinstallunidist[all]# Install unidist with dependencies for all the backendspipinstallunidist[mpi]# Install unidist with dependencies for MPI backendpipinstallunidist[dask]# Install unidist with dependencies for Dask backendpipinstallunidist[ray]# Install unidist with dependencies for Ray backendunidist automatically detects which execution backends are installed and uses that for scheduling computation.Note:There are different MPI implementations, each of which can be used as a backend in unidist.
Mappingunidist[mpi]installsmpi4pypackage, which is just a Python wrapper for MPI.
To enable unidist on MPI execution you need to have a working MPI implementation and certain software installed beforehand.
Refer toInstallationpage of thempi4pydocumentation for details.
Also, you can find some instructions onMPI backendpage.Using condaFor installing unidist with dependencies for MPI and Dask execution backends into a conda environment
the following command should be used:condainstallunidist-mpiunidist-dask-cconda-forgeAll set of backends could be available in a conda environment by specifying:condainstallunidist-all-cconda-forgeor explicitly:condainstallunidist-mpiunidist-daskunidist-ray-cconda-forgeNote:There are different MPI implementations, each of which can be used as a backend in unidist.
By default, mappingunidist-mpiinstalls a default MPI implementation, which comes withmpi4pypackage and is ready to use.
The conda dependency solver decides on which MPI implementation is to be installed. If you want to use a specific version of MPI,
you can install the core dependencies for MPI backend and the specific version of MPI asconda install unidist-mpi <mpi>as shown in theInstallationpage ofmpi4pydocumentation. That said, it is highly encouraged to use your own MPI binaries as stated in theUsing External MPI Librariessection of the conda-forge documentation in order to get ultimate performance.For more information refer toInstallationsection.Choosing an execution backendIf you want to choose a specific execution backend to run on,
you can set the environment variableUNIDIST_BACKENDand unidist will do computation with that backend:exportUNIDIST_BACKEND=mpi# unidist will use MPIexportUNIDIST_BACKEND=dask# unidist will use DaskexportUNIDIST_BACKEND=ray# unidist will use RayThis can also be done within a notebook/interpreter before you initialize unidist:fromunidist.configimportBackendBackend.put("mpi")# unidist will use MPIBackend.put("dask")# unidist will use DaskBackend.put("ray")# unidist will use RayIf you have installed all the execution backends and haven't specified any of the execution backends, MPI is used by default.
Currently, almost all MPI implementations requirempiexeccommand to be used when running an MPI program.
If you use a backend other than MPI, you run a program as a regular python script (see below).Usage# script.pyimportunidistunidist.init()# MPI backend is used by default@unidist.remotedeffoo(x):returnx*x# This will run `foo` on a pool of workers in parallel;# `refs` will contain object references to actual datarefs=[foo.remote(i)foriinrange(5)]# To get the data call `unidist.get(...)`print(unidist.get(refs))Run thescript.pywith:$mpiexec-n1pythonscript.py# for MPI backend# $ python script.py  # for any other supported backend[0,1,4,9,16]# outputFor more examples refer toGetting Startedsection
in our documentation.Powered by unidistunidist is meant to be used not only directly by users to get better performance in their workloads,
but also be a core component of other libraries to power those with the performant execution backends.
Refer toLibraries powered by unidistsection ofUsing Unidistpage
to get more information on which libraries have already been using unidist.Full DocumentationVisit the complete documentation on readthedocs:https://unidist.readthedocs.io.