Package: pathos
About the Pathos Frameworkpathosis a framework for heterogeneous computing. It provides a consistent
high-level interface for configuring and launching parallel computations
across heterogeneous resources.pathosprovides configurable launchers for
parallel and distributed computing, where each launcher contains the
syntactic logic to configure and launch jobs in an execution environment.
Examples of launchers that plug intopathosare: a queue-less MPI-based
launcher (inpyina), a ssh-based launcher (inpathos), and a multi-process
launcher (inmultiprocess).pathosprovides a consistent interface for parallel and/or distributed
versions ofmapandapplyfor each launcher, thus lowering the barrier
for users to extend their code to parallel and/or distributed resources.
The guiding design principle behindpathosis thatmapandapplyshould be drop-in replacements in otherwise serial code, and thus switching
to one or more of thepathoslaunchers is all that is needed to enable
code to leverage the selected parallel or distributed computing resource.
This not only greatly reduces the time to convert a code to parallel, but it
also enables a single code-base to be maintained instead of requiring
parallel, serial, and distributed versions of a code.pathosmaps can be
nested, thus hierarchical heterogeneous computing is possible by merely
selecting the desired hierarchy ofmapandpipe(apply) objects.Thepathosframework is composed of several interoperating packages:dill: serialize all of Pythonpox: utilities for filesystem exploration and automated buildsklepto: persistent caching to memory, disk, or databasemultiprocess: better multiprocessing and multithreading in Pythonppft: distributed and parallel Pythonpyina: MPI parallelmapand cluster schedulingpathos: graph management and execution in heterogeneous computingAbout PathosThepathospackage provides a few basic tools to make parallel and
distributed computing more accessible to the end user. The goal ofpathosis to enable the user to extend their own code to parallel and distributed
computing with minimal refactoring.pathosprovides methods for configuring, launching, monitoring, and
controlling a service on a remote host. One of the most basic features
ofpathosis the ability to configure and launch a RPC-based service
on a remote host.pathosseeds the remote host with theportpickerscript, which allows the remote host to inform the localhost of a port
that is available for communication.Beyond the ability to establish a RPC service, and then post requests,
is the ability to launch code in parallel. Unlike parallel computing
performed at the node level (typically with MPI),pathosenables the
user to launch jobs in parallel across heterogeneous distributed resources.pathosprovides distributedmapandpipealgorithms, where a mix of
local processors and distributed workers can be selected.pathosalso provides a very basic automated load balancing service, as well as
the ability for the user to directly select the resources.The high-levelpool.mapinterface, yields amapimplementation that
hides the RPC internals from the user. Withpool.map, the user can launch
their code in parallel, and as a distributed service, using standard Python
and without writing a line of server or parallel batch code.RPC servers and communication in general is known to be insecure.  However,
instead of attempting to make the RPC communication itself secure,pathosprovides the ability to automatically wrap any distributes service or
communication in a ssh-tunnel. Ssh is a universally trusted method.
Using ssh-tunnels,pathoshas launched several distributed calculations
on national lab clusters, and to date has performed test calculations
that utilize node-to-node communication between several national lab clusters
and a user’s laptop.pathosallows the user to configure and launch
at a very atomistic level, through raw access to ssh and scp.pathosis the core of a Python framework for heterogeneous computing.pathosis in active development, so any user feedback, bug reports, comments,
or suggestions are highly appreciated.  A list of issues is located athttps://github.com/uqfoundation/pathos/issues, with a legacy list maintained athttps://uqfoundation.github.io/project/pathos/query.Major Featurespathosprovides a configurable distributed parallelmapinterface
to launching RPC service calls, with:amapinterface that meets and extends the Pythonmapstandardthe ability to submit service requests to a selection of serversthe ability to tunnel server communications with sshThepathoscore is built on low-level communication to remote hosts using
ssh. The interface to ssh, scp, and ssh-tunneled connections can:configure and launch remote processes with sshconfigure and copy file objects with scpestablish an tear-down a ssh-tunnelTo get up and running quickly,pathosalso provides infrastructure to:easily establish a ssh-tunneled connection to a RPC serverCurrent ReleaseThe latest released version ofpathosis available from:https://pypi.org/project/pathospathosis distributed under a 3-clause BSD license.Development VersionYou can get the latest development version with all the shiny new features at:https://github.com/uqfoundationIf you have a new contribution, please submit a pull request.Installationpathoscan be installed withpip:$ pip install pathosRequirementspathosrequires:python(orpypy),>=3.8setuptools,>=42pox,>=0.3.4dill,>=0.3.8ppft,>=1.7.6.8multiprocess,>=0.70.16More InformationProbably the best way to get started is to look at the documentation athttp://pathos.rtfd.io. Also seepathos.testsandhttps://github.com/uqfoundation/pathos/tree/master/examplesfor a set of scripts that demonstrate the
configuration and launching of communications with ssh and scp, and demonstrate
the configuration and execution of jobs in a hierarchical parallel workflow.
You can run the test suite withpython-mpathos.tests. Tunnels and other
connections to remote servers can be established with thepathos_connectscript (or withpython-mpathos). Seepathos_connect--helpfor more
information.pathosalso provides aportpickerscript to select an
open port (also available withpython-mpathos.portpicker). The source
code is generally well documented, so further questions may be resolved by
inspecting the code itself.  Please feel free to submit a ticket on github,
or ask a question on stackoverflow (@Mike McKerns). If you would like to
share how you usepathosin your work, please send an email (tommckerns
at uqfoundation dot org).Important classes and functions are found here:pathos.abstract_launcher[the worker pool API definition]pathos.pools[all of the pathos worker pools]pathos.core[the high-level command interface]pathos.hosts[the hostname registry interface]pathos.serial.SerialPool[the serial Python worker pool]pathos.parallel.ParallelPool[the parallelpython worker pool]pathos.multiprocessing.ProcessPool[the multiprocessing worker pool]pathos.threading.ThreadPool[the multithreading worker pool]pathos.connection.Pipe[the launcher base class]pathos.secure.Pipe[the secure launcher base class]pathos.secure.Copier[the secure copier  base class]pathos.secure.Tunnel[the secure tunnel base class]pathos.selector.Selector[the selector base class]pathos.server.Server[the server base class]pathos.profile[profiling in threads and processes]pathos.maps[standalone map instances]pathosalso provides two convenience scripts that are used to establish
secure distributed connections. These scripts are installed to a directory
on the user’s$PATH, and thus can be run from anywhere:portpicker[get the portnumber of an open port]pathos_connect[establish tunnel and/or RPC server]Typing--helpas an argument to any of the above scripts will print out an
instructive help message.CitationIf you usepathosto do research that leads to publication, we ask that you
acknowledge use ofpathosby citing the following in your publication:M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,
"Building a framework for predictive science", Proceedings of
the 10th Python in Science Conference, 2011;
http://arxiv.org/pdf/1202.1056

Michael McKerns and Michael Aivazis,
"pathos: a framework for heterogeneous computing", 2010- ;
https://uqfoundation.github.io/project/pathosPlease seehttps://uqfoundation.github.io/project/pathosorhttp://arxiv.org/pdf/1202.1056for further information.