Repository: uqfoundation/pathos (language: python)

Issue:
Refactor setup.py
I looked into setup.py, its code is very strange. It definitely needs refactoring.


Discussion / hints:
Any progress on it?

Fix patch:
diff --git a/.codecov.yml b/.codecov.yml
index 2f73002..b109729 100644
--- a/.codecov.yml
+++ b/.codecov.yml
@@ -20,3 +20,9 @@ coverage:
         target: auto
         threshold: 1%
  
+fixes:
+  # reduces pip-installed path to git root and
+  # remove dist-name from setup-installed path
+  - "*/site-packages/::"
+  - "*/site-packages/pathos-*::"
+
diff --git a/.coveragerc b/.coveragerc
index a7bd17b..f06270b 100644
--- a/.coveragerc
+++ b/.coveragerc
@@ -1,6 +1,10 @@
 [run]
 # source = pathos
-include = */pathos/*
+include =
+    */pathos/*
+    */pathos/helpers/*
+    */pathos/secure/*
+    */pathos/xmlrpc/*
 omit = 
     */tests/*
     */info.py
@@ -11,7 +15,27 @@ branch = true
 # data_file = $TRAVIS_BUILD_DIR/.coverage
 # debug = trace
 
+[paths]
+source =
+    pathos
+    pathos/helpers
+    pathos/secure
+    pathos/xmlrpc
+    */site-packages/pathos
+    */site-packages/pathos/helpers
+    */site-packages/pathos/secure
+    */site-packages/pathos/xmlrpc
+    */site-packages/pathos-*/pathos
+    */site-packages/pathos-*/pathos/helpers
+    */site-packages/pathos-*/pathos/secure
+    */site-packages/pathos-*/pathos/xmlrpc
+
 [report]
+include =
+    */pathos/*
+    */pathos/helpers/*
+    */pathos/secure/*
+    */pathos/xmlrpc/*
 exclude_lines =
     pragma: no cover
     raise NotImplementedError
diff --git a/.readthedocs.yml b/.readthedocs.yml
index 80edfea..1541a7f 100644
--- a/.readthedocs.yml
+++ b/.readthedocs.yml
@@ -8,7 +8,7 @@ sphinx:
 
 # build
 python:
-  version: "3.7"
+  version: "3.8"
   install:
     - method: pip
       path: .
diff --git a/.travis.yml b/.travis.yml
index b36f1b5..c70d49e 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -10,13 +10,13 @@ matrix:
 
         - python: '3.7'
           env:
-            - COVERAGE="true"
 
         - python: '3.8'
           env:
 
         - python: '3.9'
           env:
+            - COVERAGE="true"
 
         - python: '3.10'
           env:
@@ -51,7 +51,7 @@ before_install:
     - if [[ $MULTIPROCESS == "true" ]]; then pip install multiprocess --no-binary multiprocess; fi
 
 install:
-    - python setup.py build && python setup.py install
+    - python -m pip install .
 
 script:
     - for test in tests/__init__.py; do echo $test ; if [[ $COVERAGE == "true" ]]; then coverage run -a $test > /dev/null; else python $test > /dev/null; fi ; done
@@ -59,3 +59,4 @@ script:
 
 after_success:
     - if [[ $COVERAGE == "true" ]]; then bash <(curl -s https://codecov.io/bash); else echo ''; fi
+    - if [[ $COVERAGE == "true" ]]; then coverage report; fi
diff --git a/MANIFEST.in b/MANIFEST.in
index e07adb0..36efe70 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -1,5 +1,7 @@
 include LICENSE
 include README*
+include MANIFEST.in
+include pyproject.toml
 include tox.ini
 recursive-include applications *
 recursive-include docs *
diff --git a/README.md b/README.md
index 014740f..03024c8 100644
--- a/README.md
+++ b/README.md
@@ -124,6 +124,26 @@ You can get the latest development version with all the shiny new features at:
 If you have a new contribution, please submit a pull request.
 
 
+Installation
+------------
+``pathos`` can be installed with ``pip``::
+
+    $ pip install pathos
+
+
+Requirements
+------------
+``pathos`` requires:
+
+* ``python`` (or ``pypy``), **==2.7** or **>=3.7**
+* ``setuptools``, **>=42**
+* ``wheel``, **>=0.1**
+* ``pox``, **>=0.3.0**
+* ``dill``, **>=0.3.4**
+* ``ppft``, **>=1.6.6.4**
+* ``multiprocess``, **>=0.70.12.1**
+
+
 More Information
 ----------------
 Probably the best way to get started is to look at the documentation at
diff --git a/docs/Makefile b/docs/Makefile
index 673bb52..3853655 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -24,4 +24,5 @@ html:
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)
 	-rm -f $(BUILDDIR)/../../scripts/_*py
 	-rm -f $(BUILDDIR)/../../scripts/_*pyc
+	-rm -rf $(BUILDDIR)/../../scripts/__pycache__
 
diff --git a/pathos/__init__.py b/pathos/__init__.py
index f573232..9c7b347 100644
--- a/pathos/__init__.py
+++ b/pathos/__init__.py
@@ -7,21 +7,264 @@
 # License: 3-clause BSD.  The full license text is available at:
 #  - https://github.com/uqfoundation/pathos/blob/master/LICENSE
 
-# get version numbers, license, and long description
-try:
-    from .info import this_version as __version__
-    from .info import readme as __doc__, license as __license__
-except ImportError:
-    msg = """First run 'python setup.py build' to build pathos."""
-    raise ImportError(msg)
-
+# author, version, license, and long description
+__version__ = '0.2.9.dev0'
 __author__ = 'Mike McKerns'
 
 __doc__ = """
-""" + __doc__
+--------------------------------------------------------------------------
+pathos: parallel graph management and execution in heterogeneous computing
+--------------------------------------------------------------------------
+
+About the Pathos Framework
+==========================
+
+``pathos`` is a framework for heterogeneous computing. It provides a consistent
+high-level interface for configuring and launching parallel computations
+across heterogeneous resources. ``pathos`` provides configurable launchers for
+parallel and distributed computing, where each launcher contains the
+syntactic logic to configure and launch jobs in an execution environment.
+Examples of launchers that plug into ``pathos`` are: a queue-less MPI-based
+launcher (in ``pyina``), a ssh-based launcher (in ``pathos``), and a multi-process
+launcher (in ``multiprocess``).
+
+``pathos`` provides a consistent interface for parallel and/or distributed
+versions of ``map`` and ``apply`` for each launcher, thus lowering the barrier
+for users to extend their code to parallel and/or distributed resources.
+The guiding design principle behind ``pathos`` is that ``map`` and ``apply``
+should be drop-in replacements in otherwise serial code, and thus switching
+to one or more of the ``pathos`` launchers is all that is needed to enable
+code to leverage the selected parallel or distributed computing resource.
+This not only greatly reduces the time to convert a code to parallel, but it
+also enables a single code-base to be maintained instead of requiring
+parallel, serial, and distributed versions of a code. ``pathos`` maps can be
+nested, thus hierarchical heterogeneous computing is possible by merely
+selecting the desired hierarchy of ``map`` and ``pipe`` (``apply``) objects.
+
+The ``pathos`` framework is composed of several interoperating packages:
+
+    - ``dill``: a utility to serialize all of python
+    - ``pox``: utilities for filesystem exploration and automated builds
+    - ``klepto``: persistent caching to memory, disk, or database
+    - ``multiprocess``: better multiprocessing and multithreading in python
+    - ``ppft``: distributed and parallel python
+    - ``pyina``: MPI parallel ``map`` and cluster scheduling
+    - ``pathos``: graph management and execution in heterogeneous computing
+
+
+About Pathos
+============
+
+The ``pathos`` package provides a few basic tools to make parallel and
+distributed computing more accessible to the end user. The goal of ``pathos``
+is to enable the user to extend their own code to parallel and distributed
+computing with minimal refactoring.
+
+``pathos`` provides methods for configuring, launching, monitoring, and
+controlling a service on a remote host. One of the most basic features
+of ``pathos`` is the ability to configure and launch a RPC-based service
+on a remote host. ``pathos`` seeds the remote host with the  ``portpicker``
+script, which allows the remote host to inform the localhost of a port
+that is available for communication.
+
+Beyond the ability to establish a RPC service, and then post requests,
+is the ability to launch code in parallel. Unlike parallel computing
+performed at the node level (typically with MPI), ``pathos`` enables the
+user to launch jobs in parallel across heterogeneous distributed resources.
+``pathos`` provides distributed ``map`` and ``pipe`` algorithms, where a mix of
+local processors and distributed workers can be selected.  ``pathos``
+also provides a very basic automated load balancing service, as well as
+the ability for the user to directly select the resources.
+
+The high-level ``pool.map`` interface, yields a ``map`` implementation that
+hides the RPC internals from the user. With ``pool.map``, the user can launch
+their code in parallel, and as a distributed service, using standard python
+and without writing a line of server or parallel batch code.
+
+RPC servers and communication in general is known to be insecure.  However,
+instead of attempting to make the RPC communication itself secure, ``pathos``
+provides the ability to automatically wrap any distributes service or
+communication in a ssh-tunnel. Ssh is a universally trusted method.
+Using ssh-tunnels, ``pathos`` has launched several distributed calculations
+on national lab clusters, and to date has performed test calculations
+that utilize node-to-node communication between several national lab clusters
+and a user's laptop.  ``pathos`` allows the user to configure and launch
+at a very atomistic level, through raw access to ssh and scp. 
+
+``pathos`` is the core of a python framework for heterogeneous computing.
+``pathos`` is in active development, so any user feedback, bug reports, comments,
+or suggestions are highly appreciated.  A list of issues is located at https://github.com/uqfoundation/pathos/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query.
+
+
+Major Features
+==============
+
+``pathos`` provides a configurable distributed parallel ``map`` interface
+to launching RPC service calls, with:
+
+    - a ``map`` interface that meets and extends the python ``map`` standard
+    - the ability to submit service requests to a selection of servers
+    - the ability to tunnel server communications with ssh
+
+The ``pathos`` core is built on low-level communication to remote hosts using
+ssh. The interface to ssh, scp, and ssh-tunneled connections can:
+
+    - configure and launch remote processes with ssh
+    - configure and copy file objects with scp
+    - establish an tear-down a ssh-tunnel
+
+To get up and running quickly, ``pathos`` also provides infrastructure to:
+
+    - easily establish a ssh-tunneled connection to a RPC server
+
+
+Current Release
+===============
+
+The latest released version of ``pathos`` is available from:
+
+    https://pypi.org/project/pathos
+
+``pathos`` is distributed under a 3-clause BSD license.
+
+
+Development Version
+===================
+
+You can get the latest development version with all the shiny new features at:
+
+    https://github.com/uqfoundation
+
+If you have a new contribution, please submit a pull request.
+
+
+Installation
+============
+
+``pathos`` can be installed with ``pip``::
+
+    $ pip install pathos
+
+
+Requirements
+============
+
+``pathos`` requires:
+
+    - ``python`` (or ``pypy``), **==2.7** or **>=3.7**
+    - ``setuptools``, **>=42**
+    - ``wheel``, **>=0.1**
+    - ``pox``, **>=0.3.0**
+    - ``dill``, **>=0.3.4**
+    - ``ppft``, **>=1.6.6.4**
+    - ``multiprocess``, **>=0.70.12.1**
+
+
+More Information
+================
+
+Probably the best way to get started is to look at the documentation at
+http://pathos.rtfd.io. Also see ``pathos.tests`` and ``pathos.examples``
+for a set of scripts that demonstrate the configuration and launching of
+communications with ssh and scp, and demonstrate the configuration and
+execution of jobs in a hierarchical parallel workflow. You can run the test
+suite with ``python -m pathos.tests``. Tunnels and other connections to
+remote servers can be established with the ``pathos_connect`` script (or with
+``python -m pathos``). See ``pathos_connect --help`` for more information.
+``pathos`` also provides a ``portpicker`` script to select an open port
+(also available with ``python -m pathos.portpicker``). The source code is 
+generally well documented, so further questions may be resolved by inspecting
+the code itself.  Please feel free to submit a ticket on github, or ask a
+question on stackoverflow (**@Mike McKerns**).
+If you would like to share how you use ``pathos`` in your work, please send
+an email (to **mmckerns at uqfoundation dot org**).
+
+Important classes and functions are found here:
+
+    - ``pathos.abstract_launcher``           [the worker pool API definition]
+    - ``pathos.pools``                       [all of the pathos worker pools]
+    - ``pathos.core``                        [the high-level command interface] 
+    - ``pathos.hosts``                       [the hostname registry interface] 
+    - ``pathos.serial.SerialPool``           [the serial python worker pool]
+    - ``pathos.parallel.ParallelPool``       [the parallelpython worker pool]
+    - ``pathos.multiprocessing.ProcessPool`` [the multiprocessing worker pool]
+    - ``pathos.threading.ThreadPool``        [the multithreading worker pool]
+    - ``pathos.connection.Pipe``             [the launcher base class]
+    - ``pathos.secure.Pipe``                 [the secure launcher base class]
+    - ``pathos.secure.Copier``               [the secure copier  base class]
+    - ``pathos.secure.Tunnel``               [the secure tunnel base class]
+    - ``pathos.selector.Selector``           [the selector base class]
+    - ``pathos.server.Server``               [the server base class]
+    - ``pathos.profile``                     [profiling in threads and processes]
+
+``pathos`` also provides two convenience scripts that are used to establish
+secure distributed connections. These scripts are installed to a directory
+on the user's ``$PATH``, and thus can be run from anywhere:
+
+    - ``portpicker``                         [get the portnumber of an open port]
+    - ``pathos_connect``                     [establish tunnel and/or RPC server]
+
+Typing ``--help`` as an argument to any of the above scripts will print out an
+instructive help message.
+
+
+Citation
+========
+
+If you use ``pathos`` to do research that leads to publication, we ask that you
+acknowledge use of ``pathos`` by citing the following in your publication::
+
+    M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,
+    "Building a framework for predictive science", Proceedings of
+    the 10th Python in Science Conference, 2011;
+    http://arxiv.org/pdf/1202.1056
+
+    Michael McKerns and Michael Aivazis,
+    "pathos: a framework for heterogeneous computing", 2010- ;
+    https://uqfoundation.github.io/project/pathos
+
+Please see https://uqfoundation.github.io/project/pathos or
+http://arxiv.org/pdf/1202.1056 for further information.
+
+"""
 
 __license__ = """
-""" + __license__
+Copyright (c) 2004-2016 California Institute of Technology.
+Copyright (c) 2016-2022 The Uncertainty Quantification Foundation.
+All rights reserved.
+
+This software is available subject to the conditions and terms laid
+out below. By downloading and using this software you are agreeing
+to the following conditions.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met::
+
+    - Redistribution of source code must retain the above copyright
+      notice, this list of conditions and the following disclaimer.
+
+    - Redistribution in binary form must reproduce the above copyright
+      notice, this list of conditions and the following disclaimer in the
+      documentations and/or other materials provided with the distribution.
+
+    - Neither the names of the copyright holders nor the names of any of
+      the contributors may be used to endorse or promote products derived
+      from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
+OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
+OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+"""
 
 # logger
 def logger(level=None, handler=None, **kwds):
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..50804a4
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,7 @@
+[build-system]
+# Further build requirements come from setup.py via the PEP 517 interface
+requires = [
+    "setuptools>=42",
+    "wheel",
+]
+build-backend = "setuptools.build_meta"
diff --git a/setup.cfg b/setup.cfg
index 46edaab..01378d8 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,4 +1,7 @@
 [egg_info]
 tag_build = .dev0
-#tag_svn_revision = yes
-#tag_date = yes
+
+[bdist_wheel]
+#universal = 1
+#python-tag = py3
+#plat-name = manylinux1_x86_64
diff --git a/setup.py b/setup.py
index eef2002..c709d19 100644
--- a/setup.py
+++ b/setup.py
@@ -17,330 +17,68 @@
 if unsupported:
     raise ValueError(unsupported)
 
-# set version numbers
-stable_version = '0.2.8'
-target_version = '0.2.9'
-is_release = stable_version == target_version
-
-# check if easy_install is available
+# get distribution meta info
+here = os.path.abspath(os.path.dirname(__file__))
+meta_fh = open(os.path.join(here, 'pathos/__init__.py'))
+try:
+    meta = {}
+    for line in meta_fh:
+        if line.startswith('__version__'):
+            VERSION = line.split()[-1].strip("'").strip('"')
+            break
+    meta['VERSION'] = VERSION
+    for line in meta_fh:
+        if line.startswith('__author__'):
+            AUTHOR = line.split(' = ')[-1].strip().strip("'").strip('"')
+            break
+    meta['AUTHOR'] = AUTHOR
+    LONG_DOC = ""
+    DOC_STOP = "FAKE_STOP_12345"
+    for line in meta_fh:
+        if LONG_DOC:
+            if line.startswith(DOC_STOP):
+                LONG_DOC += DOC_STOP.rstrip()
+                break
+            else:
+                LONG_DOC += line
+        elif line.startswith('__doc__'):
+            DOC_STOP = line.split(' = ')[-1]
+            LONG_DOC += DOC_STOP
+    meta['LONG_DOC'] = LONG_DOC
+finally:
+    meta_fh.close()
+
+# get version numbers, long_description, etc
+AUTHOR = meta['AUTHOR']
+VERSION = meta['VERSION']
+LONG_DOC = meta['LONG_DOC'] #FIXME: near-duplicate of README.md
+#LICENSE = meta['LICENSE'] #FIXME: duplicate of LICENSE
+AUTHOR_EMAIL = 'mmckerns@uqfoundation.org'
+
+# check if setuptools is available
 try:
-#   import __force_distutils__ #XXX: uncomment to force use of distutills
     from setuptools import setup
+    from setuptools.dist import Distribution
     has_setuptools = True
 except ImportError:
     from distutils.core import setup
+    Distribution = object
     has_setuptools = False
 
-# generate version number
-if os.path.exists('pathos/info.py'):
-    # is a source distribution, so use existing version
-    os.chdir('pathos')
-    with open('info.py','r') as f:
-        f.readline() # header
-        this_version = f.readline().split()[-1].strip("'")
-    os.chdir('..')
-elif stable_version == target_version:
-    # we are building a stable release
-    this_version = target_version
-else:
-    # we are building a distribution
-    this_version = target_version + '.dev0'
-    if is_release:
-      from datetime import date
-      today = "".join(date.isoformat(date.today()).split('-'))
-      this_version += "-" + today
-
-# get the license info
-with open('LICENSE') as file:
-    license_text = file.read()
-
-# generate the readme text
-long_description = \
-"""--------------------------------------------------------------------------
-pathos: parallel graph management and execution in heterogeneous computing
---------------------------------------------------------------------------
-
-About the Pathos Framework
-==========================
-
-``pathos`` is a framework for heterogeneous computing. It provides a consistent
-high-level interface for configuring and launching parallel computations
-across heterogeneous resources. ``pathos`` provides configurable launchers for
-parallel and distributed computing, where each launcher contains the
-syntactic logic to configure and launch jobs in an execution environment.
-Examples of launchers that plug into ``pathos`` are: a queue-less MPI-based
-launcher (in ``pyina``), a ssh-based launcher (in ``pathos``), and a multi-process
-launcher (in ``multiprocess``).
-
-``pathos`` provides a consistent interface for parallel and/or distributed
-versions of ``map`` and ``apply`` for each launcher, thus lowering the barrier
-for users to extend their code to parallel and/or distributed resources.
-The guiding design principle behind ``pathos`` is that ``map`` and ``apply``
-should be drop-in replacements in otherwise serial code, and thus switching
-to one or more of the ``pathos`` launchers is all that is needed to enable
-code to leverage the selected parallel or distributed computing resource.
-This not only greatly reduces the time to convert a code to parallel, but it
-also enables a single code-base to be maintained instead of requiring
-parallel, serial, and distributed versions of a code. ``pathos`` maps can be
-nested, thus hierarchical heterogeneous computing is possible by merely
-selecting the desired hierarchy of ``map`` and ``pipe`` (``apply``) objects.
-
-The ``pathos`` framework is composed of several interoperating packages:
-
-    - ``dill``: a utility to serialize all of python
-    - ``pox``: utilities for filesystem exploration and automated builds
-    - ``klepto``: persistent caching to memory, disk, or database
-    - ``multiprocess``: better multiprocessing and multithreading in python
-    - ``ppft``: distributed and parallel python
-    - ``pyina``: MPI parallel ``map`` and cluster scheduling
-    - ``pathos``: graph management and execution in heterogeneous computing
-
-
-About Pathos
-============
-
-The ``pathos`` package provides a few basic tools to make parallel and
-distributed computing more accessible to the end user. The goal of ``pathos``
-is to enable the user to extend their own code to parallel and distributed
-computing with minimal refactoring.
-
-``pathos`` provides methods for configuring, launching, monitoring, and
-controlling a service on a remote host. One of the most basic features
-of ``pathos`` is the ability to configure and launch a RPC-based service
-on a remote host. ``pathos`` seeds the remote host with the  ``portpicker``
-script, which allows the remote host to inform the localhost of a port
-that is available for communication.
-
-Beyond the ability to establish a RPC service, and then post requests,
-is the ability to launch code in parallel. Unlike parallel computing
-performed at the node level (typically with MPI), ``pathos`` enables the
-user to launch jobs in parallel across heterogeneous distributed resources.
-``pathos`` provides distributed ``map`` and ``pipe`` algorithms, where a mix of
-local processors and distributed workers can be selected.  ``pathos``
-also provides a very basic automated load balancing service, as well as
-the ability for the user to directly select the resources.
-
-The high-level ``pool.map`` interface, yields a ``map`` implementation that
-hides the RPC internals from the user. With ``pool.map``, the user can launch
-their code in parallel, and as a distributed service, using standard python
-and without writing a line of server or parallel batch code.
-
-RPC servers and communication in general is known to be insecure.  However,
-instead of attempting to make the RPC communication itself secure, ``pathos``
-provides the ability to automatically wrap any distributes service or
-communication in a ssh-tunnel. Ssh is a universally trusted method.
-Using ssh-tunnels, ``pathos`` has launched several distributed calculations
-on national lab clusters, and to date has performed test calculations
-that utilize node-to-node communication between several national lab clusters
-and a user's laptop.  ``pathos`` allows the user to configure and launch
-at a very atomistic level, through raw access to ssh and scp. 
-
-``pathos`` is the core of a python framework for heterogeneous computing.
-``pathos`` is in active development, so any user feedback, bug reports, comments,
-or suggestions are highly appreciated.  A list of issues is located at https://github.com/uqfoundation/pathos/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query.
-
-
-Major Features
-==============
-
-``pathos`` provides a configurable distributed parallel ``map`` interface
-to launching RPC service calls, with:
-
-    - a ``map`` interface that meets and extends the python ``map`` standard
-    - the ability to submit service requests to a selection of servers
-    - the ability to tunnel server communications with ssh
-
-The ``pathos`` core is built on low-level communication to remote hosts using
-ssh. The interface to ssh, scp, and ssh-tunneled connections can:
-
-    - configure and launch remote processes with ssh
-    - configure and copy file objects with scp
-    - establish an tear-down a ssh-tunnel
-
-To get up and running quickly, ``pathos`` also provides infrastructure to:
-
-    - easily establish a ssh-tunneled connection to a RPC server
-
-
-Current Release
-===============
-
-This documentation is for version ``pathos-%(thisver)s``.
-
-The latest released version of ``pathos`` is available from:
-
-    https://pypi.org/project/pathos
-
-``pathos`` is distributed under a 3-clause BSD license.
-
-    >>> import pathos
-    >>> pathos.license()
-
-
-Development Version
-===================
-
-You can get the latest development version with all the shiny new features at:
-
-    https://github.com/uqfoundation
-
-If you have a new contribution, please submit a pull request.
-
-
-Installation
-============
-
-``pathos`` is packaged to install from source, so you must
-download the tarball, unzip, and run the installer::
-
-    [download]
-    $ tar -xvzf pathos-%(relver)s.tar.gz
-    $ cd pathos-%(relver)s
-    $ python setup py build
-    $ python setup py install
-
-You will be warned of any missing dependencies and/or settings
-after you run the "build" step above.  ``pathos`` depends on ``dill`` and
-``pox``, each of which are essentially subpackages of ``pathos`` but are
-released independently. ``pathos`` also depends on ``multiprocess`` and
-``ppft``.  You must install all of the ``pathos`` framework packages for
-``pathos`` to provide the full functionality for heterogeneous computing. 
-
-Alternately, ``pathos`` can be installed with ``pip`` or ``easy_install``::
-
-    $ pip install pathos
-
-
-Requirements
-============
-
-``pathos`` requires:
-
-    - ``python`` (or ``pypy``), **version == 2.7** or **version >= 3.7**
-    - ``dill``, **version >= 0.3.4**
-    - ``pox``, **version >= 0.3.0**
-    - ``ppft``, **version >= 1.6.6.4**
-    - ``multiprocess``, **version >= 0.70.12.1**
-
-Optional requirements:
-
-    - ``setuptools``, **version >= 40.6.0**
-    - ``pyina``, **version >= 0.2.5**
-    - ``rpyc``, **version >= 3.0.6**
-    - ``mystic``, **version >= 0.3.8**
-
-
-More Information
-================
-
-Probably the best way to get started is to look at the documentation at
-http://pathos.rtfd.io. Also see ``pathos.tests`` and ``pathos.examples``
-for a set of scripts that demonstrate the configuration and launching of
-communications with ssh and scp, and demonstrate the configuration and
-execution of jobs in a hierarchical parallel workflow. You can run the test
-suite with ``python -m pathos.tests``. Tunnels and other connections to
-remote servers can be established with the ``pathos_connect`` script (or with
-``python -m pathos``). See ``pathos_connect --help`` for more information.
-``pathos`` also provides a ``portpicker`` script to select an open port
-(also available with ``python -m pathos.portpicker``). The source code is 
-generally well documented, so further questions may be resolved by inspecting
-the code itself.  Please feel free to submit a ticket on github, or ask a
-question on stackoverflow (**@Mike McKerns**).
-If you would like to share how you use ``pathos`` in your work, please send
-an email (to **mmckerns at uqfoundation dot org**).
-
-Important classes and functions are found here:
-
-    - ``pathos.abstract_launcher``           [the worker pool API definition]
-    - ``pathos.pools``                       [all of the pathos worker pools]
-    - ``pathos.core``                        [the high-level command interface] 
-    - ``pathos.hosts``                       [the hostname registry interface] 
-    - ``pathos.serial.SerialPool``           [the serial python worker pool]
-    - ``pathos.parallel.ParallelPool``       [the parallelpython worker pool]
-    - ``pathos.multiprocessing.ProcessPool`` [the multiprocessing worker pool]
-    - ``pathos.threading.ThreadPool``        [the multithreading worker pool]
-    - ``pathos.connection.Pipe``             [the launcher base class]
-    - ``pathos.secure.Pipe``                 [the secure launcher base class]
-    - ``pathos.secure.Copier``               [the secure copier  base class]
-    - ``pathos.secure.Tunnel``               [the secure tunnel base class]
-    - ``pathos.selector.Selector``           [the selector base class]
-    - ``pathos.server.Server``               [the server base class]
-    - ``pathos.profile``                     [profiling in threads and processes]
-
-``pathos`` also provides two convenience scripts that are used to establish
-secure distributed connections. These scripts are installed to a directory
-on the user's ``$PATH``, and thus can be run from anywhere:
-
-    - ``portpicker``                         [get the portnumber of an open port]
-    - ``pathos_connect``                     [establish tunnel and/or RPC server]
-
-Typing ``--help`` as an argument to any of the above scripts will print out an
-instructive help message.
-
-
-Citation
-========
-
-If you use ``pathos`` to do research that leads to publication, we ask that you
-acknowledge use of ``pathos`` by citing the following in your publication::
-
-    M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,
-    "Building a framework for predictive science", Proceedings of
-    the 10th Python in Science Conference, 2011;
-    http://arxiv.org/pdf/1202.1056
-
-    Michael McKerns and Michael Aivazis,
-    "pathos: a framework for heterogeneous computing", 2010- ;
-    https://uqfoundation.github.io/project/pathos
-
-Please see https://uqfoundation.github.io/project/pathos or
-http://arxiv.org/pdf/1202.1056 for further information.
-
-""" % {'relver' : stable_version, 'thisver' : this_version}
-
-# write readme file
-with open('README', 'w') as file:
-    file.write(long_description)
-
-# generate 'info' file contents
-def write_info_py(filename='pathos/info.py'):
-    contents = """# THIS FILE GENERATED FROM SETUP.PY
-this_version = '%(this_version)s'
-stable_version = '%(stable_version)s'
-readme = '''%(long_description)s'''
-license = '''%(license_text)s'''
-"""
-    with open(filename, 'w') as file:
-        file.write(contents % {'this_version' : this_version,
-                               'stable_version' : stable_version,
-                               'long_description' : long_description,
-                               'license_text' : license_text })
-    return
-
-# write info file
-write_info_py()
-
-# platform-specific instructions
-from sys import platform, version_info
-if platform[:3] == 'win':
-    pass
-else: #platform = linux or mac
-    if platform[:6] == 'darwin':
-        pass
-    pass
-
 # build the 'setup' call
-setup_code = """
-setup(name="pathos",
-    version='%s',
+setup_kwds = dict(
+    name="pathos",
+    version=VERSION,
     description="parallel graph management and execution in heterogeneous computing",
-    long_description = '''%s''',
-    author = 'Mike McKerns',
-    maintainer = 'Mike McKerns',
+    long_description = LONG_DOC,
+    author = AUTHOR,
+    author_email = AUTHOR_EMAIL,
+    maintainer = AUTHOR,
+    maintainer_email = AUTHOR_EMAIL,
     license = '3-clause BSD',
     platforms = ['Linux', 'Windows', 'Mac'],
     url = 'https://github.com/uqfoundation/pathos',
-    download_url = 'https://github.com/uqfoundation/pathos/releases/download/pathos-%s/pathos-%s.tar.gz',
+    download_url = 'https://pypi.org/project/pathos/#files',
     python_requires = '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*',
     classifiers = ['Development Status :: 5 - Production/Stable',
                    'Intended Audience :: Developers',
@@ -356,7 +94,6 @@ def write_info_py(filename='pathos/info.py'):
                    'Programming Language :: Python :: Implementation :: PyPy',
                    'Topic :: Scientific/Engineering',
                    'Topic :: Software Development'],
-
     packages=['pathos','pathos.tests',\
               'pathos.helpers','pathos.secure','pathos.xmlrpc'],
     package_dir={'pathos':'pathos', 'pathos.tests':'tests', \
@@ -364,82 +101,58 @@ def write_info_py(filename='pathos/info.py'):
                  'pathos.secure':'pathos/secure', \
                  'pathos.xmlrpc':'pathos/xmlrpc', \
                 },
-""" % (target_version, long_description, stable_version, stable_version)
-
-'''
-# check for 'processing'
-try: #NOTE: odd... if processing is installed, *don't* install multiprocess
-    from processing import __version__ as processing_version
-    if processing_version >= '0.52-pathos': # NOTE: modified redistribution
-        processing_version = '=='+processing_version
-        mp_version = ''
-    else: raise AttributeError('multiprocess')
-except Exception:
-    mp_version = '>=0.70.12.1' # 0.70a1 py25-py33, 0.52 on py25, None on py34
-    processing_version = ''
-'''
-
+    scripts=['scripts/pathos_connect', 'scripts/portpicker'],
+)
+
+# force python-, abi-, and platform-specific naming of bdist_wheel
+class BinaryDistribution(Distribution):
+    """Distribution which forces a binary package with platform name"""
+    def has_ext_modules(foo):
+        return True
+
+# define dependencies
+ppft_version = 'ppft>=1.6.6.4'
+dill_version = 'dill>=0.3.4'
+pox_version = 'pox>=0.3.0'
+mp_version = 'multiprocess>=0.70.12.1'
+pyina_version = 'pyina>=0.2.5'
+mystic_version = 'mystic>=0.3.8'
 # add dependencies
-ppft_version = '>=1.6.6.4'
-dill_version = '>=0.3.4'
-pox_version = '>=0.3.0'
-mp_version = '>=0.70.12.1' if version_info >= (2,6) else '>=0.52.0'
-pyina_version = '>=0.2.5'
-rpyc_version = '>=3.0.6'
-deps = [ppft_version, dill_version, pox_version]
-if mp_version:
-    deps = tuple(deps + ["'multiprocess%s']," % mp_version])
-else:
-    deps = tuple(deps + ["],"])
+depend = [ppft_version, dill_version, pox_version, mp_version]
+extras = {'examples': [mystic_version, pyina_version]}
+# update setup kwds
 if has_setuptools:
-    setup_code += """
-        zip_safe = False,
-        install_requires = ['ppft%s','dill%s','pox%s',%s
-""" % deps
+    setup_kwds.update(
+        zip_safe=False,
+        # distclass=BinaryDistribution,
+        install_requires=depend,
+        # extras_require=extras,
+    )
 
-# add the scripts, and close 'setup' call
-setup_code += """
-    scripts=['scripts/pathos_connect',
-             'scripts/portpicker'])
-"""
-
-# exec the 'setup' code
-exec(setup_code)
+# call setup
+setup(**setup_kwds)
 
 # if dependencies are missing, print a warning
 try:
     import ppft
     import dill
     import pox
-    if version_info >= (2,6):
-        import multiprocess
-    else:
-        import processing
-   #try:
-   #    import processing
-   #except ImportError:
-   #    import multiprocess
-   #    if getattr(multiprocess, '__version__', '0.70a1') == '0.70a1':
-   #        raise ImportError
+    import multiprocess
+    #import mystic
+    #import pyina
 except ImportError:
     print("\n***********************************************************")
     print("WARNING: One of the following dependencies is unresolved:")
-    print("    ppft %s" % ppft_version)
-    print("    dill %s" % dill_version)
-    print("    pox %s" % pox_version)
-    print("    multiprocess %s" % mp_version)
+    print("    %s" % ppft_version)
+    print("    %s" % dill_version)
+    print("    %s" % pox_version)
+    print("    %s" % mp_version)
+    #print("    %s (optional)" % mystic_version)
+    #print("    %s (optional)" % pyina_version)
     print("***********************************************************\n")
 
-#    print("""
-#If '%s' is installed, '%s' will be regarded as optional, and thus will
-#not be installed.  Note that '%s' is not available through a standard
-#install, however it may be downloaded from:
-#  http://dev.danse.us/packages/
-#or found in the "external" directory included in the pathos source distribution
-#""" % ('processing','multiprocess','processing'))
-
 
 if __name__=='__main__':
     pass
 
-# End of file
+# end of file
diff --git a/tox.ini b/tox.ini
index 6bfc586..b6f4f32 100644
--- a/tox.ini
+++ b/tox.ini
@@ -25,7 +25,6 @@ deps =
 whitelist_externals =
     bash
 commands =
-    {envpython} setup.py build
-    {envpython} setup.py install
+    {envpython} -m pip install .
     bash -c "failed=0; for test in tests/*.py; do echo $test; \
              {envpython} $test || failed=1; done; exit $failed"


Test patch:
diff --git a/tests/test_join.py b/tests/test_join.py
index e0a588d..b1d90a0 100644
--- a/tests/test_join.py
+++ b/tests/test_join.py
@@ -14,8 +14,8 @@
 
 import sys
 import dill
-P33 = (sys.hexversion >= 0x30300f0) #FIXME: below oddity due to travis
-P37 = (sys.hexversion >= 0x30700f0) and sys.version_info[:3] != (3,7,7)
+P33 = (sys.hexversion >= 0x30300f0)
+P37 = (sys.hexversion >= 0x30700f0)
 PYPY38 = (sys.hexversion >= 0x30800f0) and dill._dill.IS_PYPY
 PoolClosedError = ValueError if P33 else AssertionError
 PoolRunningError = ValueError if P37 else AssertionError