Repository: pantsbuild/pants

Issue:
setup-py loads __init__.py with errors
Hi,

I'm evaluating pants and it's looking awesome so far! However it seems I run into a possible issue. I'm using python 3.6 and noticed that when running `./pants setup-py` I get an error message like `Exception message: invalid syntax (__init__.py, line 5)`. I'm using pants 1.8.0. 

While investigating, I noticed that if I disable some python 3 features like type hinting and f-strings I don't get the error. Given that both these features were introduced in later python versions, is it possible that pants uses system interpreter (or a wrong interpreter) for loading `__init__.py`?

I created a repo to reproduce the problem: https://github.com/ifoukarakis/pants-typehints. 
Python 3 - setup_py swaps `setupwheels` for `wheel`
## Issue
`test_setup_py.py`'s `test_setuptools_version` fails  because it swaps `setupwheels` with `wheel`:
```python
                                 # The 1st two elements of the sys.path should be our custom SetupPyRunner Installer's
                                 # setuptools and wheel mixins, which should match the setuptools and wheel versions
                                 # specified by the PythonSetup subsystem.
                                 init_subsystem(PythonSetup)
                                 python_setup = PythonSetup.global_instance()
                     >           assert_extra('setuptools', python_setup.setuptools_version)

                     .pants.d/pyprep/sources/6c832546dc456d2f1cf12129c6c77494ae93ba24/pants_test/backend/python/tasks/test_setup_py.py:133:
                     _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
                     .pants.d/pyprep/sources/6c832546dc456d2f1cf12129c6c77494ae93ba24/pants_test/backend/python/tasks/test_setup_py.py:125: in assert_extra
                         self.assertEqual(name, package.name)
                     E   AssertionError: 'setuptools' != 'wheel'
                     E   - setuptools
                     E   + wheel
```

Note that the following assertion is valid, though:
```python
assert_extra('wheel', python_setup.wheel_version)
```

## Reproduce
1. Apply this diff to avoid running every unit test in the folder:
```
diff --git a/tests/python/pants_test/backend/python/tasks/BUILD b/tests/python/pants_test/backend/python/tasks/BUILD
index 4325fa90b..b453222c8 100644
--- a/tests/python/pants_test/backend/python/tasks/BUILD
+++ b/tests/python/pants_test/backend/python/tasks/BUILD
@@ -50,7 +50,7 @@ python_tests(
 )

 python_tests(
-  sources=globs('test_*.py', exclude=([globs('*_integration.py')] + python_native_code_test_files)),
+  sources='test_setup_py.py',
   dependencies=[
     ':python_task_test_base',
     '3rdparty/python/twitter/commons:twitter.common.collections',
```
2. `./pants clean-all`
3. `./pants --python-setup-interpreter-constraints='["CPython>=3.5"]' test tests/python/pants_test/backend/python/tasks:tasks`


Discussion:
Turning on stacktraces:
```
jsirois@gill ~/dev/ifoukarakis/pants-typehints (master) $ ./pants --print-exception-stacktrace setup-py src/python/package:my-package

09:51:16 00:00 [main]
               (To run a reporting server: ./pants server)
09:51:16 00:00   [setup]
09:51:16 00:00     [parse]
               Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> pyprep -> setup-py
09:51:16 00:00   [bootstrap]
09:51:16 00:00     [substitute-aliased-targets]
09:51:16 00:00     [jar-dependency-management]
09:51:17 00:01     [bootstrap-jvm-tools]
09:51:17 00:01     [provide-tools-jar]
09:51:17 00:01   [imports]
09:51:17 00:01     [ivy-imports]
09:51:17 00:01   [unpack-jars]
09:51:17 00:01     [unpack-jars]
09:51:17 00:01   [deferred-sources]
09:51:17 00:01     [deferred-sources]
09:51:17 00:01   [gen]
09:51:17 00:01     [antlr-java]
09:51:17 00:01     [antlr-py]
09:51:17 00:01     [jaxb]
09:51:17 00:01     [protoc]
09:51:17 00:01     [ragel]
09:51:17 00:01     [thrift-java]
09:51:17 00:01     [thrift-py]
09:51:17 00:01     [wire]
09:51:17 00:01   [pyprep]
09:51:17 00:01     [interpreter]
09:51:17 00:01     [build-local-dists]
09:51:17 00:01     [requirements]
09:51:17 00:01     [sources]
09:51:17 00:01   [setup-py]
09:51:17 00:01     [setup-py]
                   Creating setup.py project for PythonLibrary(BuildFileAddress(src/python/package/BUILD, my-package))
               Waiting for background workers to finish.
09:51:17 00:01   [complete]
               FAILURE
Exception caught: (<type 'exceptions.SyntaxError'>)
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/bin/pants", line 11, in <module>
    sys.exit(main())
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 69, in main
    PantsLoader.run()
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 65, in run
    cls.load_and_execute(entrypoint)
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 58, in load_and_execute
    entrypoint_main()
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 39, in main
    PantsRunner(exiter, start_time=start_time).run()
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 53, in run
    return runner.run()
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 46, in run
    self._run()
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 94, in _run
    goal_runner_result = goal_runner.run()
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 222, in run
    result = self._execute_engine()
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 210, in _execute_engine
    result = engine.execute(self._context, self._goals)
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/engine/legacy_engine.py", line 26, in execute
    self.attempt(context, goals)
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/engine/round_engine.py", line 233, in attempt
    goal_executor.attempt(explain)
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/engine/round_engine.py", line 49, in attempt
    task.execute()
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/backend/python/tasks/setup_py.py", line 644, in execute
    create(exported_python_target)
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/backend/python/tasks/setup_py.py", line 636, in create
    setup_dir, dependencies = self.create_setup_py(subject, dist_dir)
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/backend/python/tasks/setup_py.py", line 605, in create_setup_py
    self.write_setup(target, reduced_deps, chroot)
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/backend/python/tasks/setup_py.py", line 539, in write_setup
    packages, namespace_packages, resources = self.find_packages(chroot, self.context.log)
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/backend/python/tasks/setup_py.py", line 444, in find_packages
    if cls.declares_namespace_package(real_filename):
  File "/home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0/lib/python2.7/site-packages/pants/backend/python/tasks/setup_py.py", line 397, in declares_namespace_package
    init_py = ast.parse(fp.read(), filename)
  File "/usr/lib64/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)

Exception message: invalid syntax (__init__.py, line 5)
```
Does this repro in Pants source? @onioni discovered an internal bug that had this same behavior
It does repro. I have a fix here: https://github.com/pantsbuild/pants/pull/6439/files#diff-bba62d80c6a1d7cc16929787f9e166f1R129

I did not see this issue, but now that I have I'll break out that fix.

Resolving patch:
diff --git a/src/python/pants/backend/python/interpreter_cache.py b/src/python/pants/backend/python/interpreter_cache.py
--- a/src/python/pants/backend/python/interpreter_cache.py
+++ b/src/python/pants/backend/python/interpreter_cache.py
@@ -36,13 +36,13 @@ class UnsatisfiableInterpreterConstraintsError(TaskError):
     """Indicates a python interpreter matching given constraints could not be located."""
 
   @staticmethod
-  def _matches(interpreter, filters):
-    return any(interpreter.identity.matches(filt) for filt in filters)
+  def _matches(interpreter, filters=()):
+    return not filters or any(interpreter.identity.matches(filt) for filt in filters)
 
   @classmethod
-  def _matching(cls, interpreters, filters):
+  def _matching(cls, interpreters, filters=()):
     for interpreter in interpreters:
-      if cls._matches(interpreter, filters):
+      if cls._matches(interpreter, filters=filters):
         yield interpreter
 
   @classmethod
@@ -100,7 +100,7 @@ def select_interpreter_for_targets(self, targets):
     # Return the lowest compatible interpreter.
     return min(allowed_interpreters)
 
-  def _interpreter_from_path(self, path, filters):
+  def _interpreter_from_path(self, path, filters=()):
     try:
       executable = os.readlink(os.path.join(path, 'python'))
     except OSError:
@@ -116,29 +116,29 @@ def _setup_interpreter(self, interpreter, cache_target_path):
       os.symlink(interpreter.binary, os.path.join(safe_path, 'python'))
       return self._resolve(interpreter, safe_path)
 
-  def _setup_cached(self, filters):
+  def _setup_cached(self, filters=()):
     """Find all currently-cached interpreters."""
     for interpreter_dir in os.listdir(self._cache_dir):
-      if os.path.isdir(interpreter_dir):
-        path = os.path.join(self._cache_dir, interpreter_dir)
-        pi = self._interpreter_from_path(path, filters)
+      path = os.path.join(self._cache_dir, interpreter_dir)
+      if os.path.isdir(path):
+        pi = self._interpreter_from_path(path, filters=filters)
         if pi:
           self._logger('Detected interpreter {}: {}'.format(pi.binary, str(pi.identity)))
           yield pi
 
-  def _setup_paths(self, paths, filters):
+  def _setup_paths(self, paths, filters=()):
     """Find interpreters under paths, and cache them."""
-    for interpreter in self._matching(PythonInterpreter.all(paths), filters):
+    for interpreter in self._matching(PythonInterpreter.all(paths), filters=filters):
       identity_str = str(interpreter.identity)
       cache_path = os.path.join(self._cache_dir, identity_str)
-      pi = self._interpreter_from_path(cache_path, filters)
+      pi = self._interpreter_from_path(cache_path, filters=filters)
       if pi is None:
         self._setup_interpreter(interpreter, cache_path)
-        pi = self._interpreter_from_path(cache_path, filters)
+        pi = self._interpreter_from_path(cache_path, filters=filters)
       if pi:
         yield pi
 
-  def setup(self, paths=(), filters=(b'',)):
+  def setup(self, paths=(), filters=()):
     """Sets up a cache of python interpreters.
 
     :param paths: The paths to search for a python interpreter; the system ``PATH`` by default.
@@ -165,14 +165,14 @@ def unsatisfied_filters(interpreters):
 
     interpreters = []
     with OwnerPrintingInterProcessFileLock(path=os.path.join(self._cache_dir, '.file_lock')):
-      interpreters.extend(self._setup_cached(filters))
+      interpreters.extend(self._setup_cached(filters=filters))
       if unsatisfied_filters(interpreters):
-        interpreters.extend(self._setup_paths(setup_paths, filters))
+        interpreters.extend(self._setup_paths(setup_paths, filters=filters))
 
     for filt in unsatisfied_filters(interpreters):
       self._logger('No valid interpreters found for {}!'.format(filt))
 
-    matches = list(self._matching(interpreters, filters))
+    matches = list(self._matching(interpreters, filters=filters))
     if len(matches) == 0:
       self._logger('Found no valid interpreters!')
 
diff --git a/src/python/pants/backend/python/tasks/setup_py.py b/src/python/pants/backend/python/tasks/setup_py.py
--- a/src/python/pants/backend/python/tasks/setup_py.py
+++ b/src/python/pants/backend/python/tasks/setup_py.py
@@ -4,18 +4,23 @@
 
 from __future__ import absolute_import, division, print_function, unicode_literals
 
-import ast
+import inspect
+import io
 import itertools
 import os
-import pprint
 import shutil
+import textwrap
 from abc import abstractmethod
-from builtins import map, object, str, zip
+from builtins import bytes, map, object, str, zip
 from collections import OrderedDict, defaultdict
 
-from pex.compatibility import string, to_bytes
+from future.moves import collections
+from future.utils import PY2
 from pex.installer import InstallerBase, Packager
 from pex.interpreter import PythonInterpreter
+from pex.pex import PEX
+from pex.pex_builder import PEXBuilder
+from pex.pex_info import PexInfo
 from twitter.common.collections import OrderedSet
 from twitter.common.dirutil.chroot import Chroot
 
@@ -26,14 +31,17 @@
 from pants.base.build_environment import get_buildroot
 from pants.base.exceptions import TargetDefinitionException, TaskError
 from pants.base.specs import SiblingAddresses
+from pants.base.workunit import WorkUnitLabel
 from pants.build_graph.address_lookup_error import AddressLookupError
 from pants.build_graph.build_graph import sort_targets
 from pants.build_graph.resources import Resources
 from pants.task.task import Task
-from pants.util.dirutil import safe_rmtree, safe_walk
+from pants.util.contextutil import temporary_file
+from pants.util.dirutil import safe_concurrent_creation, safe_rmtree, safe_walk
 from pants.util.memo import memoized_property
 from pants.util.meta import AbstractClass
-from pants.util.strutil import safe_shlex_split
+from pants.util.process_handler import subprocess
+from pants.util.strutil import ensure_binary, ensure_text, safe_shlex_split
 
 
 SETUP_BOILERPLATE = """
@@ -42,12 +50,65 @@
 
 from setuptools import setup
 
-setup(**
-{setup_dict}
-)
+setup(**{setup_dict})
 """
 
 
+# Distutils does not support unicode strings in setup.py, so we must explicitly convert to binary
+# strings as pants uses unicode_literals. A natural and prior technique was to use `pprint.pformat`,
+# but that embeds u's in the string itself during conversion. For that reason we roll out own
+# literal pretty-printer here.
+#
+# For more information, see http://bugs.python.org/issue13943
+def distutils_repr(obj):
+  output = io.StringIO()
+  linesep = os.linesep
+
+  def _write(data):
+    output.write(ensure_text(data))
+
+  def _write_repr(o, indent=False, level=0):
+    pad = ' ' * 4 * level
+    if indent:
+      _write(pad)
+    level += 1
+
+    if isinstance(o, (bytes, str)):
+      # The py2 repr of str (unicode) is `u'...'` and we don't want the `u` prefix; likewise,
+      # the py3 repr of bytes is `b'...'` and we don't want the `b` prefix so we hand-roll a
+      # repr here.
+      if linesep in o:
+        _write('"""{}"""'.format(ensure_text(o.replace('"""', r'\"\"\"'))))
+      else:
+        _write("'{}'".format(ensure_text(o.replace("'", r"\'"))))
+    elif isinstance(o, collections.Mapping):
+      _write('{' + linesep)
+      for k, v in o.items():
+        _write_repr(k, indent=True, level=level)
+        _write(': ')
+        _write_repr(v, indent=False, level=level)
+        _write(',' + linesep)
+      _write(pad + '}')
+    elif isinstance(o, collections.Iterable):
+      if isinstance(o, collections.MutableSequence):
+        open_collection, close_collection = '[]'
+      elif isinstance(o, collections.Set):
+        open_collection, close_collection = '{}'
+      else:
+        open_collection, close_collection = '()'
+
+      _write(open_collection + linesep)
+      for i in o:
+        _write_repr(i, indent=True, level=level)
+        _write(',' + linesep)
+      _write(pad + close_collection)
+    else:
+      _write(repr(o))  # Numbers and bools.
+
+  _write_repr(obj)
+  return output.getvalue()
+
+
 class SetupPyRunner(InstallerBase):
   _EXTRAS = ('setuptools', 'wheel')
 
@@ -297,10 +358,30 @@ def collect_reduced_dependencies(current):
     return OrderedSet(d for d in reduced_dependencies if d.is_original)
 
 
+def declares_namespace_package(filename):
+  """Given a filename, walk its ast and determine if it declares a namespace package."""
+
+  import ast
+
+  with open(filename) as fp:
+    init_py = ast.parse(fp.read(), filename)
+  calls = [node for node in ast.walk(init_py) if isinstance(node, ast.Call)]
+  for call in calls:
+    if len(call.args) != 1:
+      continue
+    if isinstance(call.func, ast.Attribute) and call.func.attr != 'declare_namespace':
+      continue
+    if isinstance(call.func, ast.Name) and call.func.id != 'declare_namespace':
+      continue
+    if isinstance(call.args[0], ast.Name) and call.args[0].id == '__name__':
+      return True
+  return False
+
+
 class SetupPy(Task):
   """Generate setup.py-based Python projects."""
 
-  SOURCE_ROOT = b'src'
+  SOURCE_ROOT = 'src'
 
   PYTHON_DISTS_PRODUCT = 'python_dists'
 
@@ -366,26 +447,6 @@ def iter_entry_points(cls, target):
             'Cannot add a binary to a PythonArtifact if it does not contain an entry_point.')
       yield name, concrete_target.entry_point
 
-  @classmethod
-  def declares_namespace_package(cls, filename):
-    """Given a filename, walk its ast and determine if it is declaring a namespace package.
-
-    Intended only for __init__.py files though it will work for any .py.
-    """
-    with open(filename) as fp:
-      init_py = ast.parse(fp.read(), filename)
-    calls = [node for node in ast.walk(init_py) if isinstance(node, ast.Call)]
-    for call in calls:
-      if len(call.args) != 1:
-        continue
-      if isinstance(call.func, ast.Attribute) and call.func.attr != 'declare_namespace':
-        continue
-      if isinstance(call.func, ast.Name) and call.func.id != 'declare_namespace':
-        continue
-      if isinstance(call.args[0], ast.Name) and call.args[0].id == '__name__':
-        return True
-    return False
-
   @classmethod
   def nearest_subpackage(cls, package, all_packages):
     """Given a package, find its nearest parent in all_packages."""
@@ -396,8 +457,66 @@ def shared_prefix(candidate):
     shared_packages = [_f for _f in map(shared_prefix, all_packages) if _f]
     return '.'.join(max(shared_packages, key=len)) if shared_packages else package
 
-  @classmethod
-  def find_packages(cls, chroot, log=None):
+  @memoized_property
+  def nsutil_pex(self):
+    interpreter = self.context.products.get_data(PythonInterpreter)
+    chroot = os.path.join(self.workdir, 'nsutil', interpreter.version_string)
+    if not os.path.exists(chroot):
+      pex_info = PexInfo.default(interpreter=interpreter)
+      with safe_concurrent_creation(chroot) as scratch:
+        builder = PEXBuilder(path=scratch, interpreter=interpreter, pex_info=pex_info, copy=True)
+        with temporary_file(binary_mode=False) as fp:
+          declares_namespace_package_code = inspect.getsource(declares_namespace_package)
+          fp.write(textwrap.dedent("""
+            import sys
+
+
+            {declares_namespace_package_code}
+
+
+            if __name__ == '__main__':
+              for path in sys.argv[1:]:
+                if declares_namespace_package(path):
+                  print(path)
+          """).strip().format(declares_namespace_package_code=declares_namespace_package_code))
+          fp.close()
+          builder.set_executable(filename=fp.name, env_filename='main.py')
+          builder.freeze()
+    return PEX(pex=chroot, interpreter=interpreter)
+
+  def filter_namespace_packages(self, root_target, inits):
+    args = list(inits)
+    with self.context.new_workunit(name='find-namespace-packages',
+                                   cmd=' '.join(self.nsutil_pex.cmdline(args=args)),
+                                   labels=[WorkUnitLabel.TOOL]) as workunit:
+
+      process = self.nsutil_pex.run(args=args,
+                                    stdout=subprocess.PIPE,
+                                    stderr=subprocess.PIPE,
+                                    blocking=False)
+
+      stdout, stderr = process.communicate()
+
+      # TODO(John Sirois): Find a way to tee a workunit output instead of buffering up all output
+      # and then writing it out after the process has finished like we do here.
+      def write(stream_name, data):
+        stream = workunit.output(stream_name)
+        stream.write(ensure_binary(data) if PY2 else ensure_text(data))
+        stream.flush()
+
+      write('stdout', stdout)
+      write('stderr', stderr)
+
+      exit_code = process.returncode
+      if exit_code != 0:
+        raise TaskError('Failure trying to detect namespace packages when constructing setup.py '
+                        'project for {}:\n{}'.format(root_target.address.reference(), stderr),
+                        exit_code=exit_code,
+                        failed_targets=[root_target])
+
+      return ensure_text(stdout).splitlines()
+
+  def find_packages(self, root_target, chroot):
     """Detect packages, namespace packages and resources from an existing chroot.
 
     :returns: a tuple of:
@@ -405,7 +524,7 @@ def find_packages(cls, chroot, log=None):
                 set(namespace_packages)
                 map(package => set(files))
     """
-    base = os.path.join(chroot.path(), cls.SOURCE_ROOT)
+    base = os.path.join(chroot.path(), self.SOURCE_ROOT)
     packages, namespace_packages = set(), set()
     resources = defaultdict(set)
 
@@ -416,12 +535,15 @@ def iter_files():
           yield module, filename, os.path.join(root, filename)
 
     # establish packages, namespace packages in first pass
+    inits_to_check = {}
     for module, filename, real_filename in iter_files():
       if filename != '__init__.py':
         continue
       packages.add(module)
-      if cls.declares_namespace_package(real_filename):
-        namespace_packages.add(module)
+      inits_to_check[real_filename] = module
+    namespace_packages = {inits_to_check[init]
+                          for init in self.filter_namespace_packages(root_target,
+                                                                     inits_to_check.keys())}
 
     # second pass establishes non-source content (resources)
     for module, filename, real_filename in iter_files():
@@ -431,11 +553,11 @@ def iter_files():
           # hygiene.
           # raise cls.UndefinedSource('{} is source but does not belong to a package!'
           #                           .format(filename))
-          if log:
-            log.warn('{} is source but does not belong to a package.'.format(real_filename))
+          self.context.log.warn('{} is source but does not belong to a package.'
+                                .format(real_filename))
         else:
           continue
-      submodule = cls.nearest_subpackage(module, packages)
+      submodule = self.nearest_subpackage(module, packages)
       if submodule == module:
         resources[submodule].add(filename)
       else:
@@ -509,13 +631,10 @@ def write_setup(self, root_target, reduced_dependencies, chroot):
 
     Must be run after writing the contents to the chroot.
     """
-    # NB: several explicit str conversions below force non-unicode strings in order to comply
-    # with setuptools expectations.
-
     setup_keywords = root_target.provides.setup_py_keywords.copy()
 
-    package_dir = {b'': self.SOURCE_ROOT}
-    packages, namespace_packages, resources = self.find_packages(chroot, self.context.log)
+    package_dir = {'': self.SOURCE_ROOT}
+    packages, namespace_packages, resources = self.find_packages(root_target, chroot)
 
     if namespace_packages:
       setup_keywords['namespace_packages'] = list(sorted(namespace_packages))
@@ -537,44 +656,12 @@ def write_setup(self, root_target, reduced_dependencies, chroot):
       setup_keywords['entry_points']['console_scripts'].append(
           '{} = {}'.format(binary_name, entry_point))
 
-    # From http://stackoverflow.com/a/13105359
-    def convert(input):
-      if isinstance(input, dict):
-        out = dict()
-        for key, value in input.items():
-          out[convert(key)] = convert(value)
-        return out
-      elif isinstance(input, list):
-        return [convert(element) for element in input]
-      elif isinstance(input, string):
-        return to_bytes(input)
-      else:
-        return input
-
-    # Distutils does not support unicode strings in setup.py, so we must
-    # explicitly convert to binary strings as pants uses unicode_literals.
-    # Ideally we would write the output stream with an encoding, however,
-    # pprint.pformat embeds u's in the string itself during conversion.
-    # For that reason we convert each unicode string independently.
-    #
-    # jsirois@gill ~ $ python2
-    # Python 2.7.13 (default, Jul 21 2017, 03:24:34)
-    # [GCC 7.1.1 20170630] on linux2
-    # Type "help", "copyright", "credits" or "license" for more information.
-    # >>> import pprint
-    # >>> data = {u'entry_points': {u'console_scripts': [u'pants = pants.bin.pants_exe:main']}}
-    # >>> pprint.pformat(data, indent=4)
-    # "{   u'entry_points': {   u'console_scripts': [   u'pants = pants.bin.pants_exe:main']}}"
-    # >>>
-    #
-    # For more information, see http://bugs.python.org/issue13943
-    chroot.write(self._setup_boilerplate().format(
-      setup_dict=pprint.pformat(convert(setup_keywords), indent=4),
-      setup_target=repr(root_target)
-    ), 'setup.py')
-
-    # make sure that setup.py is included
-    chroot.write('include *.py'.encode('utf8'), 'MANIFEST.in')
+    setup_py = self._setup_boilerplate().format(setup_dict=distutils_repr(setup_keywords),
+                                                setup_target=root_target.address.reference())
+    chroot.write(ensure_binary(setup_py), 'setup.py')
+
+    # Make sure that `setup.py` is included.
+    chroot.write('include *.py', 'MANIFEST.in', mode='w')
 
   def create_setup_py(self, target, dist_dir):
     chroot = Chroot(dist_dir, name=target.provides.name)
diff --git a/src/python/pants/util/contextutil.py b/src/python/pants/util/contextutil.py
--- a/src/python/pants/util/contextutil.py
+++ b/src/python/pants/util/contextutil.py
@@ -194,7 +194,7 @@ def temporary_file_path(root_dir=None, cleanup=True, suffix='', permissions=None
 
 
 @contextmanager
-def temporary_file(root_dir=None, cleanup=True, suffix='', permissions=None):
+def temporary_file(root_dir=None, cleanup=True, suffix='', permissions=None, binary_mode=True):
   """
     A with-context that creates a temporary file and returns a writeable file descriptor to it.
 
@@ -207,8 +207,10 @@ def temporary_file(root_dir=None, cleanup=True, suffix='', permissions=None):
                        if you need one, put it at the beginning of suffix.
                        See :py:class:`tempfile.NamedTemporaryFile`.
     :param int permissions: If provided, sets the file to use these permissions.
+    :param bool binary_mode: Whether file opens in binary or text mode.
   """
-  with tempfile.NamedTemporaryFile(suffix=suffix, dir=root_dir, delete=False) as fd:
+  mode = 'w+b' if binary_mode else 'w+'  # tempfile's default is 'w+b'
+  with tempfile.NamedTemporaryFile(suffix=suffix, dir=root_dir, delete=False, mode=mode) as fd:
     try:
       if permissions is not None:
         os.chmod(fd.name, permissions)


Test patch:
diff --git a/contrib/mypy/tests/python/pants_test/contrib/mypy/tasks/BUILD b/contrib/mypy/tests/python/pants_test/contrib/mypy/tasks/BUILD
--- a/contrib/mypy/tests/python/pants_test/contrib/mypy/tasks/BUILD
+++ b/contrib/mypy/tests/python/pants_test/contrib/mypy/tasks/BUILD
@@ -5,6 +5,7 @@ python_tests(
   name='mypy_integration',
   sources=['test_mypy_integration.py'],
   dependencies=[
+    'tests/python/pants_test/backend/python:interpreter_selection_utils',
     'tests/python/pants_test:int-test',
   ],
   tags={'integration'},
diff --git a/contrib/mypy/tests/python/pants_test/contrib/mypy/tasks/test_mypy_integration.py b/contrib/mypy/tests/python/pants_test/contrib/mypy/tasks/test_mypy_integration.py
--- a/contrib/mypy/tests/python/pants_test/contrib/mypy/tasks/test_mypy_integration.py
+++ b/contrib/mypy/tests/python/pants_test/contrib/mypy/tasks/test_mypy_integration.py
@@ -4,6 +4,7 @@
 
 from __future__ import absolute_import, division, print_function, unicode_literals
 
+from pants_test.backend.python.interpreter_selection_utils import PY_3, has_python_version
 from pants_test.pants_run_integration_test import PantsRunIntegrationTest
 
 
@@ -15,7 +16,7 @@ def test_mypy(self):
       '--',
       '--follow-imports=silent'
     ]
-    if self.has_python_version('3'):
+    if has_python_version(PY_3):
       # Python 3.x is available. Test that we see an error in this integration test.
       with self.pants_results(cmd) as pants_run:
         self.assert_success(pants_run)
diff --git a/testprojects/src/python/interpreter_selection/python_3_selection_testing/main_py2.py b/testprojects/src/python/interpreter_selection/python_3_selection_testing/main_py2.py
--- a/testprojects/src/python/interpreter_selection/python_3_selection_testing/main_py2.py
+++ b/testprojects/src/python/interpreter_selection/python_3_selection_testing/main_py2.py
@@ -5,6 +5,7 @@
 from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
                         unicode_literals, with_statement)
 
+import os
 import sys
 
 from interpreter_selection.python_3_selection_testing.lib_py2 import say_something
@@ -15,7 +16,7 @@
 
 def main():
   v = sys.version_info
-  print(sys.executable)
+  print(os.path.realpath(sys.executable))
   print('%d.%d.%d' % v[0:3])
   return say_something()
 
diff --git a/tests/python/pants_test/python/__init__.py b/testprojects/src/python/interpreter_selection/python_3_selection_testing/main_py23.py
similarity index 100%
rename from tests/python/pants_test/python/__init__.py
rename to testprojects/src/python/interpreter_selection/python_3_selection_testing/main_py23.py
diff --git a/testprojects/src/python/interpreter_selection/python_3_selection_testing/main_py3.py b/testprojects/src/python/interpreter_selection/python_3_selection_testing/main_py3.py
--- a/testprojects/src/python/interpreter_selection/python_3_selection_testing/main_py3.py
+++ b/testprojects/src/python/interpreter_selection/python_3_selection_testing/main_py3.py
@@ -2,6 +2,7 @@
 # Copyright 2017 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
+import os
 import sys
 
 from interpreter_selection.python_3_selection_testing.lib_py3 import say_something
@@ -12,7 +13,7 @@
 
 def main():
   v = sys.version_info
-  print(sys.executable)
+  print(os.path.realpath(sys.executable))
   print('%d.%d.%d' % v[0:3])
   return say_something()
 
diff --git a/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/BUILD b/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/BUILD
--- a/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/BUILD
+++ b/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/BUILD
@@ -1,7 +1,7 @@
 python_binary(
   name='test_bin',
   source='main.py',
-  compatibility=['CPython>3'],
+  compatibility=['CPython>=3.6'],
   dependencies=[
     ':reqlib'
   ]
diff --git a/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/import_futures.py b/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/import_futures.py
--- a/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/import_futures.py
+++ b/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/import_futures.py
@@ -1,3 +1,5 @@
 from concurrent.futures import Future
+
+
 print(Future)
 print('Successful.')
diff --git a/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/main.py b/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/main.py
--- a/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/main.py
+++ b/testprojects/src/python/interpreter_selection/resolver_blacklist_testing/main.py
@@ -1,3 +1,5 @@
 import jupyter
+
+
 print(jupyter)
 print('Successful.')
diff --git a/tests/python/pants_test/backend/python/BUILD b/tests/python/pants_test/backend/python/BUILD
--- a/tests/python/pants_test/backend/python/BUILD
+++ b/tests/python/pants_test/backend/python/BUILD
@@ -31,13 +31,14 @@ python_tests(
     '3rdparty/python:future',
     '3rdparty/python:mock',
     '3rdparty/python:pex',
-    'src/python/pants/backend/python:interpreter_cache',
+    ':interpreter_selection_utils',
     'src/python/pants/backend/python/subsystems',
+    'src/python/pants/backend/python:interpreter_cache',
     'src/python/pants/util:contextutil',
-    'tests/python/pants_test:test_base',
-    'tests/python/pants_test:int-test',
     'tests/python/pants_test/testutils:git_util',
     'tests/python/pants_test/testutils:pexrc_util',
+    'tests/python/pants_test:int-test',
+    'tests/python/pants_test:test_base',
   ],
   timeout=1200
 )
@@ -62,3 +63,11 @@ python_library(
     'tests/python/pants_test:int-test',
   ]
 )
+
+python_library(
+  name='interpreter_selection_utils',
+  sources=['interpreter_selection_utils.py'],
+  dependencies=[
+    'src/python/pants/util:process_handler'
+  ]
+)
\ No newline at end of file
diff --git a/tests/python/pants_test/backend/python/interpreter_selection_utils.py b/tests/python/pants_test/backend/python/interpreter_selection_utils.py
new file mode 100644
--- /dev/null
+++ b/tests/python/pants_test/backend/python/interpreter_selection_utils.py
@@ -0,0 +1,79 @@
+# coding=utf-8
+# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import os
+from unittest import skipIf
+
+from pants.util.process_handler import subprocess
+
+
+PY_27 = '2.7'
+PY_3 = '3'
+PY_36 = '3.6'
+
+
+def has_python_version(version):
+  """Returns `True` if the current system has the specified version of python.
+
+  :param string version: A python version string, such as 2.7, 3.
+  """
+  return python_interpreter_path(version) is not None
+
+
+def python_interpreter_path(version):
+  """Returns the interpreter path if the current system has the specified version of python.
+
+  :param string version: A python version string, such as 2.7, 3.
+  :returns: the normalized path to the interpreter binary if found; otherwise `None`
+  :rtype: string
+  """
+  try:
+    command = ['python{}'.format(version), '-c', 'import sys; print(sys.executable)']
+    py_path = subprocess.check_output(command).decode('utf-8').strip()
+    return os.path.realpath(py_path)
+  except subprocess.CalledProcessError:
+    return None
+
+
+def skip_unless_pythons(*versions):
+  """A decorator that only runs the decorated test method if the specified pythons are present.
+
+  :param string *versions: Python version strings, such as 2.7, 3.
+  """
+  missing_versions = [v for v in versions if not has_python_version(v)]
+  if len(missing_versions) == 1:
+    return skipIf(True, 'Could not find python {} on system. Skipping.'.format(missing_versions[0]))
+  elif len(missing_versions) > 1:
+    return skipIf(True,
+                  'Skipping due to the following missing required pythons: {}'
+                  .format(', '.join(missing_versions)))
+  else:
+    return skipIf(False, 'All required pythons present, continuing with test!')
+
+
+def skip_unless_python27(func):
+  """A test skip decorator that only runs a test method if python2.7 is present."""
+  return skip_unless_pythons(PY_27)(func)
+
+
+def skip_unless_python3(func):
+  """A test skip decorator that only runs a test method if python3 is present."""
+  return skip_unless_pythons(PY_3)(func)
+
+
+def skip_unless_python36(func):
+  """A test skip decorator that only runs a test method if python3.6 is present."""
+  return skip_unless_pythons(PY_36)(func)
+
+
+def skip_unless_python27_and_python3(func):
+  """A test skip decorator that only runs a test method if python2.7 and python3 are present."""
+  return skip_unless_pythons(PY_27, PY_3)(func)
+
+
+def skip_unless_python27_and_python36(func):
+  """A test skip decorator that only runs a test method if python2.7 and python3.6 are present."""
+  return skip_unless_pythons(PY_27, PY_36)(func)
diff --git a/tests/python/pants_test/backend/python/tasks/BUILD b/tests/python/pants_test/backend/python/tasks/BUILD
--- a/tests/python/pants_test/backend/python/tasks/BUILD
+++ b/tests/python/pants_test/backend/python/tasks/BUILD
@@ -52,19 +52,19 @@ python_tests(
 python_tests(
   sources=globs('test_*.py', exclude=([globs('*_integration.py')] + python_native_code_test_files)),
   dependencies=[
-    ':python_task_test_base',
     '3rdparty/python/twitter/commons:twitter.common.collections',
     '3rdparty/python/twitter/commons:twitter.common.dirutil',
-    '3rdparty/python:future',
     '3rdparty/python:coverage',
+    '3rdparty/python:future',
     '3rdparty/python:mock',
     '3rdparty/python:pex',
+    ':python_task_test_base',
     'src/python/pants/backend/native',
-    'src/python/pants/backend/python:plugin',
     'src/python/pants/backend/python/subsystems',
     'src/python/pants/backend/python/targets',
     'src/python/pants/backend/python/tasks',
     'src/python/pants/backend/python:interpreter_cache',
+    'src/python/pants/backend/python:plugin',
     'src/python/pants/backend/python:python_artifact',
     'src/python/pants/backend/python:python_requirement',
     'src/python/pants/base:build_root',
@@ -76,10 +76,11 @@ python_tests(
     'src/python/pants/util:contextutil',
     'src/python/pants/util:dirutil',
     'src/python/pants/util:process_handler',
+    'tests/python/pants_test/backend/python:interpreter_selection_utils',
     'tests/python/pants_test/engine:scheduler_test_base',
     'tests/python/pants_test/subsystem:subsystem_utils',
-    'tests/python/pants_test:task_test_base',
     'tests/python/pants_test/tasks:task_test_base',
+    'tests/python/pants_test:task_test_base',
   ],
   timeout=600
 )
@@ -93,9 +94,10 @@ python_tests(
     'src/python/pants/base:build_environment',
     'src/python/pants/util:contextutil',
     'src/python/pants/util:process_handler',
-    'tests/python/pants_test:int-test',
-    'tests/python/pants_test/backend/python:pants_requirement_integration_test_base',
     'tests/python/pants_test/backend/python/tasks:python_task_test_base',
+    'tests/python/pants_test/backend/python:interpreter_selection_utils',
+    'tests/python/pants_test/backend/python:pants_requirement_integration_test_base',
+    'tests/python/pants_test:int-test',
   ],
   tags={'integration'},
   timeout=2400
diff --git a/tests/python/pants_test/python/test_interpreter_selection_integration.py b/tests/python/pants_test/backend/python/tasks/test_interpreter_selection_integration.py
similarity index 61%
rename from tests/python/pants_test/python/test_interpreter_selection_integration.py
rename to tests/python/pants_test/backend/python/tasks/test_interpreter_selection_integration.py
--- a/tests/python/pants_test/python/test_interpreter_selection_integration.py
+++ b/tests/python/pants_test/backend/python/tasks/test_interpreter_selection_integration.py
@@ -8,6 +8,8 @@
 
 from pants.util.contextutil import temporary_dir
 from pants.util.process_handler import subprocess
+from pants_test.backend.python.interpreter_selection_utils import (PY_3, PY_27, skip_unless_python3,
+                                                                   skip_unless_python27)
 from pants_test.pants_run_integration_test import PantsRunIntegrationTest
 
 
@@ -20,23 +22,20 @@ def test_conflict(self):
     self.assert_failure(pants_run,
                         'Unexpected successful build of {binary}.'.format(binary=binary_target))
 
+  @skip_unless_python3
   def test_select_3(self):
-    self._maybe_test_version('3')
+    self._test_version(PY_3)
 
+  @skip_unless_python27
   def test_select_27(self):
-    self._maybe_test_version('2.7')
+    self._test_version(PY_27)
 
-  def _maybe_test_version(self, version):
-    if self.has_python_version(version):
-      print('Found python {}. Testing interpreter selection against it.'.format(version))
-      echo = self._echo_version(version)
-      v = echo.split('.')  # E.g., 2.7.13.
-      self.assertTrue(len(v) > 2, 'Not a valid version string: {}'.format(v))
-      expected_components = version.split('.')
-      self.assertEquals(expected_components, v[:len(expected_components)])
-    else:
-      print('No python {} found. Skipping.'.format(version))
-      self.skipTest('No python {} on system'.format(version))
+  def _test_version(self, version):
+    echo = self._echo_version(version)
+    v = echo.split('.')  # E.g., 2.7.13.
+    self.assertTrue(len(v) > 2, 'Not a valid version string: {}'.format(v))
+    expected_components = version.split('.')
+    self.assertEqual(expected_components, v[:len(expected_components)])
 
   def _echo_version(self, version):
     with temporary_dir() as distdir:
@@ -56,10 +55,14 @@ def _echo_version(self, version):
       (stdout_data, _) = proc.communicate()
       return stdout_data
 
-  def _build_pex(self, binary_target, config=None):
-    # Avoid some known-to-choke-on interpreters.
-    command = ['binary',
-               binary_target,
-               '--python-setup-interpreter-constraints=CPython>=2.7,<3',
-               '--python-setup-interpreter-constraints=CPython>=3.3']
+  def _build_pex(self, binary_target, config=None, args=None, version=PY_27):
+    # By default, Avoid some known-to-choke-on interpreters.
+    if version == PY_3:
+      constraint = '["CPython>=3.4,<4"]'
+    else:
+      constraint = '["CPython>=2.7,<3"]'
+    args = list(args) if args is not None else [
+          '--python-setup-interpreter-constraints={}'.format(constraint)
+        ]
+    command = ['binary', binary_target] + args
     return self.run_pants(command=command, config=config)
diff --git a/tests/python/pants_test/backend/python/tasks/test_pytest_run_integration.py b/tests/python/pants_test/backend/python/tasks/test_pytest_run_integration.py
--- a/tests/python/pants_test/backend/python/tasks/test_pytest_run_integration.py
+++ b/tests/python/pants_test/backend/python/tasks/test_pytest_run_integration.py
@@ -5,10 +5,14 @@
 from __future__ import absolute_import, division, print_function, unicode_literals
 
 import os
-import sys
 import time
 
 from pants.util.contextutil import temporary_dir
+from pants_test.backend.python.interpreter_selection_utils import (PY_3, PY_27,
+                                                                   python_interpreter_path,
+                                                                   skip_unless_python3,
+                                                                   skip_unless_python27,
+                                                                   skip_unless_python27_and_python3)
 from pants_test.pants_run_integration_test import PantsRunIntegrationTest
 from pants_test.testutils.pexrc_util import setup_pexrc_with_pex_python_path
 
@@ -96,7 +100,7 @@ def test_pytest_run_killed_by_signal(self):
     self.assertIn("FAILURE: Test was killed by signal", pants_run.stdout_data)
 
   def test_pytest_explicit_coverage(self):
-    with temporary_dir(cleanup=False) as coverage_dir:
+    with temporary_dir() as coverage_dir:
       pants_run = self.run_pants(['clean-all',
                                   'test.pytest',
                                   '--coverage=pants',
@@ -118,32 +122,70 @@ def test_pytest_with_profile(self):
       # current process started.
       self.assertTrue(os.path.exists('{}.0'.format(prof)))
 
+  @skip_unless_python27_and_python3
   def test_pants_test_interpreter_selection_with_pexrc(self):
     """Test the pants test goal with intepreters selected from a PEX_PYTHON_PATH
     defined in a pexrc file on disk.
-
     """
-    py27 = '2.7'
-    py3 = '3'
-    if self.has_python_version(py27) and self.has_python_version(py3):
-      print('Found both python {} and python {}. Running test.'.format(py27, py3))
-      py27_path, py3_path = self.python_interpreter_path(py27), self.python_interpreter_path(py3)
-      with setup_pexrc_with_pex_python_path(os.path.join(os.path.dirname(sys.argv[0]), '.pexrc'),
-                                            [py27_path, py3_path]):
-        with temporary_dir() as interpreters_cache:
-          pants_ini_config = {'python-setup': {'interpreter_cache_dir': interpreters_cache}}
-          pants_run_27 = self.run_pants(
-            command=['test', '{}:test_py2'.format(os.path.join(self.testproject,
-                                                               'python_3_selection_testing'))],
-            config=pants_ini_config
-          )
-          self.assert_success(pants_run_27)
-          pants_run_3 = self.run_pants(
-            command=['test', '{}:test_py3'.format(os.path.join(self.testproject,
-                                                               'python_3_selection_testing'))],
+    py27_path, py3_path = python_interpreter_path(PY_27), python_interpreter_path(PY_3)
+    with setup_pexrc_with_pex_python_path([py27_path, py3_path]):
+      with temporary_dir() as interpreters_cache:
+        pants_ini_config = {'python-setup': {'interpreter_cache_dir': interpreters_cache}}
+        pants_run_27 = self.run_pants(
+          command=['test', '{}:test_py2'.format(os.path.join(self.testproject,
+                                                             'python_3_selection_testing'))],
+          config=pants_ini_config
+        )
+        self.assert_success(pants_run_27)
+        pants_run_3 = self.run_pants(
+          command=['test', '{}:test_py3'.format(os.path.join(self.testproject,
+                                                             'python_3_selection_testing'))],
+          config=pants_ini_config
+        )
+        self.assert_success(pants_run_3)
+
+  @skip_unless_python27
+  def test_pants_test_interpreter_selection_with_option_2(self):
+    """
+    Test that the pants test goal properly constrains the SelectInterpreter task to Python 2
+    using the '--python-setup-interpreter-constraints' option.
+    """
+    with temporary_dir() as interpreters_cache:
+      pants_ini_config = {
+        'python-setup': {
+          'interpreter_constraints': ['CPython>=2.7,<4'],
+          'interpreter_cache_dir': interpreters_cache,
+        }
+      }
+      pants_run_2 = self.run_pants(
+        command=[
+          'test',
+          '{}:test_py2'.format(os.path.join(self.testproject,'python_3_selection_testing')),
+          '--python-setup-interpreter-constraints=["CPython<3"]',
+        ],
+        config=pants_ini_config
+      )
+      self.assert_success(pants_run_2)
+
+  @skip_unless_python3
+  def test_pants_test_interpreter_selection_with_option_3(self):
+    """
+    Test that the pants test goal properly constrains the SelectInterpreter task to Python 3
+    using the '--python-setup-interpreter-constraints' option.
+    """
+    with temporary_dir() as interpreters_cache:
+      pants_ini_config = {
+        'python-setup': {
+          'interpreter_constraints': ['CPython>=2.7,<4'],
+          'interpreter_cache_dir': interpreters_cache,
+        }
+      }
+      pants_run_3 = self.run_pants(
+            command=[
+              'test',
+              '{}:test_py3'.format(os.path.join(self.testproject, 'python_3_selection_testing')),
+              '--python-setup-interpreter-constraints=["CPython>=3"]',
+            ],
             config=pants_ini_config
           )
-          self.assert_success(pants_run_3)
-    else:
-      print('Could not find both python {} and python {} on system. Skipping.'.format(py27, py3))
-      self.skipTest('Missing neccesary Python interpreters on system.')
+      self.assert_success(pants_run_3)
diff --git a/tests/python/pants_test/python/test_python_binary_integration.py b/tests/python/pants_test/backend/python/tasks/test_python_binary_integration.py
similarity index 100%
rename from tests/python/pants_test/python/test_python_binary_integration.py
rename to tests/python/pants_test/backend/python/tasks/test_python_binary_integration.py
diff --git a/tests/python/pants_test/backend/python/tasks/test_python_run_integration.py b/tests/python/pants_test/backend/python/tasks/test_python_run_integration.py
--- a/tests/python/pants_test/backend/python/tasks/test_python_run_integration.py
+++ b/tests/python/pants_test/backend/python/tasks/test_python_run_integration.py
@@ -5,11 +5,16 @@
 from __future__ import absolute_import, division, print_function, unicode_literals
 
 import os
-import sys
 
 from pex.pex_bootstrapper import get_pex_info
 
 from pants.util.contextutil import temporary_dir
+from pants_test.backend.python.interpreter_selection_utils import (PY_3, PY_27,
+                                                                   python_interpreter_path,
+                                                                   skip_unless_python3,
+                                                                   skip_unless_python27,
+                                                                   skip_unless_python27_and_python3,
+                                                                   skip_unless_python36)
 from pants_test.pants_run_integration_test import PantsRunIntegrationTest, ensure_daemon
 from pants_test.testutils.pexrc_util import setup_pexrc_with_pex_python_path
 
@@ -17,18 +22,41 @@
 class PythonRunIntegrationTest(PantsRunIntegrationTest):
   testproject = 'testprojects/src/python/interpreter_selection'
 
+  @skip_unless_python3
   @ensure_daemon
   def test_run_3(self):
-    self._maybe_run_version('3')
+    self._run_version(PY_3)
 
+  @skip_unless_python27
   @ensure_daemon
   def test_run_27(self):
-    self._maybe_run_version('2.7')
+    self._run_version(PY_27)
 
-  def test_run_27_and_then_3(self):
-    if self.skip_if_no_python('2.7') or self.skip_if_no_python('3'):
-      return
+  def _run_version(self, version):
+    echo = self._run_echo_version(version)
+    v = echo.split('.')  # E.g., 2.7.13.
+    self.assertTrue(len(v) > 2, 'Not a valid version string: {}'.format(v))
+    expected_components = version.split('.')
+    self.assertEqual(expected_components, v[:len(expected_components)])
+
+  def _run_echo_version(self, version):
+    binary_name = 'echo_interpreter_version_{}'.format(version)
+    binary_target = '{}:{}'.format(self.testproject, binary_name)
+    # Build a pex.
+    # Avoid some known-to-choke-on interpreters.
+    if version == PY_3:
+      constraint = '["CPython>=3.4,<3.7"]'
+    else:
+      constraint = '["CPython>=2.7,<3"]'
+    command = ['run',
+               binary_target,
+               '--python-setup-interpreter-constraints={}'.format(constraint),
+               '--quiet']
+    pants_run = self.run_pants(command=command)
+    return pants_run.stdout_data.splitlines()[0].strip()
 
+  @skip_unless_python27_and_python3
+  def test_run_27_and_then_3(self):
     with temporary_dir() as interpreters_cache:
       pants_ini_config = {'python-setup': {'interpreter_cache_dir': interpreters_cache}}
       pants_run_27 = self.run_pants(
@@ -39,16 +67,39 @@ def test_run_27_and_then_3(self):
       pants_run_3 = self.run_pants(
         command=['run', '{}:echo_interpreter_version_3'.format(self.testproject),
                  '--python-setup-interpreter-constraints=CPython>=2.7,<3',
-                 '--python-setup-interpreter-constraints=CPython>=3.3'],
+                 '--python-setup-interpreter-constraints=CPython>=3.4,<3.7'],
+        config=pants_ini_config
+      )
+      self.assert_success(pants_run_3)
+
+  @skip_unless_python3
+  def test_run_3_by_option(self):
+    with temporary_dir() as interpreters_cache:
+      pants_ini_config = {'python-setup': {'interpreter_constraints': ["CPython>=2.7,<3.7"],
+        'interpreter_cache_dir': interpreters_cache}}
+      pants_run_3 = self.run_pants(
+        command=['run', '{}:echo_interpreter_version_3'.format(self.testproject),
+        '--python-setup-interpreter-constraints=["CPython>=3"]'],
         config=pants_ini_config
       )
       self.assert_success(pants_run_3)
 
+  @skip_unless_python27
+  def test_run_2_by_option(self):
+    with temporary_dir() as interpreters_cache:
+      pants_ini_config = {'python-setup': {'interpreter_constraints': ["CPython>=2.7,<3.7"],
+        'interpreter_cache_dir': interpreters_cache}}
+      pants_run_2 = self.run_pants(
+        command=['run', '{}:echo_interpreter_version_2.7'.format(self.testproject),
+        '--python-setup-interpreter-constraints=["CPython<3"]'],
+        config=pants_ini_config
+      )
+      self.assert_success(pants_run_2)
+
   def test_die(self):
     command = ['run',
                '{}:die'.format(self.testproject),
-               '--python-setup-interpreter-constraints=CPython>=2.7,<3',
-               '--python-setup-interpreter-constraints=CPython>=3.3',
+               '--python-setup-interpreter-constraints=["CPython>=2.7,<3", ">=3.4,<3.7"]',
                '--quiet']
     pants_run = self.run_pants(command=command)
     assert pants_run.returncode == 57
@@ -63,52 +114,44 @@ def test_get_env_var(self):
                var_key]
     pants_run = self.run_pants(command=command, extra_env={var_key: var_val})
     self.assert_success(pants_run)
-    self.assertEquals(var_val, pants_run.stdout_data.strip())
+    self.assertEqual(var_val, pants_run.stdout_data.strip())
 
+  @skip_unless_python27_and_python3
   def test_pants_run_interpreter_selection_with_pexrc(self):
-    py27 = '2.7'
-    py3 = '3'
-    if self.skip_if_no_python(py27) or self.skip_if_no_python(py3):
-      return
-
-    py27_path, py3_path = self.python_interpreter_path(py27), self.python_interpreter_path(py3)
-    with setup_pexrc_with_pex_python_path(os.path.join(os.path.dirname(sys.argv[0]), '.pexrc'), [py27_path, py3_path]):
+    py27_path, py3_path = python_interpreter_path(PY_27), python_interpreter_path(PY_3)
+    with setup_pexrc_with_pex_python_path([py27_path, py3_path]):
       with temporary_dir() as interpreters_cache:
         pants_ini_config = {'python-setup': {'interpreter_cache_dir': interpreters_cache}}
         pants_run_27 = self.run_pants(
-          command=['run', '{}:main_py2'.format(os.path.join(self.testproject, 'python_3_selection_testing'))],
+          command=['run', '{}:main_py2'.format(os.path.join(self.testproject,
+                                                            'python_3_selection_testing'))],
           config=pants_ini_config
         )
         self.assert_success(pants_run_27)
-        # Interpreter selection for Python 2 is problematic in CI due to multiple virtualenvs in play.
-        if not os.getenv('CI'):
-          self.assertIn(py27_path.split(py27)[0], pants_run_27.stdout_data)
+        self.assertIn(py27_path, pants_run_27.stdout_data)
         pants_run_3 = self.run_pants(
-          command=['run', '{}:main_py3'.format(os.path.join(self.testproject, 'python_3_selection_testing'))],
+          command=['run', '{}:main_py3'.format(os.path.join(self.testproject,
+                                                            'python_3_selection_testing'))],
           config=pants_ini_config
         )
         self.assert_success(pants_run_3)
-        # Protection for when the sys.executable path underlies a symlink pointing to 'python' without '3'
-        # at the end of the basename.
-        self.assertIn(py3_path.split(py3)[0], pants_run_3.stdout_data)
+        self.assertIn(py3_path, pants_run_3.stdout_data)
 
+  @skip_unless_python27_and_python3
   def test_pants_binary_interpreter_selection_with_pexrc(self):
-    py27 = '2.7'
-    py3 = '3'
-    if self.skip_if_no_python(py27) or self.skip_if_no_python(py3):
-      return
-
-    py27_path, py3_path = self.python_interpreter_path(py27), self.python_interpreter_path(py3)
-    with setup_pexrc_with_pex_python_path(os.path.join(os.path.dirname(sys.argv[0]), '.pexrc')  , [py27_path, py3_path]):
+    py27_path, py3_path = python_interpreter_path(PY_27), python_interpreter_path(PY_3)
+    with setup_pexrc_with_pex_python_path([py27_path, py3_path]):
       with temporary_dir() as interpreters_cache:
         pants_ini_config = {'python-setup': {'interpreter_cache_dir': interpreters_cache}}
         pants_run_27 = self.run_pants(
-          command=['binary', '{}:main_py2'.format(os.path.join(self.testproject, 'python_3_selection_testing'))],
+          command=['binary', '{}:main_py2'.format(os.path.join(self.testproject,
+                                                               'python_3_selection_testing'))],
           config=pants_ini_config
         )
         self.assert_success(pants_run_27)
         pants_run_3 = self.run_pants(
-          command=['binary', '{}:main_py3'.format(os.path.join(self.testproject, 'python_3_selection_testing'))],
+          command=['binary', '{}:main_py3'.format(os.path.join(self.testproject,
+                                                               'python_3_selection_testing'))],
           config=pants_ini_config
         )
         self.assert_success(pants_run_3)
@@ -125,26 +168,31 @@ def test_pants_binary_interpreter_selection_with_pexrc(self):
     os.remove(py2_pex)
     os.remove(py3_pex)
 
+  @skip_unless_python3
   def test_target_constraints_with_no_sources(self):
     with temporary_dir() as interpreters_cache:
+      pants_ini_config = {
+          'python-setup': {
+            'interpreter_cache_dir': interpreters_cache,
+            'interpreter_constraints': ['CPython>3'],
+          }
+        }
       # Run task.
-      py3 = '3'
-      if not self.skip_if_no_python(py3):
-        pants_ini_config = {'python-setup': {'interpreter_cache_dir': interpreters_cache}}
-        pants_run_3 = self.run_pants(
-          command=['run', '{}:test_bin'.format(os.path.join(self.testproject, 'test_target_with_no_sources'))],
-          config=pants_ini_config
-        )
-        self.assert_success(pants_run_3)
-        self.assertIn('python3', pants_run_3.stdout_data)
+      pants_run = self.run_pants(
+        command=['run', '{}:test_bin'.format(os.path.join(self.testproject,
+                                                          'test_target_with_no_sources'))],
+        config=pants_ini_config
+      )
+      self.assert_success(pants_run)
+      self.assertIn('python3', pants_run.stdout_data)
 
       # Binary task.
-      pants_ini_config = {'python-setup': {'interpreter_cache_dir': interpreters_cache}}
-      pants_run_27 = self.run_pants(
-        command=['binary', '{}:test_bin'.format(os.path.join(self.testproject, 'test_target_with_no_sources'))],
+      pants_run = self.run_pants(
+        command=['binary', '{}:test_bin'.format(os.path.join(self.testproject,
+                                                             'test_target_with_no_sources'))],
         config=pants_ini_config
       )
-      self.assert_success(pants_run_27)
+      self.assert_success(pants_run)
 
     # Ensure proper interpreter constraints were passed to built pexes.
     py2_pex = os.path.join(os.getcwd(), 'dist', 'test_bin.pex')
@@ -153,41 +201,8 @@ def test_target_constraints_with_no_sources(self):
     # Cleanup.
     os.remove(py2_pex)
 
-  def skip_if_no_python(self, version):
-    if not self.has_python_version(version):
-      msg = 'No python {} found. Skipping.'.format(version)
-      print(msg)
-      self.skipTest(msg)
-      return True
-    return False
-
-  def _maybe_run_version(self, version):
-    if self.skip_if_no_python(version):
-      return
-
-    echo = self._run_echo_version(version)
-    v = echo.split('.')  # E.g., 2.7.13.
-    self.assertTrue(len(v) > 2, 'Not a valid version string: {}'.format(v))
-    expected_components = version.split('.')
-    self.assertEquals(expected_components, v[:len(expected_components,)])
-
-  def _run_echo_version(self, version):
-    binary_name = 'echo_interpreter_version_{}'.format(version)
-    binary_target = '{}:{}'.format(self.testproject, binary_name)
-    # Build a pex.
-    # Avoid some known-to-choke-on interpreters.
-    command = ['run',
-               binary_target,
-               '--python-setup-interpreter-constraints=CPython>=2.7,<3',
-               '--python-setup-interpreter-constraints=CPython>=3.3',
-               '--quiet']
-    pants_run = self.run_pants(command=command)
-    return pants_run.stdout_data.rstrip().split('\n')[-1]
-
+  @skip_unless_python36
   def test_pex_resolver_blacklist_integration(self):
-    py3 = '3'
-    if self.skip_if_no_python(py3):
-      return
     pex = os.path.join(os.getcwd(), 'dist', 'test_bin.pex')
     try:
       pants_ini_config = {'python-setup': {'resolver_blacklist': {'functools32': 'CPython>3'}}}
diff --git a/tests/python/pants_test/backend/python/tasks/test_setup_py.py b/tests/python/pants_test/backend/python/tasks/test_setup_py.py
--- a/tests/python/pants_test/backend/python/tasks/test_setup_py.py
+++ b/tests/python/pants_test/backend/python/tasks/test_setup_py.py
@@ -15,8 +15,9 @@
 from twitter.common.dirutil.chroot import Chroot
 
 from pants.backend.python.subsystems.python_setup import PythonSetup
+from pants.backend.python.targets.python_library import PythonLibrary
 from pants.backend.python.tasks.select_interpreter import SelectInterpreter
-from pants.backend.python.tasks.setup_py import SetupPy
+from pants.backend.python.tasks.setup_py import SetupPy, declares_namespace_package
 from pants.base.exceptions import TaskError
 from pants.build_graph.build_file_aliases import BuildFileAliases
 from pants.build_graph.prep_command import PrepCommand
@@ -25,6 +26,7 @@
 from pants.fs.archive import TGZ
 from pants.util.contextutil import environment_as, temporary_dir, temporary_file
 from pants.util.dirutil import safe_mkdir
+from pants_test.backend.python.interpreter_selection_utils import skip_unless_python36
 from pants_test.backend.python.tasks.python_task_test_base import PythonTaskTestBase
 from pants_test.subsystem.subsystem_util import init_subsystem
 
@@ -40,15 +42,18 @@ def setUp(self):
     self.set_options(pants_distdir=self.distdir)
     init_subsystem(Target.Arguments)
 
-  @contextmanager
-  def run_execute(self, target, recursive=False):
+  def prepare_setup_py(self, target, recursive=False):
     self.set_options(recursive=recursive)
     si_task_type = self.synthesize_task_subtype(SelectInterpreter, 'si_scope')
     context = self.context(for_task_types=[si_task_type], target_roots=[target])
     si_task_type(context, os.path.join(self.pants_workdir, 'si')).execute()
-    setup_py = self.create_task(context)
+    return self.create_task(context)
+
+  @contextmanager
+  def run_execute(self, target, recursive=False):
+    setup_py = self.prepare_setup_py(target, recursive=recursive)
     setup_py.execute()
-    yield context.products.get_data(SetupPy.PYTHON_DISTS_PRODUCT)
+    yield setup_py.context.products.get_data(SetupPy.PYTHON_DISTS_PRODUCT)
 
 
 class TestSetupPyInterpreter(SetupPyTestBase):
@@ -118,10 +123,15 @@ def run(self):
     sdist_srcdir = os.path.join(self.distdir, 'foo-0.0.0', 'src')
     with environment_as(PYTHONPATH=sdist_srcdir):
       with self.run_execute(foo):
-        with open(os.path.join(sdist_srcdir, 'foo', 'commands', 'sys_path.txt')) as fp:
+        with open(os.path.join(sdist_srcdir, 'foo', 'commands', 'sys_path.txt'), 'r') as fp:
+          load_package = lambda: Package.from_href(fp.readline().strip())
+          # We don't care about the ordering of `wheel` and `setuptools` on the `sys.path`, just
+          # that they are 1st as a group.
+          extras = {p.name: p for p in (load_package(), load_package())}
+
           def assert_extra(name, expected_version):
-            package = Package.from_href(fp.readline().strip())
-            self.assertEqual(name, package.name)
+            package = extras.get(name)
+            self.assertIsNotNone(package)
             self.assertEqual(expected_version, package.raw_version)
 
           # The 1st two elements of the sys.path should be our custom SetupPyRunner Installer's
@@ -603,7 +613,7 @@ def has_ns(stmt):
     with temporary_file() as fp:
       fp.write(stmt)
       fp.flush()
-      return SetupPy.declares_namespace_package(fp.name)
+      return declares_namespace_package(fp.name)
 
   assert not has_ns('')
   assert not has_ns('add(1, 2); foo(__name__); self.shoot(__name__)')
@@ -613,70 +623,80 @@ def has_ns(stmt):
   assert has_ns('from pkg_resources import declare_namespace; declare_namespace(__name__)')
 
 
-@contextmanager
-def yield_chroot(packages, namespace_packages, resources):
-  def to_path(package):
-    return package.replace('.', os.path.sep)
-
-  with temporary_dir() as td:
-    def write(package, name, content):
-      package_path = os.path.join(td, SetupPy.SOURCE_ROOT, to_path(package))
-      safe_mkdir(os.path.dirname(os.path.join(package_path, name)))
-      with open(os.path.join(package_path, name), 'w') as fp:
-        fp.write(content)
-    for package in packages:
-      write(package, '__init__.py', '')
-    for package in namespace_packages:
-      write(package, '__init__.py', '__import__("pkg_resources").declare_namespace(__name__)')
-    for package, resource_list in resources.items():
-      for resource in resource_list:
-        write(package, resource, 'asdfasdf')
-
-    chroot_mock = Mock(spec=Chroot)
-    chroot_mock.path.return_value = td
-    yield chroot_mock
-
-
-def test_find_packages():
-  def assert_single_chroot(packages, namespace_packages, resources):
-    with yield_chroot(packages, namespace_packages, resources) as chroot:
-      p, n_p, r = SetupPy.find_packages(chroot)
-      assert p == set(packages + namespace_packages)
-      assert n_p == set(namespace_packages)
-      assert r == dict((k, set(v)) for (k, v) in resources.items())
-
-  # assert both packages and namespace packages work
-  assert_single_chroot(['foo'], [], {})
-  assert_single_chroot(['foo'], ['foo'], {})
-
-  # assert resources work
-  assert_single_chroot(['foo'], [], {'foo': ['blork.dat']})
-
-  resources = {
-    'foo': [
-      'f0',
-      os.path.join('bar', 'baz', 'f1'),
-      os.path.join('bar', 'baz', 'f2'),
-    ]
-  }
-  assert_single_chroot(['foo'], [], resources)
-
-  # assert that nearest-submodule is honored
-  with yield_chroot(['foo', 'foo.bar'], [], resources) as chroot:
-    _, _, r = SetupPy.find_packages(chroot)
-    assert r == {
-      'foo': {'f0'},
-      'foo.bar': {os.path.join('baz', 'f1'), os.path.join('baz', 'f2')}
+class TestSetupPyFindPackages(SetupPyTestBase):
+  @contextmanager
+  def yield_chroot(self, packages, namespace_packages, resources, py3=False):
+    def to_path(package):
+      return package.replace('.', os.path.sep)
+
+    with temporary_dir() as td:
+      def write(package, name, content):
+        package_path = os.path.join(td, SetupPy.SOURCE_ROOT, to_path(package))
+        safe_mkdir(os.path.dirname(os.path.join(package_path, name)))
+        with open(os.path.join(package_path, name), 'w') as fp:
+          fp.write(content)
+      for package in packages:
+        write(package, '__init__.py', 'a = "b"; b = f"{a}"' if py3 else '')
+      for package in namespace_packages:
+        write(package, '__init__.py', '__import__("pkg_resources").declare_namespace(__name__)')
+      for package, resource_list in resources.items():
+        for resource in resource_list:
+          write(package, resource, 'asdfasdf')
+
+      chroot_mock = Mock(spec=Chroot)
+      chroot_mock.path.return_value = td
+      yield chroot_mock
+
+  def assert_find_packages(self, py36):
+    compatibility = 'CPython>=3.6,<3.7' if py36 else 'CPython>=2.7,<3'
+    dummy_target = self.make_target(spec='src/python/foo',
+                                    target_type=PythonLibrary,
+                                    sources=[],
+                                    compatibility=compatibility)
+    setup_py = self.prepare_setup_py(dummy_target)
+
+    def assert_single_chroot(packages, namespace_packages, resources):
+      with self.yield_chroot(packages, namespace_packages, resources, py3=py36) as chroot:
+        p, n_p, r = setup_py.find_packages(dummy_target, chroot)
+        assert p == set(packages + namespace_packages)
+        assert n_p == set(namespace_packages)
+        assert r == dict((k, set(v)) for (k, v) in resources.items())
+
+    # assert both packages and namespace packages work
+    assert_single_chroot(['foo'], [], {})
+    assert_single_chroot(['foo'], ['foo'], {})
+
+    # assert resources work
+    assert_single_chroot(['foo'], [], {'foo': ['blork.dat']})
+
+    resources = {
+      'foo': [
+        'f0',
+        os.path.join('bar', 'baz', 'f1'),
+        os.path.join('bar', 'baz', 'f2'),
+      ]
     }
-
-  # assert that nearest submodule splits on module prefixes
-  with yield_chroot(
-      ['foo', 'foo.bar'],
-      [],
-      {'foo.bar1': ['f0']}) as chroot:
-
-    _, _, r = SetupPy.find_packages(chroot)
-    assert r == {'foo': {'bar1/f0'}}
+    assert_single_chroot(['foo'], [], resources)
+
+    # assert that nearest-submodule is honored
+    with self.yield_chroot(['foo', 'foo.bar'], [], resources, py3=py36) as chroot:
+      _, _, r = setup_py.find_packages(dummy_target, chroot)
+      assert r == {
+        'foo': {'f0'},
+        'foo.bar': {os.path.join('baz', 'f1'), os.path.join('baz', 'f2')}
+      }
+
+    # assert that nearest submodule splits on module prefixes
+    with self.yield_chroot(['foo', 'foo.bar'], [], {'foo.bar1': ['f0']}, py3=py36) as chroot:
+      _, _, r = setup_py.find_packages(dummy_target, chroot)
+      assert r == {'foo': {'bar1/f0'}}
+
+  def test_find_packages(self):
+    self.assert_find_packages(py36=False)
+
+  @skip_unless_python36
+  def test_find_packages_py3(self):
+    self.assert_find_packages(py36=True)
 
 
 def test_nearest_subpackage():
diff --git a/tests/python/pants_test/backend/python/test_interpreter_cache.py b/tests/python/pants_test/backend/python/test_interpreter_cache.py
--- a/tests/python/pants_test/backend/python/test_interpreter_cache.py
+++ b/tests/python/pants_test/backend/python/test_interpreter_cache.py
@@ -9,14 +9,19 @@
 from contextlib import contextmanager
 
 import mock
+from future.utils import PY3
 from pex.package import EggPackage, Package, SourcePackage
 from pex.resolver import Unsatisfiable, resolve
 
 from pants.backend.python.interpreter_cache import PythonInterpreter, PythonInterpreterCache
 from pants.backend.python.subsystems.python_repos import PythonRepos
 from pants.backend.python.subsystems.python_setup import PythonSetup
+from pants.subsystem.subsystem import Subsystem
 from pants.util.contextutil import temporary_dir
-from pants_test.pants_run_integration_test import PantsRunIntegrationTest
+from pants_test.backend.python.interpreter_selection_utils import (PY_27, PY_36,
+                                                                   python_interpreter_path,
+                                                                   skip_unless_python27_and_python36)
+from pants_test.subsystem.subsystem_util import global_subsystem_instance
 from pants_test.test_base import TestBase
 from pants_test.testutils.pexrc_util import setup_pexrc_with_pex_python_path
 
@@ -28,42 +33,50 @@ def _make_bad_requirement(requirement):
 
     E.g. 'CPython==2.7.5' becomes 'CPython==99.7.5'
     """
-    return str(requirement).replace('==2.', '==99.')
+    if PY3:
+      return str(requirement).replace('==3', '==99')
+    else:
+      return str(requirement).replace('==2.', '==99')
 
   def setUp(self):
     super(TestInterpreterCache, self).setUp()
     self._interpreter = PythonInterpreter.get()
 
-  @contextmanager
-  def _setup_test(self, constraints=None, mock_setup_paths_interpreters=None):
-    mock_setup = mock.MagicMock().return_value
-    type(mock_setup).interpreter_constraints = mock.PropertyMock(return_value=constraints)
+  def create_python_subsystems(self, setup_options=None, repos_options=None):
+    Subsystem.reset(reset_options=True)
+    def create_subsystem(subsystem_type, options=None):
+      return global_subsystem_instance(subsystem_type,
+                                       options={subsystem_type.options_scope: options or {}})
+    return (create_subsystem(PythonSetup, setup_options),
+            create_subsystem(PythonRepos, repos_options))
 
+  @contextmanager
+  def _setup_cache(self, constraints=None):
     with temporary_dir() as path:
-      mock_setup.interpreter_cache_dir = path
-      cache = PythonInterpreterCache(mock_setup, mock.MagicMock())
-      cache._setup_cached = mock.Mock(return_value=[self._interpreter])
-      if mock_setup_paths_interpreters:
-        cache._setup_paths = mock.Mock(return_value=[PythonInterpreter.from_binary(mock_setup_paths_interpreters[0]),
-                                                     PythonInterpreter.from_binary(mock_setup_paths_interpreters[1])])
-      else:
-        cache._setup_paths = mock.Mock(return_value=[])
+      cache = self._setup_cache_at(path, constraints=constraints)
       yield cache, path
 
-  def _do_test(self, constraints, filters, expected):
-    with self._setup_test(constraints) as (cache, _):
-      self.assertEqual(cache.setup(filters=filters), expected)
+  def _setup_cache_at(self, path, constraints=None):
+    setup_options = {'interpreter_cache_dir': path}
+    if constraints:
+      setup_options.update(interpreter_constraints=constraints)
+    python_setup, python_repos = self.create_python_subsystems(setup_options=setup_options)
+    return PythonInterpreterCache(python_setup=python_setup, python_repos=python_repos)
 
   def test_cache_setup_with_no_filters_uses_repo_default_excluded(self):
-    self._do_test([self._make_bad_requirement(self._interpreter.identity.requirement)], [], [])
+    bad_interpreter_requirement = self._make_bad_requirement(self._interpreter.identity.requirement)
+    with self._setup_cache(constraints=[bad_interpreter_requirement]) as (cache, _):
+      self.assertEqual([], cache.setup())
 
   def test_cache_setup_with_no_filters_uses_repo_default(self):
-    self._do_test((b'',), [], [self._interpreter])
+    with self._setup_cache(constraints=['']) as (cache, _):
+      self.assertIn(self._interpreter, cache.setup())
 
   def test_cache_setup_with_filter_overrides_repo_default(self):
-    self._do_test([self._make_bad_requirement(self._interpreter.identity.requirement)],
-                  (str(self._interpreter.identity.requirement), ),
-                  [self._interpreter])
+    bad_interpreter_requirement = self._make_bad_requirement(self._interpreter.identity.requirement)
+    with self._setup_cache(constraints=[bad_interpreter_requirement]) as (cache, _):
+      self.assertIn(self._interpreter,
+                    cache.setup(filters=(str(self._interpreter.identity.requirement),)))
 
   def test_setup_using_eggs(self):
     def link_egg(repo_root, requirement):
@@ -91,20 +104,20 @@ def link_egg(repo_root, requirement):
 
       interpreter_requirement = self._interpreter.identity.requirement
 
-      self.context(for_subsystems=[PythonSetup, PythonRepos], options={
-        PythonSetup.options_scope: {
+      python_setup, python_repos = self.create_python_subsystems(
+        setup_options={
           'interpreter_cache_dir': None,
           'pants_workdir': os.path.join(root, 'workdir'),
           'constraints': [interpreter_requirement],
           'setuptools_version': setuptools_version,
           'wheel_version': wheel_version,
         },
-        PythonRepos.options_scope: {
+        repos_options={
           'indexes': [],
           'repos': [egg_dir],
         }
-      })
-      cache = PythonInterpreterCache(PythonSetup.global_instance(), PythonRepos.global_instance())
+      )
+      cache = PythonInterpreterCache(python_setup=python_setup, python_repos=python_repos)
 
       interpereters = cache.setup(paths=[os.path.dirname(self._interpreter.binary)],
                                   filters=[str(interpreter_requirement)])
@@ -125,7 +138,7 @@ def test_setup_resolve_failure_cleanup(self):
     See https://github.com/pantsbuild/pants/issues/2038 for more info.
     """
     with mock.patch.object(PythonInterpreterCache, '_resolve') as mock_resolve, \
-         self._setup_test() as (cache, cache_path):
+         self._setup_cache() as (cache, cache_path):
       mock_resolve.side_effect = Unsatisfiable('nope')
 
       with self.assertRaises(Unsatisfiable):
@@ -138,49 +151,33 @@ def test_setup_resolve_failure_cleanup(self):
       self.assertFalse('.tmp.' in ' '.join(os.listdir(cache_path)),
                        'interpreter cache path contains tmp dirs!')
 
+  @skip_unless_python27_and_python36
   def test_pex_python_paths(self):
     """Test pex python path helper method of PythonInterpreterCache."""
-    py27 = '2'
-    py3 = '3'
-    if PantsRunIntegrationTest.has_python_version(py27) and PantsRunIntegrationTest.has_python_version(py3):
-      print('Found both python {} and python {}. Running test.'.format(py27, py3))
-      py27_path = PantsRunIntegrationTest.python_interpreter_path(py27)
-      py3_path = PantsRunIntegrationTest.python_interpreter_path(py3)
-      with self._setup_test() as (cache, cache_path):
-        with setup_pexrc_with_pex_python_path('~/.pexrc', [py27_path, py3_path]):
-          pex_python_paths = cache.pex_python_paths()
-          self.assertEqual(pex_python_paths, [py27_path, py3_path])
-
-  def test_interpereter_cache_setup_using_pex_python_paths(self): 
+    py27_path, py36_path = python_interpreter_path(PY_27), python_interpreter_path(PY_36)
+    with setup_pexrc_with_pex_python_path([py27_path, py36_path]):
+      with self._setup_cache() as (cache, _):
+        pex_python_paths = cache.pex_python_paths()
+        self.assertEqual(pex_python_paths, [py27_path, py36_path])
+
+  @skip_unless_python27_and_python36
+  def test_interpereter_cache_setup_using_pex_python_paths(self):
     """Test cache setup using interpreters from a mocked PEX_PYTHON_PATH."""
-    py27 = '2'
-    py3 = '3'
-    if PantsRunIntegrationTest.has_python_version(py27) and PantsRunIntegrationTest.has_python_version(py3):
-      print('Found both python {} and python {}. Running test.'.format(py27, py3))
-      py27_path = PantsRunIntegrationTest.python_interpreter_path(py27)
-      py3_path = PantsRunIntegrationTest.python_interpreter_path(py3)
-      with setup_pexrc_with_pex_python_path('~/.pexrc', [py27_path, py3_path]):
-        # Target python 2 for interpreter cache.
-        with self._setup_test(constraints=['CPython>=2.7'],
-                              mock_setup_paths_interpreters=(py27_path, py3_path)) as (cache, cache_path):
-          interpreters = cache.setup()
-          # The majority of systems are able to satisfy the above constraint without needing to use setup paths,
-          # so checking for presence of compatible interpreters is sufficient. Constraints are typcially met
-          # by a system interpreter and as a result, the Pants venv interpreter from `py27_path` does not get
-          # added to the interpreter cache.
-          self.assertGreater(len(interpreters), 0)
-        # Target python 3 for interpreter cache.
-        with self._setup_test(constraints=['CPython>=3'],
-                              mock_setup_paths_interpreters=(py27_path, py3_path)) as (cache, cache_path):
-          interpreters = cache.setup()
-          self.assertGreater(len(interpreters), 0)
-          # Travis CI does not support `python3.6` from the command line, so we just have to check
-          # that the interpreter path of `py3_path` lines up with a cached interpreter. For this
-          # case,`py3_path` is just missing '.6' at the end of the binary's path basename.
-          # Ex:
-          #   p36_path = /path/to/python3
-          #   pi.binary = /path/to/python3.6
-          assert any([py3_path in pi.binary for pi in interpreters])
-    else:
-      print('Could not find both python {} and python {} on system. Skipping.'.format(py27, py3))
-      self.skipTest('Missing neccesary Python interpreters on system.')
+    py27_path, py36_path = python_interpreter_path(PY_27), python_interpreter_path(PY_36)
+    with setup_pexrc_with_pex_python_path([py27_path, py36_path]):
+      with self._setup_cache(constraints=['CPython>=2.7,<3']) as (cache, _):
+        self.assertIn(py27_path, {pi.binary for pi in cache.setup()})
+      with self._setup_cache(constraints=['CPython>=3.6,<4']) as (cache, _):
+        self.assertIn(py36_path, {pi.binary for pi in cache.setup()})
+
+  def test_setup_cached_warm(self):
+    with self._setup_cache() as (cache, path):
+      interpreters = cache.setup()
+      self.assertGreater(len(interpreters), 0)
+
+      cache = self._setup_cache_at(path)
+      self.assertEqual(sorted(interpreters), sorted(list(cache._setup_cached())))
+
+  def test_setup_cached_cold(self):
+    with self._setup_cache() as (cache, _):
+      self.assertEqual([], list(cache._setup_cached()))
diff --git a/tests/python/pants_test/pants_run_integration_test.py b/tests/python/pants_test/pants_run_integration_test.py
--- a/tests/python/pants_test/pants_run_integration_test.py
+++ b/tests/python/pants_test/pants_run_integration_test.py
@@ -112,28 +112,6 @@ def hermetic_env_whitelist(cls):
         'PANTS_PROFILE',
       ]
 
-  @classmethod
-  def has_python_version(cls, version):
-    """Returns true if the current system has the specified version of python.
-
-    :param version: A python version string, such as 2.7, 3.
-    """
-    return cls.python_interpreter_path(version) is not None
-
-  @classmethod
-  def python_interpreter_path(cls, version):
-    """Returns the interpreter path if the current system has the specified version of python.
-
-    :param version: A python version string, such as 2.7, 3.
-    """
-    try:
-      py_path = subprocess.check_output(['python%s' % version,
-                                         '-c',
-                                         'import sys; print(sys.executable)']).strip()
-      return os.path.realpath(py_path)
-    except OSError:
-      return None
-
   def setUp(self):
     super(PantsRunIntegrationTest, self).setUp()
     # Some integration tests rely on clean subsystem state (e.g., to set up a DistributionLocator).
diff --git a/tests/python/pants_test/python/BUILD b/tests/python/pants_test/python/BUILD
deleted file mode 100644
--- a/tests/python/pants_test/python/BUILD
+++ /dev/null
@@ -1,25 +0,0 @@
-# Copyright 2014 Pants project contributors (see CONTRIBUTORS.md).
-# Licensed under the Apache License, Version 2.0 (see LICENSE).
-
-
-python_tests(
-  name = 'interpreter_selection_integration',
-  sources = ['test_interpreter_selection_integration.py'],
-  dependencies = [
-    'src/python/pants/util:contextutil',
-    'src/python/pants/util:process_handler',
-    'tests/python/pants_test:int-test',
-  ],
-  tags = {'integration'},
-)
-
-python_tests(
-  name = 'python_binary_integration',
-  sources = ['test_python_binary_integration.py'],
-  dependencies = [
-    '3rdparty/python:pex',
-    'src/python/pants/util:contextutil',
-    'tests/python/pants_test:int-test',
-  ],
-  tags = {'integration'},
-)
diff --git a/tests/python/pants_test/testutils/pexrc_util.py b/tests/python/pants_test/testutils/pexrc_util.py
--- a/tests/python/pants_test/testutils/pexrc_util.py
+++ b/tests/python/pants_test/testutils/pexrc_util.py
@@ -7,27 +7,30 @@
 import os
 from contextlib import contextmanager
 
+from pants.base.build_environment import get_pants_cachedir
+from pants.util.contextutil import environment_as, temporary_dir
+from pants.util.dirutil import safe_mkdir_for
+
 
 @contextmanager
-def setup_pexrc_with_pex_python_path(pexrc_path, interpreter_paths):
-  """A helper function for writing interpreter paths to a PEX_PYTHON_PATH variable
-  in a .pexrc file. This function raise an error if a .pexrc file already exists at
-  `pexrc_path`.
-
-  :param pexrc_path (str): a path to a temporary .pexrc to write for testing purposes.
-  :param interpreter_paths (list): a list of paths to interpreter binaries to include on 
-  PEX_PYTHON_PATH.
+def setup_pexrc_with_pex_python_path(interpreter_paths):
+  """A helper function for writing interpreter paths to a PEX_PYTHON_PATH variable in a .pexrc file.
+
+  NB: Mutates HOME and XDG_CACHE_HOME to ensure a `~/.pexrc` that won't trample any existing file
+  and will also be found.
+
+  :param list interpreter_paths: a list of paths to interpreter binaries to include on
+                                 PEX_PYTHON_PATH.
   """
-  pexrc_path = os.path.expanduser(pexrc_path)
-  if os.path.exists(pexrc_path):
-    raise RuntimeError("A pexrc file already exists in {}".format(pexrc_path))
-
-  # Write a temp .pexrc in pexrc_dir.
-  with open(pexrc_path, 'w') as pexrc:
-    pexrc.write("PEX_PYTHON_PATH=%s" % ':'.join(interpreter_paths))
-  
-  try:
-    yield
-  finally:
-    # Cleanup temporary .pexrc.
-    os.remove(pexrc_path)
+  cache_dir = get_pants_cachedir()
+  with temporary_dir() as home:
+    xdg_cache_home = os.path.join(home, '.cache')
+    with environment_as(HOME=home, XDG_CACHE_HOME=xdg_cache_home):
+      target = os.path.join(xdg_cache_home, os.path.basename(cache_dir))
+      safe_mkdir_for(target)
+      os.symlink(cache_dir, target)
+
+      with open(os.path.join(home, '.pexrc'), 'w') as pexrc:
+        pexrc.write('PEX_PYTHON_PATH={}'.format(':'.join(interpreter_paths)))
+
+      yield