Repository: pex-tool/pex

Issue:
Top level environment markers are lost in built PEXes.
When any requirements passed to the Pex CLI have environment markers, these should be recorded in the resulting PEX / PEX-INFO but currently are not. This can lead to erroneous runtime activation failure:
```
$ python -mpex --python=python2.7 --python=python3.6 "subprocess32==3.2.7; python_version<'3'" -o maybe_subprocess32.pex
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
$ ./maybe_subprocess32.pex 
Python 2.7.17 (default, Oct 22 2019, 09:14:09) 
[GCC 9.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> 
$ python3.6 ./maybe_subprocess32.pex 
Failed to execute PEX file. Needed manylinux2014_x86_64-cp-36-cp36m compatible dependencies for:
 1: subprocess32==3.2.7
    But this pex only contains:
      subprocess32-3.2.7-cp27-cp27mu-linux_x86_64.whl
$ unzip -qc maybe_subprocess32.pex PEX-INFO | jq .
{
  "always_write_cache": false,
  "build_properties": {
    "class": "CPython",
    "pex_version": "2.1.3",
    "platform": "manylinux2014_x86_64",
    "version": [
      2,
      7,
      17
    ]
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "subprocess32-3.2.7-cp27-cp27mu-linux_x86_64.whl": "9c46d1bcfb13ee6660cbd282edbf46be7e941e1f"
  },
  "emit_warnings": true,
  "ignore_errors": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_path": null,
  "requirements": [
    "subprocess32==3.2.7"
  ],
  "zip_safe": true
}
```

Similar to #898 


Discussion:
Fixing this bug will require parsing requirements. In the switch to pip we ~dropped this and started just passing the requirements through to pip as strings.
The full requirements file spec: https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format
And the implementation: https://github.com/pypa/pip/blob/master/src/pip/_internal/req/req_file.py
https://github.com/sarugaku/requirementslib might help
@jhermann requirementslib would be good but it has fairly out of control transitive dependencies for what Pex needs - just requirements.txt parsing:
```
$ pip wheel -w requirementslib requirementslib
$ du -sh requirementslib/
3.4M	requirementslib/
$ ls -1 requirementslib/
appdirs-1.4.3-py2.py3-none-any.whl
attrs-19.3.0-py2.py3-none-any.whl
cached_property-1.5.1-py2.py3-none-any.whl
Cerberus-1.3.2-py3-none-any.whl
certifi-2019.11.28-py2.py3-none-any.whl
chardet-3.0.4-py2.py3-none-any.whl
colorama-0.4.3-py2.py3-none-any.whl
distlib-0.3.0-py3-none-any.whl
first-2.0.2-py2.py3-none-any.whl
idna-2.9-py2.py3-none-any.whl
orderedmultidict-1.0.1-py2.py3-none-any.whl
packaging-20.1-py2.py3-none-any.whl
pep517-0.8.1-py2.py3-none-any.whl
pip-20.0.2-py2.py3-none-any.whl
pip_shims-0.5.0-py2.py3-none-any.whl
plette-0.2.3-py2.py3-none-any.whl
pyparsing-2.4.6-py2.py3-none-any.whl
requests-2.23.0-py2.py3-none-any.whl
requirementslib-1.5.3-py2.py3-none-any.whl
setuptools-45.2.0-py3-none-any.whl
six-1.14.0-py2.py3-none-any.whl
toml-0.10.0-py2.py3-none-any.whl
tomlkit-0.5.8-py2.py3-none-any.whl
urllib3-1.25.8-py2.py3-none-any.whl
vistir-0.5.0-py2.py3-none-any.whl
wheel-0.34.2-py2.py3-none-any.whl
```

I think rolling our own might be the best of bad choices here.
I'd ask them if they're willing to spin off a helper library before that. I found other projects with more of a library character, but they were unmaintained – having a current parsing package would be useful for many.
I stumbled across `pip-api` which is a good deal slimmer: https://github.com/di/pip-api

Resolving patch:
diff --git a/pex/environment.py b/pex/environment.py
--- a/pex/environment.py
+++ b/pex/environment.py
@@ -9,7 +9,7 @@
 import site
 import sys
 import zipfile
-from collections import OrderedDict, defaultdict
+from collections import OrderedDict, defaultdict, namedtuple
 
 from pex import dist_metadata, pex_builder, pex_warnings
 from pex.bootstrap import Bootstrap
@@ -19,13 +19,22 @@
 from pex.orderedset import OrderedSet
 from pex.pex_info import PexInfo
 from pex.third_party.packaging import tags
-from pex.third_party.pkg_resources import DistributionNotFound, Environment, Requirement, WorkingSet
+from pex.third_party.pkg_resources import Distribution, Requirement
 from pex.tracer import TRACER
-from pex.typing import TYPE_CHECKING
+from pex.typing import TYPE_CHECKING, cast
 from pex.util import CacheHelper, DistributionHelper
 
 if TYPE_CHECKING:
-    from typing import Container, Iterator, Optional, Tuple, Iterable
+    from typing import (
+        Container,
+        DefaultDict,
+        Iterable,
+        Iterator,
+        List,
+        Optional,
+        Tuple,
+        Union,
+    )
 
 
 def _import_pkg_resources():
@@ -42,7 +51,40 @@ def _import_pkg_resources():
         return pkg_resources, True
 
 
-class PEXEnvironment(Environment):
+class _RankedDistribution(namedtuple("_RankedDistribution", ["rank", "distribution"])):
+    # N.B.: A distribution implements rich comparison with the leading component being the
+    # `parsed_version`; as such, a _RankedDistribution sorts as a whole 1st by `rank` (which is a
+    # rank of the distribution's tags specificity for the target interpreter), then by version and
+    # finally by redundant components of distribution metadata we never get to since they are
+    # encoded in the tag specificity rank value.
+
+    @classmethod
+    def maximum(cls, distribution=None):
+        # type: (Optional[Distribution]) -> _RankedDistribution
+        return cls(rank=sys.maxsize, distribution=distribution)
+
+    @property
+    def distribution(self):
+        # type: () -> Distribution
+        return cast(Distribution, super(_RankedDistribution, self).distribution)
+
+    def satisfies(self, requirement):
+        # type: (Requirement) -> bool
+        return self.distribution in requirement
+
+
+class _DistributionNotFound(namedtuple("_DistributionNotFound", ["requirement", "required_by"])):
+    @classmethod
+    def create(
+        cls,
+        requirement,  # type: Requirement
+        required_by=None,  # type: Optional[Distribution]
+    ):
+        # type: (...) -> _DistributionNotFound
+        return cls(requirement=requirement, required_by=required_by)
+
+
+class PEXEnvironment(object):
     class _CachingZipImporter(object):
         class _CachingLoader(object):
             def __init__(self, delegate):
@@ -203,14 +245,10 @@ def _load_internal_cache(cls, pex, pex_info):
                 return
 
             if os.path.isdir(pex):
-                search_path = [
-                    os.path.join(internal_cache, dist_chroot)
-                    for dist_chroot in os.listdir(internal_cache)
-                ]
-                internal_env = Environment(search_path=search_path)
-                for dist_name in internal_env:
-                    for dist in internal_env[dist_name]:
-                        yield dist
+                for distribution_name in pex_info.distributions:
+                    yield DistributionHelper.distribution_from_path(
+                        os.path.join(internal_cache, distribution_name)
+                    )
             else:
                 with open_zip(pex) as zf:
                     for dist in cls._write_zipped_internal_cache(zf, pex_info):
@@ -225,12 +263,21 @@ def __init__(
         # type: (...) -> None
         self._pex = pex
         self._pex_info = pex_info or PexInfo.from_pex(pex)
-        self._internal_cache = os.path.join(self._pex, self._pex_info.internal_cache)
-        self._activated = False
-        self._working_set = None
         self._interpreter = interpreter or PythonInterpreter.get()
-        self._inherit_path = self._pex_info.inherit_path
-        self._supported_tags = frozenset(self._interpreter.identity.supported_tags)
+
+        self._available_ranked_dists_by_key = defaultdict(
+            list
+        )  # type: DefaultDict[str, List[_RankedDistribution]]
+        self._activated_dists = None  # type: Optional[Iterable[Distribution]]
+
+        # The supported_tags come ordered most specific (platform specific) to least specific
+        # (universal). We want to rank most specific highest; so we need to reverse iteration order
+        # here.
+        self._supported_tags_to_rank = {
+            tag: rank
+            for rank, tag in enumerate(reversed(self._interpreter.identity.supported_tags))
+        }
+
         self._target_interpreter_env = self._interpreter.identity.env_markers
 
         # For the bug this works around, see: https://bitbucket.org/pypy/pypy/issues/1686
@@ -238,111 +285,220 @@ def __init__(
         if self._interpreter.identity.python_tag.startswith("pp") and zipfile.is_zipfile(self._pex):
             self._install_pypy_zipimporter_workaround(self._pex)
 
-        super(PEXEnvironment, self).__init__(
-            search_path=[] if self._pex_info.inherit_path == InheritPath.FALSE else sys.path,
-            platform=self._interpreter.identity.platform_tag,
-        )
-        TRACER.log(
-            "E: tags for %r x %r -> %s" % (self.platform, self._interpreter, self._supported_tags),
-            V=9,
-        )
-
     def _update_candidate_distributions(self, distribution_iter):
+        # type: (Iterable[Distribution]) -> None
         for dist in distribution_iter:
-            if self.can_add(dist):
+            ranked_dist = self._can_add(dist)
+            if ranked_dist is not None:
                 with TRACER.timed("Adding %s" % dist, V=2):
-                    self.add(dist)
+                    self._available_ranked_dists_by_key[dist.key].append(ranked_dist)
 
-    def can_add(self, dist):
+    def _can_add(self, dist):
+        # type: (Distribution) -> Optional[_RankedDistribution]
         filename, ext = os.path.splitext(os.path.basename(dist.location))
         if ext.lower() != ".whl":
             # This supports resolving pex's own vendored distributions which are vendored in directory
             # directory with the project name (`pip/` for pip) and not the corresponding wheel name
             # (`pip-19.3.1-py2.py3-none-any.whl/` for pip). Pex only vendors universal wheels for all
             # platforms it supports at buildtime and runtime so this is always safe.
-            return True
+            return _RankedDistribution.maximum(dist)
 
         # Wheel filename format: https://www.python.org/dev/peps/pep-0427/#file-name-convention
         # `{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl`
         wheel_components = filename.split("-")
         if len(wheel_components) < 3:
-            return False
+            return None
 
-        wheel_tags = "-".join(wheel_components[-3:])  # `{python tag}-{abi tag}-{platform tag}`
-        if self._supported_tags.isdisjoint(tags.parse_tag(wheel_tags)):
-            return False
+        # `{python tag}-{abi tag}-{platform tag}`
+        wheel_tags = tags.parse_tag("-".join(wheel_components[-3:]))
+        # There will be multiple parsed tags for compressed tag sets. Ensure we grab the parsed tag
+        # with highest rank from that expanded set.
+        rank = max(self._supported_tags_to_rank.get(tag, -1) for tag in wheel_tags)
+        if rank == -1:
+            return None
 
         python_requires = dist_metadata.requires_python(dist)
-        if not python_requires:
-            return True
+        if python_requires and self._interpreter.identity.version_str not in python_requires:
+            return None
 
-        return self._interpreter.identity.version_str in python_requires
+        return _RankedDistribution(rank, dist)
 
     def activate(self):
-        if not self._activated:
+        # type: () -> Iterable[Distribution]
+        if self._activated_dists is None:
             with TRACER.timed("Activating PEX virtual environment from %s" % self._pex):
-                self._working_set = self._activate()
-            self._activated = True
+                self._activated_dists = self._activate()
+        return self._activated_dists
 
-        return self._working_set
+    def _evaluate_marker(
+        self,
+        requirement,  # type: Requirement
+        extras=None,  # type: Optional[Tuple[str, ...]]
+    ):
+        # type: (...) -> bool
+        if requirement.marker is None:
+            return True
+        if not extras:
+            # Provide an empty extra to safely evaluate the markers without matching any extra.
+            extras = ("",)
+        for extra in extras:
+            environment = self._target_interpreter_env.copy()
+            environment["extra"] = extra
+            if requirement.marker.evaluate(environment=environment):
+                return True
+        TRACER.log(
+            "Skipping activation of `{}` due to environment marker de-selection".format(requirement)
+        )
+        return False
 
-    def _resolve(self, working_set, reqs):
-        environment = self._target_interpreter_env.copy()
-        environment["extra"] = list(set(itertools.chain(*(req.extras for req in reqs))))
+    def _resolve_requirement(
+        self,
+        requirement,  # type: Requirement
+        extras=None,  # type: Optional[Tuple[str, ...]]
+        required_by=None,  # type: Optional[Distribution]
+    ):
+        # type: (...) -> Iterator[Union[Distribution, _DistributionNotFound]]
+        if not self._evaluate_marker(requirement, extras=extras):
+            return
+
+        available_distributions = [
+            ranked_dist
+            for ranked_dist in self._available_ranked_dists_by_key.get(requirement.key, [])
+            if ranked_dist.satisfies(requirement)
+        ]
+        if not available_distributions:
+            yield _DistributionNotFound.create(requirement, required_by=required_by)
+
+        resolved_distribution = sorted(available_distributions, reverse=True)[0].distribution
+        if len(available_distributions) > 1:
+            TRACER.log(
+                "Resolved {req} to {dist} and discarded {discarded}.".format(
+                    req=requirement,
+                    dist=resolved_distribution,
+                    discarded=", ".join(
+                        str(ranked_dist.distribution) for ranked_dist in available_distributions[1:]
+                    ),
+                ),
+                V=9,
+            )
 
-        reqs_by_key = OrderedDict()
+        yield resolved_distribution
+        for dep_requirement in dist_metadata.requires_dists(resolved_distribution):
+            # A note regarding extras and why they're passed down one level (we don't pass / use
+            # dep_requirement.extras for example):
+            #
+            # Say we're resolving the `requirement` 'requests[security]==2.25.1'. That means
+            # `resolved_distribution` is the requests distribution. It will have metadata that
+            # looks like so:
+            #
+            # $ grep Requires-Dist requests-2.25.1.dist-info/METADATA | grep security -C1
+            # Requires-Dist: certifi (>=2017.4.17)
+            # Requires-Dist: pyOpenSSL (>=0.14) ; extra == 'security'
+            # Requires-Dist: cryptography (>=1.3.4) ; extra == 'security'
+            # Requires-Dist: PySocks (!=1.5.7,>=1.5.6) ; extra == 'socks'
+            #
+            # We want to recurse and resolve all standard requests requirements but also those that
+            # are part of the 'security' extra. In order to resolve the latter we need to include
+            # the 'security' extra environment marker.
+            for dependency in self._resolve_requirement(
+                dep_requirement, extras=requirement.extras, required_by=resolved_distribution
+            ):
+                yield dependency
+
+    def _root_requirements_iter(self, reqs):
+        # type: (Iterable[Requirement]) -> (Iterator[Requirement])
+
+        # We want to pick one requirement for each key (required project) to then resolve
+        # recursively.
+
+        # First, the selected requirement clearly needs to be applicable (its environment markers
+        # must apply to our interpreter). For example, for a Python 3.6 interpreter this would
+        # select just "isort==5.6.4; python_version>='3.6'" from the input set:
+        # {
+        #   "isort==4.3.21; python_version<'3.6'",
+        #   "setuptools==44.1.1; python_version<'3.6'",
+        #   "isort==5.6.4; python_version>='3.6'",
+        # }
+        reqs_by_key = OrderedDict()  # type: OrderedDict[str, List[Requirement]]
         for req in reqs:
-            if req.marker and not req.marker.evaluate(environment=environment):
+            if not self._evaluate_marker(req):
+                continue
+            requirements = reqs_by_key.get(req.key)
+            if requirements is None:
+                reqs_by_key[req.key] = requirements = []
+            requirements.append(req)
+
+        # Next, from among the remaining applicable requirements for a given project, we want to
+        # select the most tailored (highest ranked) available distribution. That distribution's
+        # transitive requirements will later fill in the full resolve.
+        for key, requirements in reqs_by_key.items():
+            ranked_dists = self._available_ranked_dists_by_key.get(key)
+            if ranked_dists is None:
+                # This can only happen in a multi-platform PEX where the original requirement had
+                # an environment marker and this environment does not satisfy that marker.
                 TRACER.log(
-                    "Skipping activation of `%s` due to environment marker de-selection" % req
+                    "A distribution for {} will not be resolved in this environment.".format(key)
                 )
                 continue
-            reqs_by_key.setdefault(req.key, []).append(req)
+            candidates = [
+                (ranked_dist, requirement)
+                for requirement in requirements
+                for ranked_dist in ranked_dists
+                if ranked_dist.satisfies(requirement)
+            ]
+            ranked_dist, requirement = sorted(candidates, key=lambda tup: tup[0], reverse=True)[0]
+            if len(candidates) > 1:
+                TRACER.log(
+                    "Selected {dist} via {req} and discarded {discarded}.".format(
+                        req=requirement,
+                        dist=ranked_dist.distribution,
+                        discarded=", ".join(
+                            "{dist} via {req}".format(req=req, dist=ranked_dist.distribution)
+                            for ranked_dist, req in candidates[1:]
+                        ),
+                    ),
+                    V=9,
+                )
+            yield requirement
 
-        unresolved_reqs = OrderedDict()
+    def _resolve(self, reqs):
+        # type: (Iterable[Requirement]) -> Iterable[Distribution]
+
+        unresolved_reqs = OrderedDict()  # type: OrderedDict[Requirement, OrderedSet]
         resolveds = OrderedSet()
 
-        # Resolve them one at a time so that we can figure out which ones we need to elide should
-        # there be an interpreter incompatibility.
-        for key, reqs in reqs_by_key.items():
-            with TRACER.timed("Resolving {} from {}".format(key, reqs), V=2):
-                # N.B.: We resolve the bare requirement with no version specifiers since the resolve process
-                # used to build this pex already did so. There may be multiple distributions satisfying any
-                # particular key (e.g.: a Python 2 specific version and a Python 3 specific version for a
-                # multi-python PEX) and we want the working set to pick the most appropriate one.
-                req = Requirement.parse(key)
-                try:
-                    resolveds.update(working_set.resolve([req], env=self))
-                except DistributionNotFound as e:
-                    TRACER.log("Failed to resolve a requirement: %s" % e)
-                    requirers = unresolved_reqs.setdefault(e.req, OrderedSet())
-                    if e.requirers:
-                        for requirer in e.requirers:
-                            requirers.update(reqs_by_key[requirer])
+        for req in self._root_requirements_iter(reqs):
+            with TRACER.timed("Resolving {}".format(req), V=2):
+                for dependency in self._resolve_requirement(req):
+                    if isinstance(dependency, Distribution):
+                        resolveds.add(dependency)
+                    else:
+                        TRACER.log(
+                            "Failed to resolve a requirement: {}".format(dependency.requirement)
+                        )
+                        requirers = unresolved_reqs.get(dependency.requirement)
+                        if requirers is None:
+                            requirers = OrderedSet()
+                        requirers.add(dependency.required_by)
 
         if unresolved_reqs:
             TRACER.log("Unresolved requirements:")
             for req in unresolved_reqs:
-                TRACER.log("  - %s" % req)
+                TRACER.log("  - {}".format(req))
 
             TRACER.log("Distributions contained within this pex:")
-            distributions_by_key = defaultdict(list)
             if not self._pex_info.distributions:
                 TRACER.log("  None")
             else:
-                for dist_name, dist_digest in self._pex_info.distributions.items():
-                    TRACER.log("  - %s" % dist_name)
-                    distribution = DistributionHelper.distribution_from_path(
-                        path=os.path.join(self._pex_info.install_cache, dist_digest, dist_name)
-                    )
-                    distributions_by_key[distribution.as_requirement().key].append(distribution)
+                for dist_name in self._pex_info.distributions:
+                    TRACER.log("  - {}".format(dist_name))
 
             if not self._pex_info.ignore_errors:
                 items = []
                 for index, (requirement, requirers) in enumerate(unresolved_reqs.items()):
                     rendered_requirers = ""
                     if requirers:
-                        rendered_requirers = ("\n    Required by:" "\n      {requirers}").format(
+                        rendered_requirers = "\n    Required by:" "\n      {requirers}".format(
                             requirers="\n      ".join(map(str, requirers))
                         )
 
@@ -355,16 +511,17 @@ def _resolve(self, working_set, reqs):
                             requirement=requirement,
                             rendered_requirers=rendered_requirers,
                             distributions="\n      ".join(
-                                os.path.basename(d.location)
-                                for d in distributions_by_key[requirement.key]
+                                os.path.basename(ranked_dist.distribution.location)
+                                for ranked_dist in self._available_ranked_dists_by_key[
+                                    requirement.key
+                                ]
                             ),
                         )
                     )
 
                 die(
-                    "Failed to execute PEX file. Needed {platform} compatible dependencies for:\n{items}".format(
-                        platform=self._interpreter.platform, items="\n".join(items)
-                    )
+                    "Failed to execute PEX file. Needed {platform} compatible dependencies for:\n"
+                    "{items}".format(platform=self._interpreter.platform, items="\n".join(items))
                 )
 
         return resolveds
@@ -379,7 +536,8 @@ def _get_namespace_packages(cls, dist):
             return []
 
     @classmethod
-    def declare_namespace_packages(cls, resolved_dists):
+    def _declare_namespace_packages(cls, resolved_dists):
+        # type: (Iterable[Distribution]) -> None
         namespace_packages_by_dist = OrderedDict()
         for dist in resolved_dists:
             namespace_packages = cls._get_namespace_packages(dist)
@@ -434,7 +592,7 @@ def declare_namespace_packages(cls, resolved_dists):
                 pkg_resources.declare_namespace(pkg)
 
     def _activate(self):
-        # type: () -> WorkingSet
+        # type: () -> Iterable[Distribution]
         pex_file = os.path.realpath(self._pex)
 
         self._update_candidate_distributions(self._load_internal_cache(pex_file, self._pex_info))
@@ -456,14 +614,11 @@ def _activate(self):
 
         all_reqs = [Requirement.parse(req) for req in self._pex_info.requirements]
 
-        working_set = WorkingSet([])
-        resolved = self._resolve(working_set, all_reqs)
+        resolved = self._resolve(all_reqs)
 
         for dist in resolved:
             with TRACER.timed("Activating %s" % dist, V=2):
-                working_set.add(dist)
-
-                if self._inherit_path == InheritPath.FALLBACK:
+                if self._pex_info.inherit_path == InheritPath.FALLBACK:
                     # Prepend location to sys.path.
                     #
                     # This ensures that bundled versions of libraries will be used before system-installed
@@ -482,4 +637,4 @@ def _activate(self):
                 with TRACER.timed("Adding sitedir", V=2):
                     site.addsitedir(dist.location)
 
-        return working_set
+        return resolved
diff --git a/pex/interpreter.py b/pex/interpreter.py
--- a/pex/interpreter.py
+++ b/pex/interpreter.py
@@ -220,6 +220,7 @@ def version_str(self):
 
     @property
     def supported_tags(self):
+        # type: () -> Tuple[tags.Tag, ...]
         return self._supported_tags
 
     @property
diff --git a/pex/pex.py b/pex/pex.py
--- a/pex/pex.py
+++ b/pex/pex.py
@@ -11,7 +11,6 @@
 from site import USER_SITE
 from types import ModuleType
 
-import pex.third_party.pkg_resources as pkg_resources
 from pex import third_party
 from pex.bootstrap import Bootstrap
 from pex.common import die
@@ -23,7 +22,7 @@
 from pex.interpreter import PythonInterpreter
 from pex.orderedset import OrderedSet
 from pex.pex_info import PexInfo
-from pex.third_party.pkg_resources import EntryPoint, WorkingSet, find_distributions
+from pex.third_party.pkg_resources import Distribution, EntryPoint, find_distributions
 from pex.tracer import TRACER
 from pex.typing import TYPE_CHECKING
 from pex.util import iter_pth_paths, named_temporary_file
@@ -79,7 +78,7 @@ def __init__(
         self._pex_info_overrides = PexInfo.from_env(env=env)
         self._vars = env
         self._envs = []  # type: List[PEXEnvironment]
-        self._working_set = None  # type: Optional[WorkingSet]
+        self._activated_dists = None  # type: Optional[Iterable[Distribution]]
         if verify_entry_point:
             self._do_entry_point_verification()
 
@@ -96,8 +95,7 @@ def interpreter(self):
         return self._interpreter
 
     def _activate(self):
-        # type: () -> WorkingSet
-        working_set = WorkingSet([])
+        # type: () -> Iterable[Distribution]
 
         # set up the local .pex environment
         pex_info = self.pex_info()
@@ -114,24 +112,23 @@ def _activate(self):
                 self._envs.append(PEXEnvironment(pex_path, pex_info, interpreter=self._interpreter))
 
         # activate all of them
+        activated_dists = []  # type: List[Distribution]
         for env in self._envs:
-            for dist in env.activate():
-                working_set.add(dist)
+            activated_dists.extend(env.activate())
 
         # Ensure that pkg_resources is not imported until at least every pex environment
         # (i.e. PEX_PATH) has been merged into the environment
-        PEXEnvironment.declare_namespace_packages(working_set)
-        self.patch_pkg_resources(working_set)
-        return working_set
+        PEXEnvironment._declare_namespace_packages(activated_dists)
+        return activated_dists
 
     def activate(self):
-        # type: () -> WorkingSet
-        if not self._working_set:
+        # type: () -> Iterable[Distribution]
+        if self._activated_dists is None:
             # 1. Scrub the sys.path to present a minimal Python environment.
             self.patch_sys()
             # 2. Activate all code and distributions in the PEX.
-            self._working_set = self._activate()
-        return self._working_set
+            self._activated_dists = self._activate()
+        return self._activated_dists
 
     @classmethod
     def _extras_paths(cls):
@@ -361,15 +358,6 @@ def minimum_sys(cls, inherit_path):
 
         return sys_path, sys_path_importer_cache, sys_modules
 
-    @classmethod
-    def patch_pkg_resources(cls, working_set):
-        """Patch pkg_resources given a new working set."""
-        pkg_resources.working_set = working_set
-        pkg_resources.require = working_set.require
-        pkg_resources.iter_entry_points = working_set.iter_entry_points
-        pkg_resources.run_script = pkg_resources.run_main = working_set.run_script
-        pkg_resources.add_activation_listener = working_set.subscribe
-
     # Thar be dragons -- when this function exits, the interpreter is potentially in a wonky state
     # since the patches here (minimum_sys_modules for example) actually mutate global state.
     def patch_sys(self):


Test patch:
diff --git a/tests/test_environment.py b/tests/test_environment.py
--- a/tests/test_environment.py
+++ b/tests/test_environment.py
@@ -12,7 +12,7 @@
 from pex import resolver
 from pex.common import open_zip, temporary_dir
 from pex.compatibility import PY2, nested, to_bytes
-from pex.environment import PEXEnvironment
+from pex.environment import PEXEnvironment, _RankedDistribution
 from pex.inherit_path import InheritPath
 from pex.interpreter import PythonInterpreter
 from pex.pex import PEX
@@ -32,7 +32,7 @@
 from pex.typing import TYPE_CHECKING
 
 if TYPE_CHECKING:
-    from typing import Any, Dict, Iterable, Iterator, Optional, Tuple
+    from typing import Any, Callable, Dict, Iterable, Iterator, Optional, Tuple
 
 
 @contextmanager
@@ -436,6 +436,13 @@ def create_dist(
     return Distribution(location=location, version=version)
 
 
+@pytest.fixture
+def cpython_35_environment(python_35_interpreter):
+    return PEXEnvironment(
+        pex="", pex_info=PexInfo.default(python_35_interpreter), interpreter=python_35_interpreter
+    )
+
+
 @pytest.mark.parametrize(
     ("wheel_distribution", "wheel_is_linux"),
     [
@@ -462,19 +469,71 @@ def create_dist(
     ],
 )
 def test_can_add_handles_optional_build_tag_in_wheel(
-    python_35_interpreter, wheel_distribution, wheel_is_linux
+    cpython_35_environment, wheel_distribution, wheel_is_linux
 ):
-    # type: (PythonInterpreter, str, bool) -> None
-    pex_environment = PEXEnvironment(
-        pex="", pex_info=PexInfo.default(python_35_interpreter), interpreter=python_35_interpreter
-    )
+    # type: (PEXEnvironment, str, bool) -> None
     native_wheel = IS_LINUX and wheel_is_linux
-    assert pex_environment.can_add(wheel_distribution) is native_wheel
+    added = cpython_35_environment._can_add(wheel_distribution) is not None
+    assert added is native_wheel
 
 
-def test_can_add_handles_invalid_wheel_filename(python_35_interpreter):
-    # type: (PythonInterpreter) -> None
-    pex_environment = PEXEnvironment(
-        pex="", pex_info=PexInfo.default(python_35_interpreter), interpreter=python_35_interpreter
+def test_can_add_handles_invalid_wheel_filename(cpython_35_environment):
+    # type: (PEXEnvironment) -> None
+    assert cpython_35_environment._can_add(create_dist("pep427-invalid.whl")) is None
+
+
+@pytest.fixture
+def assert_cpython_35_environment_can_add(cpython_35_environment):
+    # type: (PEXEnvironment) -> Callable[[Distribution], _RankedDistribution]
+    def assert_can_add(dist):
+        # type: (Distribution) -> _RankedDistribution
+        rank = cpython_35_environment._can_add(dist)
+        assert rank is not None
+        return rank
+
+    return assert_can_add
+
+
+def test_can_add_ranking_platform_tag_more_specific(assert_cpython_35_environment_can_add):
+    # type: (Callable[[Distribution], _RankedDistribution]) -> None
+    ranked_specific = assert_cpython_35_environment_can_add(
+        create_dist("foo-1.0.0-cp35-cp35m-macosx_10_9_x86_64.linux_x86_64.whl", "1.0.0")
+    )
+    ranked_universal = assert_cpython_35_environment_can_add(
+        create_dist("foo-2.0.0-py2.py3-none-any.whl", "2.0.0")
+    )
+    assert ranked_specific > ranked_universal
+
+    ranked_almost_py3universal = assert_cpython_35_environment_can_add(
+        create_dist("foo-2.0.0-py3-none-any.whl", "2.0.0")
+    )
+    assert ranked_universal.rank == ranked_almost_py3universal.rank, (
+        "Expected the 'universal' compressed tag set to be expanded into two tags and the more "
+        "specific tag picked from those two for ranking."
+    )
+
+
+def test_can_add_ranking_version_newer_tie_break(assert_cpython_35_environment_can_add):
+    # type: (Callable[[Distribution], _RankedDistribution]) -> None
+    ranked_v1 = assert_cpython_35_environment_can_add(
+        create_dist("foo-1.0.0-cp35-cp35m-macosx_10_9_x86_64.linux_x86_64.whl", "1.0.0")
+    )
+    ranked_v2 = assert_cpython_35_environment_can_add(
+        create_dist("foo-2.0.0-cp35-cp35m-macosx_10_9_x86_64.linux_x86_64.whl", "2.0.0")
+    )
+    assert ranked_v2 > ranked_v1
+
+
+def test_ranking_platform_tag_maximum(cpython_35_environment):
+    # type: (PEXEnvironment) -> None
+    dist = create_dist("foo-1.0.0-cp35-cp35m-macosx_10_9_x86_64.linux_x86_64.whl", "1.0.0")
+
+    minimum_tag_rank = min(cpython_35_environment._supported_tags_to_rank.values())
+    maximum_tag_rank = max(cpython_35_environment._supported_tags_to_rank.values())
+    assert maximum_tag_rank > minimum_tag_rank
+
+    maximum_rank = _RankedDistribution.maximum(dist)
+    bigger_than_naturally_possible_rank = _RankedDistribution(
+        rank=maximum_tag_rank + 1, distribution=dist
     )
-    assert pex_environment.can_add(create_dist("pep427-invalid.whl")) is False
+    assert maximum_rank > bigger_than_naturally_possible_rank
diff --git a/tests/test_integration.py b/tests/test_integration.py
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -1529,10 +1529,10 @@ def test_setup_python_multiple_direct_markers():
                 stderr=subprocess.STDOUT,
                 env=make_env(PATH=os.path.dirname(py36_interpreter)),
             )
-        # TODO: Is the intention of this test that this should fail on platform incompatibility?
-        #  Or should we be working around the platform issue so that the error we see here is
-        #  b"ModuleNotFoundError: No module named 'subprocess32'" ?
-        assert re.search(b"Needed .* compatible dependencies", err.value.output) is not None
+        assert (
+            re.search(b"ModuleNotFoundError: No module named 'subprocess32'", err.value.output)
+            is not None
+        )
 
         subprocess.check_call(
             py2_only_program, env=make_env(PATH=os.path.dirname(py27_interpreter))