Repository: pex-tool/pex

Issue:
pex 1.5.1 bombs with unhandled `pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName` on certain resolves
repro:

```
[omerta show]$ wget -q https://github.com/pantsbuild/pex/releases/download/v1.5.1/pex27 -O pex27-1.5.1
[omerta show]$ ./pex27-1.5.1 apache-beam[gcp]==2.7.0
Traceback (most recent call last):
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/_pex/pex.py", line 349, in execute
    working_set = self._activate()
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/_pex/pex.py", line 90, in _activate
    for dist in env.activate():
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/_pex/environment.py", line 152, in activate
    self._working_set = self._activate()
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/_pex/environment.py", line 211, in _activate
    resolved = self._resolve(working_set, all_reqs)
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/_pex/environment.py", line 165, in _resolve
    if req.marker and not req.marker.evaluate(environment=self._target_interpreter_env):
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/pkg_resources/_vendor/packaging/markers.py", line 301, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/pkg_resources/_vendor/packaging/markers.py", line 226, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/private/var/folders/kd/k5b__qgs0gl5yzp48gctmhz80000gn/T/tmp_XY7Uj/.bootstrap/pkg_resources/_vendor/packaging/markers.py", line 208, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.
```

vs 1.4.4:

```
[omerta show]$ wget -q https://github.com/pantsbuild/pex/releases/download/v1.4.4/pex27 -O pex27-1.4.4
[omerta show]$ ./pex27-1.4.4 apache-beam[gcp]==2.7.0
Python 2.7.15 (default, Jun 20 2018, 17:40:09) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> 
```


Discussion:
noting the shape of the markers in the failure case is (from the apache-beam wheel):

```
Provides-Extra: gcp
Requires-Dist: google-apitools (<=0.5.20,>=0.5.18); extra == 'gcp'
Requires-Dist: proto-google-cloud-datastore-v1 (<=0.90.4,>=0.90.0); extra == 'gcp'
Requires-Dist: googledatastore (==7.0.1); extra == 'gcp'
Requires-Dist: google-cloud-pubsub (==0.26.0); extra == 'gcp'
Requires-Dist: proto-google-cloud-pubsub-v1 (==0.15.4); extra == 'gcp'
Requires-Dist: google-cloud-bigquery (==0.25.0); extra == 'gcp'
```
OK - thanks for the report. Easy repro and the fix appears to be tiny.
The bug is not on the resolution side, but in the activation side where we filter out embedded dists not appropriate to the current enviornment the pex is running in.

Resolving patch:
diff --git a/pex/environment.py b/pex/environment.py
--- a/pex/environment.py
+++ b/pex/environment.py
@@ -159,10 +159,13 @@ def _resolve(self, working_set, reqs):
     unresolved_reqs = set()
     resolveds = set()
 
+    environment = self._target_interpreter_env.copy()
+    environment['extra'] = list(set(itertools.chain(*(req.extras for req in reqs))))
+
     # Resolve them one at a time so that we can figure out which ones we need to elide should
     # there be an interpreter incompatibility.
     for req in reqs:
-      if req.marker and not req.marker.evaluate(environment=self._target_interpreter_env):
+      if req.marker and not req.marker.evaluate(environment=environment):
         TRACER.log('Skipping activation of `%s` due to environment marker de-selection' % req)
         continue
       with TRACER.timed('Resolving %s' % req, V=2):


Test patch:
diff --git a/tests/test_environment.py b/tests/test_environment.py
--- a/tests/test_environment.py
+++ b/tests/test_environment.py
@@ -163,3 +163,24 @@ def bad_interpreter(include_site_extras=True):
       release, _, _ = platform.mac_ver()
       major_minor = '.'.join(release.split('.')[:2])
       assert to_bytes('macosx-{}-intel'.format(major_minor)) == stdout.strip()
+
+
+def test_activate_extras_issue_615():
+  with yield_pex_builder() as pb:
+    for resolved_dist in resolver.resolve(['pex[requests]==1.5.1']):
+      pb.add_requirement(resolved_dist.requirement)
+      pb.add_dist_location(resolved_dist.distribution.location)
+    pb.set_script('pex')
+    pb.freeze()
+    env = os.environ.copy()
+    env['PEX_VERBOSE'] = '9'
+    process = PEX(pb.path()).run(args=['--version'],
+                                 env=env,
+                                 blocking=False,
+                                 stdout=subprocess.PIPE,
+                                 stderr=subprocess.PIPE)
+    stdout, stderr = process.communicate()
+    assert 0 == process.returncode, (
+      'Process failed with exit code {} and stderr:\n{}'.format(process.returncode, stderr)
+    )
+    assert to_bytes('{} 1.5.1'.format(os.path.basename(pb.path()))) == stdout.strip()