Repository: pex-tool/pex

Issue:
Implement long-term solution for cross-interpreter requirement handling the in pex resolver
When a setup.py file specifies `install_requires` requirements with interpreter-specific constraints, the pex resolver cannot parse and handle these constraints to produce a cross-interpreter pex. See https://github.com/Julian/jsonschema/blob/master/setup.py#L43 for an example of such a constraint. 

Interpreter constraints like this are typically for stdlib backport libraries and the like that only make sense to include in a universal dist, so this problem is not as significant when the pex is targeting a specific interpreter version. In the targeted case, the resolver only needs to ignore requirements that are pinned to non-targeted interpreter versions. The short term workaround for the targeted-interpreter case is to allow the caller of `pex.resolver#resolve` to supply a blacklist dict that maps package name string -> interpreter constraint string, allowing the resolver to effectively skip them as implemented in https://github.com/pantsbuild/pex/pull/457. 

The scope of this ticket is to remove the blacklist implementation and replace it with smarter handling for the "pex-targeting-X-interpreter-version" case, specifically where all requirements are compared against interpreter constraints for a pex. Then at run time, the pex environment selectively activates packaged dependencies in accordance with a given pex's interpreter constraints. If no constraints are specified and the pex is intended to run with Python 2 or 3, then there should be support for determining this at runtime and handling env activation accordingly as well. 

This approach would solve the cross-interpreter case. This is a similar problem to the one surfaced in #455, however that issue deals with cross-platform specifications, but this is effectively the same concept.


Discussion:
Is there any workaround for this currently available?
I tried something like:
```sh
$ pex jupyter --python=python2.7 --python=python3.4 -o jup_pex.pex
```
Which results in:
```
**** Failed to install functools32-3.2.3-2 (caused by: NonZeroExit("received exit code 1 during execution of `['/Users/kieran/.pyenv/versions/3.4.8/bin/python3.4', '-', 'bdist_wheel', '--dist-dir=/var/folders/7d/cnv90sjj7rx4nhq38nvx44nr0000gn/T/tmpZEG5fd']` while trying to execute `['/Users/kieran/.pyenv/versions/3.4.8/bin/python3.4', '-', 'bdist_wheel', '--dist-dir=/var/folders/7d/cnv90sjj7rx4nhq38nvx44nr0000gn/T/tmpZEG5fd']`",)
):
stdout:

stderr:
This backport is for Python 2.7 only.


Traceback (most recent call last):
  File "/Users/kieran/.pyenv/versions/2.7.13/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/bin/pex.py", line 663, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/bin/pex.py", line 601, in build_pex
    for dist in resolveds:
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/resolver.py", line 438, in resolve_multi
    allow_prereleases):
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/resolver.py", line 376, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder))
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/resolver.py", line 209, in resolve
    dist = self.build(package, resolvable.options)
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/resolver.py", line 282, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/Users/kieran/.pyenv/versions/2.7.13/lib/python2.7/site-packages/pex/resolver.py", line 177, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage('file:///Users/kieran/.pex/build/functools32-3.2.3-2.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
```
Due to, I believe, the specific line in jsonschema's setup py (a nested subdependency of jupyter) that you referenced above.

Tried flattering my dependencies and trying to specify a environment marker directly in `requirements.txt` in an attempt to help pex's resolver out, so something like:
```sh
$ echo "functools32;python_version<'3.0'\njupyter" >> requirements.txt
$ pex -r requirements.txt --python=python2.7 --python=python3.4 -o jup_pex.pex
```
Which lead to the same error. So it would seem that pex fails to parse environment markers in `requirements.txt` ([PEP508](https://www.python.org/dev/peps/pep-0508/#environment-markers)) as well as in `install_requires` of sub-packages.

Anyone have a different work around idea?
I was unable to reproduce the error with that command, but it appears to be the same issue that I am describing here. There is no workaround that I am aware of, so unfortunately you will need to either build multiple pex files or target only one interpreter for the time being.
Hm, wish I could.
My use case is bundling python dependencies for a spark apps on a cluster where (clients, don't ask me why) some of the data nodes have varying versions of python installed.

My plan was to create a multi-interpreter pex file, ship it, and then specify `PYSPARK_PYTHON=/path/to/multi_interpreter.pex`, which would then act as the node's py interpreter, picking up on the actual python installed on the node but with the correct libraries depended on by a given app. Specifying different paths for nodes on a version dependent basis can be a bit messy.

Simple tests of this technique with `SciPy` worked, so I figured if it could bundle all that c and fortran, other dependencies would be fine. Whoops 😅

I'm still very interested in pursuing this approach. What does the timeline look like on this work?

I'll give the source a closer read and see if I can be of assistance as well.
I see. In terms of a timeline on my availability to knock this out, I would say it would be ~1 month to a merged fix. The long-term solution boils down to providing the pex resolver module with the intelligence to parse environmental markers correctly and propagate its decisions to include/exclude a requirement through to the PEX-INFO metadata. That second part is to ensure that there are no requirements that are registered but actually excluded due to an incompatible marker.
As discovered in https://github.com/pantsbuild/pants/pull/6613 there is a bit more work needed here to filter out top-level requirements passed to pex. The work in #582 only hit transitive requirements.

Resolving patch:
diff --git a/pex/environment.py b/pex/environment.py
--- a/pex/environment.py
+++ b/pex/environment.py
@@ -130,6 +130,7 @@ def __init__(self, pex, pex_info, interpreter=None, **kw):
       platform=platform_name,
       **kw
     )
+    self._target_interpreter_env = self._interpreter.identity.pkg_resources_env(platform_name)
     self._supported_tags.extend(platform.supported_tags(self._interpreter))
     TRACER.log(
       'E: tags for %r x %r -> %s' % (self.platform, self._interpreter, self._supported_tags),
@@ -161,7 +162,7 @@ def _resolve(self, working_set, reqs):
     # Resolve them one at a time so that we can figure out which ones we need to elide should
     # there be an interpreter incompatibility.
     for req in reqs:
-      if req.marker and not req.marker.evaluate():
+      if req.marker and not req.marker.evaluate(environment=self._target_interpreter_env):
         TRACER.log('Skipping activation of `%s` due to environment marker de-selection' % req)
         continue
       with TRACER.timed('Resolving %s' % req, V=2):
diff --git a/pex/resolver.py b/pex/resolver.py
--- a/pex/resolver.py
+++ b/pex/resolver.py
@@ -220,6 +220,8 @@ def __init__(self, allow_prereleases=None, interpreter=None, platform=None, use_
     self._interpreter = interpreter or PythonInterpreter.get()
     self._platform = self._maybe_expand_platform(self._interpreter, platform)
     self._allow_prereleases = allow_prereleases
+    platform_name = self._platform.platform
+    self._target_interpreter_env = self._interpreter.identity.pkg_resources_env(platform_name)
     self._supported_tags = self._platform.supported_tags(
       self._interpreter,
       use_manylinux
@@ -259,8 +261,17 @@ def build(self, package, options):
         'Could not get distribution for %s on platform %s.' % (package, self._platform))
     return dist
 
+  def is_resolvable_in_target_interpreter_env(self, resolvable):
+    if not isinstance(resolvable, ResolvableRequirement):
+      return True
+    elif resolvable.requirement.marker is None:
+      return True
+    else:
+      return resolvable.requirement.marker.evaluate(environment=self._target_interpreter_env)
+
   def resolve(self, resolvables, resolvable_set=None):
-    resolvables = [(resolvable, None) for resolvable in resolvables]
+    resolvables = [(resolvable, None) for resolvable in resolvables
+                   if self.is_resolvable_in_target_interpreter_env(resolvable)]
     resolvable_set = resolvable_set or _ResolvableSet()
     processed_resolvables = set()
     processed_packages = {}
@@ -297,9 +308,7 @@ def resolve(self, resolvables, resolvable_set=None):
         new_parent = '%s->%s' % (parent, resolvable) if parent else str(resolvable)
         # We patch packaging.markers.default_environment here so we find optional reqs for the
         # platform we're building the PEX for, rather than the one we're on.
-        with patched_packing_env(
-          self._interpreter.identity.pkg_resources_env(self._platform.platform)
-        ):
+        with patched_packing_env(self._target_interpreter_env):
           resolvables.extend(
             (ResolvableRequirement(req, resolvable.options), new_parent) for req in
             distribution.requires(extras=resolvable_set.extras(resolvable.name))


Test patch:
diff --git a/tests/test_integration.py b/tests/test_integration.py
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -1112,7 +1112,7 @@ def test_setup_interpreter_constraint():
 @pytest.mark.skipif(IS_PYPY,
                     reason='Our pyenv interpreter setup fails under pypy: '
                            'https://github.com/pantsbuild/pex/issues/477')
-def test_setup_python_multiple():
+def test_setup_python_multiple_transitive_markers():
   py27_interpreter = ensure_python_interpreter(PY27)
   py36_interpreter = ensure_python_interpreter(PY36)
   with temporary_dir() as out:
@@ -1143,3 +1143,50 @@ def test_setup_python_multiple():
 
       stdout = subprocess.check_output(both_program)
       assert to_bytes(os.path.realpath(py36_interpreter)) == stdout.strip()
+
+
+@pytest.mark.skipif(IS_PYPY,
+                    reason='Our pyenv interpreter setup fails under pypy: '
+                           'https://github.com/pantsbuild/pex/issues/477')
+def test_setup_python_direct_markers():
+  py36_interpreter = ensure_python_interpreter(PY36)
+  with temporary_dir() as out:
+    pex = os.path.join(out, 'pex.pex')
+    results = run_pex_command(['subprocess32==3.2.7; python_version<"3"',
+                               '--disable-cache',
+                               '--python-shebang=#!/usr/bin/env python',
+                               '--python={}'.format(py36_interpreter),
+                               '-o', pex])
+    results.assert_success()
+
+    py2_only_program = [pex, '-c', 'import subprocess32']
+
+    with environment_as(PATH=os.path.dirname(py36_interpreter)):
+      with pytest.raises(subprocess.CalledProcessError):
+        subprocess.check_call(py2_only_program)
+
+
+@pytest.mark.skipif(IS_PYPY,
+                    reason='Our pyenv interpreter setup fails under pypy: '
+                           'https://github.com/pantsbuild/pex/issues/477')
+def test_setup_python_multiple_direct_markers():
+  py36_interpreter = ensure_python_interpreter(PY36)
+  py27_interpreter = ensure_python_interpreter(PY27)
+  with temporary_dir() as out:
+    pex = os.path.join(out, 'pex.pex')
+    results = run_pex_command(['subprocess32==3.2.7; python_version<"3"',
+                               '--disable-cache',
+                               '--python-shebang=#!/usr/bin/env python',
+                               '--python={}'.format(py36_interpreter),
+                               '--python={}'.format(py27_interpreter),
+                               '-o', pex])
+    results.assert_success()
+
+    py2_only_program = [pex, '-c', 'import subprocess32']
+
+    with environment_as(PATH=os.path.dirname(py36_interpreter)):
+      with pytest.raises(subprocess.CalledProcessError):
+        subprocess.check_call(py2_only_program)
+
+    with environment_as(PATH=os.path.dirname(py27_interpreter)):
+      subprocess.check_call(py2_only_program)