Repository: pex-tool/pex

Issue:
specify (interpreter-constraint) x (platform), and pass --python-version to pip regardless of a local binary
When i run `pex.resolver.resolve_multi()`, it asks for both `interpreters` and `platforms` separately. A few questions:
1. Is it possible to construct a `PythonInterpreter` without actually having a matching interpreter binary on the local disk?
2. If the answer to (1) is "no", is there any way to create a pex file with requirements downloaded for an interpreter I don't have on my local machine?
3. If we *are* restricting `PythonInterpreter` to only local files as per (1), why wouldn't the `platform` argument be something we could infer from the interpreter binary path? It looks like `PythonInterpreter` has this `supported_platforms` property. Is there a reason we wouldn't just use that, instead of having to spread that information over a separate `platforms` kwarg to `resolve_multi()`?

It seems like `(interpreter-constraint) x (platform)` are the axes that pip actually uses to resolve, and I'm wondering why we specify those separately, and why we require an interpreter necessarily on the local disk. One alternative might be:
```python
# Monolithic resolve method, instead of a separate resolve_multi().
def resolve(..., matching_interpreters=None):
    if matching_interpreters is None:
        if interpreter is None and platform is None:
            matching_interpreters = [MatchingInterpreter.from_interpreter(PythonInterpreter.get())]
        else:
            # Produces a list of `MatchingInterpreter`s with all combinations of interpreter and platform.
            matching_interpreters = (InterpreterSet(interpreters=[interpreter])
                                     .for_platforms(platforms=[platform]))
    ...

# Create a declarative interpreter spec matching a python 3.6 interpreter 
# on the local machine  -- this is possible currently by iterating over interpreters and checking
# their compatibility. This is a translation of what is possible today.
# This line *does* touch the filesystem, to iterate over interpreters until a matching one is found.
whatever_the_current_36_is = MatchingInterpreter.from_constraints(['CPython==3.6.*']).

# This line, and all the rest, do *not* touch the filesystem or perform any syscalls.
# If this is in the `matched_interpreters` list when calling resolve(), a pip job will be executed with:
# --python-version 3.6.11 --platform linux_x86_64 --abi cp36m
linux_cp_36_m = MatchingInterpreter('CPython==3.6.11', platform=Platform.create('linux_x86_64-cp-36-m'))

# If we pretend we found a 3.6.11 python interpreter in the first example, the above line would be equivalent
# to the below line, which uses a utility method `.copy()`. This is the direct translation of the current
# method of creating a multiplatform pex, which depends on finding an interpreter on the local disk, and
# then providing a set of other platforms to fetch for (totally separate from that local interpreter version)
# discarding the `.platform` of that local interpreter (necessitating the redundant 'current' shorthand).
linux_cp_36_m = whatever_the_current_36_is.copy(platform=Platform.create('linux_x86_64-cp-36-m'))

# Ask for wheels matching the most recent macosx_10_XY_x86_64 platform,
# because that has the widest compatibility, and many wheels are simply published
# with a platform matching the publisher's laptop, which may be very recent, for no reason.
osx_36_abi3 = MatchingInterpreter('CPython==3.6.11', platform=Platform.most_recent_osx(abi_tag='abi3'))

# Note that *only* == interpreter constraints are allowed when using the normal constructor.
# This is a possible convenience method to match CPython specifically, with shorter strings.
# This should produce the exact same result as the previous line.
 osx_36_abi3 = MatchingInterpreter.CPython('3.6.11', platform=Platform.create('macosx_10_14_x86_64-cp36-abi3'))
```

If this was to be entertained, it seems that it could subsume the functionality of the existing `resolve()` method entirely (translating `interpreters` and `platforms` to a list of `MatchingInterpreter`s), and therefore could avoid breaking any existing users of the pex library API at all (while closing #969).

We would have to figure out how to appropriately error when:
1. the user specifies an interpreter version (or platform) that *doesn't* exist on their machine, and
2. some requirements are platform-specific, and there are no prebuilt wheels for that `(interpreter-version) x (platform)` pair.

However, I believe we already error out today in this case, if we request e.g. `linux_86_64-cp36-abi3` in `platforms`, and there are no prebuilt wheels available for that platform. So we should be able to extend or keep the same error message as before when things don't match up -- we would just be adding a new failure state (which just errors out much earlier today, at interpreter selection time), by allowing the user to resolve for interpreters they don't have on their system.

Regarding the Pex CLI, I believe we could consider an `--explicit-python-version` option corresponding to the first argument of `MatchingInterpreter(...)`, aka:
```bash
> pex --explicit-python-version CPython==3.6.11 ...
```
This flag would be mutually exclusive with both `--python` and `--interpreter-constraints`, but could be applied multiple times to resolve for multiple explicit interpreter versions.


Discussion:
It would actually be very very nice if we could make the `abi_tag` into a python 3 enum (along with values for `PEX_*` config vars and cli args, but that would be more an internal-facing change). For now it's fine to just have a nice error message for invalid tags.
Hmm. Upon reviewing open issues I find that the problem this tries to solve is very similar to #1020, but could be said to extend the goals of that ticket to the Pex library API and CLI, and perhaps clinching a solution to #969.
1st - to be clear - Pex and Pip are offering two unequal forms of resolving. I'll stick to Pex terminology but its a 1-1 map into Pip:
1. Full featured: You have a local interpreter that can be used to build a wheel when no wheel is resolvable but an sdist is.
2. "Foreign": Either the platform you're resolving for is foreign (You're trying to build a platform-specific OSX PEX on linux) or its not, but the targeted interpreter is not present (You're trying to build a Python 2.7 PEX for linux on a linux machine with no Python 2.7 interpreter).

> 1. Is it possible to construct a PythonInterpreter without actually having a matching interpreter binary on the local disk?

No.

> 2. If the answer to (1) is "no", is there any way to create a pex file with requirements downloaded for an interpreter I don't have on my local machine?

Yes - specify the exact platform of that interpreter. IE: the most specific wheel tag it supports.

I'll stop there since it may be the case that the preamble above clears up misconceptions about the role of PythonInterpreter vs Platform when resolving distributions. In short, PythonInterpreters and Platforms should form a disjoint set of DistributionTargets you're attempting to build the PEX for.

> ... and perhaps clinching a solution to #969

#969 already has a solution fully outlined. Its just a name change and the collapse of two functions, one that takes lists and one that takes single items into the one that takes lists since its the most general.
After reading through all your stuff, I think your use of `(interpreter-constraint) x (platform)` sums up the confusion. It is not in fact a cross product that's used to resolve, its `(interpreter-constraint) | (platform)` - ie a heterogenous set of individual targets to resolve for, some local interpreters, some foreign platforms. The code is here:
https://github.com/pantsbuild/pex/blob/b9217232a913e8d32fb78e8c1625f7b56e05f613/pex/resolver.py#L958-L975

If I'm understanding the confusion correctly, maybe you could contribute an API doc fix to clear things up for you?
I think you have absolutely understood the confusion correctly and an API doc fix sounds just right. Would that be in the docstring for the `resolve/resolve_multi()` methods?

> Yes - specify the exact platform of that interpreter. IE: the most specific wheel tag it supports.

This is exactly the answer I was missing earlier! I think that the two methods of resolution you specified map directly onto the operations I was thinking of.

Thanks so much for your extremely thorough response!
> Would that be in the docstring for the resolve/resolve_multi() methods?

I think so, yes. You're the API reader here though so you know better where the words should go and what the words should have been,
Great! I will think about what would seem most effective!

Resolving patch:
diff --git a/pex/bin/pex.py b/pex/bin/pex.py
--- a/pex/bin/pex.py
+++ b/pex/bin/pex.py
@@ -501,8 +501,8 @@ def configure_clp_pex_environment(parser):
         help="The platform for which to build the PEX. This option can be passed multiple times "
         "to create a multi-platform pex. To use the platform corresponding to the current "
         "interpreter you can pass `current`. To target any other platform you pass a string "
-        "composed of fields: <platform>-<python impl abbr>-<python version>-<abi>. These fields "
-        "stem from wheel name conventions as outlined in "
+        "composed of fields: <platform>-<python impl abbr>-<python version>-<abi>. "
+        "These fields stem from wheel name conventions as outlined in "
         "https://www.python.org/dev/peps/pep-0427#file-name-convention and influenced by "
         "https://www.python.org/dev/peps/pep-0425. For the current interpreter at {} the full "
         "platform string is {}. To find out more, try `{} --platform explain`.".format(
@@ -518,8 +518,9 @@ def configure_clp_pex_environment(parser):
         callback=parse_bool,
         help="When --platforms are specified, attempt to resolve a local interpreter that matches "
         "each platform specified. If found, use the interpreter to resolve distributions; if "
-        "not, resolve for the platform only allowing matching binary distributions and failing "
-        "if only sdists or non-matching binary distributions can be found.",
+        "not (or if this option is not specified), resolve for each platform only allowing "
+        "matching binary distributions and failing if only sdists or non-matching binary "
+        "distributions can be found.",
     )
 
     group.add_option(
@@ -533,7 +534,8 @@ def configure_clp_pex_environment(parser):
             "compatible Python version. Normally, when multiple interpreters match, Pex will "
             "resolve requirements for each interpreter; this allows the resulting Pex to be "
             "compatible with more interpreters, such as different Python versions. However, "
-            "resolving for multiple interpreters results in worse performance."
+            "resolving for multiple interpreters will take longer to build, and the resulting PEX "
+            "may be larger."
         ),
     )
 
diff --git a/pex/resolver.py b/pex/resolver.py
--- a/pex/resolver.py
+++ b/pex/resolver.py
@@ -733,11 +733,15 @@ def resolve(
       prerelease or development versions will override this setting.
     :keyword bool transitive: Whether to resolve transitive dependencies of requirements.
       Defaults to ``True``.
-    :keyword interpreter: The interpreter to use for building distributions and for testing
-      distribution compatibility. Defaults to the current interpreter.
+    :keyword interpreter: If specified, distributions will be resolved for this interpreter, and
+      non-wheel distributions will be built against this interpreter. If both `interpreter` and
+      `platform` are ``None`` (the default), this defaults to the current interpreter.
     :type interpreter: :class:`pex.interpreter.PythonInterpreter`
-    :keyword str platform: The exact target platform to resolve distributions for. If ``None`` or
-      ``'current'``, resolve for distributions appropriate for `interpreter`.
+    :keyword str platform: The exact PEP425-compatible platform string to resolve distributions for,
+      in addition to the platform of the given interpreter, if provided. If any distributions need
+      to be built, use the interpreter argument instead, providing the corresponding interpreter.
+      However, if the platform matches the current interpreter, the current interpreter will be used
+      to build any non-wheels.
     :keyword indexes: A list of urls or paths pointing to PEP 503 compliant repositories to search for
       distributions. Defaults to ``None`` which indicates to use the default pypi index. To turn off
       use of all indexes, pass an empty list.
@@ -765,6 +769,8 @@ def resolve(
     :raises Unsatisfiable: If ``requirements`` is not transitively satisfiable.
     :raises Untranslatable: If no compatible distributions could be acquired for
       a particular requirement.
+    :raises ValueError: If a foreign `platform` was provided, and `use_wheel=False`.
+    :raises ValueError: If `build=False` and `use_wheel=False`.
     """
     # TODO(https://github.com/pantsbuild/pex/issues/969): Deprecate resolve with a single interpreter
     #  or platform and rename resolve_multi to resolve for a single API entrypoint to a full resolve.
@@ -824,12 +830,16 @@ def resolve_multi(
       prerelease or development versions will override this setting.
     :keyword bool transitive: Whether to resolve transitive dependencies of requirements.
       Defaults to ``True``.
-    :keyword interpreters: The interpreters to use for building distributions and for testing
-      distribution compatibility. Defaults to the current interpreter.
+    :keyword interpreters: If specified, distributions will be resolved for these interpreters, and
+      non-wheel distributions will be built against each interpreter. If both `interpreters` and
+      `platforms` are ``None`` (the default) or an empty iterable, this defaults to a list
+      containing only the current interpreter.
     :type interpreters: list of :class:`pex.interpreter.PythonInterpreter`
     :keyword platforms: An iterable of PEP425-compatible platform strings to resolve distributions
-      for. If ``None`` (the default) or an empty iterable, use the platforms of the given
-      interpreters.
+      for, in addition to the platforms of any given interpreters. If any distributions need to be
+      built, use the interpreters argument instead, providing the corresponding interpreter.
+      However, if any platform matches the current interpreter, the current interpreter will be used
+      to build any non-wheels for that platform.
     :type platforms: list of str
     :keyword indexes: A list of urls or paths pointing to PEP 503 compliant repositories to search for
       distributions. Defaults to ``None`` which indicates to use the default pypi index. To turn off
@@ -858,6 +868,8 @@ def resolve_multi(
     :raises Unsatisfiable: If ``requirements`` is not transitively satisfiable.
     :raises Untranslatable: If no compatible distributions could be acquired for
       a particular requirement.
+    :raises ValueError: If a foreign platform was provided in `platforms`, and `use_wheel=False`.
+    :raises ValueError: If `build=False` and `use_wheel=False`.
     """
 
     # A resolve happens in four stages broken into two phases:
@@ -1047,12 +1059,12 @@ def download(
       prerelease or development versions will override this setting.
     :keyword bool transitive: Whether to resolve transitive dependencies of requirements.
       Defaults to ``True``.
-    :keyword interpreters: The interpreters to use for building distributions and for testing
-      distribution compatibility. Defaults to the current interpreter.
+    :keyword interpreters: If specified, distributions will be resolved for these interpreters.
+      If both `interpreters` and `platforms` are ``None`` (the default) or an empty iterable, this
+      defaults to a list containing only the current interpreter.
     :type interpreters: list of :class:`pex.interpreter.PythonInterpreter`
     :keyword platforms: An iterable of PEP425-compatible platform strings to resolve distributions
-      for. If ``None`` (the default) or an empty iterable, use the platforms of the given
-      interpreters.
+      for, in addition to the platforms of any given interpreters.
     :type platforms: list of str
     :keyword indexes: A list of urls or paths pointing to PEP 503 compliant repositories to search for
       distributions. Defaults to ``None`` which indicates to use the default pypi index. To turn off
@@ -1075,8 +1087,10 @@ def download(
     :keyword str dest: A directory path to download distributions to.
     :keyword int max_parallel_jobs: The maximum number of parallel jobs to use when resolving,
       building and installing distributions in a resolve. Defaults to the number of CPUs available.
-    :raises Unsatisfiable: If the resolution of download of distributions fails for any reason.
     :returns: List of :class:`LocalDistribution` instances meeting ``requirements``.
+    :raises Unsatisfiable: If the resolution of download of distributions fails for any reason.
+    :raises ValueError: If a foreign platform was provided in `platforms`, and `use_wheel=False`.
+    :raises ValueError: If `build=False` and `use_wheel=False`.
     """
 
     local_distributions, download_results = _download_internal(


Test patch: