Repository: pex-tool/pex

Issue:
Slow builds in 2.0 - how does --python interact with --platform?
I want to build a pex that supports Linux armv7l and x86_64, on Python versions 3.5 through 3.7.

Before pex 2.0, I used

`--platform linux_armv7l --platform linux_x86_64 --python python3.5 --python python3.6 --python python3.7 --python-shebang "/usr/bin/env python3"`

And that produced a pex file meeting those requirements in about ten seconds.

As of 2.0 I'm not sure what I need to do. pex seems to resolve requirements for every combination of platform and python, so if I specify all six relevant `--platform` options (`--platform linux_armv7l-cp-35-m`, `--platform linux_x86_64-cp-36-m`, etc) plus all three `--python` options it takes **200s**.

For my use case I can simplify and specify just `--platform any-py-<35/36/37>-none` and the `--python`s. That takes **90s**.

Ultimately the dependencies are in this case resolved to py3-none-any wheels, so I figure I can probably get away with specifying `--platform any-py-35-none` and the `--python`s. That takes **60s**.

If I omit `--python` entirely with the single `--platform` arg, it takes about **15s** - close to the pre-2.0 time. For every `--python` I add, including the default interpreter, it adds another ~15s.

In all these cases the actual resolved and bundled dependencies are identical. What effect does `--python` actually have on the pex over and above the platform specifier? Is there anything I can do to get the build time down while ensuring that I'm producing a file that will work where it needs to?


Discussion:
Could you give a bit more concrete example? Preferrably one we can share in whittling away the perf difference? For example, platforms like `--platform linux_armv7l` and `--platform linux_x86_64` are no longer accepted. Any `--platform` supplied must be exact:
```
$ ./pex-2.0.2.pex --platform linux_armv7l
Usage: pex-2.0.2.pex [-o OUTPUT.PEX] [options] [-- arg1 arg2 ...]

pex-2.0.2.pex builds a PEX (Python Executable) file based on the given specifications: sources, requirements, their dependencies and other options.

pex-2.0.2.pex: error: The --platform option is invalid:
Not a valid platform specifier: linux_armv7l

Platform strings must be in one of two forms:
1. Canonical: <platform>-<python impl abbr>-<python version>-<abi>
2. Abbreviated: <platform>-<python impl abbr>-<python version>-<abbr abi>

Given a canonical platform string for CPython 3.7.5 running on 64 bit linux of:
  linux-x86_64-cp-37-cp37m

Where the fields above are:
+ <platform>: linux-x86_64 
+ <python impl abbr>: cp
+ <python version>: 37
+ <abi>: cp37m

The abbreviated platform string is:
  linux-x86_64-cp-37-m

These fields stem from wheel name conventions as outlined in
https://www.python.org/dev/peps/pep-0427#file-name-convention and influenced by
https://www.python.org/dev/peps/pep-0425.
```

Once we have a shared example, the obvious places to compare will be correctness changes and missed caching opportunities after that. I'm pretty sure pex pre 2.0.0 incorrectly handled multi-interpreter / multi-platform.

In pex<2.0.0:
1. The list of local interpreters and the list of platforms were combined in a cross-product, and, for each pair in th cross-product, a resolve was done.
2. When a wheel for a foreign platform was not found, the active local interpreter in the cross-product was used to build a wheel - this is incorrect.

In pex>=2.0.0:
1. A full resolve is done for each local interpreter, and then for each platform that is foreign, a full resolve is done.
2. For foreign platforms, no wheel building is allowed. Either a wheel for the foreign platform is found, or the resolve fails - this is correct.



Sorry yes, I've edited my initial post to make that clear. I was effectively doing the cross-product manually to provide the six relevant `--platform` arguments in 2.0.
Do you have a public distribution I can try this against that has pre-built wheels? It would be extremely useful to have an exact command line we can both execute to work this problem. I don't have access to an arm box, so if you can't provide a public example I'll have to simulate with a Linux / OSX example.
For a similar repro case I'm using numpy:
```
$ time ~/Downloads/pex-1.6.12.pex -vvv --platform=manylinux1-x86_64-cp-35-m --platform=manylinux1-x86_64-cp-36-m --platform=manylinux1-x86_64-cp-37-m --platform=macosx-10.9-x86_64-cp-35-m --platform=macosx-10.9-x86_64-cp-36-m --platform=macosx-10.9-x86_64-cp-37-m numpy==1.17.4 --python-shebang "/usr/bin/env python3" -o numpy-pex1.6.2.pex
...
Saving PEX file to numpy-pex1.6.2.pex

real	0m20.963s
user	0m19.862s
sys	0m0.852s
```
vs.
```
$ time ~/Downloads/pex-2.0.2.pex -vvv --platform=manylinux1-x86_64-cp-35-m --platform=manylinux1-x86_64-cp-36-m --platform=manylinux1-x86_64-cp-37-m --platform=macosx-10.9-x86_64-cp-35-m --platform=macosx-10.9-x86_64-cp-36-m --platform=macosx-10.9-x86_64-cp-37-m numpy==1.17.4 --python-shebang "/usr/bin/env python3" -o numpy-pex2.0.2.pex
...
Saving PEX file to numpy-pex2.0.2.pex

real	0m27.946s
user	0m24.947s
sys	0m1.935s
```

Thats ~33% slower with pex 2.0.2.

Noting a few things:
1. Since platforms are now required to be exact, there is no need when building a multiplatform pex to specify both `--python` and `--platform`, `--platform` is enough. Note though, `--platform` now requires all wheels be available for that platform pre-built. So, in your arm example, if you don't have arm wheels available, you'd need to run pex on an arm machine and use ~ `./pex --python=3.{5,6,7} --platform=linux-86_64-cp-3{5,6,7}-m ...`; ie supply 3 local arm python interpreters and 3 foreign linux platforms instead of supplying 6 platforms.
2. Looking at the verbose output of both its clear pex caches the resolve better in 1.6.2 than it does in 2.0.2 accounting for most of the speed difference.

I'll look at the caching 1st.
A reasonable change has the above example down to 23s in he pex 2.0 case or ~10% slower. These measurements are with warm caches. There is one further obvious change for the cold cache case to parallelize installation of wheels in chroots. I'll get out seperate PRs for both here.

Resolving patch:
diff --git a/pex/bin/pex.py b/pex/bin/pex.py
--- a/pex/bin/pex.py
+++ b/pex/bin/pex.py
@@ -538,7 +538,7 @@ def walk_and_do(fn, src_dir):
   if options.indexes != [_PYPI] and options.indexes is not None:
     indexes = [str(index) for index in options.indexes]
 
-  with TRACER.timed('Resolving distributions ({})'.format(reqs)):
+  with TRACER.timed('Resolving distributions ({})'.format(reqs + options.requirement_files)):
     try:
       resolveds = resolve_multi(requirements=reqs,
                                 requirement_files=options.requirement_files,
diff --git a/pex/interpreter.py b/pex/interpreter.py
--- a/pex/interpreter.py
+++ b/pex/interpreter.py
@@ -330,7 +330,7 @@ def _create_isolated_cmd(cls, binary, args=None, pythonpath=None, env=None):
     rendered_command = ' '.join(cmd)
     if pythonpath:
       rendered_command = 'PYTHONPATH={} {}'.format(env['PYTHONPATH'], rendered_command)
-    TRACER.log('Executing: {}'.format(rendered_command))
+    TRACER.log('Executing: {}'.format(rendered_command), V=3)
 
     return cmd, env
 
diff --git a/pex/resolver.py b/pex/resolver.py
--- a/pex/resolver.py
+++ b/pex/resolver.py
@@ -6,13 +6,13 @@
 import functools
 import json
 import os
-import shutil
 import subprocess
 from collections import defaultdict, namedtuple
 from textwrap import dedent
+from uuid import uuid4
 
 from pex import third_party
-from pex.common import safe_mkdir, safe_mkdtemp
+from pex.common import safe_mkdtemp
 from pex.interpreter import PythonInterpreter
 from pex.orderedset import OrderedSet
 from pex.pip import PipError, build_wheels, download_distributions, install_wheel
@@ -172,27 +172,29 @@ def resolve(requirements=None,
     return []
 
   workspace = safe_mkdtemp()
+  cache = cache or workspace
   resolved_dists = os.path.join(workspace, 'resolved')
   built_wheels = os.path.join(workspace, 'wheels')
-  installed_chroots = os.path.join(workspace, 'chroots')
+  installed_chroots = os.path.join(cache, 'chroots')
 
   # 1. Resolve
-  try:
-    download_distributions(target=resolved_dists,
-                           requirements=requirements,
-                           requirement_files=requirement_files,
-                           constraint_files=constraint_files,
-                           allow_prereleases=allow_prereleases,
-                           transitive=transitive,
-                           interpreter=interpreter,
-                           platform=parsed_platform(platform),
-                           indexes=indexes,
-                           find_links=find_links,
-                           cache=cache,
-                           build=build,
-                           use_wheel=use_wheel)
-  except PipError as e:
-    raise Unsatisfiable(str(e))
+  with TRACER.timed('Resolving and downloading distributions'):
+    try:
+      download_distributions(target=resolved_dists,
+                             requirements=requirements,
+                             requirement_files=requirement_files,
+                             constraint_files=constraint_files,
+                             allow_prereleases=allow_prereleases,
+                             transitive=transitive,
+                             interpreter=interpreter,
+                             platform=parsed_platform(platform),
+                             indexes=indexes,
+                             find_links=find_links,
+                             cache=cache,
+                             build=build,
+                             use_wheel=use_wheel)
+    except PipError as e:
+      raise Unsatisfiable(str(e))
 
   # 2. Build
   to_build = []
@@ -205,57 +207,64 @@ def resolve(requirements=None,
     for requirement_file in requirement_files:
       to_build.extend(local_projects_from_requirement_file(requirement_file))
 
-  to_copy = []
+  to_install = []
   if os.path.exists(resolved_dists):
     for distribution in os.listdir(resolved_dists):
       path = os.path.join(resolved_dists, distribution)
       if os.path.isfile(path) and path.endswith('.whl'):
-        to_copy.append(path)
+        to_install.append(path)
       else:
         to_build.append(path)
 
-  if not any((to_build, to_copy)):
+  if not any((to_build, to_install)):
     # Nothing to build or install.
     return []
 
-  safe_mkdir(built_wheels)
-
   if to_build:
-    try:
-      build_wheels(distributions=to_build,
-                   target=built_wheels,
-                   cache=cache,
-                   interpreter=interpreter)
-    except PipError as e:
-      raise Untranslateable('Failed to build at least one of {}:\n\t{}'.format(to_build, str(e)))
-
-  if to_copy:
-    for wheel in to_copy:
-      dest = os.path.join(built_wheels, os.path.basename(wheel))
-      TRACER.log('Copying downloaded wheel from {} to {}'.format(wheel, dest))
-      shutil.copy(wheel, dest)
+    with TRACER.timed('Building distributions:\n  {}'.format('\n  '.join(to_build))):
+      try:
+        build_wheels(distributions=to_build,
+                     target=built_wheels,
+                     cache=cache,
+                     interpreter=interpreter)
+      except PipError as e:
+        raise Untranslateable('Failed to build at least one of {}:\n  {}'
+                              .format(', '.join(to_build), str(e)))
+      to_install.extend(os.path.join(built_wheels, wheel) for wheel in os.listdir(built_wheels))
 
   # 3. Chroot
   resolved_distributions = []
 
-  for wheel_file in os.listdir(built_wheels):
-    chroot = os.path.join(installed_chroots, wheel_file)
-    try:
-      install_wheel(wheel=os.path.join(built_wheels, wheel_file),
-                    target=chroot,
-                    compile=compile,
-                    overwrite=True,
-                    cache=cache,
-                    interpreter=interpreter)
-    except PipError as e:
-      raise Untranslateable('Failed to install {}:\n\t{}'.format(wheel_file, str(e)))
-
-    environment = Environment(search_path=[chroot])
-    for dist_project_name in environment:
-      resolved_distributions.extend(environment[dist_project_name])
-
-  markers_by_req_key = _calculate_dependency_markers(resolved_distributions,
-                                                     interpreter=interpreter)
+  with TRACER.timed('Installing distributions:\n  {}'.format('\n  '.join(to_install))):
+    for wheel_file_path in to_install:
+      wheel_file = os.path.basename(wheel_file_path)
+      chroot = os.path.join(installed_chroots, wheel_file)
+      if os.path.exists(chroot):
+        TRACER.log('Using cached installation of {} at {}'.format(wheel_file, chroot))
+      else:
+        TRACER.log('Installing {} in {}'.format(wheel_file_path, chroot))
+        tmp_chroot = '{}.{}'.format(chroot, uuid4().hex)
+        try:
+          install_wheel(wheel=wheel_file_path,
+                        target=tmp_chroot,
+                        compile=compile,
+                        overwrite=True,
+                        cache=cache,
+                        interpreter=interpreter)
+        except PipError as e:
+          raise Untranslateable('Failed to install {}:\n\t{}'.format(wheel_file_path, e))
+        os.rename(tmp_chroot, chroot)
+
+      environment = Environment(search_path=[chroot])
+      for dist_project_name in environment:
+        resolved_distributions.extend(environment[dist_project_name])
+
+  with TRACER.timed('Determining environment markers of installed distributions:\n  {}'
+                    .format('\n  '.join(map(str, resolved_distributions)))):
+    markers_by_req_key = _calculate_dependency_markers(
+      resolved_distributions,
+      interpreter=interpreter
+    )
 
   def to_requirement(dist):
     req = dist.as_requirement()


Test patch:
diff --git a/tests/test_resolver.py b/tests/test_resolver.py
--- a/tests/test_resolver.py
+++ b/tests/test_resolver.py
@@ -6,6 +6,7 @@
 import pytest
 
 from pex.common import safe_copy, safe_mkdtemp, temporary_dir
+from pex.compatibility import nested
 from pex.interpreter import PythonInterpreter
 from pex.resolver import resolve_multi
 from pex.testing import (
@@ -50,6 +51,29 @@ def test_simple_local_resolve():
     assert len(resolved_dists) == 1
 
 
+def test_resolve_cache():
+  project_wheel = build_wheel(name='project')
+
+  with nested(temporary_dir(), temporary_dir()) as (td, cache):
+    safe_copy(project_wheel, os.path.join(td, os.path.basename(project_wheel)))
+
+    # Without a cache, each resolve should be isolated, but otherwise identical.
+    resolved_dists1 = local_resolve_multi(['project'], find_links=[td])
+    resolved_dists2 = local_resolve_multi(['project'], find_links=[td])
+    assert resolved_dists1 != resolved_dists2
+    assert len(resolved_dists1) == 1
+    assert len(resolved_dists2) == 1
+    assert resolved_dists1[0].requirement == resolved_dists2[0].requirement
+    assert resolved_dists1[0].distribution.location != resolved_dists2[0].distribution.location
+
+    # With a cache, each resolve should be identical.
+    resolved_dists3 = local_resolve_multi(['project'], find_links=[td], cache=cache)
+    resolved_dists4 = local_resolve_multi(['project'], find_links=[td], cache=cache)
+    assert resolved_dists1 != resolved_dists3
+    assert resolved_dists2 != resolved_dists3
+    assert resolved_dists3 == resolved_dists4
+
+
 def test_diamond_local_resolve_cached():
   # This exercises the issue described here: https://github.com/pantsbuild/pex/issues/120
   project1_wheel = build_wheel(name='project1', install_reqs=['project2<1.0.0'])