Repository: pex-tool/pex

Issue:
Constrain the matrix of shebang, platforms and ICs as much as possible.
Building and booting a PEX can involve the interplay of 4 concepts currently:
1. The interpreter(s) the Pex CLI is run with.
2. The platforms specified.
3. The interpreter constraints specified.
4. The shebang the resulting PEX file is built with.

These all need to be in alignment to produce a PEX that "boots" properly. Further, they need to be particular beyond that in order to squeeze performance out of some combinations of those items.

Pex cannot ensure alignment, let along maximally performing alignment, in all cases. It can probably catch a few out-of-aligment cases though and - preferably - automatically bring them in alignment, or - less preferably - warn or potential issues. For the latter, care will be needed not to provided unwanted nannying for a knowledgeable user or a too sharp knife (the current situation) for the casual user.

An example of automated alignment would be basing the default shebang for a single --platform PEX, or else multiplatform PEX where all platforms share the same major / minor versions in their platform tag, off that major / minor version pair. I.E.: for `--platform linux-x86_64-cp-37-cp37m --platform macosx-10.13-x86_64-cp-37-m` use a default shebang of `#!/usr/bin/env python3.7`. This is in contrast to the behavior today, which is the shebang defaulting to `#!/usr/bin/env pythonX.Y` where X / Y are the major / minor versions of the interpreter used to run the Pex CLI, and which need bear no relation to the platforms selected.

The answer to what can be and should be constrained will change once #1020 is implemented, but even shy of that the status quo can be improved to prevent this style of unambiguous mis-aligment.


Discussion:
> 4. The shebang the resulting PEX file is built with.

Lightly related, but a thing I've wondered about: how awkward would it be to replace the bootstrap/`shebang` entrypoint with a bash script to locate Pythons, rather than a Python script? The bootstrap code is already intentionally small I presume, and that would remove one dimension here.
What cases that aren't currently handled would be your goal to handle with such a bash header? It does seem that `/usr/bin/env bash` is more universally available than `/usr/bin/env python` or `/usr/bin/env python3`; so it would concretely start to launch successfully on more machines - but it seems that particular issue has been a very narrow corner of issues. So do I have that right, you're proposing:
1. bash header finds python compatible with Pex bootstrap code - today Python 2.7 & 3.5+ then re-execs using that
2. PEX bootstrap now potentially re-execs again to match ICs or after its run #1020 logic
3. PEX finally runs using correct interpreter
That sounds right, although I suppose that I hadn't considered that both steps 1 and 2 would be necessary (implementing interpreter constraints in bash would be a pain). The goal would mostly be to remove a dimension from the description, since `shebang` setting has been a non-trivial complication (since a user has to choose exactly one, and even `/usr/bin/env python` is not universally available, since some systems only install `python3`).
> ... setting has been a non-trivial complication (since a user has to choose exactly one ...

Yeah - I think the error here is auto-selecting a hashbang. Since these aren't in fact universal, Pex should probably force you to pick one to make it clear you should think about this and know the right answer for your fleet. There are exactly 2 cases I'm aware of where users can't answer this because the fleet is not theirs - the Pex PEX and Pants' PEX. For the former, Pants uses your suggestion (externalized though) to find the bootstrap Python to run the Pex PEX with.
I realized step 1 in [this list above](https://github.com/pantsbuild/pex/issues/1540#issuecomment-996997194) could be done in Python code that emits a bash header. That simplifies the bash greatly.

I still think that Pex needs to keep its Python 1st tradition to both not break current users and to make sure PEX by default can ship to a machine with Python and work - it would be bad to land on a machine with Python in the PATH but not a shell and fail to run. It would also hinder the many-times-stalled effort to run out of the box on Windows.

So, if put behind a flag though, this is viable. In fact it cleans up a whole mess of code over in Pants designed to avoid ~50ms of leftover overhead in the PEX zip figuring out it has an already installed `~/.pex/venvs/...` and re-execing into it and it effectively puts that code in the PEX where it belongs and where non-Pants users can benefit too:

Numbers look good. Does not affect the cold case but is snappy for the `--seed` (Pants use case) and warm cases:

Cold:
```
$ hyperfine \
  -p 'python -mpex cowsay -c cowsay -o cowsay.pex && rm -rf ~/.pex' \
  -p 'python -mpex cowsay -c cowsay -o cowsay.venv.pex --venv && rm -rf ~/.pex' \
  -p 'python -mpex cowsay -c cowsay -o cowsay.bash.pex --bash-boot && rm -rf ~/.pex' \
  -p 'python -mpex cowsay -c cowsay -o cowsay.bash.venv.pex --bash-boot --venv && rm -rf ~/.pex' \
  './cowsay.pex Moo' \
  './cowsay.venv.pex Moo' \
  './cowsay.bash.pex Moo' \
  './cowsay.bash.venv.pex Moo'
Benchmark 1: ./cowsay.pex Moo
  Time (mean ± σ):     358.6 ms ±   4.7 ms    [User: 329.0 ms, System: 29.9 ms]
  Range (min … max):   352.7 ms … 368.6 ms    10 runs
 
Benchmark 2: ./cowsay.venv.pex Moo
  Time (mean ± σ):     607.9 ms ±   5.7 ms    [User: 557.2 ms, System: 50.8 ms]
  Range (min … max):   600.5 ms … 618.7 ms    10 runs
 
Benchmark 3: ./cowsay.bash.pex Moo
  Time (mean ± σ):     362.7 ms ±   5.0 ms    [User: 335.5 ms, System: 27.7 ms]
  Range (min … max):   355.3 ms … 370.4 ms    10 runs
 
Benchmark 4: ./cowsay.bash.venv.pex Moo
  Time (mean ± σ):     614.0 ms ±  11.1 ms    [User: 556.1 ms, System: 58.1 ms]
  Range (min … max):   604.6 ms … 642.7 ms    10 runs
 
Summary
  './cowsay.pex Moo' ran
    1.01 ± 0.02 times faster than './cowsay.bash.pex Moo'
    1.70 ± 0.03 times faster than './cowsay.venv.pex Moo'
    1.71 ± 0.04 times faster than './cowsay.bash.venv.pex Moo'
```

Seed:
```
$ hyperfine \
  -p 'rm -rf ~/.pex && python -mpex cowsay -c cowsay -o cowsay.pex --seed' \
  -p 'rm -rf ~/.pex && python -mpex cowsay -c cowsay -o cowsay.venv.pex --venv --seed' \
  -p 'rm -rf ~/.pex && python -mpex cowsay -c cowsay -o cowsay.bash.pex --bash-boot --seed' \
  -p 'rm -rf ~/.pex && python -mpex cowsay -c cowsay -o cowsay.bash.venv.pex --bash-boot --venv --seed' \
  './cowsay.pex Moo' \
  './cowsay.venv.pex Moo' \
  './cowsay.bash.pex Moo' \
  './cowsay.bash.venv.pex Moo'
Benchmark 1: ./cowsay.pex Moo
  Time (mean ± σ):     321.6 ms ±   3.1 ms    [User: 295.4 ms, System: 26.7 ms]
  Range (min … max):   313.8 ms … 323.7 ms    10 runs
 
Benchmark 2: ./cowsay.venv.pex Moo
  Time (mean ± σ):      75.1 ms ±   0.9 ms    [User: 66.9 ms, System: 9.1 ms]
  Range (min … max):    74.0 ms …  76.6 ms    10 runs
 
Benchmark 3: ./cowsay.bash.pex Moo
  Time (mean ± σ):     285.9 ms ±   4.5 ms    [User: 261.9 ms, System: 24.5 ms]
  Range (min … max):   280.2 ms … 295.0 ms    10 runs
 
Benchmark 4: ./cowsay.bash.venv.pex Moo
  Time (mean ± σ):      16.0 ms ±   0.6 ms    [User: 13.0 ms, System: 3.0 ms]
  Range (min … max):    15.6 ms …  17.7 ms    10 runs
 
Summary
  './cowsay.bash.venv.pex Moo' ran
    4.68 ± 0.19 times faster than './cowsay.venv.pex Moo'
   17.82 ± 0.74 times faster than './cowsay.bash.pex Moo'
   20.04 ± 0.79 times faster than './cowsay.pex Moo'
```

Warm:
```
$ hyperfine \
  -w 2 \
  -p 'python -mpex cowsay -c cowsay -o cowsay.pex' \
  -p 'python -mpex cowsay -c cowsay -o cowsay.venv.pex --venv' \
  -p 'python -mpex cowsay -c cowsay -o cowsay.bash.pex --bash-boot' \
  -p 'python -mpex cowsay -c cowsay -o cowsay.bash.venv.pex --bash-boot --venv' \
  -p 'python -mpex cowsay -c cowsay -o cowsay.bash.venv.pex --bash-boot --venv' \
  './cowsay.pex Moo' \
  './cowsay.venv.pex Moo' \
  './cowsay.bash.pex Moo' \
  './cowsay.bash.venv.pex Moo' \
  '/home/jsirois/.pex/venvs/s/bccd8b55/venv/pex Moo'
Benchmark 1: ./cowsay.pex Moo
  Time (mean ± σ):     201.1 ms ±   2.2 ms    [User: 180.1 ms, System: 21.8 ms]
  Range (min … max):   196.6 ms … 203.8 ms    10 runs
 
Benchmark 2: ./cowsay.venv.pex Moo
  Time (mean ± σ):      73.5 ms ±   0.5 ms    [User: 63.1 ms, System: 11.4 ms]
  Range (min … max):    73.0 ms …  74.7 ms    10 runs
 
Benchmark 3: ./cowsay.bash.pex Moo
  Time (mean ± σ):     163.6 ms ±   3.6 ms    [User: 145.0 ms, System: 19.5 ms]
  Range (min … max):   160.0 ms … 170.9 ms    10 runs
 
Benchmark 4: ./cowsay.bash.venv.pex Moo
  Time (mean ± σ):      14.0 ms ±   0.3 ms    [User: 12.1 ms, System: 1.9 ms]
  Range (min … max):    13.6 ms …  14.5 ms    10 runs
 
Benchmark 5: /home/jsirois/.pex/venvs/s/bccd8b55/venv/pex Moo
  Time (mean ± σ):      12.4 ms ±   0.2 ms    [User: 11.9 ms, System: 0.5 ms]
  Range (min … max):    12.0 ms …  12.5 ms    10 runs
 
Summary
  '/home/jsirois/.pex/venvs/s/bccd8b55/venv/pex Moo' ran
    1.13 ± 0.03 times faster than './cowsay.bash.venv.pex Moo'
    5.95 ± 0.08 times faster than './cowsay.venv.pex Moo'
   13.24 ± 0.33 times faster than './cowsay.bash.pex Moo'
   16.28 ± 0.27 times faster than './cowsay.pex Moo'
```

N.B.: I threw in direct execution of the venv pex script in the warm perf comparison. So we lose ~2ms by not executing the venv pex script directly. This is the smae penalty Pants experiences today - it just writes its own bash re-directory script instead of using one embedded in the PEX header like this.

The interpreter selection for `--bash-boot` is very robust and fast despite the large list of binary names to try. In my tests, removing the `p` from the binary names so that the bash boot script has to rip through all of them before failing, I find that takes 2-3ms:
```sh
$ head -71 cowsay.bash.venv.pex
#!/usr/bin/env sh
# N.B.: This script should stick to syntax defined for POSIX `sh` and
# avoid non-builtins:
#   https://pubs.opengroup.org/onlinepubs/9699919799/idx/shell.html
set -eu

VENV="1"

# N.B.: This ensures tilde-expansion of the DEFAULT_PEX_ROOT value.
DEFAULT_PEX_ROOT="$(echo ~/.pex)"

PEX_ROOT="${PEX_ROOT:-${DEFAULT_PEX_ROOT}}"
PEX="${PEX_ROOT}/venvs/346aee797b51ee3f468179886b3a5db7c6d723a3/0c2af63c3815d1d03077ee9c1f2cbc64e6c7925d/pex"

if [ -n "${VENV}" -a -x "${PEX}" ]; then
    exec "${PEX}" "$@"
fi

find_python() {
    for python in \
"python3.10" \
"python2.7" \
"python3.5" \
"python3.6" \
"python3.7" \
"python3.8" \
"python3.9" \
"python3.11" \
"python3" \
"python2" \
"python" \
    ; do
        if command -v "${python}" 2>/dev/null; then
            return
        fi
    done
}

python_exe="$(find_python)"
if [ -n "${python_exe}" ]; then
    if [ -z "${VENV}" -a -e "${PEX}" ]; then
        exec "${python_exe}" "${PEX}" "$@"
    else
        # The slow path, run the PEX zipapp so it can rebuild its fast
        # path layout under the PEX_ROOT.
        exec "${python_exe}" "$0" "$@"
    fi
else
    echo >&2 "Failed to find any of these python binaries on the PATH:"
    for python in \
"python3.10" \
"python2.7" \
"python3.5" \
"python3.6" \
"python3.7" \
"python3.8" \
"python3.9" \
"python3.11" \
"python3" \
"python2" \
"python" \
    ; do
        echo >&2 "${python}"
    done
    echo >&2 "Either adjust your $PATH which is currently:"
    echo >&2 "${PATH}"
    echo >&2 -n "Or else install an appropriate Python that provides "
    echo >&2 "one of the binaries in this list."
    exit 1
fi
P!
  .bootstrap/P!.bootstrap/pex/P!�};
```


And busybox crushes all comers:
```
$ hyperfine -L interpreter 'sh,bash,ksh,dash,zsh,busybox ash,python' '{interpreter} ./cowsay.bash.venv.pex Moo'
Benchmark 1: sh ./cowsay.bash.venv.pex Moo
  Time (mean ± σ):      13.5 ms ±   0.4 ms    [User: 11.4 ms, System: 2.2 ms]
  Range (min … max):    12.8 ms …  15.9 ms    188 runs
 
Benchmark 2: bash ./cowsay.bash.venv.pex Moo
  Time (mean ± σ):      13.5 ms ±   0.3 ms    [User: 11.9 ms, System: 1.8 ms]
  Range (min … max):    12.8 ms …  14.7 ms    195 runs
 
Benchmark 3: ksh ./cowsay.bash.venv.pex Moo
  Time (mean ± σ):      13.2 ms ±   0.3 ms    [User: 11.6 ms, System: 1.7 ms]
  Range (min … max):    12.5 ms …  14.3 ms    196 runs
 
Benchmark 4: dash ./cowsay.bash.venv.pex Moo
  Time (mean ± σ):      12.3 ms ±   0.4 ms    [User: 10.4 ms, System: 2.1 ms]
  Range (min … max):    11.8 ms …  13.5 ms    207 runs
 
Benchmark 5: zsh ./cowsay.bash.venv.pex Moo
  Time (mean ± σ):      13.6 ms ±   0.4 ms    [User: 11.2 ms, System: 2.4 ms]
  Range (min … max):    12.9 ms …  15.4 ms    193 runs
 
Benchmark 6: busybox ash ./cowsay.bash.venv.pex Moo
  Time (mean ± σ):      12.1 ms ±   0.3 ms    [User: 10.5 ms, System: 1.7 ms]
  Range (min … max):    11.4 ms …  13.2 ms    217 runs
 
Benchmark 7: python ./cowsay.bash.venv.pex Moo
  Time (mean ± σ):      74.2 ms ±   0.8 ms    [User: 64.4 ms, System: 10.7 ms]
  Range (min … max):    72.7 ms …  75.9 ms    38 runs
 
Summary
  'busybox ash ./cowsay.bash.venv.pex Moo' ran
    1.02 ± 0.04 times faster than 'dash ./cowsay.bash.venv.pex Moo'
    1.09 ± 0.04 times faster than 'ksh ./cowsay.bash.venv.pex Moo'
    1.11 ± 0.04 times faster than 'bash ./cowsay.bash.venv.pex Moo'
    1.11 ± 0.04 times faster than 'sh ./cowsay.bash.venv.pex Moo'
    1.12 ± 0.04 times faster than 'zsh ./cowsay.bash.venv.pex Moo'
    6.10 ± 0.18 times faster than 'python ./cowsay.bash.venv.pex Moo'
```

Super awesome.

In the spirit of making progress on this ticket (and understanding that what you said about wanting to support environments without shells installed means that python-shebang will need to stick around as an option), it seems like maybe `--bash-boot` would be a reasonable default... it's certainly easier to get right than shebang setting.
Pants can definitely choose that as a default for its users. Pex cannot until the Pex 3 release - my policy is to only fix bugs and add features but not break users without bumping major.

Resolving patch:
diff --git a/pex/bin/pex.py b/pex/bin/pex.py
--- a/pex/bin/pex.py
+++ b/pex/bin/pex.py
@@ -872,8 +872,24 @@ def build_pex(
             filename=options.executable, env_filename="__pex_executable__.py"
         )
 
-    if options.python_shebang:
-        pex_builder.set_shebang(options.python_shebang)
+    specific_shebang = options.python_shebang or targets.compatible_shebang()
+    if specific_shebang:
+        pex_builder.set_shebang(specific_shebang)
+    else:
+        # TODO(John Sirois): Consider changing fallback to `#!/usr/bin/env python` in Pex 3.x.
+        pex_warnings.warn(
+            "Could not calculate a targeted shebang for:\n"
+            "{targets}\n"
+            "\n"
+            "Using shebang: {default_shebang}\n"
+            "If this is not appropriate, you can specify a custom shebang using the "
+            "--python-shebang option.".format(
+                targets="\n".join(
+                    sorted(target.render_description() for target in targets.unique_targets())
+                ),
+                default_shebang=pex_builder.shebang,
+            )
+        )
 
     return pex_builder
 
diff --git a/pex/pex_builder.py b/pex/pex_builder.py
--- a/pex/pex_builder.py
+++ b/pex/pex_builder.py
@@ -457,6 +457,11 @@ def set_entry_point(self, entry_point):
         self._ensure_unfrozen("Setting an entry point")
         self._pex_info.entry_point = entry_point
 
+    @property
+    def shebang(self):
+        # type: () -> str
+        return self._shebang
+
     def set_shebang(self, shebang):
         """Set the exact shebang line for the PEX file.
 
diff --git a/pex/targets.py b/pex/targets.py
--- a/pex/targets.py
+++ b/pex/targets.py
@@ -5,6 +5,7 @@
 
 import os
 
+from pex import pex_warnings
 from pex.dist_metadata import Requirement
 from pex.interpreter import PythonInterpreter, calculate_binary_name
 from pex.orderedset import OrderedSet
@@ -377,3 +378,16 @@ def require_at_most_one_target(self, purpose):
             return cast(Target, next(iter(resolved_targets)))
         except StopIteration:
             return None
+
+    def compatible_shebang(self):
+        # type: () -> Optional[str]
+        pythons = {
+            (target.platform.impl, target.platform.version_info[:2])
+            for target in self.unique_targets()
+        }
+        if len(pythons) == 1:
+            impl, version = pythons.pop()
+            return "#!/usr/bin/env {python}{version}".format(
+                python="pypy" if impl == "pp" else "python", version=".".join(map(str, version))
+            )
+        return None


Test patch:
diff --git a/tests/integration/test_issue_1540.py b/tests/integration/test_issue_1540.py
new file mode 100644
--- /dev/null
+++ b/tests/integration/test_issue_1540.py
@@ -0,0 +1,85 @@
+# Copyright 2023 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+from __future__ import absolute_import
+
+import os.path
+from textwrap import dedent
+
+from pex.interpreter import PythonInterpreter
+from pex.typing import TYPE_CHECKING
+from testing import run_pex_command
+
+if TYPE_CHECKING:
+    from typing import Any
+
+
+def test_derive_consistent_shebang_platforms(
+    tmpdir,  # type: Any
+    current_interpreter,  # type: PythonInterpreter
+):
+    # type: (...) -> None
+
+    pex = os.path.join(str(tmpdir), "pex")
+
+    def read_pex_shebang():
+        # type: () -> bytes
+        with open(pex, "rb") as fp:
+            return fp.readline()
+
+    run_pex_command(args=["--platform", "linux_x86_64-cp-311-cp311", "-o", pex]).assert_success()
+    assert b"#!/usr/bin/env python3.11\n" == read_pex_shebang()
+
+    run_pex_command(
+        args=[
+            "--platform",
+            "linux_x86_64-cp-311-cp311",
+            "--platform",
+            "macosx_10.9_x86_64-cp-311-cp311",
+            "-o",
+            pex,
+        ]
+    ).assert_success()
+    assert b"#!/usr/bin/env python3.11\n" == read_pex_shebang()
+
+    run_pex_command(
+        args=[
+            "--platform",
+            "linux_x86_64-cp-3.11.5-cp311",
+            "--platform",
+            "macosx_10.9_x86_64-cp-311-cp311",
+            "-o",
+            pex,
+        ]
+    ).assert_success()
+    assert b"#!/usr/bin/env python3.11\n" == read_pex_shebang()
+
+    result = run_pex_command(
+        args=[
+            "--platform",
+            "linux_x86_64-cp-310-cp310",
+            "--platform",
+            "macosx_10.9_x86_64-cp-311-cp311",
+            "-o",
+            pex,
+        ]
+    )
+    result.assert_success()
+    current_interpreter_shebang = current_interpreter.identity.hashbang()
+    assert (
+        "{shebang}\n".format(shebang=current_interpreter_shebang).encode("utf-8")
+        == read_pex_shebang()
+    )
+    assert (
+        dedent(
+            """\
+            PEXWarning: Could not calculate a targeted shebang for:
+            abbreviated platform cp310-cp310-linux_x86_64
+            abbreviated platform cp311-cp311-macosx_10_9_x86_64
+
+            Using shebang: {shebang}
+            If this is not appropriate, you can specify a custom shebang using the --python-shebang option.
+            """
+        ).format(shebang=current_interpreter_shebang)
+        in result.error
+    )
diff --git a/tests/test_targets.py b/tests/test_targets.py
--- a/tests/test_targets.py
+++ b/tests/test_targets.py
@@ -24,6 +24,7 @@
 )
 from pex.third_party.packaging.specifiers import SpecifierSet
 from pex.typing import TYPE_CHECKING
+from testing import IS_PYPY, PY39, PY310, PY_VER, ensure_python_interpreter
 
 if TYPE_CHECKING:
     from typing import Optional
@@ -261,3 +262,70 @@ def test_from_target_complete_platform(current_interpreter):
     assert tgts.interpreter is None
     assert OrderedSet([complete_platform]) == tgts.unique_targets(only_explicit=False)
     assert OrderedSet([complete_platform]) == tgts.unique_targets(only_explicit=True)
+
+
+def test_compatible_shebang(
+    current_interpreter,  # type: PythonInterpreter
+    current_platform,  # type: Platform
+):
+    # type: (...) -> None
+
+    current_python = "pypy" if IS_PYPY else "python"
+    current_version = ".".join(map(str, PY_VER))
+    current_python_shebang = "#!/usr/bin/env {python}{version}".format(
+        python=current_python, version=current_version
+    )
+    assert (
+        current_python_shebang
+        == Targets.from_target(LocalInterpreter.create(current_interpreter)).compatible_shebang()
+    )
+    assert (
+        current_python_shebang
+        == Targets.from_target(AbbreviatedPlatform.create(current_platform)).compatible_shebang()
+    )
+    current_complete_platform = CompletePlatform.from_interpreter(current_interpreter)
+    assert (
+        current_python_shebang
+        == Targets.from_target(current_complete_platform).compatible_shebang()
+    )
+    assert (
+        current_python_shebang
+        == Targets(
+            interpreters=(current_interpreter,),
+            complete_platforms=(current_complete_platform,),
+            platforms=(current_platform,),
+        ).compatible_shebang()
+    )
+
+    incompatible_interpreter = PythonInterpreter.from_binary(
+        ensure_python_interpreter(PY39 if PY_VER == (3, 10) else PY310)
+    )
+    assert (
+        Targets(interpreters=(current_interpreter, incompatible_interpreter)).compatible_shebang()
+        is None
+    )
+
+    incompatible_version_info = (3, 9) if PY_VER == (3, 10) else (3, 10)
+    incompatible_platform_version = Platform(
+        platform=current_platform.platform,
+        impl=current_platform.impl,
+        version_info=incompatible_version_info,
+        version=".".join(map(str, incompatible_version_info)),
+        abi=current_platform.abi,
+    )
+    assert (
+        Targets(platforms=(current_platform, incompatible_platform_version)).compatible_shebang()
+        is None
+    )
+
+    incompatible_platform_impl = Platform(
+        platform=current_platform.platform,
+        impl="cp" if current_platform.impl == "pp" else "pp",
+        version_info=current_platform.version_info,
+        version=current_platform.version,
+        abi=current_platform.abi,
+    )
+    assert (
+        Targets(platforms=(current_platform, incompatible_platform_impl)).compatible_shebang()
+        is None
+    )