Repository: pantsbuild/pants

Issue:
pex python interpreter path contains absolute path from previous remote execution sandbox
Protobuf code generation is currently failing when run in remote execution against Buildfarm even after the fix in https://github.com/pantsbuild/pants/pull/13077 for archive extraction in remote execution.

Using the [example-python repository](https://github.com/pantsbuild/example-python/) for the test, the error that occurs when running `./pants test helloworld::` (with appropriate options to enable remote execution) is: 

```
Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generating Python sources from helloworld/util/proto/config.proto.' failed with exit code 1.
stdout:

stderr:
/tmp/worker/remote-execution/operations/28150c45-177d-4894-9fcb-7b88fc57f9c7/.cache/pex_root/venvs/8ccfdd776b06121b883ca8d3331e429b87fe042d/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy: 2: exec: /tmp/worker/remote-execution/operations/c79cf097-80ab-4212-b27f-b999f573cfc7/.cache/pex_root/venvs/short/02a3c5ef/bin/python3.8: not found
--mypy_out: protoc-gen-mypy: Plugin failed with status code 127.
```

Note that the path to `protoc-gen-mypy` and the path to `python3.8` are different: `28150c45-177d-4894-9fcb-7b88fc57f9c7` vs `c79cf097-80ab-4212-b27f-b999f573cfc7`. The shebang for the `python3.8` in the venv pex  has the path that was valid in a prior remote execution sandbox.

This occurs on the current Pants `main`. It is unclear when it was introduced since the bug was obscured by the archive extraction bug fixed by https://github.com/pantsbuild/pants/pull/13077.



Discussion:
To reproduce:

1. Clone Pants, [remote-apis-testing](https://gitlab.com/remote-apis-testing/remote-apis-testing/), and [example-python](https://github.com/pantsbuild/example-python/).
2. Bring up Buildfarm using the configs in remote-apis-testing:
  ```
  cd remote-apis-testing
  cd docker-compose
  python3 -m venv .venv
  .venv/bin/pip install jinja2 ruamel.yaml
  .venv/bin/python ./create_docker_compose_yaml.py 
  docker-compose -f docker-compose-buildfarm.yml up
  ```
3. Add the following script to example-python (and `chmod +x` it):
  ```
  #!/bin/bash
  ./pants_from_sources --no-pantsd -ldebug \
    --remote-execution \
    --remote-execution-address=grpc://127.0.0.1:8980 \
    --remote-store-address=grpc://127.0.0.1:8980 \
    --remote-instance-name=remote-execution \
    --remote-execution-extra-platform-properties=OSFamily=linux \
    --pex-verbosity=3 \
    test helloworld::
  ```
4. Run `do-test.sh`.
I tried changing the `link_input_directories` option of Buildfarm's worker from `false` to `true`, but no change in the error. (I had thought that maybe https://github.com/bazelbuild/bazel-buildfarm/issues/492#issuecomment-672914689 was relevant, but the config used in remote-apis-testing already sets `link_input_directories` to `false`.)
> This occurs on the current Pants `main`. It is unclear when it was introduced since the bug was obscured by the archive extraction bug fixed by #13077.

Does it also occur with the latest `2.7.1` (will cut an rc today)? The largest change(s) on `main` were cherrypicked as 2512aa2e33e91604781ef06cf63077faca18ed1c.
> > 
> 
> Does it also occur with the latest `2.7.1` (will cut an rc today)? The largest change(s) on `main` were cherrypicked as [2512aa2](https://github.com/pantsbuild/pants/commit/2512aa2e33e91604781ef06cf63077faca18ed1c).

It does. The regression occurred some time after 2.3.x and I have not bisected yet. Given the length of time / number of changes, probably not worth bisecting.
This doesn't make alot of sense since we have no append only caches facility when remoting. As such, the file `/tmp/worker/remote-execution/operations/28150c45-177d-4894-9fcb-7b88fc57f9c7/.cache/pex_root/venvs/8ccfdd776b06121b883ca8d3331e429b87fe042d/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy` is created on-the-fly in every execution of the source PEX file since `/tmp/worker/remote-execution/operations/28150c45-177-4894-9fcb-7b88fc57f9c7/.cache/pex_root` is not a symlink to a shared cache in the remoting case like it is in the local case. I'll continue to investigate...
Info on Buildfarm that may or may not be relevant: Buildfarm hard links all of the files in the input root into its local CAS cache. See discussion at https://buildteamworld.slack.com/archives/C9C4H1SN7/p1633710791295800. 
Thanks - that shouldn't be relevant since the relevant input root is:
```
mypy_protobuf.pex
mypy_protobuf.pex_bin_protoc-gen-mypy_shim.sh
mypy_protobuf.pex_bin_python_shim.sh
mypy_protobuf.pex_pex_shim.sh
```
The relevant packed wheel contains:
```console
$ unzip -qc mypy_protobuf.pex/.deps/mypy_protobuf-2.4-py3-none-any.whl bin/protoc-gen-mypy | head -1
#!python
```

And the relevant shim script contains:
```bash
$ cat mypy_protobuf.pex_bin_protoc-gen-mypy_shim.sh
#!/usr/bin/bash
set -euo pipefail

# N.B.: We convert all sandbox root relative paths to absolute paths so this script
# works when run with a cwd set elsewhere.

# N.B.: This relies on BASH_SOURCE which has been available since bash-3.0, released in
# 2004. In turn, our use of BASH_SOURCE relies on the fact that this script is executed
# by the engine via its absolute path.
ABS_SANDBOX_ROOT="${BASH_SOURCE%/*}"

function ensure_absolute() {
    local value0="$1"
    shift
    if [ "${value0:0:1}" == "/" ]; then
        echo "${value0}" "$@"
    else
        echo "${ABS_SANDBOX_ROOT}/${value0}" "$@"
    fi
}

export PATH=/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.unknown.py37.venv/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/jsirois/go/bin:/home/jsirois/.bash.d/bin PEX_IGNORE_RCFILES=true PEX_ROOT=.cache/pex_root LANG=en_US.UTF-8
export PEX_ROOT="$(ensure_absolute ${PEX_ROOT})"

execute_pex_args="$(ensure_absolute /usr/bin/python3.6) $(ensure_absolute mypy_protobuf.pex)"
target_venv_executable="$(ensure_absolute .cache/pex_root/venvs/89c5e62b4572637dc3b077e8d663e561c89a2920/061a3a67f6a451d33806aa2696288de4ee1b175c/bin/protoc-gen-mypy)"
venv_dir="$(ensure_absolute .cache/pex_root/venvs/89c5e62b4572637dc3b077e8d663e561c89a2920/061a3a67f6a451d33806aa2696288de4ee1b175c)"

# Let PEX_TOOLS invocations pass through to the original PEX file since venvs don't come
# with tools support.
if [ -n "${PEX_TOOLS:-}" ]; then
  exec ${execute_pex_args} "$@"
fi

# If the seeded venv has been removed from the PEX_ROOT, we re-seed from the original
# `--venv` mode PEX file.
if [ ! -e "${target_venv_executable}" ]; then
    rm -rf "${venv_dir}" || true
    PEX_INTERPRETER=1 ${execute_pex_args} -c ''
fi

exec "${target_venv_executable}" "$@"
```

No references to any absolute paths from yore in sight.

It's as-if the `.cache/pex_root` dir tree from a prior run is getting saved somehow and restored in later remote process executions.
Oh, hm - `execute_pex_args="$(ensure_absolute /usr/bin/python3.6) $(ensure_absolute mypy_protobuf.pex)"`.

My examples above were all taken from a local process execution sandbox. Continuing to dig.
> Oh, hm - `execute_pex_args="$(ensure_absolute /usr/bin/python3.6) $(ensure_absolute mypy_protobuf.pex)"`.
> 
> My examples above were all taken from a local process execution sandbox. Continuing to dig.

I would try running it in Buildfarm (using the config in remote-apis-testing). This error only happens in Buildfarm. The code works fine running in Buildbarn and Buildgrid.
Yeah - I am. Using your setup above. I just used a local run too to easily grok what's going on. Not as easy to do that in BuildFarm without re-familiarizing myself with its knobs to observe / keep around sandboxes.
I verified that the error occurs with v2.5.2 and Buildfarm. (Tested with latest 2.5.x branch plus had to cherry-pick the archive extraction fix from 2.7.x of `4358c62335d287939242f847d0df1818e12b099e`.)
Works for 2.4.2. So the error was introduced somewhere between 2.4.2 and 2.5.2. `pex` went from 2.1.35 to 2.1.42.
I reproduced the failure by extracting the input root into its own directory and running the applicable command:

```
vagrant@vagrant:~/input-root$ PANTS_CACHE_KEY_TARGET_PLATFORM=none ./bin/protoc --python_out _generated_files --plugin=protoc-gen-mypy=./mypy_protobuf.pex_bin_protoc-gen-mypy_shim.sh --mypy_out _generated_files helloworld/util/proto/config.proto
/home/vagrant/input-root/.cache/pex_root/venvs/89aa3d0e44d3459d7d29cab9699baf5549aadf63/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy: 2: exec: /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8: not found
--mypy_out: protoc-gen-mypy: Plugin failed with status code 127.
vagrant@vagrant:~/input-root$ head .cache/pex_root/venvs/89aa3d0e44d3459d7d29cab9699baf5549aadf63/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from mypy_protobuf.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
```

So `protoc-gen-mypy` was built with the sandbox's absolute path in its shebang. So whatever else is going on, the issue is probably not in this build action, but in the prior build action that would have built `protoc-gen-mypy`.

Indeed happened for all of the similar scripts:

```
vagrant@vagrant:~/input-root$ rg /tmp/worker -- .cache/
.cache/pex_root/venvs/89aa3d0e44d3459d7d29cab9699baf5549aadf63/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy_grpc
2:'''exec' /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8 "$0" "$@"

.cache/pex_root/venvs/89aa3d0e44d3459d7d29cab9699baf5549aadf63/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy
2:'''exec' /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8 "$0" "$@"

.cache/pex_root/installed_wheels/9b0919465dea5f6e283da6280356b0c5ae234313/mypy_protobuf-2.4-py3-none-any.whl/bin/protoc-gen-mypy_grpc
2:'''exec' /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8 "$0" "$@"

.cache/pex_root/installed_wheels/9b0919465dea5f6e283da6280356b0c5ae234313/mypy_protobuf-2.4-py3-none-any.whl/bin/protoc-gen-mypy
2:'''exec' /tmp/worker/remote-execution/operations/0cfd5474-d62e-4b0a-bc5b-213f0200383d/.cache/pex_root/venvs/short/d3393b3e/bin/python3.8 "$0" "$@"
```

The `.deps/mypy_protobuf-2.4-py3-none-any.whl` has the paths embedded in the contained `bin` scripts:

```
vagrant@vagrant:~/mypy_protobuf_unpacked$ unzip ../pex-tree-output/.deps/mypy_protobuf-2.4-py3-none-any.whl
Archive:  ../pex-tree-output/.deps/mypy_protobuf-2.4-py3-none-any.whl
warning:  skipped "../" path component(s) in ../
   creating: mypy_protobuf-2.4.dist-info/
  inflating: mypy_protobuf-2.4.dist-info/INSTALLER
  inflating: mypy_protobuf-2.4.dist-info/LICENSE
  inflating: mypy_protobuf-2.4.dist-info/METADATA
  inflating: mypy_protobuf-2.4.dist-info/RECORD
  inflating: mypy_protobuf-2.4.dist-info/REQUESTED
  inflating: mypy_protobuf-2.4.dist-info/WHEEL
  inflating: mypy_protobuf-2.4.dist-info/entry_points.txt
  inflating: mypy_protobuf-2.4.dist-info/top_level.txt
   creating: bin/
  inflating: bin/protoc-gen-mypy
  inflating: bin/protoc-gen-mypy_grpc
  inflating: bin/protoc_gen_mypy.bat
   creating: mypy_protobuf/
  inflating: mypy_protobuf/__init__.py
  inflating: mypy_protobuf/extensions_pb2.py
  inflating: mypy_protobuf/main.py
vagrant@vagrant:~/mypy_protobuf_unpacked$ head -3 bin/*
==> bin/protoc-gen-mypy <==
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/8379cb0b-99d5-48cf-800e-9adcaa1c4c0e/.cache/pex_root/venvs/short/906abfa3/bin/python3.8 "$0" "$@"
' '''

==> bin/protoc_gen_mypy.bat <==
@echo off
%~dp0\python -u %0\..\protoc-gen-mypy

==> bin/protoc-gen-mypy_grpc <==
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/8379cb0b-99d5-48cf-800e-9adcaa1c4c0e/.cache/pex_root/venvs/short/906abfa3/bin/python3.8 "$0" "$@"
' '''
```
@jsirois: Any pointers on where to look to see how that wheel in https://github.com/pantsbuild/pants/issues/13170#issuecomment-947361138 is made?
Welp:
```console
$ rm -rf ~/.pex
$ pex mypy-protobuf==2.4 --layout packed -omypy-protobuf.pex
$ ls -l mypy-protobuf.pex/.deps
total 1060
-rw-r--r-- 2 jsirois jsirois   15296 Oct 20 08:57 mypy_protobuf-2.4-py3-none-any.whl
-rw-r--r-- 2 jsirois jsirois 1067135 Oct 20 08:57 protobuf-3.18.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
$ zipinfo mypy-protobuf.pex/.deps/mypy_protobuf-2.4-py3-none-any.whl 
Archive:  mypy-protobuf.pex/.deps/mypy_protobuf-2.4-py3-none-any.whl
Zip file size: 15296 bytes, number of entries: 18
drwxr-xr-x  2.0 unx        0 b- stor 80-Jan-01 00:00 ../
drwxr-xr-x  2.0 unx        0 b- stor 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/
-rw-r--r--  2.0 unx        4 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/INSTALLER
-rw-r--r--  2.0 unx    11348 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/LICENSE
-rw-r--r--  2.0 unx      438 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/METADATA
-rw-r--r--  2.0 unx     1171 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/RECORD
-rw-r--r--  2.0 unx        0 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/REQUESTED
-rw-r--r--  2.0 unx       92 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/WHEEL
-rw-r--r--  2.0 unx      108 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/entry_points.txt
-rw-r--r--  2.0 unx       14 b- defN 80-Jan-01 00:00 mypy_protobuf-2.4.dist-info/top_level.txt
drwxr-xr-x  2.0 unx        0 b- stor 80-Jan-01 00:00 mypy_protobuf/
-rw-r--r--  2.0 unx        0 b- defN 80-Jan-01 00:00 mypy_protobuf/__init__.py
-rw-r--r--  2.0 unx     2959 b- defN 80-Jan-01 00:00 mypy_protobuf/extensions_pb2.py
-rw-r--r--  2.0 unx    32486 b- defN 80-Jan-01 00:00 mypy_protobuf/main.py
drwxr-xr-x  2.0 unx        0 b- stor 80-Jan-01 00:00 bin/
-rwxr-xr-x  2.0 unx      207 b- defN 80-Jan-01 00:00 bin/protoc-gen-mypy
-rwxr-xr-x  2.0 unx      207 b- defN 80-Jan-01 00:00 bin/protoc-gen-mypy_grpc
-rwxr-xr-x  2.0 unx       48 b- defN 80-Jan-01 00:00 bin/protoc_gen_mypy.bat
18 files, 49082 bytes uncompressed, 12928 bytes compressed:  73.7%
$ unzip -qc mypy-protobuf.pex/.deps/mypy_protobuf-2.4-py3-none-any.whl bin/protoc-gen-mypy
#!python
# -*- coding: utf-8 -*-
import re
import sys
from mypy_protobuf.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
```

You'll notice the `#!python` special shebang which means fill me in later when you create a venv.

That's done here when Pex uses Pip to install each wheel into a `~/.pex/installed_wheels/<hash>/...` chroot. That chroot is then packed here: https://github.com/pantsbuild/pex/blob/d6235f55282521d28448422d773ec58280f85fdc/pex/pex_builder.py#L690-L713 No sheabng manipulations are performed, just a straight zipping of the wheel chroot; so the `#!python` psuedo-shebang is preserved.

The place where the psuedo-shebang is reified is creating a venv in `--venv` mode here: https://github.com/pantsbuild/pex/blob/0ab02beb91b406563b88049fd470beca82a12d7d/pex/tools/commands/venv.py#L329-L331

Note though that that venv re-write occurs inside `~/.pex/venvs` which should not persist between sandboxes on the remote since the remexec doesn't support named_caches.
And to restate the issue for clarity, once that wheel is packed with the old paths, those paths will persist in all subsequent venv builds using that wheel?
>No sheabng manipulations are performed, just a straight zipping of the wheel chroot; so the #!python psuedo-shebang is preserved.

As demonstrated by https://github.com/pantsbuild/pants/issues/13170#issuecomment-947361138, the wheel captured from the build_pex step does have the fully-substituted shebang paths and not `#!python`. Any ideas where that could be coming from, or is the protobuf rule doing something wrong?
> And to restate the issue for clarity, once that wheel is packed with the old paths, those paths will persist in all subsequent venv builds using that wheel?

Yes, but "once that wheel is packed with the old paths" is the part that I can't currently concede. https://github.com/pantsbuild/pants/issues/13170#issuecomment-947816216 shows that mechanism.

> As demonstrated by #13170 (comment), the wheel captured from the build_pex step does have the fully-substituted shebang paths and not #!python. Any ideas where that could be coming from, or is the protobuf rule doing something wrong?

Well, it has a `#!/bin/sh...`shebang which is nothing Pants or Pex generates in any of their codebases. So still no clue what is generating that thing.
Yeah, so for this remexec request / response pair:
1. Request:
```
12:33:13.72 [DEBUG] built REv2 request (build_id=pants_run_2021_11_23_12_32_53_722_902fdd593d1b428cb53078fe1d0e8ea1): action=Action { command_digest: Some(Digest { hash: "9fcbb93300f2d28aafe21f5658c1bea0856279031ccc4877cc981361abe6da3a", size_bytes: 1758 }), input_root_digest: Some(Digest { hash: "8a0f7034ac7f7fb4ed5d68f8de9ba16766fba86797ccbb618b7ab3fb8f9b8f1c", size_bytes: 351 }), timeout: None, do_not_cache: false, salt: b"", platform: None }; command=Command { arguments: ["/usr/bin/python3.8", "./pex", "--python-path", "/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.unknown.py37.venv/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/jvm/default/bin:/home/jsirois/go/bin:/home/jsirois/.pyenv/versions/2.7.18/bin:/home/jsirois/.pyenv/versions/3.10.0/bin:/home/jsirois/.pyenv/versions/3.5.10/bin:/home/jsirois/.pyenv/versions/3.6.15/bin:/home/jsirois/.pyenv/versions/3.7.12/bin:/home/jsirois/.pyenv/versions/3.8.12/bin:/home/jsirois/.pyenv/versions/3.9.7/bin:/home/jsirois/.pyenv/versions/pypy2.7-7.3.1/bin:/home/jsirois/.pyenv/versions/pypy3.6-7.3.3/bin:/home/jsirois/.pyenv/versions/pypy3.7-7.3.5/bin", "--tmpdir", ".tmp", "-vvv", "--output-file", "mypy_protobuf.pex", "--venv", "--seed", "verbose", "--no-pypi", "--index=https://pypi.org/simple/", "--resolver-version", "pip-2020-resolver", "--python", "/usr/bin/python3.8", "--no-emit-warnings", "--jobs", "8", "--manylinux", "manylinux2014", "--sources-directory=source_files", "--requirement", "mypy-protobuf_default_lockfile.txt", "--no-transitive", "--layout", "packed"], environment_variables: [EnvironmentVariable { name: "CPPFLAGS", value: "" }, EnvironmentVariable { name: "LANG", value: "en_US.UTF-8" }, EnvironmentVariable { name: "LDFLAGS", value: "" }, EnvironmentVariable { name: "PANTS_CACHE_KEY_TARGET_PLATFORM", value: "linux_x86_64" }, EnvironmentVariable { name: "PATH", value: "/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.unknown.py37.venv/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/jsirois/go/bin" }, EnvironmentVariable { name: "PEX_IGNORE_RCFILES", value: "true" }, EnvironmentVariable { name: "PEX_ROOT", value: ".cache/pex_root" }], output_files: [], output_directories: ["mypy_protobuf.pex"], output_paths: [], platform: Some(Platform { properties: [Property { name: "OSFamily", value: "linux" }] }), working_directory: "", output_node_properties: [] }; execute_request=ExecuteRequest { instance_name: "remote-execution", skip_cache_lookup: false, action_digest: Some(Digest { hash: "9f9cc8a24e068cd32dd9dd1832da8cd204312876068c48bf59d6d21df7dd8f20", size_bytes: 142 }), execution_policy: None, results_cache_policy: None }
```
2. Response:
```
12:33:20.09 [DEBUG] Got (nested) execute response: ExecuteResponse { result: Some(ActionResult { output_files: [], output_file_symlinks: [], output_symlinks: [], output_directories: [OutputDirectory { path: "mypy_protobuf.pex", tree_digest: Some(Digest { hash: "f07520de14528d331d759be2fcd5e1b1f4c3578f60965a685e2035737cfab35d", size_bytes: 601 }) }], output_directory_symlinks: [], exit_code: 0, stdout_raw: b"{\"pex_root\": \"/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root\", \"python\": \"/usr/bin/python3.8\", \"pex\": \"/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6/pex\"}\n", stdout_digest: Some(Digest { hash: "4be490816903666cede665f4bbd1e2a40627992388fda26c24e01a52306aaabb", size_bytes: 336 }), stderr_raw: b"pex: Resolving interpreters\npex: Resolving interpreters: 0.1ms\npex: Building pex\npex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt)\npex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt) :: Resolving requirements.\npex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt) :: Resolving requirements. :: Resolving for:\n  DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10))))\npex: Hashing pex\npex: Hashing pex: 19.4ms\npex: Isolating pex\npex: Isolating pex :: Extracting pex to /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/isolated/3ec5c9313da2b2b650cd788c530bdd67b417628d\npex: Isolating pex: 49.9ms\npex:   Extracting pex to /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/isolated/3ec5c9313da2b2b650cd788c530bdd67b417628d: 49.7ms\npex: Vendoring attr from VendorSpec(key='attrs', requirement='attrs==21.2.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/attrs\npex: Vendoring packaging from VendorSpec(key='packaging', requirement='packaging==20.8', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/packaging\npex: Vendoring pyparsing from VendorSpec(key='packaging', requirement='packaging==20.8', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/packaging\npex: Vendoring pkg_resources from VendorSpec(key='setuptools', requirement='setuptools==44.0.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools\npex: Vendoring setuptools from VendorSpec(key='setuptools', requirement='setuptools==44.0.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools\npex: Vendoring wheel from VendorSpec(key='wheel', requirement='wheel==0.36.2', rewrite=False, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/wheel\npex: Executing: /usr/bin/python3.8 -s -E /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmp9r05ljdg\npex: Executing: /usr/bin/python3.8 -s -E -m venv --without-pip /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee --prompt pex\npex: Executing: PYTHONPATH=/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/isolated/3ec5c9313da2b2b650cd788c530bdd67b417628d /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee/bin/python3.8 -s -c import os\nimport sys\n\nfrom pex.common import atomic_directory, safe_open\nfrom pex.interpreter import PythonIdentity\n\n\nencoded_identity = PythonIdentity.get(binary='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee/bin/python3.8').encode()\nwith atomic_directory('/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/interpreters/1767c82461266558947a7891785e6e3a0441205f/a311155cf6dcd5e39bbe622992d50b4097308d16/15ae6261d159651d4661ae21d260f3090ec77280', exclusive=False) as cache_dir:\n    if not cache_dir.is_finalized:\n        with safe_open(\n            os.path.join(cache_dir.work_dir, 'INTERP-INFO'), 'w'\n        ) as fp:\n            fp.write(encoded_identity)\n\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee/bin/activate\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee/bin/activate.csh\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6.da35e8f2ca9c4369bb8cef9b40b155ee/bin/activate.fish\npex: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d/.deps\npex: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d/.deps :: Adding pip 20.3.4\npex: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d/.deps :: Adding setuptools 44.0.0\npex: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d/.deps :: Adding wheel 0.36.2\npex: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d/.deps: 0.7ms\npex:   Adding pip 20.3.4: 0.0ms\npex:   Adding setuptools 44.0.0: 0.0ms\npex:   Adding wheel 0.36.2: 0.0ms\npex: Resolving pip==20.3.4\npex: Resolving pip==20.3.4: 0.4ms\npex: Resolving setuptools==44.0.0\npex: Skipping activation of `certifi==2016.9.26; extra == \"certs\"` due to environment marker de-selection\npex: Skipping activation of `wincertstore==0.2; sys_platform == \"win32\" and extra == \"ssl\"` due to environment marker de-selection\npex: Resolving setuptools==44.0.0: 1.6ms\npex: Resolving wheel==0.36.2\npex: Skipping activation of `pytest>=3.0.0; extra == \"test\"` due to environment marker de-selection\npex: Skipping activation of `pytest-cov; extra == \"test\"` due to environment marker de-selection\npex: Resolving wheel==0.36.2: 1.2ms\n/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_bootstrapper.py:483: PEXWarning: The venv for /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/pip.pex/3ec5c9313da2b2b650cd788c530bdd67b417628d at /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/3dcf6b31c378238a00ac6d4c266d107abd64cf6e/ddab8011daaee380698ac2fb9701af18c90c03f6 has script shebangs of '#!/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/s/89607fa6/venv/bin/python3.8 -sE' with 134 characters. On some systems this may be too long and cause problems running the venv scripts. You may be able adjust PEX_ROOT from /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root to a shorter path as a work-around.\n  pex_warnings.warn(\npex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt) :: Resolving requirements. :: Calculating project names for direct requirements:\n  PyPIRequirement(line=LogicalLine(raw_text='mypy-protobuf==2.4; python_version >= \"3.6\" \\\\\\n    --hash=sha256:77e10c476cdd3ee14535c2357e64deac6b1a69f33eb500d795b064acda48c66f \\\\\\n    --hash=sha256:1986dc8a7994a4e549be5c8ac586aa8247caf92456c67159d71dbe5005809651\\n', processed_text='mypy-protobuf==2.4; python_version >= \"3.6\" --hash=sha256:77e10c476cdd3ee14535c2357e64deac6b1a69f33eb500d795b064acda48c66f --hash=sha256:1986dc8a7994a4e549be5c8ac586aa8247caf92456c67159d71dbe5005809651', source='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/mypy-protobuf_default_lockfile.txt', start_line=17, end_line=19), requirement=Requirement.parse('mypy-protobuf==2.4; python_version >= \"3.6\"'), editable=False)\n  PyPIRequirement(line=LogicalLine(raw_text='protobuf==3.19.1; python_version >= \"3.6\" \\\\\\n    --hash=sha256:d80f80eb175bf5f1169139c2e0c5ada98b1c098e2b3c3736667f28cbbea39fc8 \\\\\\n    --hash=sha256:a529e7df52204565bcd33738a7a5f288f3d2d37d86caa5d78c458fa5fabbd54d \\\\\\n    --hash=sha256:28ccea56d4dc38d35cd70c43c2da2f40ac0be0a355ef882242e8586c6d66666f \\\\\\n    --hash=sha256:8b30a7de128c46b5ecb343917d9fa737612a6e8280f440874e5cc2ba0d79b8f6 \\\\\\n    --hash=sha256:5935c8ce02e3d89c7900140a8a42b35bc037ec07a6aeb61cc108be8d3c9438a6 \\\\\\n    --hash=sha256:74f33edeb4f3b7ed13d567881da8e5a92a72b36495d57d696c2ea1ae0cfee80c \\\\\\n    --hash=sha256:038daf4fa38a7e818dd61f51f22588d61755160a98db087a046f80d66b855942 \\\\\\n    --hash=sha256:8e51561d72efd5bd5c91490af1f13e32bcba8dab4643761eb7de3ce18e64a853 \\\\\\n    --hash=sha256:6e8ea9173403219239cdfd8d946ed101f2ab6ecc025b0fda0c6c713c35c9981d \\\\\\n    --hash=sha256:db3532d9f7a6ebbe2392041350437953b6d7a792de10e629c1e4f5a6b1fe1ac6 \\\\\\n    --hash=sha256:615b426a177780ce381ecd212edc1e0f70db8557ed72560b82096bd36b01bc04 \\\\\\n    --hash=sha256:d8919368410110633717c406ab5c97e8df5ce93020cfcf3012834f28b1fab1ea \\\\\\n    --hash=sha256:71b0250b0cfb738442d60cab68abc166de43411f2a4f791d31378590bfb71bd7 \\\\\\n    --hash=sha256:3cd0458870ea7d1c58e948ac8078f6ba8a7ecc44a57e03032ed066c5bb318089 \\\\\\n    --hash=sha256:655264ed0d0efe47a523e2255fc1106a22f6faab7cc46cfe99b5bae085c2a13e \\\\\\n    --hash=sha256:b691d996c6d0984947c4cf8b7ae2fe372d99b32821d0584f0b90277aa36982d3 \\\\\\n    --hash=sha256:e7e8d2c20921f8da0dea277dfefc6abac05903ceac8e72839b2da519db69206b \\\\\\n    --hash=sha256:fd390367fc211cc0ffcf3a9e149dfeca78fecc62adb911371db0cec5c8b7472d \\\\\\n    --hash=sha256:d83e1ef8cb74009bebee3e61cc84b1c9cd04935b72bca0cbc83217d140424995 \\\\\\n    --hash=sha256:36d90676d6f426718463fe382ec6274909337ca6319d375eebd2044e6c6ac560 \\\\\\n    --hash=sha256:e7b24c11df36ee8e0c085e5b0dc560289e4b58804746fb487287dda51410f1e2 \\\\\\n    --hash=sha256:77d2fadcf369b3f22859ab25bd12bb8e98fb11e05d9ff9b7cd45b711c719c002 \\\\\\n    --hash=sha256:e813b1c9006b6399308e917ac5d298f345d95bb31f46f02b60cd92970a9afa17 \\\\\\n    --hash=sha256:62a8e4baa9cb9e064eb62d1002eca820857ab2138440cb4b3ea4243830f94ca7\\n', processed_text='protobuf==3.19.1; python_version >= \"3.6\" --hash=sha256:d80f80eb175bf5f1169139c2e0c5ada98b1c098e2b3c3736667f28cbbea39fc8 --hash=sha256:a529e7df52204565bcd33738a7a5f288f3d2d37d86caa5d78c458fa5fabbd54d --hash=sha256:28ccea56d4dc38d35cd70c43c2da2f40ac0be0a355ef882242e8586c6d66666f --hash=sha256:8b30a7de128c46b5ecb343917d9fa737612a6e8280f440874e5cc2ba0d79b8f6 --hash=sha256:5935c8ce02e3d89c7900140a8a42b35bc037ec07a6aeb61cc108be8d3c9438a6 --hash=sha256:74f33edeb4f3b7ed13d567881da8e5a92a72b36495d57d696c2ea1ae0cfee80c --hash=sha256:038daf4fa38a7e818dd61f51f22588d61755160a98db087a046f80d66b855942 --hash=sha256:8e51561d72efd5bd5c91490af1f13e32bcba8dab4643761eb7de3ce18e64a853 --hash=sha256:6e8ea9173403219239cdfd8d946ed101f2ab6ecc025b0fda0c6c713c35c9981d --hash=sha256:db3532d9f7a6ebbe2392041350437953b6d7a792de10e629c1e4f5a6b1fe1ac6 --hash=sha256:615b426a177780ce381ecd212edc1e0f70db8557ed72560b82096bd36b01bc04 --hash=sha256:d8919368410110633717c406ab5c97e8df5ce93020cfcf3012834f28b1fab1ea --hash=sha256:71b0250b0cfb738442d60cab68abc166de43411f2a4f791d31378590bfb71bd7 --hash=sha256:3cd0458870ea7d1c58e948ac8078f6ba8a7ecc44a57e03032ed066c5bb318089 --hash=sha256:655264ed0d0efe47a523e2255fc1106a22f6faab7cc46cfe99b5bae085c2a13e --hash=sha256:b691d996c6d0984947c4cf8b7ae2fe372d99b32821d0584f0b90277aa36982d3 --hash=sha256:e7e8d2c20921f8da0dea277dfefc6abac05903ceac8e72839b2da519db69206b --hash=sha256:fd390367fc211cc0ffcf3a9e149dfeca78fecc62adb911371db0cec5c8b7472d --hash=sha256:d83e1ef8cb74009bebee3e61cc84b1c9cd04935b72bca0cbc83217d140424995 --hash=sha256:36d90676d6f426718463fe382ec6274909337ca6319d375eebd2044e6c6ac560 --hash=sha256:e7b24c11df36ee8e0c085e5b0dc560289e4b58804746fb487287dda51410f1e2 --hash=sha256:77d2fadcf369b3f22859ab25bd12bb8e98fb11e05d9ff9b7cd45b711c719c002 --hash=sha256:e813b1c9006b6399308e917ac5d298f345d95bb31f46f02b60cd92970a9afa17 --hash=sha256:62a8e4baa9cb9e064eb62d1002eca820857ab2138440cb4b3ea4243830f94ca7', source='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/mypy-protobuf_default_lockfile.txt', start_line=20, end_line=44), requirement=Requirement.parse('protobuf==3.19.1; python_version >= \"3.6\"'), editable=False)\npex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt) :: Resolving requirements. :: Installing:\n  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10)))), wheel_path='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl', fingerprint='60974b0bc64909a12b2201d5d0093478d955e90a')\n  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10)))), wheel_path='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', fingerprint='48658658ca878b5d7683a9c90b3f7d92169ca45b')\npex: Installing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl in /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl\npex: Installing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl in /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/48658658ca878b5d7683a9c90b3f7d92169ca45b/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl\nUsing pip 20.3.4 from /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/s/89607fa6/venv/lib/python3.8/site-packages/pip (python 3.8)\nNon-user install due to --prefix or --target option\nUsing pip 20.3.4 from /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/s/89607fa6/venv/lib/python3.8/site-packages/pip (python 3.8)\nNon-user install due to --prefix or --target option\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-target-dv089yax\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-target-pgdxjh3n\nIgnoring indexes: https://pypi.org/simple\nIgnoring indexes: https://pypi.org/simple\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-ephem-wheel-cache-scfu0jpx\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-ephem-wheel-cache-ckp1vntx\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq\nInitialized build tracking at /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd\nInitialized build tracking at /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq\nCreated build tracker: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd\nCreated build tracker: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq\nEntered build tracker: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd\nEntered build tracker: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-install-soa8ffvm\nCreated temporary directory: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-install-q3rlkryu\nProcessing ./.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl\nProcessing ./.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl\n  Added protobuf==3.19.1 from file:///tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl to build tracker '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd'\n  Added mypy-protobuf==2.4 from file:///tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl to build tracker '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq'\n  Removed protobuf==3.19.1 from file:///tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl from build tracker '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd'\n  Removed mypy-protobuf==2.4 from file:///tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl from build tracker '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq'\nInstalling collected packages: mypy-protobuf\nInstalling collected packages: protobuf\n  changing mode of /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-target-pgdxjh3n/bin/protoc-gen-mypy to 755\n  changing mode of /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-target-pgdxjh3n/bin/protoc-gen-mypy_grpc to 755\nSuccessfully installed mypy-protobuf-2.4\nRemoved build tracker: '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-hdb539sq'\nSuccessfully installed protobuf-3.19.1\nRemoved build tracker: '/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/pip-req-tracker-ijvsnxgd'\npex: Building pex: 2209.1ms\npex:   Resolving distributions (mypy-protobuf_default_lockfile.txt): 2208.8ms\npex:     Resolving requirements.: 2205.3ms\npex:       Resolving for:\n  DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10)))): 1993.7ms\npex:       Calculating project names for direct requirements:\n  PyPIRequirement(line=LogicalLine(raw_text='mypy-protobuf==2.4; python_version >= \"3.6\" \\\\\\n    --hash=sha256:77e10c476cdd3ee14535c2357e64deac6b1a69f33eb500d795b064acda48c66f \\\\\\n    --hash=sha256:1986dc8a7994a4e549be5c8ac586aa8247caf92456c67159d71dbe5005809651\\n', processed_text='mypy-protobuf==2.4; python_version >= \"3.6\" --hash=sha256:77e10c476cdd3ee14535c2357e64deac6b1a69f33eb500d795b064acda48c66f --hash=sha256:1986dc8a7994a4e549be5c8ac586aa8247caf92456c67159d71dbe5005809651', source='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/mypy-protobuf_default_lockfile.txt', start_line=17, end_line=19), requirement=Requirement.parse('mypy-protobuf==2.4; python_version >= \"3.6\"'), editable=False)\n  PyPIRequirement(line=LogicalLine(raw_text='protobuf==3.19.1; python_version >= \"3.6\" \\\\\\n    --hash=sha256:d80f80eb175bf5f1169139c2e0c5ada98b1c098e2b3c3736667f28cbbea39fc8 \\\\\\n    --hash=sha256:a529e7df52204565bcd33738a7a5f288f3d2d37d86caa5d78c458fa5fabbd54d \\\\\\n    --hash=sha256:28ccea56d4dc38d35cd70c43c2da2f40ac0be0a355ef882242e8586c6d66666f \\\\\\n    --hash=sha256:8b30a7de128c46b5ecb343917d9fa737612a6e8280f440874e5cc2ba0d79b8f6 \\\\\\n    --hash=sha256:5935c8ce02e3d89c7900140a8a42b35bc037ec07a6aeb61cc108be8d3c9438a6 \\\\\\n    --hash=sha256:74f33edeb4f3b7ed13d567881da8e5a92a72b36495d57d696c2ea1ae0cfee80c \\\\\\n    --hash=sha256:038daf4fa38a7e818dd61f51f22588d61755160a98db087a046f80d66b855942 \\\\\\n    --hash=sha256:8e51561d72efd5bd5c91490af1f13e32bcba8dab4643761eb7de3ce18e64a853 \\\\\\n    --hash=sha256:6e8ea9173403219239cdfd8d946ed101f2ab6ecc025b0fda0c6c713c35c9981d \\\\\\n    --hash=sha256:db3532d9f7a6ebbe2392041350437953b6d7a792de10e629c1e4f5a6b1fe1ac6 \\\\\\n    --hash=sha256:615b426a177780ce381ecd212edc1e0f70db8557ed72560b82096bd36b01bc04 \\\\\\n    --hash=sha256:d8919368410110633717c406ab5c97e8df5ce93020cfcf3012834f28b1fab1ea \\\\\\n    --hash=sha256:71b0250b0cfb738442d60cab68abc166de43411f2a4f791d31378590bfb71bd7 \\\\\\n    --hash=sha256:3cd0458870ea7d1c58e948ac8078f6ba8a7ecc44a57e03032ed066c5bb318089 \\\\\\n    --hash=sha256:655264ed0d0efe47a523e2255fc1106a22f6faab7cc46cfe99b5bae085c2a13e \\\\\\n    --hash=sha256:b691d996c6d0984947c4cf8b7ae2fe372d99b32821d0584f0b90277aa36982d3 \\\\\\n    --hash=sha256:e7e8d2c20921f8da0dea277dfefc6abac05903ceac8e72839b2da519db69206b \\\\\\n    --hash=sha256:fd390367fc211cc0ffcf3a9e149dfeca78fecc62adb911371db0cec5c8b7472d \\\\\\n    --hash=sha256:d83e1ef8cb74009bebee3e61cc84b1c9cd04935b72bca0cbc83217d140424995 \\\\\\n    --hash=sha256:36d90676d6f426718463fe382ec6274909337ca6319d375eebd2044e6c6ac560 \\\\\\n    --hash=sha256:e7b24c11df36ee8e0c085e5b0dc560289e4b58804746fb487287dda51410f1e2 \\\\\\n    --hash=sha256:77d2fadcf369b3f22859ab25bd12bb8e98fb11e05d9ff9b7cd45b711c719c002 \\\\\\n    --hash=sha256:e813b1c9006b6399308e917ac5d298f345d95bb31f46f02b60cd92970a9afa17 \\\\\\n    --hash=sha256:62a8e4baa9cb9e064eb62d1002eca820857ab2138440cb4b3ea4243830f94ca7\\n', processed_text='protobuf==3.19.1; python_version >= \"3.6\" --hash=sha256:d80f80eb175bf5f1169139c2e0c5ada98b1c098e2b3c3736667f28cbbea39fc8 --hash=sha256:a529e7df52204565bcd33738a7a5f288f3d2d37d86caa5d78c458fa5fabbd54d --hash=sha256:28ccea56d4dc38d35cd70c43c2da2f40ac0be0a355ef882242e8586c6d66666f --hash=sha256:8b30a7de128c46b5ecb343917d9fa737612a6e8280f440874e5cc2ba0d79b8f6 --hash=sha256:5935c8ce02e3d89c7900140a8a42b35bc037ec07a6aeb61cc108be8d3c9438a6 --hash=sha256:74f33edeb4f3b7ed13d567881da8e5a92a72b36495d57d696c2ea1ae0cfee80c --hash=sha256:038daf4fa38a7e818dd61f51f22588d61755160a98db087a046f80d66b855942 --hash=sha256:8e51561d72efd5bd5c91490af1f13e32bcba8dab4643761eb7de3ce18e64a853 --hash=sha256:6e8ea9173403219239cdfd8d946ed101f2ab6ecc025b0fda0c6c713c35c9981d --hash=sha256:db3532d9f7a6ebbe2392041350437953b6d7a792de10e629c1e4f5a6b1fe1ac6 --hash=sha256:615b426a177780ce381ecd212edc1e0f70db8557ed72560b82096bd36b01bc04 --hash=sha256:d8919368410110633717c406ab5c97e8df5ce93020cfcf3012834f28b1fab1ea --hash=sha256:71b0250b0cfb738442d60cab68abc166de43411f2a4f791d31378590bfb71bd7 --hash=sha256:3cd0458870ea7d1c58e948ac8078f6ba8a7ecc44a57e03032ed066c5bb318089 --hash=sha256:655264ed0d0efe47a523e2255fc1106a22f6faab7cc46cfe99b5bae085c2a13e --hash=sha256:b691d996c6d0984947c4cf8b7ae2fe372d99b32821d0584f0b90277aa36982d3 --hash=sha256:e7e8d2c20921f8da0dea277dfefc6abac05903ceac8e72839b2da519db69206b --hash=sha256:fd390367fc211cc0ffcf3a9e149dfeca78fecc62adb911371db0cec5c8b7472d --hash=sha256:d83e1ef8cb74009bebee3e61cc84b1c9cd04935b72bca0cbc83217d140424995 --hash=sha256:36d90676d6f426718463fe382ec6274909337ca6319d375eebd2044e6c6ac560 --hash=sha256:e7b24c11df36ee8e0c085e5b0dc560289e4b58804746fb487287dda51410f1e2 --hash=sha256:77d2fadcf369b3f22859ab25bd12bb8e98fb11e05d9ff9b7cd45b711c719c002 --hash=sha256:e813b1c9006b6399308e917ac5d298f345d95bb31f46f02b60cd92970a9afa17 --hash=sha256:62a8e4baa9cb9e064eb62d1002eca820857ab2138440cb4b3ea4243830f94ca7', source='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/mypy-protobuf_default_lockfile.txt', start_line=20, end_line=44), requirement=Requirement.parse('protobuf==3.19.1; python_version >= \"3.6\"'), editable=False): 0.1ms\npex:       Installing:\n  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10)))), wheel_path='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/mypy_protobuf-2.4-py3-none-any.whl', fingerprint='60974b0bc64909a12b2201d5d0093478d955e90a')\n  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/usr/bin/python3.8', PythonIdentity('/usr/bin/python3.8', 'cp38', 'cp38', 'manylinux_2_31_x86_64', (3, 8, 10)))), wheel_path='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmppaz3wf9q/usr.bin.python3.8/protobuf-3.19.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', fingerprint='48658658ca878b5d7683a9c90b3f7d92169ca45b'): 205.4ms\npex: Vendoring attr from VendorSpec(key='attrs', requirement='attrs==21.2.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/attrs\npex: Vendoring packaging from VendorSpec(key='packaging', requirement='packaging==20.8', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/packaging\npex: Vendoring pyparsing from VendorSpec(key='packaging', requirement='packaging==20.8', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/packaging\npex: Vendoring pkg_resources from VendorSpec(key='setuptools', requirement='setuptools==44.0.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools\npex: Vendoring setuptools from VendorSpec(key='setuptools', requirement='setuptools==44.0.0', rewrite=True, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools\npex: Vendoring wheel from VendorSpec(key='wheel', requirement='wheel==0.36.2', rewrite=False, constrain=True) @ /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/vendor/_vendored/wheel\nSaving PEX file to mypy_protobuf.pex\npex: Seeding local caches for mypy_protobuf.pex\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex\npex: Executing: /usr/bin/python3.8 -s -E -m venv --without-pip /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5 --prompt mypy_protobuf.pex\npex: Executing: PYTHONPATH=/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/isolated/3ec5c9313da2b2b650cd788c530bdd67b417628d /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5/bin/python3.8 -s -c import os\nimport sys\n\nfrom pex.common import atomic_directory, safe_open\nfrom pex.interpreter import PythonIdentity\n\n\nencoded_identity = PythonIdentity.get(binary='/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5/bin/python3.8').encode()\nwith atomic_directory('/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/interpreters/1767c82461266558947a7891785e6e3a0441205f/a311155cf6dcd5e39bbe622992d50b4097308d16/1ebcaf932da6f6bd92ef15c004d857d1cb282c4e', exclusive=False) as cache_dir:\n    if not cache_dir.is_finalized:\n        with safe_open(\n            os.path.join(cache_dir.work_dir, 'INTERP-INFO'), 'w'\n        ) as fp:\n            fp.write(encoded_identity)\n\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5/bin/activate\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5/bin/activate.csh\npex: Re-writing /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6.b9d4f6bf1f29430e9dc38d8fbadd22e5/bin/activate.fish\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex :: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmplp_c_xp1/.deps\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex :: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmplp_c_xp1/.deps :: Adding mypy-protobuf 2.4\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex :: Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmplp_c_xp1/.deps :: Adding protobuf 3.19.1\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex :: Resolving mypy-protobuf==2.4; python_version >= \"3.6\"\npex: Seeding local caches for mypy_protobuf.pex :: Creating venv from mypy_protobuf.pex :: Resolving protobuf==3.19.1; python_version >= \"3.6\"\n/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_bootstrapper.py:483: PEXWarning: The venv for /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmplp_c_xp1 at /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/039f8633766b446a76d78ba9001ca2cc0bdf99c2/ddab8011daaee380698ac2fb9701af18c90c03f6 has script shebangs of '#!/tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/s/7f892a2b/venv/bin/python3.8 -sE' with 134 characters. On some systems this may be too long and cause problems running the venv scripts. You may be able adjust PEX_ROOT from /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root to a shorter path as a work-around.\n  pex_warnings.warn(\npex: Seeding local caches for mypy_protobuf.pex: 131.1ms\npex:   Creating venv from mypy_protobuf.pex: 131.0ms\npex:     Searching dependency cache: /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.tmp/tmplp_c_xp1/.deps: 1.1ms\npex:       Adding mypy-protobuf 2.4: 0.0ms\npex:       Adding protobuf 3.19.1: 0.0ms\npex:     Resolving mypy-protobuf==2.4; python_version >= \"3.6\": 0.3ms\npex:     Resolving protobuf==3.19.1; python_version >= \"3.6\": 0.0ms\n", stderr_digest: Some(Digest { hash: "bfffa759c19cd0b770877c5a0aefe767ebdbd0ceea14ab9788dd12b590c6a5c1", size_bytes: 33230 }), execution_metadata: Some(ExecutedActionMetadata { worker: "7abe2cb28c6c", queued_timestamp: Some(Timestamp { seconds: 1637699595, nanos: 412000000 }), worker_start_timestamp: Some(Timestamp { seconds: 1637699595, nanos: 413000000 }), worker_completed_timestamp: Some(Timestamp { seconds: 1637699600, nanos: 90000000 }), input_fetch_start_timestamp: Some(Timestamp { seconds: 1637699595, nanos: 413000000 }), input_fetch_completed_timestamp: Some(Timestamp { seconds: 1637699595, nanos: 422000000 }), execution_start_timestamp: Some(Timestamp { seconds: 1637699596, nanos: 926000000 }), execution_completed_timestamp: Some(Timestamp { seconds: 1637699600, nanos: 81000000 }), output_upload_start_timestamp: Some(Timestamp { seconds: 1637699600, nanos: 81000000 }), output_upload_completed_timestamp: Some(Timestamp { seconds: 1637699600, nanos: 90000000 }), auxiliary_metadata: [] }) }), cached_result: false, status: Some(Status { code: 0, message: "", details: [] }), server_logs: {}, message: "" }
```

If I 1st materialize the input and then run that command:
```
$ src/rust/engine/target/debug/fs_util --server-address grpc://127.0.0.1:8980 directory materialize 8a0f7034ac7f7fb4ed5d68f8de9ba16766fba86797ccbb618b7ab3fb8f9b8f1c 351 input
$ cd input/
$ CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= PANTS_CACHE_KEY_TARGET_PLATFORM=linux_x86_64 PATH=/home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.unknown.py37.venv/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/jsirois/go/bin PEX_IGNORE_RCFILES=true PEX_ROOT=.cache/pex_root /usr/bin/python3.8 ./pex --python-path /home/jsirois/.cache/pants/pants_dev_deps/Linux.x86_64.unknown.py37.venv/bin:/home/jsirois/.cargo/bin:/home/jsirois/.pyenv/shims:/home/jsirois/.pyenv/bin:/home/jsirois/bin:/home/jsirois/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/jvm/default/bin:/home/jsirois/go/bin:/home/jsirois/.pyenv/versions/2.7.18/bin:/home/jsirois/.pyenv/versions/3.10.0/bin:/home/jsirois/.pyenv/versions/3.5.10/bin:/home/jsirois/.pyenv/versions/3.6.15/bin:/home/jsirois/.pyenv/versions/3.7.12/bin:/home/jsirois/.pyenv/versions/3.8.12/bin:/home/jsirois/.pyenv/versions/3.9.7/bin:/home/jsirois/.pyenv/versions/pypy2.7-7.3.1/bin:/home/jsirois/.pyenv/versions/pypy3.6-7.3.3/bin:/home/jsirois/.pyenv/versions/pypy3.7-7.3.5/bin --tmpdir .tmp -vvv --output-file mypy_protobuf.pex --venv --seed verbose --no-pypi --index=https://pypi.org/simple/ --resolver-version pip-2020-resolver --python /usr/bin/python3.8 --no-emit-warnings --jobs 8 --manylinux manylinux2014 --sources-directory=source_files --requirement mypy-protobuf_default_lockfile.txt --no-transitive --layout packed
pex: Resolving interpreters: 0.1ms
pex: Building pex :: Resolving distributions (mypy-protobuf_default_lockfile.txt) :: Resolving requirements. :: Resolving for:
pex: Hashing pex                                                                                                                                                                                                                                                                 pex: Hashing pex: 20.0ms
pex: Isolating pex: 55.0ms                                                                                                                                
pex:   Extracting pex to /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/isolated/3ec5c9313da2b2b650cd788c530bdd67b417628d: 54.8ms
...
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6.a6024aab610848749ad0960a65d149d3/bin/activate
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6.a6024aab610848749ad0960a65d149d3/bin/activate.csh
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6.a6024aab610848749ad0960a65d149d3/bin/activate.fish
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6.a6024aab610848749ad0960a65d149d3/bin/protoc-gen-mypy
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6.a6024aab610848749ad0960a65d149d3/bin/protoc-gen-mypy_grpc
pex: Seeding local caches for mypy_protobuf.pex: 136.8ms
pex:   Creating venv from mypy_protobuf.pex: 136.7ms
pex:     Searching dependency cache: /home/jsirois/dev/pantsbuild/jsirois-pants/input/.tmp/tmpxajtj1uk/.deps: 1.1ms
pex:       Adding mypy-protobuf 2.4: 0.0ms
pex:       Adding protobuf 3.19.1: 0.0ms
pex:     Resolving mypy-protobuf==2.4; python_version >= "3.6": 0.4ms
pex:     Resolving protobuf==3.19.1; python_version >= "3.6": 0.0ms
{"pex_root": "/home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root", "python": "/usr/bin/python3.8", "pex": "/home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6/pex"}
```

I find again, the shebang in the mypy_protobuf wheel zip is not expanded.:
```
$ unzip -qc mypy_protobuf.pex/.deps/mypy_protobuf-2.4-py3-none-any.whl bin/protoc-gen-mypy
#!python
# -*- coding: utf-8 -*-
import re
import sys
from mypy_protobuf.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
```

Now the venv script equivalent is - of course - re-written, but not to `#!/bin/sh`, and, crucially, not to a path that is captured as an output (`.cache/pex_root`):
```
$ head /home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/ee42c6357299c57b1663ddafc10d825a815ef185/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/protoc-gen-mypy
#!/home/jsirois/dev/pantsbuild/jsirois-pants/input/.cache/pex_root/venvs/s/b445c838/venv/bin/python3.8 -sE
# -*- coding: utf-8 -*-
import re
import sys
from mypy_protobuf.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
```

That said! The output digest for the process has:
```
$ src/rust/engine/target/debug/fs_util --server-address grpc://127.0.0.1:8980 tree materialize f07520de14528d331d759be2fcd5e1b1f4c3578f60965a685e2035737cfab35d 601 output
$ cd output/
$ unzip -qc .deps/mypy_protobuf-2.4-py3-none-any.whl bin/protoc-gen-mypy
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/180b2a48-a118-4397-82b7-75275350e7f2/.cache/pex_root/venvs/s/89607fa6/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from mypy_protobuf.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
```
So ... I'm at a complete loss.
Alright, doctored up Pex to log script re-write operations to a `/debug.txt` log file in the container and I find:
```console
$ docker container exec -it 7abe2cb28c6c bash -i
root@7abe2cb28c6c:/bazel-buildfarm# cat /debug.txt 
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.516853df8d3644309a44b255cf2f84ba/bin
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.516853df8d3644309a44b255cf2f84ba/bin/pytest
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.516853df8d3644309a44b255cf2f84ba/bin/py.test
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.d1f50e8a2a54400390e30f5aedf230f0/bin
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.d1f50e8a2a54400390e30f5aedf230f0/bin/coverage-3.8
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.d1f50e8a2a54400390e30f5aedf230f0/bin/coverage
/tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/733a416e-078e-4425-8c5b-9948e3808e1c/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.d1f50e8a2a54400390e30f5aedf230f0/bin/coverage3
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.70fd55c05eb54e46988fa82c9fe12734/bin
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.70fd55c05eb54e46988fa82c9fe12734/bin/translate-cli
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/9fa96037a1a985fe7dda81c0e1e35ae015163d88/charset_normalizer-2.0.4-py3-none-any.whl.139b639c79b94d59bd52bbb10c16348a/bin
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/9fa96037a1a985fe7dda81c0e1e35ae015163d88/charset_normalizer-2.0.4-py3-none-any.whl.139b639c79b94d59bd52bbb10c16348a/bin/normalizer
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.286876c502694a6e9c836ad5c4c4315d/bin
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.286876c502694a6e9c836ad5c4c4315d/bin/pip3.8
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.286876c502694a6e9c836ad5c4c4315d/bin/pip
/tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/096f67a9-f750-412a-9169-44281efd61e2/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.286876c502694a6e9c836ad5c4c4315d/bin/pip3
/tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.8a63762cf82b465a88e3c6e3aece3819/bin
/tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.8a63762cf82b465a88e3c6e3aece3819/bin/protoc_gen_mypy.bat
/tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.8a63762cf82b465a88e3c6e3aece3819/bin/protoc-gen-mypy
/tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/98430780dd87bc6ce504dce68c64b626286f31f7/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/af6ae367-eca4-4716-af8b-d716a01c62e8/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.8a63762cf82b465a88e3c6e3aece3819/bin/protoc-gen-mypy_grpc
/tmp/worker/remote-execution/operations/768271cd-264c-4c2f-b64a-a9714b0c54f0/.cache/pex_root/installed_wheels/c720e2bf057dc4ff88df428c9dd87f782a55dbd4/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/768271cd-264c-4c2f-b64a-a9714b0c54f0/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.e62c93e0086e4c6986c3ce51601a3c4f/bin
/tmp/worker/remote-execution/operations/768271cd-264c-4c2f-b64a-a9714b0c54f0/.cache/pex_root/installed_wheels/c720e2bf057dc4ff88df428c9dd87f782a55dbd4/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/768271cd-264c-4c2f-b64a-a9714b0c54f0/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.e62c93e0086e4c6986c3ce51601a3c4f/bin/pytest
/tmp/worker/remote-execution/operations/ac458f6a-2fe0-472e-804e-45c4ae120187/.cache/pex_root/installed_wheels/c720e2bf057dc4ff88df428c9dd87f782a55dbd4/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/ac458f6a-2fe0-472e-804e-45c4ae120187/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.1165c662e8c24c6aa2055d1ede6b90b3/bin
/tmp/worker/remote-execution/operations/ac458f6a-2fe0-472e-804e-45c4ae120187/.cache/pex_root/installed_wheels/c720e2bf057dc4ff88df428c9dd87f782a55dbd4/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/ac458f6a-2fe0-472e-804e-45c4ae120187/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.1165c662e8c24c6aa2055d1ede6b90b3/bin/translate-cli
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.6e8589b93c8b4aee8a5a31be7c61bedc/bin
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.6e8589b93c8b4aee8a5a31be7c61bedc/bin/pytest
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/venvs/s/b7f9dc7f/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/d8dbe72f24eaeea8b51e0f353336f9bd4915dc4a/pytest-6.2.5-py3-none-any.whl.6e8589b93c8b4aee8a5a31be7c61bedc/bin/py.test
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/venvs/s/b7f9dc7f/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.4f92e12264694ab2bc10e1e2b913735e/bin
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.4f92e12264694ab2bc10e1e2b913735e/bin/coverage-3.8
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/venvs/s/b7f9dc7f/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.4f92e12264694ab2bc10e1e2b913735e/bin/coverage
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/venvs/s/b7f9dc7f/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/04ac6ecd49abd3b7bd5158795b8e181899083f55/coverage-6.1.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.4f92e12264694ab2bc10e1e2b913735e/bin/coverage3
/tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/f5b19532-34e5-4dc3-80e1-d24711eb39da/.cache/pex_root/venvs/s/b7f9dc7f/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.fde2c4cc03af4679af6c1dd9304fb9eb/bin
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/48afe6378d3f200977f5a08be517e6409eefba98/translate-3.6.1-py2.py3-none-any.whl.fde2c4cc03af4679af6c1dd9304fb9eb/bin/translate-cli
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/venvs/s/a99a3aeb/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/9fa96037a1a985fe7dda81c0e1e35ae015163d88/charset_normalizer-2.0.4-py3-none-any.whl.e6a4bc3223914e7b9002f8c3b3fd753d/bin
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/9fa96037a1a985fe7dda81c0e1e35ae015163d88/charset_normalizer-2.0.4-py3-none-any.whl.e6a4bc3223914e7b9002f8c3b3fd753d/bin/normalizer
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/venvs/s/a99a3aeb/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.364eedc968a94c60af05f03f05fa0aee/bin
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.364eedc968a94c60af05f03f05fa0aee/bin/pip3.8
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/venvs/s/a99a3aeb/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.364eedc968a94c60af05f03f05fa0aee/bin/pip
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/venvs/s/a99a3aeb/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/e13b43549899a1478d60af690ea973c3943a5175/pip-21.2.2-py3-none-any.whl.364eedc968a94c60af05f03f05fa0aee/bin/pip3
/tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/679c67b0-4e40-4c6f-bdad-7a0bc88668d0/.cache/pex_root/venvs/s/a99a3aeb/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: Re-writing scripts in /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.ca6b8c8c3fe14e3cbe12e1a1f9560ebb/bin
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.ca6b8c8c3fe14e3cbe12e1a1f9560ebb/bin/protoc_gen_mypy.bat
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
@echo off
%~dp0\python -u %0\..\protoc-gen-mypy
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.ca6b8c8c3fe14e3cbe12e1a1f9560ebb/bin/protoc-gen-mypy
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/venvs/s/ea545397/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: won't re-write non-python script /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/60974b0bc64909a12b2201d5d0093478d955e90a/mypy_protobuf-2.4-py3-none-any.whl.ca6b8c8c3fe14e3cbe12e1a1f9560ebb/bin/protoc-gen-mypy_grpc
/tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/installed_wheels/99c26016ed352bc9532967eaf31424ca58576aae/pex-2.1.56-py2.py3-none-any.whl/pex/pip.py: ---
#!/bin/sh
'''exec' /tmp/worker/remote-execution/operations/7f0557df-3516-4e79-a881-600c61dd5829/.cache/pex_root/venvs/s/ea545397/venv/bin/python3.8 "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
root@7abe2cb28c6c:/bazel-buildfarm# 
```
So, under BuildFarm in the worker container, it's the `pip install ...` of the wheel that is creating the strange `#!/bin/sh` header. This is not what Pip does when run under not-BuildFarm.
BuildFarm uses hard links to setup the input root. The files are also hard linked in the CAS cache directory used by BuildFarm. Could this confuse `pip` in any way?
Maybe this could be related to the specific remote execution environment (e.g., the python / pip versions etc.) being used in remote -apis-testing and our ad hoc testing?
The only ~arbitrary thing is Python, Pip is vendored by Pex.
@#!$%:
```
root@7abe2cb28c6c:/bazel-buildfarm# python -mvenv test.venv
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt install python3.8-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/bazel-buildfarm/test.venv/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']

```

Still no straight line but this is likely it.
Actually, Pex sets up the `~/.pex/venvs` without Pip and that works fine:
```console
root@7abe2cb28c6c:/bazel-buildfarm# python -mvenv --without-pip test.venv
root@7abe2cb28c6c:/bazel-buildfarm# source test.venv/bin/activate
(test.venv) root@7abe2cb28c6c:/bazel-buildfarm# python --version
Python 3.8.10
(test.venv) root@7abe2cb28c6c:/bazel-buildfarm#
```
Ok, code generating the whacky shebang is here:
https://github.com/pantsbuild/pex/blob/196b4cd5b8dd4b4af2586460530e9a777262be7d/pex/vendor/_vendored/pip/pip/_vendor/distlib/scripts.py#L127-L156
Aha:
```python
if sys.platform == 'darwin':
    max_shebang_length = 512
else:
    max_shebang_length = 127
```
If the shebang length is > 127 then the funky `#!/bin/sh` is used. Pex also warns about this:
```
/tmp/worker/remote-execution/operations/a5e295b0-845b-4403-bf0c-a962013d4fb9/.cache/pex_root/installed_wheels/4ec2965cac6c115a0e102a23041ff575ca770a2f/pex-2.1.56-py2.py3-none-any.whl/pex/pex_bootstrapper.py:483: PEXWarning: The venv for /tmp/worker/remote-execution/operations/a5e295b0-845b-4403-bf0c-a962013d4fb9/.tmp/tmpgnt8b5wg at /tmp/worker/remote-execution/operations/a5e295b0-845b-4403-bf0c-a962013d4fb9/.cache/pex_root/venvs/58cc34e4c025a3f58981bc86a69c269d5b2d177d/ddab8011daaee380698ac2fb9701af18c90c03f6 has script shebangs of '#!/tmp/worker/remote-execution/operations/a5e295b0-845b-4403-bf0c-a962013d4fb9/.cache/pex_root/venvs/s/1b7b2da4/venv/bin/python3.8 -sE' with 134 characters. On some systems this may be too long and cause problems running the venv scripts. You may be able adjust PEX_ROOT from /tmp/worker/remote-execution/operations/a5e295b0-845b-4403-bf0c-a962013d4fb9/.cache/pex_root to a shorter path as a work-around.
```

So it looks like this is all explained now. BuildFarm uses super long paths, which triggers this bit of Pip / distlib code which produces a non python shebang which foils Pip producing a `#!python` faux-shebang, which rolls all the way down hill. I'm not sure what I can do about this. But the fix will have to be over in Pex land if possible at all. I'll link an issue once / if I can think of something here.

That said, on this image, the kernel is configured for 256 so its all unfortunate and should be moot.
Whittled down here: https://github.com/pantsbuild/pex/issues/1520
Still no clue how I work around this yet.
Great debugging! I don't think I would have found that.
The BuildFarm only was screaming from the get go, it was just hard to see what was relevantly different about BuildFarm.
This is in fact fixed by https://github.com/pantsbuild/pex/pull/1521 when using `./pants_from_sources` on a local pants clone modified to point to the Pex artifacts served up by `tox -eserve` against https://github.com/pantsbuild/pex/pull/1521.

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex_cli.py b/src/python/pants/backend/python/util_rules/pex_cli.py
--- a/src/python/pants/backend/python/util_rules/pex_cli.py
+++ b/src/python/pants/backend/python/util_rules/pex_cli.py
@@ -37,7 +37,7 @@ class PexBinary(TemplatedExternalTool):
     name = "pex"
     help = "The PEX (Python EXecutable) tool (https://github.com/pantsbuild/pex)."
 
-    default_version = "v2.1.54"
+    default_version = "v2.1.56"
     default_url_template = "https://github.com/pantsbuild/pex/releases/download/{version}/pex"
     version_constraints = ">=2.1.51,<3.0"
 
@@ -48,8 +48,8 @@ def default_known_versions(cls):
                 (
                     cls.default_version,
                     plat,
-                    "c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451",
-                    "3679238",
+                    "aff02e2ef0212db4531354e9b7b0d5f61745b3eb49665bc11142f0b603a27db9",
+                    "3688610",
                 )
             )
             for plat in ["macos_arm64", "macos_x86_64", "linux_x86_64"]


Test patch: