Repository: pex-tool/pex

Issue:
Be able to see what .pex file is run from the list of system processes
**Is your feature request related to a problem? Please describe.**

When running a PEX file:

```
$ python3 dist/project/bar.pex --input="baz"
```

you don’t see much helpful information on `ps ww` command:

```
46131 s010  S+     0:00.78 /usr/bin/python3 /Users/user-name/.pex/unzipped_pexes/e9eb91dcb789973ca601f8bfb4401555f188d841 --input=baz
```

That is, you don't know what `.pex` file is being run by looking at this process.

**Describe the solution you'd like**

The use case is that at a given point of time there may be multiple PEX files running, and it would be handy to see which ones are running from the `ps` output (or somehow else). Perhaps it would be possible to give the unzipped pex a less obscure name. So instead of `unzipped_pexes/2312ebcec214434cf2349791389503f03e1033a7` the PEX could possibly unpack it to `unzipped_pexes/2312ebcec214434cf2349791389503f03e1033a7/bar.pex`.

**Describe alternatives you've considered**

One can inspect the `/Users/user-name/.pex/unzipped_pexes/e9eb91dcb789973ca601f8bfb4401555f188d841/PEX-INFO` file's `entry_point` property to see what module is being run, but this is not visible from the `ps` output directly. This is tedious.

**Additional context**

It's possible that user may have a few long running tasks (started from `.pex` files) and it would be handy to see if any of them are running by doing `ps` and `grep`.

It is possible to set the process title using a 3rd party package (https://pypi.org/project/setproctitle) as well, however, this implies setting it for each `main()` in all the PEXed programs.

https://github.com/pantsbuild/pants/blob/4b335cdccfebcbd1f92caaa5fef7d5ca161c7226/src/python/pants/pantsd/pants_daemon.py#L173-L174


Discussion:
I think the `<PEX_ROOT>/unzipped_pexes/<PEX hash>/<PEX file basename>` idea is good for standard execution mode PEXes. The other case to handle is `<PEX_ROOT>/venvs/<PEX hash>/<interpreter hash>/pex` for `--venv` execution mode PEXes. That should go to `<PEX_ROOT>/venvs/<PEX hash>/<interpreter hash>/<PEX file basename>/pex` to provide the same `<PEX file basename` clue. The venv case is more delicate though since the venv interpreter (`<PEX_ROOT>/venvs/<PEX hash>/<interpreter hash>/<PEX file basename>/bin/python`) gets embedded in the venv `bin/` scripts where it can run into shebang length limits on Mac and Linux.

The arguably better alternative is to add an option to the Pex CLI that allow the user to choose adding an extra platform-specific dependency to their PEX ([setproctitle](https://pypi.org/project/setproctitle)) to get a crisp process name that reflects the path to the PEX file and not the re-execed unzipped or venv PEX.
@alexey-tereshenkov-oxb adding best-effort `setproctitle` use is straight forward. With that feature added, if you add `setproctitle` as a dependency of your PEX you'd then get:
```console
$ pex -Dsrc -mapp -o new.pex setproctitle
$ ./new.pex bob jones &
$ ps -ef | grep pex
jsirois    56997    2256  8 10:45 pts/8    00:00:00 /home/jsirois/.pyenv/versions/3.10.2/bin/python3.10 /home/jsirois/dev/pantsbuild/jsirois-pex/new.pex bob jones
```

Versus the current:
```console
$ pex -Dsrc -mapp -o old.pex
$ ./old.pex bob jones &
$ ps -ef | grep pex
jsirois    56935    2256 10 10:45 pts/8    00:00:00 /home/jsirois/.pyenv/versions/3.10.2/bin/python3.10 /home/jsirois/.pex/unzipped_pexes/94790b07dc3768a9926dab999b41a87e399e0aa9 bob jones
```

The size impact is minimal:
```console
$ ls -lrth *.pex
-rwxr-xr-x 1 jsirois jsirois  983K Feb 13 10:08 old.pex
-rwxr-xr-x 1 jsirois jsirois 1014K Feb 13 10:38 new.pex
```
Would that work for you? The only complication I can see is building a multi-`--platform` PEX on Linux for Mac. In that case, there are no Mac wheels published for setproctitle on PyPI and the PEX build would fail. The other way around (building a multi-`--platform` PEX for Linux on Mac) would work just fine. To be able to build a multi-`--platform` PEX on Linux for Mac you'd need to pre-build the Mac wheel(s) for `setproctitle` by hand and host them in a custom `--index` or `--find-links` repo.

Either way, I'll add this feature, but if it doesn't work for your case, I'll also add the `<PEX_ROOT>/unzipped_pexes/<PEX hash>/<PEX file basename>` feature you suggested.
Thanks @jsirois, this is great. I think using `setproctitle` should work for us. Thanks so much for looking into this; this is arguably a convenience only thing -- it's still possible to look into the unpacked directory entry point. So please don't add any of this if this will add unnecessary complexity, it's not worth it!
Ok - execllent. The setproctitle path is actually simpler and with no impact to any existing use-cases since it's best-effort. IFF `setproctitle` is part of your PEX requirements (directly or indirectly), is this convenience activated. The hardest bit is the test!

Resolving patch:
diff --git a/pex/pex.py b/pex/pex.py
--- a/pex/pex.py
+++ b/pex/pex.py
@@ -473,6 +473,26 @@ def execute(self):
                 exit_value = tools.main(pex=PEX(sys.argv[0]))
             else:
                 self.activate()
+
+                pex_file = os.environ.get("PEX", None)
+                if pex_file:
+                    try:
+                        from setproctitle import setproctitle  # type: ignore[import]
+
+                        setproctitle(
+                            "{python} {pex_file} {args}".format(
+                                python=sys.executable,
+                                pex_file=pex_file,
+                                args=" ".join(sys.argv[1:]),
+                            )
+                        )
+                    except ImportError:
+                        TRACER.log(
+                            "Not setting process title since setproctitle is not available in "
+                            "{pex_file}".format(pex_file=pex_file),
+                            V=3,
+                        )
+
                 exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
             sys.exit(exit_value)
         except Exception:
diff --git a/pex/tools/commands/venv.py b/pex/tools/commands/venv.py
--- a/pex/tools/commands/venv.py
+++ b/pex/tools/commands/venv.py
@@ -282,6 +282,17 @@ def iter_valid_venv_pythons():
                 os.environ[current_interpreter_blessed_env_var] = "1"
                 os.execv(python, [python, "-sE"] + sys.argv)
 
+            pex_file = os.environ.get("PEX", None)
+            if pex_file:
+                try:
+                    from setproctitle import setproctitle
+
+                    setproctitle("{{python}} {{pex_file}} {{args}}".format(
+                        python=sys.executable, pex_file=pex_file, args=" ".join(sys.argv[1:]))
+                    )
+                except ImportError:
+                    pass
+
             ignored_pex_env_vars = [
                 "{{}}={{}}".format(name, value)
                 for name, value in os.environ.items()


Test patch:
diff --git a/tests/integration/test_setproctitle.py b/tests/integration/test_setproctitle.py
new file mode 100644
--- /dev/null
+++ b/tests/integration/test_setproctitle.py
@@ -0,0 +1,136 @@
+# Copyright 2022 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+import os.path
+import subprocess
+import sys
+from textwrap import dedent
+from typing import Text
+
+import pytest
+
+from pex import variables
+from pex.common import safe_open
+from pex.interpreter import PythonInterpreter
+from pex.layout import Layout
+from pex.pex_info import PexInfo
+from pex.testing import run_pex_command
+from pex.typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from typing import Any, Tuple
+
+
+@pytest.mark.parametrize("venv", [pytest.param(True, id="VENV"), pytest.param(False, id="UNZIP")])
+@pytest.mark.parametrize(
+    "layout", [pytest.param(layout, id=layout.value) for layout in Layout.values()]
+)
+def test_setproctitle(
+    tmpdir,  # type: Any
+    venv,  # type: bool
+    layout,  # type: Layout.Value
+):
+    # type: (...) -> None
+
+    pid_file = os.path.join(str(tmpdir), "pid")
+    os.mkfifo(pid_file)
+
+    src = os.path.join(str(tmpdir), "src")
+    with safe_open(os.path.join(src, "app.py"), "w") as fp:
+        fp.write(
+            dedent(
+                """\
+                from __future__ import print_function
+
+                import os
+                import threading
+
+                # Indicate PEX boot is complete and we've reached user code.
+                with open({pid_file!r}, "w") as fp:
+                    print(str(os.getpid()), file=fp)
+
+                # Run forever to ensure there is time to read out our `ps` info.
+                cv = threading.Condition()
+                with cv:
+                    cv.wait()
+                """.format(
+                    pid_file=pid_file
+                )
+            )
+        )
+
+    build_pex_args = ["-D", src, "-m", "app", "--layout", layout.value]
+    if venv:
+        build_pex_args.append("--venv")
+
+    def grab_ps(
+        pex,  # type: str
+        *extra_pex_args  # type: str
+    ):
+        # type: (...) -> Tuple[Text, Text]
+        run_pex_command(args=build_pex_args + ["-o", pex] + list(extra_pex_args)).assert_success()
+
+        process = subprocess.Popen(args=[sys.executable, pex, "--some", "arguments", "here"])
+        try:
+            # N.B.: We need to block on receiving the pid via fifo to prove that the PEX runtime has
+            # finished booting and completed any and all re-execs and landed in user code.
+            with open(pid_file) as fp:
+                assert process.pid == int(fp.read().strip())
+
+            exe, args = (
+                subprocess.check_output(
+                    args=["ps", "-p", str(process.pid), "-o", "command=", "-ww"]
+                )
+                .decode("utf-8")
+                .strip()
+                .split(" ", 1)
+            )
+            return exe, args
+        finally:
+            process.kill()
+
+    def assert_expected_python(exe):
+        # type: (Text) -> None
+        assert (
+            PythonInterpreter.get().resolve_base_interpreter()
+            == PythonInterpreter.from_binary(str(exe)).resolve_base_interpreter()
+        )
+
+    pex_file = os.path.join(str(tmpdir), "pex.file")
+    exe, args = grab_ps(pex_file)
+    assert_expected_python(exe)
+
+    pex_info = PexInfo.from_pex(pex_file)
+    assert pex_info.pex_hash is not None
+
+    if Layout.LOOSE == layout and not venv:
+        # N.B.: A non-venv loose PEX runs from where it is and presents a nice ps without help.
+        assert "{pex_file} --some arguments here".format(pex_file=pex_file) == args
+    elif venv:
+        # A `--venv` mode PEX boot terminates in a final process of:
+        # ~/.pex/venvs/<venv short dir>/bin/python -sE ~/.pex/venvs/<venv long dir>/pex <args...>
+        python_args, installed_location, rest = args.split(" ", 2)
+        assert "-sE" == python_args
+        assert (
+            os.path.join(
+                variables.venv_dir(
+                    pex_file,
+                    pex_info.pex_root,
+                    pex_info.pex_hash,
+                    has_interpreter_constraints=False,
+                ),
+                "pex",
+            )
+            == installed_location
+        )
+        assert "--some arguments here" == rest
+    else:
+        # All other PEX boots terminate in an unzipped execution that looks like:
+        # <python> ~/.pex/unzipped_pexes/<unzipped pex dir> <args...>
+        installed_location, rest = args.split(" ", 1)
+        assert variables.unzip_dir(pex_info.pex_root, pex_info.pex_hash) == installed_location
+        assert "--some arguments here" == rest
+
+    setproctitle_pex_file = os.path.join(str(tmpdir), "pex.file.titled")
+    exe, args = grab_ps(setproctitle_pex_file, "setproctitle")
+    assert_expected_python(exe)
+    assert "{pex_file} --some arguments here".format(pex_file=setproctitle_pex_file) == args