Repository: pex-tool/pex

Issue:
pex fails to build pycryptodome due to filename too long
I tried building a pex file for a project that has pycryptodome in its dependency tree and pex fails to include that wheel:

```
$ pex pycryptodome==3.16.0 -o pycryptodome.pex
pid 32518 -> /home/konsti/.pex/venvs/6a452ba209173892c06990a2eddf5e60e775054e/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /home/konsti/.pex/venvs/6a452ba209173892c06990a2eddf5e60e775054e/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/konsti/.pex/pip_cache install --no-deps --no-index --only-binary :all: --no-warn-script-location --force-reinstall --ignore-installed --ignore-requires-python --prefix /home/konsti/.pex/installed_wheels/e750a21d8a265b1f9bfb1a28822995ea33511ba7db5e2b55f41fb30781d0d073/pycryptodome-3.16.0-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.a4d26b5a48ca4a8c92e788d1879f39e6.work --no-compile /home/konsti/.pex/downloads/resolver_download.e44__tz5/home.konsti.meine-stadt-transparent..venv.bin.python/pycryptodome-3.16.0-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl exited with 1 and STDERR:
ERROR: Could not install packages due to an EnvironmentError: [Errno 36] File name too long: '/home/konsti/.pex/installed_wheels/e750a21d8a265b1f9bfb1a28822995ea33511ba7db5e2b55f41fb30781d0d073/pycryptodome-3.16.0-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.a4d26b5a48ca4a8c92e788d1879f39e6.work'
```

pex 2.1.127
Ubuntu 20.04
Python 3.8.10


Discussion:
Hrm:
```
echo /home/konsti/.pex/installed_wheels/e750a21d8a265b1f9bfb1a28822995ea33511ba7db5e2b55f41fb30781d0d073/pycryptodome-3.16.0-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.a4d26b5a48ca4a8c92e788d1879f39e6.work | wc -c
254
```

I have no issues running `pex pycryptodome==3.16.0 -o pycryptodome.pex` but I have:
```
$ grep -E " (NAME_MAX|PATH_MAX) " /usr/include/linux/limits.h
#define NAME_MAX         255    /* # chars in a file name */
#define PATH_MAX        4096    /* # chars in a path name including nul */
```

I'm not sure what to do about this. At some point path names will simply be too long for some systems.

@konstin what does you kernel show for those same limits?
yeah they do:

```
$ grep -E " (NAME_MAX|PATH_MAX) " /usr/include/linux/limits.h
#define NAME_MAX         255	/* # chars in a file name */
#define PATH_MAX        4096	/* # chars in a path name including nul */
```

i'll try to make reproducer in docker later
Docker may not help much since containers run using the host kernel. You could hand me a container and if my host kernel is configured differently, I'll get a different result.
@konstin is your file system encrypted with ecryptfs? I had the same issue but stoped after switching to full-disk encryption (luks) on mine.
yes it is, never considered this to make a difference
Aha, big difference. See, for example: https://bugs.launchpad.net/ecryptfs/+bug/344878

Thanks for teasing that out of @konstin @anler. I'm going to mark this closed. There really isn't much PEX can do about a ~144 max path length. I know I switched off ecryptfs >10 years ago after hitting path name length issues in other software.
@konstin this is a bit of a pain to test from a non-eCryptFS system, but if you have a chance to try out the forthcoming PR, I'd be grateful. I'll post instructions on the PR and ping you there.

Resolving patch:
diff --git a/pex/atomic_directory.py b/pex/atomic_directory.py
--- a/pex/atomic_directory.py
+++ b/pex/atomic_directory.py
@@ -5,6 +5,7 @@
 
 import errno
 import fcntl
+import hashlib
 import os
 import threading
 from contextlib import contextmanager
@@ -68,6 +69,32 @@ def __init__(
         # type: (...) -> None
         self._target_dir = target_dir
         self._work_dir = "{}.{}.work".format(target_dir, "lck" if locked else uuid4().hex)
+        target_basename = os.path.basename(self._work_dir)
+        if len(target_basename) > 143:
+            # Guard against eCryptFS home dir encryption which restricts file names to 143
+            # characters when the underlying file system has a max file name length of 255
+            # characters, which is the common case. We can break this limit with wheels like
+            # `pycryptodome-3.16.0-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl`.
+            # That wheel name is only 116 characters long, but can have something like
+            # `.a4d26b5a48ca4a8c92e788d1879f39e6.work` appended, which is an additional 39
+            # characters totalling 155 characters when the AtomicDirectory is unlocked. In these
+            # cases we replace the basename with its sha256 hex digest which is a fixed 64
+            # characters.
+            #
+            # No attempt is made to probe for an eCryptFS partition or even guess if the OS supports
+            # this sort of thing since not many wheel names are this long and just always performing
+            # this compaction is more robust.
+            #
+            # See: https://bugs.launchpad.net/ecryptfs/+bug/344878
+            # See: https://github.com/pantsbuild/pex/issues/2087
+
+            fingerprint = hashlib.sha256(target_basename.encode("utf-8")).hexdigest()
+            self._work_dir = os.path.join(
+                os.path.dirname(self._target_dir),
+                "{prefix}...{fingerprint}".format(
+                    prefix=target_basename[: 143 - 3 - len(fingerprint)], fingerprint=fingerprint
+                ),
+            )
 
     @property
     def work_dir(self):


Test patch:
diff --git a/tests/integration/test_issue_2087.py b/tests/integration/test_issue_2087.py
new file mode 100644
--- /dev/null
+++ b/tests/integration/test_issue_2087.py
@@ -0,0 +1,50 @@
+# Copyright 2023 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+import json
+import os.path
+from textwrap import dedent
+
+from pex.compatibility import commonpath
+from pex.typing import TYPE_CHECKING
+from testing import run_pex_command
+
+if TYPE_CHECKING:
+    from typing import Any
+
+
+def test_long_wheel_names(tmpdir):
+    # type: (Any) -> None
+
+    # N.B.: The pycryptodome 3.16.0 release has a wheel with a 116 character name in
+    # pycryptodome-3.16.0-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
+    # that triggers special logic to deal with temporary work dir names being too long for some
+    # systems, notably those that house the PEX_ROOT on an eCryptFS volume.
+    pex_root = os.path.join(str(tmpdir), "pex_root")
+    result = run_pex_command(
+        args=[
+            "--pex-root",
+            pex_root,
+            "--runtime-pex-root",
+            pex_root,
+            "pycryptodome==3.16.0",
+            "--",
+            "-c",
+            dedent(
+                """\
+                import json
+                import sys
+
+                import Crypto
+
+
+                json.dump({"version": Crypto.__version__, "path": Crypto.__file__}, sys.stdout)
+                """
+            ),
+        ]
+    )
+    result.assert_success()
+
+    data = json.loads(result.output)
+    assert "3.16.0" == data["version"]
+    assert pex_root == commonpath((pex_root, data["path"]))
diff --git a/tests/test_atomic_directory.py b/tests/test_atomic_directory.py
--- a/tests/test_atomic_directory.py
+++ b/tests/test_atomic_directory.py
@@ -3,6 +3,7 @@
 
 import errno
 import os
+import re
 from contextlib import contextmanager
 
 import pytest
@@ -139,3 +140,35 @@ def test_atomic_directory_locked_mode():
         AtomicDirectory("locked", locked=True).work_dir
         == AtomicDirectory("locked", locked=True).work_dir
     )
+
+
+def test_long_file_name_issue_2087():
+    # type: () -> None
+
+    atomic_directory = AtomicDirectory(
+        "/tmp/pycryptodome-3.16.0-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64."
+        "manylinux_2_12_x86_64",
+        locked=False,
+    )
+    assert re.match(
+        r"pycryptodome-3\.16\.0-cp35-abi3-manylinux_2_5_x86_64\.manylinux1_x86_64\."
+        r"manylinux_2_12_x86_64\.[a-f0-9]+.work",
+        os.path.basename(atomic_directory.work_dir),
+    ), "Expected shorter directory names to use a workdir with the target dir as a prefix."
+
+    atomic_directory = AtomicDirectory(
+        "/tmp/pycryptodome-3.16.0-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64."
+        "manylinux_2_12_x86_64.manylinux2010_x86_64.whl",
+        locked=False,
+    )
+    assert "/tmp" == os.path.dirname(
+        atomic_directory.work_dir
+    ), "Expected the workdir to be co-located with the target dir to ensure atomic rename works."
+    assert 143 == len(
+        os.path.basename(atomic_directory.work_dir)
+    ), "Expected longer directory names to use a workdir that is 143 characters in length."
+    assert re.match(
+        r"^pycryptodome-3\.16\.0-cp35-abi3-manylinux_2_5_x86_64\.manylinux1_x86_64\.manylin"
+        r"\.\.\.[a-f0-9]{64}$",
+        os.path.basename(atomic_directory.work_dir),
+    ), "Expected longer directory names to retain their prefix with a `...<hash>` suffix."