Repository: pex-tool/pex

Issue:
`venv create` no longer includes `--sources-directory` contents when all files are nested
It seems like there was a regression from 2.1.148 -> 2.1.149 with the behaviour of `venv create` with a `--pex-repository` that was created with `--sources-directory`: those sources aren't included in the final venv.

Reproducer:

```shell
cd $(mktemp -d)

# create our dummy file
mkdir -p source_files/foo
touch source_files/foo/bar.py # NB.1
# touch source_files/qux.py # NB.2

for version in v2.1.148 v2.1.149; do
    curl -s -L https://github.com/pantsbuild/pex/releases/download/$version/pex > pex-$version
    chmod +x pex-$version

    # NB.3
    ./pex-$version --output-file=repository-$version.pex --sources-directory=source_files

    # NB.4
    PEX_SCRIPT=pex3 ./pex-$version venv create --dest-dir=dest-$version --pex-repository=repository-$version.pex --layout=flat

    # what was included?
    tree dest-$version
done
```

Running that shows that the contents of the `dest-...` directory depends on the version, without the `bar.py` file when using v2.1.149, but should be the same:

```
dest-v2.1.148
└── foo
    └── bar.py

1 directory, 1 file
dest-v2.1.149

0 directories, 0 files
```

Ablative studies:

- uncommenting `NB.2` line (to have two files) passes ✅ (both versions have both `foo/bar.py` and `qux.py`)
- _replacing_ the `NB.1` with `NB.2` (to just `qux.py` at the top level) passes ✅ 
- _always_ using v2.1.148 on line `NB.3` (create the pex) and v2.1.149 on line `NB.4` (create the venv) passes ✅ 
- v2.1.149 for `NB.3` and v2.1.148 for `NB.4` fails ❌ 
- I think third-party dependencies work okay, but haven't confirmed in this reduced setting
- This reproduces without `--layout`, but the output is simpler with `--layout=flat`

(First observed in https://github.com/pantsbuild/pants/pull/20149.)


Discussion:
Bisecting narrows it down to f9a9d949d95efd0153452dc255625656d628ee17

```shell
mkdir -p source_files/foo
touch source_files/foo/bar.py

python -m pex --output-file=repository.pex --sources-directory=source_files
python -m pex.cli venv create --dest-dir=dest --pex-repository=repository.pex --layout=flat
tree dest
```

Resolving patch:
diff --git a/pex/util.py b/pex/util.py
--- a/pex/util.py
+++ b/pex/util.py
@@ -87,7 +87,7 @@ def pex_code_hash(cls, directory):
         hashing.dir_hash(
             directory=directory,
             digest=digest,
-            dir_filter=is_pyc_dir,
+            dir_filter=lambda d: not is_pyc_dir(d),
             file_filter=lambda file_path: not is_pyc_file(file_path)
             and not file_path.startswith("."),
         )


Test patch:
diff --git a/tests/test_util.py b/tests/test_util.py
--- a/tests/test_util.py
+++ b/tests/test_util.py
@@ -6,12 +6,17 @@
 from hashlib import sha1
 from textwrap import dedent
 
+import pytest
+
 from pex.common import safe_mkdir, safe_open, temporary_dir, touch
 from pex.pex import PEX
 from pex.pex_builder import PEXBuilder
-from pex.typing import cast
+from pex.typing import TYPE_CHECKING, cast
 from pex.util import CacheHelper, DistributionHelper, named_temporary_file
 
+if TYPE_CHECKING:
+    from typing import Callable
+
 try:
     from unittest import mock
 except ImportError:
@@ -57,39 +62,49 @@ def test_hash():
         assert hash_output == empty_hash.hexdigest()
 
 
-def test_dir_hash():
-    # type: () -> None
+@pytest.mark.parametrize(
+    ("hasher", "includes_hidden_expected"),
+    [(CacheHelper.dir_hash, True), (CacheHelper.pex_code_hash, False)],
+)
+def test_directory_hasher(hasher, includes_hidden_expected):
+    # type: (Callable[[str], str], bool) -> None
     with temporary_dir() as tmp_dir:
         safe_mkdir(os.path.join(tmp_dir, "a", "b"))
         with safe_open(os.path.join(tmp_dir, "c", "d", "e.py"), "w") as fp:
             fp.write("contents1")
         with safe_open(os.path.join(tmp_dir, "f.py"), "w") as fp:
             fp.write("contents2")
-        hash1 = CacheHelper.dir_hash(tmp_dir)
+        hash1 = hasher(tmp_dir)
 
         os.rename(os.path.join(tmp_dir, "c"), os.path.join(tmp_dir, "c-renamed"))
-        assert hash1 != CacheHelper.dir_hash(tmp_dir)
+        assert hash1 != hasher(tmp_dir)
 
         os.rename(os.path.join(tmp_dir, "c-renamed"), os.path.join(tmp_dir, "c"))
-        assert hash1 == CacheHelper.dir_hash(tmp_dir)
+        assert hash1 == hasher(tmp_dir)
 
         touch(os.path.join(tmp_dir, "c", "d", "e.pyc"))
-        assert hash1 == CacheHelper.dir_hash(tmp_dir)
+        assert hash1 == hasher(tmp_dir)
         touch(os.path.join(tmp_dir, "c", "d", "e.pyc.123456789"))
-        assert hash1 == CacheHelper.dir_hash(tmp_dir)
+        assert hash1 == hasher(tmp_dir)
 
         pycache_dir = os.path.join(tmp_dir, "__pycache__")
         safe_mkdir(pycache_dir)
         touch(os.path.join(pycache_dir, "f.pyc"))
-        assert hash1 == CacheHelper.dir_hash(tmp_dir)
+        assert hash1 == hasher(tmp_dir)
         touch(os.path.join(pycache_dir, "f.pyc.123456789"))
-        assert hash1 == CacheHelper.dir_hash(tmp_dir)
+        assert hash1 == hasher(tmp_dir)
 
         touch(os.path.join(pycache_dir, "f.py"))
-        assert hash1 == CacheHelper.dir_hash(
+        assert hash1 == hasher(
             tmp_dir
         ), "All content under __pycache__ directories should be ignored."
 
+        with safe_open(os.path.join(tmp_dir, ".hidden"), "w") as fp:
+            fp.write("contents3")
+
+        includes_hidden = hash1 != hasher(tmp_dir)
+        assert includes_hidden == includes_hidden_expected
+
 
 try:
     import __builtin__ as python_builtins  # type: ignore[import]