Repository: pex-tool/pex

Issue:
"Resource deadlock avoided" when creating multiple `--venv` PEXs in parallel
Testing out some change in Pants, I create some (>10 < 60) number of processes responsible for creating a PEX venv (`--venv`) using a PEX lockfile (`--lock`) specifying some requirement strings which among the processes are likely highly overlapping yet unique subset of my lockfile.

The stacktrace is at the end of this issue. The issue is regularly reproducible in our work repo, but not reproducible in Pants' own repo with the relevant Pants code running (Pants is only relevant because that's how I'm opening the N processes).

Let me know what else you need to help debug and solve.

Stacktrace:
```
Traceback (most recent call last):                                                                                                                                            File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 504, in execute          
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)                                                                                                     File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 409, in _wrap_coverage   
    return runner(*args)                                                                                                                                                      File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 440, in _wrap_profiling  
    return runner(*args)                                                                                                                                                      File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 560, in _execute         
    return self.execute_entry(self._pex_info.entry_point)                                                                                                                   
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 696, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 728, in execute_pkg_resources
    return runner()
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/bin/pex.py", line 724, in main
    do_main(
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/bin/pex.py", line 745, in do_main
    pex_builder = build_pex(
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/bin/pex.py", line 594, in build_pex
    resolve_from_lock(
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 198, in resolve_from_lock
    pool.map(
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/result.py", line 101, in catch
    return func(*args, **kwargs)
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 46, in store_downloadable_artifact
    return self.store(artifact=downloadable_artifact.artifact)
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/resolve/lockfile/download_manager.py", line 40, in store
    with atomic_directory(download_dir, exclusive=True) as atomic_dir:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/common.py", line 470, in atomic_directory
    fcntl.lockf(lock_fd, fcntl.LOCK_EX)  # A blocking write lock.
OSError: [Errno 35] Resource deadlock avoided
```


Discussion:
The atomic directory code has been in its current form since #1126 - ~16 months of use in that form. Never a report of OSError 35 EDEADLCK since inception; that suggests looking at what's new. That is using threading in Pex. The `--lock` resolver uses a thread pool as a poor-man's way to get parallel downloads that works in Python 2.7 too and doesn't require vendoring more libraries. So; I'm surely POSIX dumb here and if I weren't there would be a well known gotcha combining threads and file locks of this style. I'm guessing the quick fix is to switch to a multipprocess pool instead of a thread pool. The overhead of that switch is likely in the noise. That though punts the why. It would be good to understand why and exactly the mechanism here.
A quick-and-dirty muck of Pants' downloaded PEX to switch from `multiprocessing.pool.ThreadPool` to `multiprocessing.pool.Pool` produces:
```
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 504, in execute          
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)                                                                                                   
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 409, in _wrap_coverage   
    return runner(*args)                                                                                                                                                    
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 440, in _wrap_profiling  
    return runner(*args)                                                                                                                                                    
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 560, in _execute         
    return self.execute_entry(self._pex_info.entry_point)                                                                                                                   
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 696, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 728, in execute_pkg_resources
    return runner()
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/bin/pex.py", line 724, in main
    do_main(
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/bin/pex.py", line 745, in do_main
    pex_builder = build_pex(
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/bin/pex.py", line 594, in build_pex
    resolve_from_lock(
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 198, in resolve_from_lock
    pool.map(
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 537, in _handle_tasks
    put(task)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle 'SSLContext' object
```
Yeah - you'll need to dig.
Another quick-and-dirty muck to make the arguments pickleable and looks like the switch works. I'll wipe my cache and try it a few more times to be ultra-sure.
Some light internet reading shows this might happen if you `fork()` and spawn threads together. 

Here's a rough sketch

1. spawn one or more threads in your program
2. have at least one of those threads make try and acquire a file lock
3. have your main thread or another thread use os.fork() to run some 
python code in a child process.
4. If the fork happened while one of your threads was within the 
behind-the-lock critical section (or anywhere else where 
the lock is acquired), your child process will deadlock as soon as 
it tries to log anything.  It inherited a held lock.

so @jsirois if I try really hard to think about this (not easy for me) I'm wondering if something is spawning a process inside the download threads. What do you think? Might be this? Something else perhaps?

Ahh I suspect the `fork` here is happening via Pants, not PEX. So it's extremely plausible this is what is happening.

Resolving patch:
diff --git a/pex/common.py b/pex/common.py
--- a/pex/common.py
+++ b/pex/common.py
@@ -464,10 +464,23 @@ def unlock():
             os.path.join(head, ".{}.atomic_directory.lck".format(tail or "here")),
             os.O_CREAT | os.O_WRONLY,
         )
-        # N.B.: Since lockf operates on an open file descriptor and these are guaranteed to be
-        # closed by the operating system when the owning process exits, this lock is immune to
-        # staleness.
-        fcntl.lockf(lock_fd, fcntl.LOCK_EX)  # A blocking write lock.
+        while True:
+            try:
+                # N.B.: Since lockf operates on an open file descriptor and these are guaranteed to be
+                # closed by the operating system when the owning process exits, this lock is immune to
+                # staleness.
+                fcntl.lockf(lock_fd, fcntl.LOCK_EX)  # A blocking write lock.
+            except OSError as e:
+                deadlock_avoided_errorno = (
+                    errno.EDEADLK  # type: ignore[attr-defined] # See https://github.com/python/typeshed/issues/7551
+                )
+                if e.errno == deadlock_avoided_errorno:
+                    # Another thread/process is doing the work. We must wait.
+                    # See https://github.com/pantsbuild/pex/issues/1693
+                    time.sleep(1)
+                    continue
+                raise
+            break
         if atomic_dir.is_finalized():
             # We lost the double-checked locking race and our work was done for us by the race
             # winner so exit early.


Test patch:
diff --git a/tests/test_common.py b/tests/test_common.py
--- a/tests/test_common.py
+++ b/tests/test_common.py
@@ -3,6 +3,7 @@
 
 import contextlib
 import errno
+import fcntl
 import os
 from contextlib import contextmanager
 
@@ -123,6 +124,31 @@ def test_atomic_directory_empty_workdir_finalized():
             ), "When the target_dir exists no work_dir should be created."
 
 
+def test_atomic_directory_deadlock():
+    # type: () -> None
+
+    def mock_lockf(fd, cmd):  # type: (int, int) -> None
+        if cmd == fcntl.LOCK_EX:
+            mock_lockf.lock_call_counter += 1  # type: ignore[attr-defined]
+            if mock_lockf.lock_call_counter < 5:  # type: ignore[attr-defined]
+                raise OSError(
+                    errno.EDEADLK,  # type: ignore[attr-defined] # See https://github.com/python/typeshed/issues/7551
+                    "Resource deadlock avoided",
+                )
+
+    # N.B. Workaround Python 2.7's lack of `nonlocal` support
+    mock_lockf.lock_call_counter = 0  # type: ignore[attr-defined]
+
+    with temporary_dir() as sandbox:
+        target_dir = os.path.join(sandbox, "target_dir")
+        with mock.patch("time.sleep", lambda i: None), mock.patch.object(
+            fcntl, "lockf", mock_lockf
+        ):
+            with atomic_directory(target_dir, exclusive=True) as work_dir:
+                assert not work_dir.is_finalized()
+            assert work_dir.is_finalized()
+
+
 def extract_perms(path):
     # type: (str) -> str
     return oct(os.stat(path).st_mode)