Repository: pex-tool/pex

Issue:
pex binary hangs on startup at atomic_directory
My pex binary is hanging on startup. Using python3.7, pex 2.1.21, ubuntu 18.04. Here's the stack:

```
Traceback (most recent call first):
  <built-in method lockf of module object at remote 0x7fdf05a08f50>
  File "/var/zt/zt_consumers/current/.bootstrap/pex/common.py", line 385, in atomic_directory
  <built-in method next of module object at remote 0x7fdf06709d10>
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/var/zt/zt_consumers/current/.bootstrap/pex/util.py", line 182, in cache_distribution
  File "/var/zt/zt_consumers/current/.bootstrap/pex/environment.py", line 173, in _write_zipped_internal_cache
  File "/var/zt/zt_consumers/current/.bootstrap/pex/environment.py", line 197, in _load_internal_cache
  File "/var/zt/zt_consumers/current/.bootstrap/pex/environment.py", line 227, in _update_candidate_distributions
  File "/var/zt/zt_consumers/current/.bootstrap/pex/environment.py", line 416, in _activate
  File "/var/zt/zt_consumers/current/.bootstrap/pex/environment.py", line 260, in activate
  File "/var/zt/zt_consumers/current/.bootstrap/pex/pex.py", line 103, in _activate
  File "/var/zt/zt_consumers/current/.bootstrap/pex/pex.py", line 444, in execute
  File "/var/zt/zt_consumers/current/.bootstrap/pex/pex_bootstrapper.py", line 360, in bootstrap_pex
  File "/var/zt/zt_consumers/current/__main__.py", line 68, in <module>
  <built-in method exec of module object at remote 0x7fdf06709d10>
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
```

It's a bit tricky to reproduce because it's timing sensitive. The basic steps should be: build a pex binary containing some wheels, launch multiple copies of the binary in parallel. All processes must use the same pex_root.

The hang happens in pex/common.py atomic_directory. What I think happens is that the first process takes a file lock for the atomic dir, finalizes it and releases the lock. If the timing is right, then another process reaches https://github.com/pantsbuild/pex/blob/v2.1.21/pex/common.py#L390 so it also grabs the lock but never releases it. 

The issue was introduced in https://github.com/pantsbuild/pex/pull/1062


Discussion:
Thanks @mbakhoff. That's a facepalm bug. Thanks for identifying, I'll get out a fix for this tomorrow.

Resolving patch:
diff --git a/pex/common.py b/pex/common.py
--- a/pex/common.py
+++ b/pex/common.py
@@ -371,6 +371,16 @@ def atomic_directory(target_dir, exclusive, source=None):
         return
 
     lock_fd = None  # type: Optional[int]
+
+    def unlock():
+        # type: () -> None
+        if lock_fd is None:
+            return
+        try:
+            fcntl.lockf(lock_fd, fcntl.LOCK_UN)
+        finally:
+            os.close(lock_fd)
+
     if exclusive:
         head, tail = os.path.split(atomic_dir.target_dir)
         if head:
@@ -388,17 +398,18 @@ def atomic_directory(target_dir, exclusive, source=None):
         if atomic_dir.is_finalized:
             # We lost the double-checked locking race and our work was done for us by the race
             # winner so exit early.
-            yield None
+            try:
+                yield None
+            finally:
+                unlock()
             return
 
-    safe_mkdir(atomic_dir.work_dir)
     try:
+        safe_mkdir(atomic_dir.work_dir)
         yield atomic_dir.work_dir
         atomic_dir.finalize(source=source)
     finally:
-        if lock_fd is not None:
-            fcntl.lockf(lock_fd, fcntl.LOCK_UN)
-            os.close(lock_fd)
+        unlock()
         atomic_dir.cleanup()
 
 


Test patch: