Repository: pex-tool/pex

Issue:
Failure to extract dependencies to ~/.pex/install if they already exist with some Python version
Specifically on a host running Python 2.7.5 we ran into the following stack trace:

```
Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 309, in execute
  File ".bootstrap/_pex/pex.py", line 78, in _activate
  File ".bootstrap/_pex/environment.py", line 132, in activate
  File ".bootstrap/_pex/environment.py", line 176, in _activate
  File ".bootstrap/_pex/environment.py", line 121, in update_candidate_distributions
  File ".bootstrap/_pex/environment.py", line 107, in load_internal_cache
  File ".bootstrap/_pex/environment.py", line 95, in write_zipped_internal_cache
  File ".bootstrap/_pex/util.py", line 177, in cache_distribution
OSError: [Errno 17] File exists
```

In this case we're on a slightly older version of pex, line 177 in util.py is: https://github.com/pantsbuild/pex/blob/ba58a854d561dcf6963c0097cad544bfe2861c5a/pex/util.py#L177

So, the rename fails because the directory exists. In this case it's raising EEXIST, whereas pex only expects ENOTEMPTY to be a continuable error code.

Pex should handle EEXIST with the same behavior as ENOTEMPTY in this case.



Discussion:
tl;dr if you're catching the exception, use the broader `OSError` and discover _why_ it happened yourself.  Error code returned in `e.errno` is filesystem-dependent and unreliable.

Python invokes system calls, in the case of `os.rename()`, that's `rename(2)` (see `man 2 rename`). This will trigger the behavior that `os.rename()` does:

```
mkdir from to
touch from/contents to/contents
strace mv -T from to
```

`OSError` will reflect in its `errno` attribute whatever the operating system (kernel) reports back when the C library invokes `rename(2)`, but this is filesystem-dependent. In the case of NFS on CentOS 5, or XFS on CentOS 7 running a stock kernel, it can be `EEXIST`, which is error 17.

```
lstat("from", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("to", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
rename("from", "to")                    = -1 EEXIST (File exists)
```

`ext3` on CentOS 5 running kernel 3.10, may raise `ENOTEMPTY` (error 39), like `util.py` catches: https://github.com/pantsbuild/pex/blob/ba58a854d561dcf6963c0097cad544bfe2861c5a/pex/util.py#L177

```
lstat("from", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("to", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
rename("from", "to")                    = -1 ENOTEMPTY (Directory not empty)
```

Anyway, you'll want a more robust approach that is kernel and filesystem agnostic.

@jcohen it'd be great to get your repro details for this. afaict, this is actually a race condition vs a simple "run pex twice on centos7" bug:

```
$ cat /etc/redhat-release
Nest Enterprise Linux release 7.2.1511 (Core)
$ rm -rf ~/.pex
$ ./pants.pex
Could not find pants.ini!
$ ./pants.pex
Could not find pants.ini!
$ ./pants.pex
Could not find pants.ini!
$ ./pants.pex
Could not find pants.ini!
```

there is logic in: https://github.com/pantsbuild/pex/blob/ba58a854d561dcf6963c0097cad544bfe2861c5a/pex/util.py#L167

which checks for existence of `target_dir` before attempting the rename. so my guess would be that another pex could be simultaneously writing this dir between that check and the problematic os.rename, causing the issue you're seeing.

hence, would be great to confirm this theory with a repro.


Resolving patch:
diff --git a/pex/common.py b/pex/common.py
--- a/pex/common.py
+++ b/pex/common.py
@@ -140,6 +140,21 @@ def safe_rmtree(directory):
     shutil.rmtree(directory, True)
 
 
+def rename_if_empty(src, dest, allowable_errors=(errno.EEXIST, errno.ENOTEMPTY)):
+  """Rename `src` to `dest` using `os.rename()`.
+
+  If an `OSError` with errno in `allowable_errors` is encountered during the rename, the `dest`
+  dir is left unchanged and the `src` directory will simply be removed.
+  """
+  try:
+    os.rename(src, dest)
+  except OSError as e:
+    if e.errno in allowable_errors:
+      safe_rmtree(src)
+    else:
+      raise
+
+
 def chmod_plus_x(path):
   """Equivalent of unix `chmod a+x path`"""
   path_mode = os.stat(path).st_mode
diff --git a/pex/util.py b/pex/util.py
--- a/pex/util.py
+++ b/pex/util.py
@@ -4,7 +4,6 @@
 from __future__ import absolute_import
 
 import contextlib
-import errno
 import os
 import shutil
 import tempfile
@@ -14,7 +13,7 @@
 
 from pkg_resources import find_distributions, resource_isdir, resource_listdir, resource_string
 
-from .common import safe_mkdir, safe_mkdtemp, safe_open, safe_rmtree
+from .common import rename_if_empty, safe_mkdir, safe_mkdtemp, safe_open
 from .finders import register_finders
 
 
@@ -176,13 +175,8 @@ def cache_distribution(cls, zf, source, target_dir):
           with contextlib.closing(zf.open(name)) as zi:
             with safe_open(os.path.join(target_dir_tmp, target_name), 'wb') as fp:
               shutil.copyfileobj(zi, fp)
-      try:
-        os.rename(target_dir_tmp, target_dir)
-      except OSError as e:
-        if e.errno == errno.ENOTEMPTY:
-          safe_rmtree(target_dir_tmp)
-        else:
-          raise
+
+      rename_if_empty(target_dir_tmp, target_dir)
 
     dist = DistributionHelper.distribution_from_path(target_dir)
     assert dist is not None, 'Failed to cache distribution %s' % source


Test patch:
diff --git a/tests/test_common.py b/tests/test_common.py
new file mode 100644
--- /dev/null
+++ b/tests/test_common.py
@@ -0,0 +1,44 @@
+# Copyright 2016 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+import errno
+import os
+from contextlib import contextmanager
+
+import pytest
+
+from pex.common import rename_if_empty
+
+try:
+  from unittest import mock
+except ImportError:
+  import mock
+
+
+@contextmanager
+def maybe_raises(exception=None):
+  @contextmanager
+  def noop():
+    yield
+
+  with (noop() if exception is None else pytest.raises(exception)):
+    yield
+
+
+def rename_if_empty_test(errno, expect_raises=None):
+  with mock.patch('os.rename', spec_set=True, autospec=True) as mock_rename:
+    mock_rename.side_effect = OSError(errno, os.strerror(errno))
+    with maybe_raises(expect_raises):
+      rename_if_empty('from.dir', 'to.dir')
+
+
+def test_rename_if_empty_eexist():
+  rename_if_empty_test(errno.EEXIST)
+
+
+def test_rename_if_empty_enotempty():
+  rename_if_empty_test(errno.ENOTEMPTY)
+
+
+def test_rename_if_empty_eperm():
+  rename_if_empty_test(errno.EPERM, expect_raises=OSError)