Repository: pypa/pip

Issue:
rmtree() failing sporadically on Windows
* Pip version: 9.0.1 as well as current development version (ce674d2ca1dfae136fb33df914085e56d100bc57), but likely older versions as well
* Python version: irrelevant, tested with python 2.7.13 & 3.6.1
* Operating system: Windows 10

### Description

On Windows I've often encountered 'false failure reports' when removing a folder. Even when running for instance `rd /s /q <folder>` on the command-line. It reports an error like the folder not being empty or an access violation error, but actually deletes the folder or at least parts of it, and just repeating the command a few times works correctly.

The underlying cause for the OS operation failures is likely either some OS or external application folder scanning logic keeping live handles referring to the files or folders being removed, and race conditions between those file handles' life times and the file/folder removal process. I've seen the issue on Windows 10 machines with no anti-virus software installed other than what the OS does internally.

This also affects pip installations where it fails to remove some of its internal folders for the same reason. In those cases it typically retries removing the folder several times but then it often happens that one of those removals fails hard because the folder was actually already removed after the previously reported failure. The most often encountered use-case has been a failure in upgrading pip itself as its installation has quite a deep folder structure, and then the final effect of the error is that the target Python environment is left without any valid pip installation and you need to restore it to some version (e.g. using `python -m ensurepip`) in order to try the upgrade again.

### What has been found thus far

All tracebacks I encontered for such cases show failures on various operations in the `rmtree_errorhandler()` error handler for the `rmtree()` utility function, e.g. on the `os.stat()` or `os.chmod()` calls, or on one of the operations coming from inside the standard library `shutil.rmtree()` operation.

### Suggested solution

And what can fix the problem is making the `rmtree_errorhandler()` error handler consider the file being already removed as a successful error resolution instead of reporting that as an error.

The only externally visible change then is that calling `rmtree()` on a non-existing folder will now be treated as success. but that does not seem like a bad change to begin with as the whole _does the thing I'm trying to remove exist_ issue is naturally riddled with race conditions and can not be answered perfectly anyway.


Discussion:
This should be straightforward. The specific change would be in `src.pip._internal.utils.misc` around [here](https://github.com/pypa/pip/blob/a9d23fadbb678338b688dac18fff4f8a3f67b969/src/pip/_internal/utils/misc.py#L180-L192). If on error the provided path does not exist then we should just return and not re-raise the exception. The check should be before `os.stat` in that function.

For tests we'd need at least one showing the new behavior - that deleting a non-existent directory doesn't fail. Another test could be of `rmtree_errorhandler` directly and should check that it doesn't raise any exception on being passed a non-existent path.

Resolving patch:
diff --git a/src/pip/_internal/utils/misc.py b/src/pip/_internal/utils/misc.py
--- a/src/pip/_internal/utils/misc.py
+++ b/src/pip/_internal/utils/misc.py
@@ -181,8 +181,13 @@ def rmtree_errorhandler(func, path, exc_info):
     """On Windows, the files in .svn are read-only, so when rmtree() tries to
     remove them, an exception is thrown.  We catch that here, remove the
     read-only attribute, and hopefully continue without problems."""
-    # if file type currently read only
-    if os.stat(path).st_mode & stat.S_IREAD:
+    try:
+        has_attr_readonly = not (os.stat(path).st_mode & stat.S_IWRITE)
+    except (IOError, OSError):
+        # it's equivalent to os.path.exists
+        return
+
+    if has_attr_readonly:
         # convert to read/write
         os.chmod(path, stat.S_IWRITE)
         # use the original function to repeat the operation


Test patch:
diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py
--- a/tests/unit/test_utils.py
+++ b/tests/unit/test_utils.py
@@ -59,6 +59,7 @@
     redact_netloc,
     remove_auth_from_url,
     rmtree,
+    rmtree_errorhandler,
     split_auth_from_netloc,
     split_auth_netloc_from_url,
     untar_file,
@@ -386,6 +387,62 @@ def test_unpack_zip(self, data):
         self.confirm_files()
 
 
+def test_rmtree_errorhandler_nonexistent_directory(tmpdir):
+    """
+    Test rmtree_errorhandler ignores the given non-existing directory.
+    """
+    nonexistent_path = str(tmpdir / 'foo')
+    mock_func = Mock()
+    rmtree_errorhandler(mock_func, nonexistent_path, None)
+    mock_func.assert_not_called()
+
+
+def test_rmtree_errorhandler_readonly_directory(tmpdir):
+    """
+    Test rmtree_errorhandler makes the given read-only directory writable.
+    """
+    # Create read only directory
+    path = str((tmpdir / 'subdir').mkdir())
+    os.chmod(path, stat.S_IREAD)
+
+    # Make sure mock_func is called with the given path
+    mock_func = Mock()
+    rmtree_errorhandler(mock_func, path, None)
+    mock_func.assert_called_with(path)
+
+    # Make sure the path is now writable
+    assert os.stat(path).st_mode & stat.S_IWRITE
+
+
+def test_rmtree_errorhandler_reraises_error(tmpdir):
+    """
+    Test rmtree_errorhandler reraises an exception
+    by the given unreadable directory.
+    """
+    # Create directory without read permission
+    path = str((tmpdir / 'subdir').mkdir())
+    os.chmod(path, stat.S_IWRITE)
+
+    mock_func = Mock()
+
+    try:
+        raise RuntimeError('test message')
+    except RuntimeError:
+        # Make sure the handler reraises an exception
+        with pytest.raises(RuntimeError, match='test message'):
+            rmtree_errorhandler(mock_func, path, None)
+
+    mock_func.assert_not_called()
+
+
+def test_rmtree_skips_nonexistent_directory():
+    """
+    Test wrapped rmtree doesn't raise an error
+    by the given nonexistent directory.
+    """
+    rmtree.__wrapped__('nonexistent-subdir')
+
+
 class Failer:
     def __init__(self, duration=1):
         self.succeed_after = time.time() + duration