Repository: pypa/pip

Issue:
pip install --upgrade stalls at the "uninstalling" phase
**Environment**

* pip version: 19.0
* Python version: 2.7.15rc1
* OS: Ubuntu 18.04.1 LTS (Kernel: 4.15.0-1031-aws)

<!-- Feel free to add more information about your environment here -->

**Description**

Running `pip install --upgrade` for any package that needs upgrading will cause pip to stall at the "Uninstalling" stage.

**Expected behavior**

Packages should be upgraded.

**How to Reproduce**

1. On a system with cryptography-2.4.2 installed...
2. Run `pip install --upgrade cryptography`
3. Pip gets as far as `Uninstalling cryptography-2.4.2` but doesn't go any further. `-vvv` does not provide any extra output.

**Output**

```
  Using version 2.5 (newest of versions: 0.1, 0.2, 0.2.1, 0.2.2, 0.3, 0.4, 0.5, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6, 0.6.1, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.9, 0.9.1, 0.9.2, 0.9.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.1.1, 1.1.2, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.4, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.6, 1.7, 1.7.1, 1.7.2, 1.8, 1.8.1, 1.8.2, 1.9, 2.0, 2.0.1, 2.0.2, 2.0.3, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2, 2.2.1, 2.2.2, 2.3, 2.3.1, 2.4, 2.4.1, 2.4.2, 2.5)
  Created temporary directory: /tmp/pip-unpack-RPH06y
  Looking up "https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl" in the cache
  Current age based on date: 5543
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 5543
  Using cached https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl
  Downloading from URL https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl#sha256=42fad67d7072216a49e34f923d8cbda9edacbf6633b19a79655e88a1b4857063 (from https://pypi.org/simple/cryptography/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*)
  Added cryptography from https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl#sha256=42fad67d7072216a49e34f923d8cbda9edacbf6633b19a79655e88a1b4857063 to build tracker '/tmp/pip-req-tracker-gljwal'
  Removed cryptography from https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl#sha256=42fad67d7072216a49e34f923d8cbda9edacbf6633b19a79655e88a1b4857063 from build tracker '/tmp/pip-req-tracker-gljwal'
Requirement already satisfied, skipping upgrade: asn1crypto>=0.21.0 in /usr/lib/python2.7/dist-packages (from cryptography) (0.24.0)
Requirement already satisfied, skipping upgrade: enum34; python_version < "3" in /usr/lib/python2.7/dist-packages (from cryptography) (1.1.6)
Requirement already satisfied, skipping upgrade: ipaddress; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from cryptography) (1.0.22)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in /usr/local/lib/python2.7/dist-packages (from cryptography) (1.11.5)
Requirement already satisfied, skipping upgrade: six>=1.4.1 in /usr/lib/python2.7/dist-packages (from cryptography) (1.11.0)
Requirement already satisfied, skipping upgrade: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi!=1.11.3,>=1.8->cryptography) (2.19)
Installing collected packages: cryptography
  Found existing installation: cryptography 2.4.2
    Uninstalling cryptography-2.4.2:
```

There is more output but it's mostly "Found link" and "Skipping link". Let me know if you need the full output.


Discussion:
@wildlyinaccurate, when you interrupt the upgrade with ctl-C, what do you see in your site directory ?
I see e.g. `/Library/Python/2.7/site-packages/-onfigparser-3.5.1.dist-info/`, `config` -> `-onfig` ?!
However `uninstall` works fine.
(pip catches / turns off the stacktrace on so I can't tell where it's looping.)
My versions: Python 2.7, pip 19.0.0, macos 10.10
We're having the same problem over at the `cibuildwheel` project when running this on Travis: https://github.com/joerick/cibuildwheel/issues/122

However, I think I've figured out what causes this:

It seems https://github.com/pypa/pip/commit/c7ae06c79827c5d4aaa4f4b4656dfc525e5e55bc added a new way of generating temporary directories to copy uninstalled packages so that they can be rolled back if the new package's installation failed.

Notice the [`_generate_names` method](https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/temp_dir.py#L108), generating potentially `len(LEADING_CHARS)**len(name)` (i.e., almost always a ridiculously enormous number) of temporary folder names. When the parent folder cannot be written to (because of permission issues, for example), the creation of these temporary folders will fail and the loop will go on checking all possibilities and never managing to find one.

See https://travis-ci.org/YannickJadoul/cibuildwheel/jobs/485146671 and https://api.travis-ci.org/v3/job/485146671/log.txt for a log of the installation and the permissions: user `travis` is a member of the `admin` group, so the group permissions are what you're looking for.

Not sure what's the best way to fix this, but I'm quite confident this is the problem.


> I see e.g. /Library/Python/2.7/site-packages/-onfigparser-3.5.1.dist-info/, config -> -onfig ?!

This new method of generating temporary folder names also explains the weird `-` names. Probably leftovers from a failed/interrupted installations/updates? Cfr. #6194, as well.
@zooba This issue seems to be related to a PR you worked on before (see the previous comment).
It's possible that my original issue was actually a permissions mismatch -- the original modules were installed as root, and running `pip install` with `sudo` proceeds as expected with no issues.
Without knowing much, it seems like the right fix might be to distinguish between permissions issues and the path already existing when handling `OSError` in the code you linked to. Errors other than the file existing should bubble up.

@wildlyinaccurate I think the confusion also stems from the fact that different permissions are involved. On the one hand, a package's folder can be unremovable (e.g., `.../my-package`), but this issue is linked to a new folder not being able to be created next to the installed package, in the parent folder (e.g., `-y-package` inside `site-packages`). That's why I never managed to reproduce the error locally.

EDIT: Seems it was too late yesterday to think clearly, and I think got the wrong folders names. `site-packages` is actually writable in my case, but its parent folder (`python3.4` in my logs) is not. So when `-ite-packages`, `~ite-packages`, `.ite-packages`, `+ite-packages`, ..., `--te-packages`, ... are tried to be created, this fails an exponential number of times.
@cjerdonek That sounds like a reasonable fix. I'll take a quick look

Resolving patch:
diff --git a/src/pip/_internal/req/req_uninstall.py b/src/pip/_internal/req/req_uninstall.py
--- a/src/pip/_internal/req/req_uninstall.py
+++ b/src/pip/_internal/req/req_uninstall.py
@@ -17,7 +17,7 @@
     FakeFile, ask, dist_in_usersite, dist_is_local, egg_link_path, is_local,
     normalize_path, renames, rmtree,
 )
-from pip._internal.utils.temp_dir import AdjacentTempDirectory
+from pip._internal.utils.temp_dir import AdjacentTempDirectory, TempDirectory
 
 logger = logging.getLogger(__name__)
 
@@ -127,7 +127,7 @@ def norm_join(*a):
         # If all the files we found are in our remaining set of files to
         # remove, then remove them from the latter set and add a wildcard
         # for the directory.
-        if len(all_files - remaining) == 0:
+        if not (all_files - remaining):
             remaining.difference_update(all_files)
             wildcards.add(root + os.sep)
 
@@ -183,6 +183,111 @@ def compress_for_output_listing(paths):
     return will_remove, will_skip
 
 
+class StashedUninstallPathSet(object):
+    """A set of file rename operations to stash files while
+    tentatively uninstalling them."""
+    def __init__(self):
+        # Mapping from source file root to [Adjacent]TempDirectory
+        # for files under that directory.
+        self._save_dirs = {}
+        # (old path, new path) tuples for each move that may need
+        # to be undone.
+        self._moves = []
+
+    def _get_directory_stash(self, path):
+        """Stashes a directory.
+
+        Directories are stashed adjacent to their original location if
+        possible, or else moved/copied into the user's temp dir."""
+
+        try:
+            save_dir = AdjacentTempDirectory(path)
+            save_dir.create()
+        except OSError:
+            save_dir = TempDirectory(kind="uninstall")
+            save_dir.create()
+        self._save_dirs[os.path.normcase(path)] = save_dir
+
+        return save_dir.path
+
+    def _get_file_stash(self, path):
+        """Stashes a file.
+
+        If no root has been provided, one will be created for the directory
+        in the user's temp directory."""
+        path = os.path.normcase(path)
+        head, old_head = os.path.dirname(path), None
+        save_dir = None
+
+        while head != old_head:
+            try:
+                save_dir = self._save_dirs[head]
+                break
+            except KeyError:
+                pass
+            head, old_head = os.path.dirname(head), head
+        else:
+            # Did not find any suitable root
+            head = os.path.dirname(path)
+            save_dir = TempDirectory(kind='uninstall')
+            save_dir.create()
+            self._save_dirs[head] = save_dir
+
+        relpath = os.path.relpath(path, head)
+        if relpath and relpath != os.path.curdir:
+            return os.path.join(save_dir.path, relpath)
+        return save_dir.path
+
+    def stash(self, path):
+        """Stashes the directory or file and returns its new location.
+        """
+        if os.path.isdir(path):
+            new_path = self._get_directory_stash(path)
+        else:
+            new_path = self._get_file_stash(path)
+
+        self._moves.append((path, new_path))
+        if os.path.isdir(path) and os.path.isdir(new_path):
+            # If we're moving a directory, we need to
+            # remove the destination first or else it will be
+            # moved to inside the existing directory.
+            # We just created new_path ourselves, so it will
+            # be removable.
+            os.rmdir(new_path)
+        renames(path, new_path)
+        return new_path
+
+    def commit(self):
+        """Commits the uninstall by removing stashed files."""
+        for _, save_dir in self._save_dirs.items():
+            save_dir.cleanup()
+        self._moves = []
+        self._save_dirs = {}
+
+    def rollback(self):
+        """Undoes the uninstall by moving stashed files back."""
+        for p in self._moves:
+            logging.info("Moving to %s\n from %s", *p)
+
+        for new_path, path in self._moves:
+            try:
+                logger.debug('Replacing %s from %s', new_path, path)
+                if os.path.isfile(new_path):
+                    os.unlink(new_path)
+                elif os.path.isdir(new_path):
+                    rmtree(new_path)
+                renames(path, new_path)
+            except OSError as ex:
+                logger.error("Failed to restore %s", new_path)
+                logger.debug("Exception: %s", ex)
+
+        self.commit()
+
+    @property
+    def can_rollback(self):
+        return bool(self._moves)
+
+
 class UninstallPathSet(object):
     """A set of file paths to be removed in the uninstallation of a
     requirement."""
@@ -191,8 +296,7 @@ def __init__(self, dist):
         self._refuse = set()
         self.pth = {}
         self.dist = dist
-        self._save_dirs = []
-        self._moved_paths = []
+        self._moved_paths = StashedUninstallPathSet()
 
     def _permitted(self, path):
         """
@@ -230,22 +334,6 @@ def add_pth(self, pth_file, entry):
         else:
             self._refuse.add(pth_file)
 
-    def _stash(self, path):
-        best = None
-        for save_dir in self._save_dirs:
-            if not path.startswith(save_dir.original + os.sep):
-                continue
-            if not best or len(save_dir.original) > len(best.original):
-                best = save_dir
-        if best is None:
-            best = AdjacentTempDirectory(os.path.dirname(path))
-            best.create()
-            self._save_dirs.append(best)
-        relpath = os.path.relpath(path, best.original)
-        if not relpath or relpath == os.path.curdir:
-            return best.path
-        return os.path.join(best.path, relpath)
-
     def remove(self, auto_confirm=False, verbose=False):
         """Remove paths in ``self.paths`` with confirmation (unless
         ``auto_confirm`` is True)."""
@@ -264,18 +352,14 @@ def remove(self, auto_confirm=False, verbose=False):
 
         with indent_log():
             if auto_confirm or self._allowed_to_proceed(verbose):
-                for path in sorted(compact(compress_for_rename(self.paths))):
-                    new_path = self._stash(path)
+                moved = self._moved_paths
+
+                for_rename = compress_for_rename(self.paths)
+
+                for path in sorted(compact(for_rename)):
+                    moved.stash(path)
                     logger.debug('Removing file or directory %s', path)
-                    self._moved_paths.append((path, new_path))
-                    if os.path.isdir(path) and os.path.isdir(new_path):
-                        # If we're moving a directory, we need to
-                        # remove the destination first or else it will be
-                        # moved to inside the existing directory.
-                        # We just created new_path ourselves, so it will
-                        # be removable.
-                        os.rmdir(new_path)
-                    renames(path, new_path)
+
                 for pth in self.pth.values():
                     pth.remove()
 
@@ -312,28 +396,20 @@ def _display(msg, paths):
 
     def rollback(self):
         """Rollback the changes previously made by remove()."""
-        if not self._save_dirs:
+        if not self._moved_paths.can_rollback:
             logger.error(
                 "Can't roll back %s; was not uninstalled",
                 self.dist.project_name,
             )
             return False
         logger.info('Rolling back uninstall of %s', self.dist.project_name)
-        for path, tmp_path in self._moved_paths:
-            logger.debug('Replacing %s', path)
-            if os.path.isdir(tmp_path) and os.path.isdir(path):
-                rmtree(path)
-            renames(tmp_path, path)
+        self._moved_paths.rollback()
         for pth in self.pth.values():
             pth.rollback()
-        for save_dir in self._save_dirs:
-            save_dir.cleanup()
 
     def commit(self):
         """Remove temporary save dir: rollback will no longer be possible."""
-        for save_dir in self._save_dirs:
-            save_dir.cleanup()
-        self._moved_paths = []
+        self._moved_paths.commit()
 
     @classmethod
     def from_dist(cls, dist):
diff --git a/src/pip/_internal/utils/temp_dir.py b/src/pip/_internal/utils/temp_dir.py
--- a/src/pip/_internal/utils/temp_dir.py
+++ b/src/pip/_internal/utils/temp_dir.py
@@ -1,5 +1,6 @@
 from __future__ import absolute_import
 
+import errno
 import itertools
 import logging
 import os.path
@@ -118,22 +119,30 @@ def _generate_names(cls, name):
         package).
         """
         for i in range(1, len(name)):
-            if name[i] in cls.LEADING_CHARS:
-                continue
             for candidate in itertools.combinations_with_replacement(
                     cls.LEADING_CHARS, i - 1):
                 new_name = '~' + ''.join(candidate) + name[i:]
                 if new_name != name:
                     yield new_name
 
+        # If we make it this far, we will have to make a longer name
+        for i in range(len(cls.LEADING_CHARS)):
+            for candidate in itertools.combinations_with_replacement(
+                    cls.LEADING_CHARS, i):
+                new_name = '~' + ''.join(candidate) + name
+                if new_name != name:
+                    yield new_name
+
     def create(self):
         root, name = os.path.split(self.original)
         for candidate in self._generate_names(name):
             path = os.path.join(root, candidate)
             try:
                 os.mkdir(path)
-            except OSError:
-                pass
+            except OSError as ex:
+                # Continue if the name exists already
+                if ex.errno != errno.EEXIST:
+                    raise
             else:
                 self.path = os.path.realpath(path)
                 break


Test patch:
diff --git a/tests/unit/test_req_uninstall.py b/tests/unit/test_req_uninstall.py
--- a/tests/unit/test_req_uninstall.py
+++ b/tests/unit/test_req_uninstall.py
@@ -5,8 +5,8 @@
 
 import pip._internal.req.req_uninstall
 from pip._internal.req.req_uninstall import (
-    UninstallPathSet, compact, compress_for_output_listing,
-    compress_for_rename, uninstallation_paths,
+    StashedUninstallPathSet, UninstallPathSet, compact,
+    compress_for_output_listing, compress_for_rename, uninstallation_paths,
 )
 from tests.lib import create_file
 
@@ -175,3 +175,96 @@ def test_detect_symlink_dirs(self, monkeypatch, tmpdir):
         ups.add(path1)
         ups.add(path2)
         assert ups.paths == {path1}
+
+
+class TestStashedUninstallPathSet(object):
+    WALK_RESULT = [
+        ("A", ["B", "C"], ["a.py"]),
+        ("A/B", ["D"], ["b.py"]),
+        ("A/B/D", [], ["c.py"]),
+        ("A/C", [], ["d.py", "e.py"]),
+        ("A/E", ["F"], ["f.py"]),
+        ("A/E/F", [], []),
+        ("A/G", ["H"], ["g.py"]),
+        ("A/G/H", [], ["h.py"]),
+    ]
+
+    @classmethod
+    def mock_walk(cls, root):
+        for dirname, subdirs, files in cls.WALK_RESULT:
+            dirname = os.path.sep.join(dirname.split("/"))
+            if dirname.startswith(root):
+                yield dirname[len(root) + 1:], subdirs, files
+
+    def test_compress_for_rename(self, monkeypatch):
+        paths = [os.path.sep.join(p.split("/")) for p in [
+            "A/B/b.py",
+            "A/B/D/c.py",
+            "A/C/d.py",
+            "A/E/f.py",
+            "A/G/g.py",
+        ]]
+
+        expected_paths = [os.path.sep.join(p.split("/")) for p in [
+            "A/B/",         # selected everything below A/B
+            "A/C/d.py",     # did not select everything below A/C
+            "A/E/",         # only empty folders remain under A/E
+            "A/G/g.py",     # non-empty folder remains under A/G
+        ]]
+
+        monkeypatch.setattr('os.walk', self.mock_walk)
+
+        actual_paths = compress_for_rename(paths)
+        assert set(expected_paths) == set(actual_paths)
+
+    @classmethod
+    def make_stash(cls, tmpdir, paths):
+        for dirname, subdirs, files in cls.WALK_RESULT:
+            root = os.path.join(tmpdir, *dirname.split("/"))
+            if not os.path.exists(root):
+                os.mkdir(root)
+            for d in subdirs:
+                os.mkdir(os.path.join(root, d))
+            for f in files:
+                with open(os.path.join(root, f), "wb"):
+                    pass
+
+        pathset = StashedUninstallPathSet()
+
+        paths = [os.path.join(tmpdir, *p.split('/')) for p in paths]
+        stashed_paths = [(p, pathset.stash(p)) for p in paths]
+
+        return pathset, stashed_paths
+
+    def test_stash(self, tmpdir):
+        pathset, stashed_paths = self.make_stash(tmpdir, [
+            "A/B/", "A/C/d.py", "A/E/", "A/G/g.py",
+        ])
+
+        for old_path, new_path in stashed_paths:
+            assert not os.path.exists(old_path)
+            assert os.path.exists(new_path)
+
+        assert stashed_paths == pathset._moves
+
+    def test_commit(self, tmpdir):
+        pathset, stashed_paths = self.make_stash(tmpdir, [
+            "A/B/", "A/C/d.py", "A/E/", "A/G/g.py",
+        ])
+
+        pathset.commit()
+
+        for old_path, new_path in stashed_paths:
+            assert not os.path.exists(old_path)
+            assert not os.path.exists(new_path)
+
+    def test_rollback(self, tmpdir):
+        pathset, stashed_paths = self.make_stash(tmpdir, [
+            "A/B/", "A/C/d.py", "A/E/", "A/G/g.py",
+        ])
+
+        pathset.rollback()
+
+        for old_path, new_path in stashed_paths:
+            assert os.path.exists(old_path)
+            assert not os.path.exists(new_path)
diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py
--- a/tests/unit/test_utils.py
+++ b/tests/unit/test_utils.py
@@ -553,6 +553,10 @@ def test_create_and_cleanup_work(self):
         "ABC.dist-info",
         "_+-",
         "_package",
+        "A......B",
+        "AB",
+        "A",
+        "2",
     ])
     def test_adjacent_directory_names(self, name):
         def names():
@@ -566,17 +570,82 @@ def names():
         # result that works, provided there are many of those
         # and that shorter names result in totally unique sets,
         # it's okay to skip part of the test.)
-        some_names = list(itertools.islice(names(), 10000))
-        assert len(some_names) == len(set(some_names))
+        some_names = list(itertools.islice(names(), 1000))
+        # We should always get at least 1000 names
+        assert len(some_names) == 1000
+
+        # Ensure original name does not appear early in the set
+        assert name not in some_names
+
+        if len(name) > 2:
+            # Names should be at least 90% unique (given the infinite
+            # range of inputs, and the possibility that generated names
+            # may already exist on disk anyway, this is a much cheaper
+            # criteria to enforce than complete uniqueness).
+            assert len(some_names) > 0.9 * len(set(some_names))
+
+            # Ensure the first few names are the same length as the original
+            same_len = list(itertools.takewhile(
+                lambda x: len(x) == len(name),
+                some_names
+            ))
+            assert len(same_len) > 10
+
+            # Check the first group are correct
+            expected_names = ['~' + name[1:]]
+            expected_names.extend('~' + c + name[2:] for c in chars)
+            for x, y in zip(some_names, expected_names):
+                assert x == y
 
-        # Ensure original name does not appear
-        assert not any(n == name for n in names())
+        else:
+            # All names are going to be longer than our original
+            assert min(len(x) for x in some_names) > 1
+
+            # All names are going to be unqiue
+            assert len(some_names) == len(set(some_names))
+
+            if len(name) == 2:
+                # All but the first name are going to end with our original
+                assert all(x.endswith(name) for x in some_names[1:])
+            else:
+                # All names are going to end with our original
+                assert all(x.endswith(name) for x in some_names)
+
+    @pytest.mark.parametrize("name", [
+        "A",
+        "ABC",
+        "ABC.dist-info",
+        "_+-",
+        "_package",
+    ])
+    def test_adjacent_directory_exists(self, name, tmpdir):
+        block_name, expect_name = itertools.islice(
+            AdjacentTempDirectory._generate_names(name), 2)
+
+        original = os.path.join(tmpdir, name)
+        blocker = os.path.join(tmpdir, block_name)
+
+        ensure_dir(original)
+        ensure_dir(blocker)
+
+        with AdjacentTempDirectory(original) as atmp_dir:
+            assert expect_name == os.path.split(atmp_dir.path)[1]
+
+    def test_adjacent_directory_permission_error(self, monkeypatch):
+        name = "ABC"
+
+        def raising_mkdir(*args, **kwargs):
+            raise OSError("Unknown OSError")
+
+        with TempDirectory() as tmp_dir:
+            original = os.path.join(tmp_dir.path, name)
+
+            ensure_dir(original)
+            monkeypatch.setattr("os.mkdir", raising_mkdir)
 
-        # Check the first group are correct
-        expected_names = ['~' + name[1:]]
-        expected_names.extend('~' + c + name[2:] for c in chars)
-        for x, y in zip(some_names, expected_names):
-            assert x == y
+            with pytest.raises(OSError):
+                with AdjacentTempDirectory(original):
+                    pass
 
 
 class TestGlibc(object):