Repository: pypa/pip

Issue:
move_wheel_files fails on setuptools dependencies
Reported in pypa/setuptools#951, it appears that when upgrading or force-reinstalling setuptools dependencies, pip calls into pkg_resources to get package metadata, but since the package is currently uninstalled, the search for its metadata fails.

I haven't looked into the pip code, but I wanted to ask this project first: Can someone provide a brief explanation for what pip is doing here and if there's an adjustment that pip or pkg_resources could make to avoid the failure?


Discussion:
Um, well ``move_wheel_file`` is use to install a wheel. I think that inside of it, it calls ``distutils_scheme()`` which creates a ``Distribution`` object to figure out where distutils would have installed the files so that we can install them to the same location (taking into account ``~/.pydistutils.cfg`` files and such).
Hmm. The Distribution object that's created is a `distutils.dist.Distribution`, which wouldn't be calling into setuptools unless that object had been monkeypatched by an import of setuptools. Searching through the pip code, it does seem that setuptools is imported at the invocation of `InstallRequirement.setup_py`.

That explains why the issue only occurs when a pure sdist package is first installed (such as `junit-xml`). It's the installation of that package that triggers the import of setuptools in the pip runtime, which triggers the monkeypatch, which triggers the enumeration of `distutils.setup_keywords` entry points, which forces the dependencies of setuptools to be `required`, but since six isn't currently installed things fall apart. I suspect this issue could also be triggered by the upgrade or downgrade of any setuptools dependency from a source dist.

My first instinct is to see if we can adapt pip not to import setuptools and thus not monkeypatch the `Distribution` object. I'm unsure if the monkeypatching is necessary.

Indeed, when I apply this patch, the failure goes away:

```
diff --git a/pip/req/req_install.py b/pip/req/req_install.py
index 1a98f377..f4716c57 100644
--- a/pip/req/req_install.py
+++ b/pip/req/req_install.py
@@ -383,17 +383,10 @@ class InstallRequirement(object):
     @property
     def setup_py(self):
         assert self.source_dir, "No source dir for %s" % self
-        try:
-            import setuptools  # noqa
-        except ImportError:
-            if get_installed_version('setuptools') is None:
-                add_msg = "Please install setuptools."
-            else:
-                add_msg = traceback.format_exc()
-            # Setuptools is not available
-            raise InstallationError(
-                "Could not import setuptools which is required to "
-                "install from a source distribution.\n%s" % add_msg
+        if get_installed_version('setuptools') is None:
+            raise InstallationError("Setuptools is required to install "
+                "from a source distribution, but none was found. Please "
+                "install setuptools."
             )
 
         setup_py = os.path.join(self.setup_py_dir, 'setup.py')
```

I can imagine this change would cause failures in environments where setuptools is importable but not installed as pip sees it. To retain the existing behavior, one could `import setuptools` in a subprocess to check for the existence of an importable setuptools... or bypass a full import and call into `find_module` instead to see if it can be found without executing it.

I'd also be open to the possibility of removing the implicit monkeypatching of distutils, such that all setup scripts would would have to be updated to reference the setuptools objects directly or call the monkeypatching explicitly. There's a lot of unknowns and possibly intractable challenges in that scenario.

What do you think about the patch above? Do late versions of pip need to detect and pass on an importable but not installed setuptools? Do you have any other ideas?
> I'd also be open to the possibility of removing the implicit monkeypatching of distutils

Another less aggressive option would be for setuptools to expose a routine to unmonkeypatch distutils, and pip could call that after the import. Another option would be for pip to import the Distribution object before it has a chance to be monkeypatched and always use that.
> Do late versions of pip need to detect and pass on an importable but not installed setuptools? 

I don't think there is any reason they need to work if setuptools isn't "installed", at least not off the top of my head.

Resolving patch:
diff --git a/pip/req/req_install.py b/pip/req/req_install.py
--- a/pip/req/req_install.py
+++ b/pip/req/req_install.py
@@ -409,13 +409,19 @@ def setup_py_dir(self):
     @property
     def setup_py(self):
         assert self.source_dir, "No source dir for %s" % self
-        try:
-            import setuptools  # noqa
-        except ImportError:
+        cmd = [sys.executable, '-c', 'import setuptools']
+        output = call_subprocess(
+            cmd,
+            show_stdout=False,
+            command_desc='python -c "import setuptools"',
+            on_returncode='ignore',
+        )
+
+        if output:
             if get_installed_version('setuptools') is None:
                 add_msg = "Please install setuptools."
             else:
-                add_msg = traceback.format_exc()
+                add_msg = output
             # Setuptools is not available
             raise InstallationError(
                 "Could not import setuptools which is required to "


Test patch:
diff --git a/tests/unit/test_req_file.py b/tests/unit/test_req_file.py
--- a/tests/unit/test_req_file.py
+++ b/tests/unit/test_req_file.py
@@ -586,15 +586,17 @@ def test_install_requirements_with_options(self, tmpdir, finder, session,
 
         req.source_dir = os.curdir
         with patch.object(subprocess, 'Popen') as popen:
-            popen.return_value.stdout.readline.return_value = ""
+            popen.return_value.stdout.readline.return_value = b""
             try:
                 req.install([])
             except:
                 pass
 
-            call = popen.call_args_list[0][0][0]
-            assert call.index(install_option) > \
-                call.index('install') > \
-                call.index(global_option) > 0
+            last_call = popen.call_args_list[-1]
+            args = last_call[0][0]
+            assert (
+                0 < args.index(global_option) < args.index('install') <
+                args.index(install_option)
+            )
         assert options.format_control.no_binary == set([':all:'])
         assert options.format_control.only_binary == set([])