Repository: pypa/pip

Issue:
Build isolation fails with system-wide pip<10 and user-site pip==10.0.0
* Pip version: 10.0.0
* Python version: 3.6.5
* Operating system: Linux/Fedora

### Description:

Apparently, the combination of system-wide installation of pip<10 and user-site installation of pip>=10 makes projects with `pyproject.toml` fail to build.

I would expect user-site installation of pip to work.

### What I've run:

```
$ python3 -mpip install --user --upgrade pip
Requirement already up-to-date: pip in ./.local/lib/python3.6/site-packages (10.0.0)
$ mkdir ~/foo
$ cd ~/foo
$ echo "assert False" > setup.py
$ cat <<EOF > pyproject.toml
> [build-system]
> requires = [
>     "wheel",
>     "setuptools",
> ]
> EOF
$ ls
pyproject.toml  setup.py
$ python3 -mpip --version
pip 10.0.0 from /home/pauli/.local/lib/python3.6/site-packages/pip (python 3.6)
$ python3 -mpip wheel .
Processing /home/pauli/foo
  Installing build dependencies ... error
  Complete output from command /usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-tlxjmy36 https://pypi.python.org/packages/1b/d2/22cde5ea9af055f81814f9f2545f5ed8a053eb749c08d186b369959189a8/wheel-0.31.0-py2.py3-none-any.whl#md5=240d714477a715bcd90e94cb2c44f28c https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc:
  
  Usage:
    /usr/bin/python3 -m pip install [options] <requirement specifier> [package-index-options] ...
    /usr/bin/python3 -m pip install [options] -r <requirements file> [package-index-options] ...
    /usr/bin/python3 -m pip install [options] [-e] <vcs project url> ...
    /usr/bin/python3 -m pip install [options] [-e] <local project path> ...
    /usr/bin/python3 -m pip install [options] <archive url/path> ...
  
  no such option: --no-user
  
  ----------------------------------------
Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-tlxjmy36 https://pypi.python.org/packages/1b/d2/22cde5ea9af055f81814f9f2545f5ed8a053eb749c08d186b369959189a8/wheel-0.31.0-py2.py3-none-any.whl#md5=240d714477a715bcd90e94cb2c44f28c https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc" failed with error code 2 in None
```

Note that the error occurs before executing setup.py


Discussion:
When you say "system-wide pip<10 and user-site pip==10.0.0" you have (different versions of) pip installed in the 2 places? That sounds unlikely to work. I'm not even sure it's something that Python supports...
The system-wide python install essentially always has some version of pip installed.
This is the default configuration pretty much on all Linux systems.

Installing newer versions of software using "pip install --user" has always worked for me so far.
Clearly, this is something that python does support. I don't know why exactly it breaks for pip
this time (it has also worked for previous pip versions), maybe it has to do with how pip sets up
the temporary environment for build dependencies.

Using the "--user" flag is even recommended here:
https://packaging.python.org/tutorials/installing-packages/
The problem seems to be that something inside pip adds `PYTHONNOUSERSITE=1` to the environment which then causes a failure at https://github.com/pypa/pip/blob/master/src/pip/_internal/operations/prepare.py#L67
Installing packages in user-site is indeed recommended. But that should be *instead* of the system site-packages, not *as well*. I don't think anyone has ever recommended having 2 different versions of a package installed in 2 separate locations on`sys.path`.

The failure is because when pip 10 tries to run `python -m pip`, it's somehow picking up an older version of pip which doesn't support the `--no-user` flag. I'm inclined to classify that as a broken environment, but if you can provide a reproducible test case, we can review whether or not it's a scenario that we should handle better.
I believe it's not correct to state that different versions in sys.path are not supported --- it's always been the case that the package that comes first wins.

This is also something that's assumed by several scenarios that deal with setting PYTHONPATH environment variable.

I already provided a reproducible test case above.
If pip does not support having different versions installed in system site-packages and user site-packages, then "pip install --upgrade --user" should simply refuse to install a newer version of the package to the user-site directory if it's already present system-wide.

However, this does not seem like it was the intent with the `--user` flag, hence I think what's broken is how the build depends setup works in pip 10, not the fact that you can do `pip install --user --upgrade pip`.
Switching to a more recent version of a system package by installing a newer version in the user site is common use case.

Anyway, I don't think the command to install the build requirements should be isolated: it's already using `--prefix` and `--ignore-installed`, anything more is going to be problematic if pip itself is installed in the user site (whether or not another version of pip is available system wide), or if one of pip's dependency is (when pip has been un-vendored).
The reason the build requirements are installed using `--no-user` is #5085. The problem is that `--user` and `--prefix` cannot be used together.

But the issue here is that for some reason, when pip runs `[sys.executable, '-m', 'pip', ...]` it's picking up a *different* version of pip (one that doesn't support the `--no-user` flag introduced in pip 10). As far as I know, that should be impossible without the user's system being broken. I think the key here will be isolating how that's happening.
The problem is not `--no-user`, but using the isolated environment (with `PYTHONNOUSERSITE=1`), no?
Oh! I see what you're saying - https://github.com/pypa/pip/blob/master/src/pip/_internal/build_env.py#L56 is where `PYTHONNOUSERSITE` is set.

I'm not the best person to comment on that code - it's dealing with situations I never really understood in detail. @pradyunsg are you able to comment?
I'm looking into it, but as stated above, I don't think we want build isolation for the command used to install the build dependencies.
Cool thanks

Resolving patch:
diff --git a/src/pip/_internal/operations/prepare.py b/src/pip/_internal/operations/prepare.py
--- a/src/pip/_internal/operations/prepare.py
+++ b/src/pip/_internal/operations/prepare.py
@@ -144,8 +144,10 @@ def format_reqs(rs):
             )
 
         if should_isolate:
-            with self.req.build_env as prefix:
-                _install_build_reqs(finder, prefix, build_requirements)
+            with self.req.build_env:
+                pass
+            _install_build_reqs(finder, self.req.build_env.path,
+                                build_requirements)
         else:
             self.req.build_env = NoOpBuildEnvironment(no_clean=False)
 


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -137,20 +137,24 @@ def isolate(tmpdir):
         )
 
 
-@pytest.yield_fixture(scope='session')
-def virtualenv_template(tmpdir_factory):
-    tmpdir = Path(str(tmpdir_factory.mktemp('virtualenv')))
-    # Copy over our source tree so that each virtual environment is self
-    # contained
-    pip_src = tmpdir.join("pip_src").abspath
+@pytest.fixture(scope='session')
+def pip_src(tmpdir_factory):
+    pip_src = Path(str(tmpdir_factory.mktemp('pip_src'))).join('pip_src')
+    # Copy over our source tree so that each use is self contained
     shutil.copytree(
         SRC_DIR,
-        pip_src,
+        pip_src.abspath,
         ignore=shutil.ignore_patterns(
             "*.pyc", "__pycache__", "contrib", "docs", "tasks", "*.txt",
             "tests", "pip.egg-info", "build", "dist", ".tox", ".git",
         ),
     )
+    return pip_src
+
+
+@pytest.yield_fixture(scope='session')
+def virtualenv_template(tmpdir_factory, pip_src):
+    tmpdir = Path(str(tmpdir_factory.mktemp('virtualenv')))
     # Create the virtual environment
     venv = VirtualEnvironment.create(
         tmpdir.join("venv_orig"),
diff --git a/tests/functional/test_install.py b/tests/functional/test_install.py
--- a/tests/functional/test_install.py
+++ b/tests/functional/test_install.py
@@ -45,6 +45,29 @@ def test_pep518_uses_build_env(script, data, original_setuptools):
         )
 
 
+def test_pep518_with_user_pip(script, virtualenv, pip_src, data):
+    virtualenv.system_site_packages = True
+    script.pip("install", "--ignore-installed", "--user", pip_src)
+    system_pip_dir = script.site_packages_path / 'pip'
+    system_pip_dir.rmtree()
+    system_pip_dir.mkdir()
+    with open(system_pip_dir / '__init__.py', 'w') as fp:
+        fp.write('raise ImportError\n')
+    to_install = data.src.join("pep518-3.0")
+    for command in ('install', 'wheel'):
+        kwargs = {}
+        if sys.version_info[:2] == (3, 3):
+            # Ignore Python 3.3 deprecation warning...
+            kwargs['expect_stderr'] = True
+        script.run(
+            "python", "-c",
+            "import pip._internal; pip._internal.main(["
+            "%r, " "'-f', %r, " "%r, "
+            "])" % (command, str(data.packages), str(to_install)),
+            **kwargs
+        )
+
+
 @pytest.mark.network
 def test_pip_second_command_line_interface_works(script, data):
     """