Repository: pypa/pip

Issue:
Unable to install wheel containing .data/headers using from non-writable pip installation into writable --prefix
With the following scenario:

 * pip installed in a virtual environment, not owned by the executing user
 * ``pip install --prefix`` to a location that *is* owned by the user

pip *is* able to install simple wheels, but fails to install wheels with header ``.data`` (didn't check for [other types of data](https://www.python.org/dev/peps/pep-0491/#the-data-directory))

I have confirmed that this is still the case on master with the following Dockerfile (placed in the pip repository root in order to copy the source and install pip):

```
FROM python

RUN python -m venv /opt/venv

# Install the latest pip (from source) into both the base Python, and the venv we just created.
COPY . /tmp/pip-repo
RUN python -m pip install /tmp/pip-repo
RUN /opt/venv/bin/python -m pip install /tmp/pip-repo

RUN useradd -ms /bin/bash not_root
USER not_root

# Fine
RUN /opt/venv/bin/python -m pip install --prefix $HOME/writable wheel

# Fine 
RUN python -m pip install --prefix $HOME/writable pybind11

# Not fine as it attempts to write to {venv}/include/site
RUN /opt/venv/bin/python -m pip install --prefix $HOME/writable pybind11

```

The output of building this Dockerfile with ``docker build --network=host .`` is:

```
$ git rev-parse HEAD
09f9b0030d0a3fe98cfe14e6fd9511100c3155e3

$ docker build --network=host .
Sending build context to Docker daemon  517.3MB
Step 1/10 : FROM python
 ---> b55669b4130e
Step 2/10 : RUN python -m venv /opt/venv
 ---> Using cache
 ---> 12c12f10ec01
Step 3/10 : COPY . /tmp/pip-repo
 ---> 6a2a8b641e26
Step 4/10 : RUN python -m pip install /tmp/pip-repo
 ---> Running in 50f5d4a53d52
Processing /tmp/pip-repo
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Building wheels for collected packages: pip
  Building wheel for pip (PEP 517): started
  Building wheel for pip (PEP 517): finished with status 'done'
  Created wheel for pip: filename=pip-20.2.dev1-py2.py3-none-any.whl size=1504387 sha256=c3f4a919e6797f6f477ab912ddc6567f8f7f845b10eb84a09e57b1f6ca612204
  Stored in directory: /root/.cache/pip/wheels/77/78/ae/d6e97e1b65b04ed7f7f18819f65a629c406fdfab1661304f08
Successfully built pip
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-20.2.dev1
Removing intermediate container 50f5d4a53d52
 ---> 62acdf090a08
Step 5/10 : RUN /opt/venv/bin/python -m pip install /tmp/pip-repo
 ---> Running in 269e4a89a909
Processing /tmp/pip-repo
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Building wheels for collected packages: pip
  Building wheel for pip (PEP 517): started
  Building wheel for pip (PEP 517): finished with status 'done'
  Created wheel for pip: filename=pip-20.2.dev1-cp38-none-any.whl size=1504387 sha256=6646a166c762c1b13bf407b81a5c428d1df949fd36b83a5c4f9b02ef43d9616d
  Stored in directory: /root/.cache/pip/wheels/3a/92/63/e66cd23efd91ab6b7552fc37293c6076634bd7c6e5925abf50
Successfully built pip
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.2.dev1
Removing intermediate container 269e4a89a909
 ---> d2b0273bd03d
Step 6/10 : RUN useradd -ms /bin/bash not_root
 ---> Running in dd9e454eed3d
Removing intermediate container dd9e454eed3d
 ---> e6ea8e7d10c6
Step 7/10 : USER not_root
 ---> Running in 17075cdc597b
Removing intermediate container 17075cdc597b
 ---> 09dae5d7fe5b
Step 8/10 : RUN /opt/venv/bin/python -m pip install --prefix $HOME/writable wheel
 ---> Running in 252992c43507
Collecting wheel
  Downloading wheel-0.34.2-py2.py3-none-any.whl (26 kB)
Installing collected packages: wheel
  WARNING: The script wheel is installed in '/home/not_root/writable/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed wheel-0.34.2
Removing intermediate container 252992c43507
 ---> 34705392a77b
Step 9/10 : RUN python -m pip install --prefix $HOME/writable pybind11
 ---> Running in 3d7d6e35d52a
Collecting pybind11
  Downloading pybind11-2.5.0-py2.py3-none-any.whl (296 kB)
Installing collected packages: pybind11
Successfully installed pybind11-2.5.0
Removing intermediate container 3d7d6e35d52a
 ---> 495db85a9fdb
Step 10/10 : RUN /opt/venv/bin/python -m pip install --prefix $HOME/writable pybind11
 ---> Running in f0287949ea8f
Collecting pybind11
  Using cached pybind11-2.5.0-py2.py3-none-any.whl (296 kB)
Installing collected packages: pybind11
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/opt/venv/include/site'
Consider using the `--user` option or check the permissions.

The command '/bin/sh -c /opt/venv/bin/python -m pip install --prefix $HOME/writable pybind11' returned a non-zero code: 1
```


The full traceback from this command is:

```
Step 10/10 : RUN /opt/venv/bin/python -m pip -vvv install --prefix $HOME/writable pybind11
 ---> Running in fcd054a27042
Non-user install due to --prefix or --target option
Created temporary directory: /tmp/pip-ephem-wheel-cache-4jufpvad
Created temporary directory: /tmp/pip-req-tracker-4g9cr9c4
Initialized build tracking at /tmp/pip-req-tracker-4g9cr9c4
Created build tracker: /tmp/pip-req-tracker-4g9cr9c4
Entered build tracker: /tmp/pip-req-tracker-4g9cr9c4
Created temporary directory: /tmp/pip-install-c8x570pg
1 location(s) to search for versions of pybind11:
* https://pypi.org/simple/pybind11/
Fetching project page and analyzing links: https://pypi.org/simple/pybind11/
Getting page https://pypi.org/simple/pybind11/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/pybind11/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/pybind11/ HTTP/1.1" 304 0
  Found link 
... snip ...
Given no hashes to check 46 links for project 'pybind11': discarding no candidates
Using version 2.5.0 (newest of versions: 1.0, 1.2, 1.3, 1.4, 1.5, 1.7, 1.8, 1.8.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.3.0, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0)
Collecting pybind11
  Created temporary directory: /tmp/pip-unpack-w_3s0md1
  Looking up "https://files.pythonhosted.org/packages/89/e3/d576f6f02bc75bacbc3d42494e8f1d063c95617d86648dba243c2cb3963e/pybind11-2.5.0-py2.py3-none-any.whl" in the cache
  Current age based on date: 2
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 2
  Using cached pybind11-2.5.0-py2.py3-none-any.whl (296 kB)
  Added pybind11 from https://files.pythonhosted.org/packages/89/e3/d576f6f02bc75bacbc3d42494e8f1d063c95617d86648dba243c2cb3963e/pybind11-2.5.0-py2.py3-none-any.whl#sha256=205d9f9615eac190811cb8c3c2b2f95f6844ddba0fa0a1d45d00793338741601 to build tracker '/tmp/pip-req-tracker-4g9cr9c4'
  Removed pybind11 from https://files.pythonhosted.org/packages/89/e3/d576f6f02bc75bacbc3d42494e8f1d063c95617d86648dba243c2cb3963e/pybind11-2.5.0-py2.py3-none-any.whl#sha256=205d9f9615eac190811cb8c3c2b2f95f6844ddba0fa0a1d45d00793338741601 from build tracker '/tmp/pip-req-tracker-4g9cr9c4'
Installing collected packages: pybind11
  Created temporary directory: /tmp/pip-unpacked-wheel-m2gac4cg

ERROR: Could not install packages due to an EnvironmentError.
Consider using the `--user` option or check the permissions.
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 393, in run
    installed = install_given_reqs(
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 71, in install_given_reqs
    requirement.install(
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 819, in install
    install_wheel(
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 683, in install_wheel
    install_unpacked_wheel(
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 497, in install_unpacked_wheel
    clobber(
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 404, in clobber
    ensure_dir(dest)  # common for the 'include' path
  File "/opt/venv/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 111, in ensure_dir
    os.makedirs(path)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/opt/venv/include/site'
Removed build tracker: '/tmp/pip-req-tracker-4g9cr9c4'
The command '/bin/sh -c /opt/venv/bin/python -m pip -vvv install --prefix $HOME/writable pybind11' returned a non-zero code: 1
```


Discussion:
From a quick glance, it seems like something in the wheel installation logic does not correctly respect `--prefix`, and wants to install into the “real” scheme of the host Python. I’m guessing similar flags like `--target` may also have the same issue.
[Here](https://github.com/pypa/pip/blob/334f06e2248c6114ee4899515c5737845a6a0fcd/src/pip/_internal/locations.py#L141) we unconditionally use `sys.prefix` when calculating the header path when running under virtualenv (or venv). That's definitely not the right thing to do when someone provides `--prefix` explicitly. I don't have a good answer for what we should do, though.
The `prefix` argument looks suspecious…

Resolving patch:
diff --git a/src/pip/_internal/locations.py b/src/pip/_internal/locations.py
--- a/src/pip/_internal/locations.py
+++ b/src/pip/_internal/locations.py
@@ -138,7 +138,7 @@ def distutils_scheme(
 
     if running_under_virtualenv():
         scheme['headers'] = os.path.join(
-            sys.prefix,
+            i.prefix,
             'include',
             'site',
             'python{}'.format(get_major_minor_version()),


Test patch:
diff --git a/tests/unit/test_locations.py b/tests/unit/test_locations.py
--- a/tests/unit/test_locations.py
+++ b/tests/unit/test_locations.py
@@ -130,3 +130,19 @@ def test_install_lib_takes_precedence(self, tmpdir, monkeypatch):
         scheme = distutils_scheme('example')
         assert scheme['platlib'] == install_lib + os.path.sep
         assert scheme['purelib'] == install_lib + os.path.sep
+
+    def test_prefix_modifies_appropriately(self):
+        prefix = os.path.abspath(os.path.join('somewhere', 'else'))
+
+        normal_scheme = distutils_scheme("example")
+        prefix_scheme = distutils_scheme("example", prefix=prefix)
+
+        def _calculate_expected(value):
+            path = os.path.join(prefix, os.path.relpath(value, sys.prefix))
+            return os.path.normpath(path)
+
+        expected = {
+            k: _calculate_expected(v)
+            for k, v in normal_scheme.items()
+        }
+        assert prefix_scheme == expected