Repository: pypa/pip

Issue:
pip 21.1 install with --no-binary fails while 21.0.1 succeeds (pep517)
### Description

Installing the pip package `pex` with `--no-binary` fails after pip version changed from 21.0.1 to 21.1.  In the output log below, it seems like pep517 is involved.  Steps below are successful if pip==21.0.1 is used.

### Expected behavior

_No response_

### pip version

21.1

### Python version

3.6.4

### OS

RHEL7 x86_64

### How to Reproduce

python3 -m venv venv
. venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install --upgrade setuptools
python3 -m pip install pex==2.1.30 --no-binary :all:

### Output

```sh-session
Cache entry deserialization failed, entry ignored
Collecting pip
  Using cached https://files.pythonhosted.org/packages/ac/cf/0cc542fc93de2f3b9b53cb979c7d1118cffb93204afb46299a9f858e113f/pip-21.1-py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-21.1

Requirement already satisfied: setuptools in ./venv/lib/python3.6/site-packages (28.8.0)
Collecting setuptools
  Using cached setuptools-56.0.0-py3-none-any.whl (784 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed setuptools-56.0.0

Collecting pex==2.1.30
  Downloading pex-2.1.30.tar.gz (3.3 MB)
     |████████████████████████████████| 3.3 MB 3.0 MB/s 
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: <path to venv>/venv/bin/python3 /tmp/pip-standalone-pip-5xbqguwg/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-3b3hrzfg/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core >=2,<3'
       cwd: None
  Complete output (67 lines):
  Collecting flit_core<3,>=2
    Using cached flit_core-2.3.0.tar.gz (22 kB)
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: <path to venv>/venv/bin/python3 /tmp/pip-standalone-pip-5xbqguwg/__env_pip__.zip/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpogp2e7mb
         cwd: /tmp/pip-install-71xy4ixf/flit-core_156db09074064d12968592391ad52e71
    Complete output (1 lines):
    <path to venv>/venv/bin/python3: can't find '__main__' module in '/tmp/pip-standalone-pip-5xbqguwg/__env_pip__.zip/pip/_vendor/pep517/in_process/_in_process.py'

^repeated error while it tries earlier versions of flit_core
```


### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
This is a side effect of pip 21.1 switching to use zipapp for PEP 517 dependency population (#9689). This crashes 3.6 in “PEP 517 inception” scenarios[see note] since `pep517` [falls back to “dumb” `__file__` manipulation when `importlib.resources` is not available](https://github.com/pypa/pip/blob/7a77484a492c8f1e1f5ef24eaf71a43df9ea47eb/src/pip/_vendor/pep517/in_process/__init__.py#L12). I should have noticed this 😞 

I think there are two solutions:

1. Fall back to the “old” way of self-invocation on 3.6. This means people on 3.6 won’t receive #9689, but it was already only fixing a fairly obscure issue anyway.
2. Vendor [importlib-resources](https://pypi.org/project/importlib-resources/) (backport of the stdlib module) and patch `pep517` to use it instead of the stdlib module.

Either way, the workaround will only be short-term, since we will be dropping 3.6 in a year.

----

Note: This will not crash without `--no-binary`, even if a package is sdist-only, as long as the build dependency has a wheel. When `pep517` is invoked to build the sdist, `_in_process.py` is a plain file, so everything works. With `--no-binary`, build dependencies required by the sdist are also only fetched by source only, and `pep517` by that point lives in a zip, causing the issue.

Resolving patch:
diff --git a/src/pip/_internal/build_env.py b/src/pip/_internal/build_env.py
--- a/src/pip/_internal/build_env.py
+++ b/src/pip/_internal/build_env.py
@@ -187,9 +187,17 @@ def install_requirements(
         prefix.setup = True
         if not requirements:
             return
-        with _create_standalone_pip() as standalone_pip:
+        with contextlib.ExitStack() as ctx:
+            # TODO: Remove this block when dropping 3.6 support. Python 3.6
+            # lacks importlib.resources and pep517 has issues loading files in
+            # a zip, so we fallback to the "old" method by adding the current
+            # pip directory to the child process's sys.path.
+            if sys.version_info < (3, 7):
+                pip_runnable = os.path.dirname(pip_location)
+            else:
+                pip_runnable = ctx.enter_context(_create_standalone_pip())
             self._install_requirements(
-                standalone_pip,
+                pip_runnable,
                 finder,
                 requirements,
                 prefix,
@@ -198,14 +206,14 @@ def install_requirements(
 
     @staticmethod
     def _install_requirements(
-        standalone_pip: str,
+        pip_runnable: str,
         finder: "PackageFinder",
         requirements: Iterable[str],
         prefix: _Prefix,
         message: str,
     ) -> None:
         args = [
-            sys.executable, standalone_pip, 'install',
+            sys.executable, pip_runnable, 'install',
             '--ignore-installed', '--no-user', '--prefix', prefix.path,
             '--no-warn-script-location',
         ]  # type: List[str]


Test patch: