Repository: pypa/pip

Issue:
virtualenv with --system-site-packages breaks pip's build isolation
It seems that if you create a virtualenv with `--system-site-packages`, the system packages (but *not* the user packages) will be on the `PYTHONPATH` in the PEP 517 isolated build environment that `pip` creates (it does not effect `python -m pep517.build`), and it seems they will be on there with *higher precedence* than the requirements installed in the build environment. It only affects `pip >= 19.0.0`.

The most common way I've seen this cause problems is with the new `setuptools.build_meta:__legacy__` backend. Since the most recent version of `pip` requires a recent version of `setuptools`, if you have an older version of `setuptools` installed on the system, `pip install` will fail due to the missing `setuptools.build_meta:__legacy__` backend. It is possible to reproduce this by crafting a deliberately bad package and installing it, then creating a wheel for a newer version of the package (which allowed me to test that this failure was actually introduced with `pip==19.0.0`), but for the MWE we can stick with `setuptools`.

To reproduce, create a package like this:

```bash
cd /tmp
mkdir demo
cd demo
mkdir badpkg

touch badpkg/pyproject.toml
echo 'from setuptools import setup; setup(name="foo", version="0.1.0")' \
    > badpkg/setup.py
```

Then install an older version of `setuptools` *on your system* (can't be `--user` or in the virtualenv), `pip install 'setuptools < 40.8.0'` (I did this in a `pyenv` environment). If your system already has an older version of `setuptools` on it, you're already good.

Next create a virtualenv with `--system-site-packages` and activate it:

```bash
virtualenv venv --system-site-packages
source venv/bin/activate
```

Finally try to install `badpkg` (`pip wheel` or `pip install -t tmp` also works to demonstrate the problem):

```bash
pip install ./badpkg
```

You should get a traceback the ends like this:

>  `AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'`

At first I thought this was because the affected packages had too loose bounds on `build-system.requires` (e.g. `requires=["setuptools"]`) and that `pip` was failing to install a more recent version in the isolated environment, but this bug still occurs even if you specify `"setuptools>=40.8.0"`, so I believe it's not a problem at install-time, it's a problem with the path resolution at build time.

At the moment it's unclear if this is a problem with `pip` or `virtualenv`, but since it does not affect `python -m pep517.build`, I'm reporting it here. It could be a combination of both.

CC: @gaborbernat 


Discussion:
Per my Bloomberg colleague @lkollar, it seems that the problem is that `pip` [creates a custom `sitecustomize.py` file](https://github.com/pypa/pip/blob/master/src/pip/_internal/build_env.py#L79-L105) in order to inject itself into the build environment.

Looks to me like it's just blacklisting the system packages with `distutils.get_python_lib`, which doesn't include the system path added by `virtualenv`. Seems like `pep517.envbuild.BuildEnvironment` doesn't have this problem - maybe switch over to using that? Or was there a reason to do it this way?
I'm also seeing this problem, when installing `cryptography` 2.5 using Python 3.7.2, even though I'm using the latest setuptools (40.8.0).

What I don't understand is why I'm seeing this even though I'm using `pip install --no-build-isolation ...`. How is that possible if the problem is related to `pip`'s build isolation feature?
I should also mention that I'm not using `virtualenv` at all (although maybe somehow the `cryptography` installation procedure does that somehow?).
@boegel Can you create a minimal-working example that reproduces this, and include your `pip` versions? I've never known anyone to be able to trigger this without `virtualenv --system-site-packages`.
@pganssle Does this suffice?

Note that this is a self-built Python 3.7.2 from source, in a custom location, not sure if that's relevant to the problem...

```
$ python -V
Python 3.7.2

$ pip -V
pip 19.0.3 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip (python 3.7)

$ easy_install --version
setuptools 40.8.0 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.8.0-py3.7.egg (Python 3.7)

$ mkdir -p /tmp/$USER/pip_debug
$ cd /tmp/$USER/pip_debug

$ curl -OL https://files.pythonhosted.org/packages/69/ed/5e97b7f54237a9e4e6291b6e52173372b7fa45ca730d36ea90b790c0059a/cryptography-2.5.tar.gz

$ tar xfz cryptography-2.5.tar.gz

$ cd cryptography-2.5

$ pip install --prefix=$PWD --no-deps  --ignore-installed  --no-build-isolation  .
Processing /tmp/user/pip_debug/cryptography-2.5
    Preparing wheel metadata ... error
    Complete output from command /prefix/Python/3.7.2-GCCcore-8.2.0/bin/python /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpkwc5sc4e:
    Traceback (most recent call last):
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 62, in prepare_metadata_for_build_wheel
        backend = _build_backend()
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
        obj = getattr(obj, path_part)
    AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

    ----------------------------------------
Command "/prefix/Python/3.7.2-GCCcore-8.2.0/bin/python /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpkwc5sc4e" failed with error code 1 in /tmp/pip-req-build-q9knj58j
```

This is on CentOS 7.6.1810, with only `python` 2.x installed on the system. We do have an old `setuptools` in the OS as well, but I think that one is irrelevant since that's only for Python 2.

```
$ rpm -qa | grep ^python[0-9]*-[0-9]
python-2.7.5-76.el7.x86_64
$ rpm -qa | grep setuptools
python-setuptools-0.9.8-7.el7.noarch
```
I should also mention that `--no-use-pep517` fixes the problem for me.
> `pip 19.0.3 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip (python 3.7)`

Looks like your pip (and your setuptools) have been installed as eggs, presumably using `easy_install`. This is obsolete, can you try with a copy of pip installed using pip?
@pfmoore Same problem when `pip` and `setuptools` were installed with `pip`, when using latest `cryptography` from PyPI. Note that there's no other Python 3 installation on the system (and hence no other `setuptools` either).

```
$ which python
/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python
$ python -V
Python 3.7.2

$ pip -V
pip 19.0.3 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip (python 3.7)

$ easy_install --version
setuptools 40.8.0 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages (Python 3.7)

$ mkdir -p /tmp/$USER/pip_debug
$ cd /tmp/$USER/pip_debug
$ curl -OL https://files.pythonhosted.org/packages/07/ca/bc827c5e55918ad223d59d299fff92f3563476c3b00d0a9157d9c0217449/cryptography-2.6.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  480k  100  480k    0     0   755k      0 --:--:-- --:--:-- --:--:--  754k

$ tar xfz cryptography-2.6.1.tar.gz
$ cd cryptography-2.6.1

$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  .
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
    Preparing wheel metadata ... error
    Complete output from command /prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_:
    Traceback (most recent call last):
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 62, in prepare_metadata_for_build_wheel
        backend = _build_backend()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
        obj = getattr(obj, path_part)
    AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

    ----------------------------------------
Command "/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_" failed with error code 1 in /tmp/pip-req-build-s_w63g1g
```

Works fine when using `--no-use-pep517`:
```
$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  . --no-use-pep517
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
Building wheels for collected packages: cryptography
  Building wheel for cryptography (setup.py) ...
...
```
> module 'setuptools.build_meta' has no attribute '__legacy__'

That indicates you're not picking up setuptools 40.8.0 - see [here](https://github.com/pypa/setuptools/blob/master/setuptools/build_meta.py#L241)

Try running

```python
import setuptools.build_meta
print(setuptools.build_meta.__legacy__)
```

You should get something like `<setuptools.build_meta._BuildMetaLegacyBackend object at 0x000000000401B2E8>`. If you're not, then there's something wrong with your system setuptools. If you are, your build environment is not picking up the same packages as your system Python - in spite of your assertion that there's only the one setuptools installed on your system :-(

Also, the fact that you're using `--prefix` may be relevant here - I don't know how that interacts (if at all) with build environments. Can you reproduce this issue without `--prefix`?
@pfmoore if virtualenv inherits the system site package (and that has a setuptoools before 40.8.0) the pip install of the setuptools build dependency is ignored (as pip does not check the version of what is installed unless -U is passed, and says any setuptools satisfies the install requirement); therefore once pip tries to get the build backend the above failure is thrown. I don't think there needs to be anything wrong with system python to fall into this issue. pip should make sure to never-ever create an isolated build environment that inherits from the system site package, which is not the case at the moment.
> if virtualenv inherits the system site package

Presumably when `--no-build-isolation` is specified? An isolated build ignores the existing installation.

> (and that has a setuptoools before 40.8.0)

Well, the point of `--no-build-isolation` is that the user is agreeing to set up the correct build environment, so that seems like it's a simple case of user error.

However, in this case @boegel is claiming that he has setuptools 40.8.0 installed, and no other setuptools is present. If that's the case, and his setuptools doesn't contain `build_meta.__legacy__`, then that setuptools installation is somehow broken, because 40.8.0 *does* contain that backend.

My suspicion is that @boegel is actually mistaken, and there's another copy of setuptools lingering around somewhere, which is getting picked up and is confusing things. The other possibility is that there's a bug somewhere - but if that *is* the case, then we're currently struggling to reduce the example to something that can easily reproduce/demonstrate that bug.

> pip should make sure to never-ever create an isolated build environment that inherits from the system site package

Well, yes, that's sort of the definition of "isolated" :-) But I'm not sure how relevant that is in this case, where `--no-build-isolation` is set.

> which is not the case at the moment

That's what @pganssle is saying in the original post in this thread, yes. But I'm responding to @boegel, who is claiming that he can reproduce the problem without a virtualenv, using `--no-build-isolation`.

If @boegel's problem turns out to be real, then I don't think it's related to this one (there's too many differences in the steps to reproduce it). So it should probably be a separate issue. But I'd be interested to understand why he *thinks* it is the same issue...
Dug a little further, it looks it's indeed a case of a broken `setuptools` installation, in some sense...

First, I should clarify my statement w.r.t. not having another `setuptools` installed. I should clarify that I mean that there's no other `setuptools` installed for Python 3.x. There is one for Python 2 (which is available in the OS), I'm assuming that one will never be picked up by Python 3.

That said, I'm pretty sure the right `setuptools` is being picked up:

```
$ python -c "import setuptools; print(setuptools.__file__); print(setuptools.version.__version__)"
/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools/__init__.py
40.8.0
```

Yet:

```
$ python -c "import setuptools.build_meta; print(setuptools.build_meta.__file__); print(setuptools.build_meta.__legacy__)"
/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools/build_meta.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'
```

So I do have `setuptools` 40.8.0, but no `__legacy__` in `setuptools.build_meta`?!

This sheds some light on it:

```
ls -ld /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools*
drwxr-xr-x 6 myuser myuser 4096 Mar  7 18:48 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:56 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.6.2.dist-info
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:57 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.8.0-py3.7.egg-info
drwxr-xr-x 3 myuser myuser 4096 Mar  7 11:58 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools_scm
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:58 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools_scm-3.2.0.dist-info
```

The `setuptools-40.6.2.dist-info` looks very suspicious, it seems like I somehow ended up with a frankenbuild of `setuptools-40.6.2` (which doesn't have `__legacy__` in `setuptools.build_meta` yet) and `setuptools-40.8.0`?! Ugh...

I got redirect to this issue via https://github.com/pypa/setuptools/issues/1694, which shows the exact same error I'm running into, so I naively assumed it's the same problem.

It seems in my case it's indeed because of a corrupted `setuptools` though, my apologies for the noise...
No problem, glad we got to the bottom of it!
Hey,

I'm running into this issue as well and I'm thoroughly confused. As a mere user, it looks totally broken with no clear path of understanding how I'm doing anything wrong.
Well, having said that. I ended up removing the system-wide setuptools package and now local pip doesn't complain.
I'm encountering the same issue, I think.  My organization defaults virtualenvs to `--system-site-packages`, and the globally installed setuptools is 40.4.3.  I cloned the `pip` git repo, and then:

In a system-site-packages virtualenv, `pip install ./pip/` gives the `AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'`.

In a virtualenv with no-site-packages, `pip install ./pip/` works correctly.

I think I agree with @gaborbernat that it would be better for "build isolation" to include ignoring `--system-site-packages`.  Or maybe there's at least some way to give a better error message?
FTR I'm hitting this with `tox` + `system-site-packages` with `setuptools>=40.8.0` in `pyproject.toml` under Python 2.7.15. I was actually hitting pypa/setuptools#1136 originally (which I knew was fixed a while back) and only after some time of playing around with removing `pyproject.toml` and then just playing with `--no-build-isolation` I've found this one...

Couldn't figure it out for a while because it was only reproducible in Travis CI which drops you into a virtualenv from the beginning automatically. Had to request debug access to their VMs to nail it down...
Okay, now I'm hitting this in https://github.com/rtfd/readthedocs.org/issues/5572 somehow.
They use explicit `--no-site-packages`: https://readthedocs.org/projects/octomachinery/builds/8869252/
@pganssle it looks like it's time to remove `--system-site-packages` from the title. I've already hit two instances of the bug with `--no-site-packages`.
@webknjaz How does it manifest with `--no-site-packages`? That may even be a distinct bug.
@pganssle it looks the same to me: setuptools fails to use a newer feature because it's not the version of setuptools I specified in `pyproject.toml`
@pganssle one is https://readthedocs.org/projects/octomachinery/builds/8869252/ (pure virtualenv) another is https://travis-ci.com/sanitizers/octomachinery/jobs/190175604#L401-L407 (tox). Both don't have system site packages.
Yea, it's another missing feature hit but the root of the issue is still the same.
I just meant we already know the cause of the problem with `--system-site-packages `, can you verify if it's the same mechanism here?
@pganssle can you point me to the comment with verification plan, please? I've scanned through and it's not obvious which one to follow...
@webknjaz Just this: https://github.com/pypa/pip/issues/6264#issuecomment-463229977

TBH I remember verifying this by setting a breakpoint or printing something, but I guess at the time it seemed obvious what to do once you knew the cause 😅.
I'd need more precise instructions because debugging this with all subprocesses and stuff is PITA and unfortunately I don't have time for this right now... Maybe later.
Hitting this issue when installing `scikit-learn` inside a --system-site-packages virtualenv that uses Python 3.6.8 on Ubuntu 18.04. My setuptools version is 41.0.1.
I also hit this problem.
In my case, downgrade pip. it works.
```
pip install --upgrade pip==19.0
pip install numpy
``` 

I also hit this problem when installing the DonkeyCar system that uses many python machine learning libraries.  I downgraded to pip 19.0 and everything seems to be working fine.  Thank you Naisy.
Same problem. Setuptools: 47.1.1, python: 3.5.2, pip: 20.1.1 virtualenv: 16.3.0. Only solution works so far is downgrade pip to 19. 
Just another data point, I hit this when installing Huggingface transformers on a Jetson Nano running Ubuntu 18.

```
$ uname -a
Linux my-hostname 4.9.140-tegra #1 SMP PREEMPT Tue Jul 16 17:04:49 PDT 2019 aarch64 aarch64 aarch64 GNU/Linux
```

Failed with pip version 20, succeeded with pip version 19 as suggested by @naisy above. Other software: virtualenv 15.1.0, python 3.6.8. Failure message from pip v20 below:

```
pip install transformers
Collecting transformers
  Using cached transformers-3.0.2-py3-none-any.whl (769 kB)
Requirement already satisfied: tqdm>=4.27 in ./venv/lib/python3.6/site-packages (from transformers) (4.48.0)
Collecting tokenizers==0.8.1.rc1
  Downloading tokenizers-0.8.1rc1.tar.gz (97 kB)
     |████████████████████████████████| 97 kB 1.1 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /whatever/venv/bin/python3 /whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpm6zfgdjl
       cwd: /tmp/pip-install-r8hkyld0/tokenizers
  Complete output (10 lines):
  Traceback (most recent call last):
    File "/whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 108, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 99, in _build_backend
      obj = getattr(obj, path_part)
  AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /whatever/venv/bin/python3 /whatever/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpm6zfgdjl Check the logs for full command output.
```
I believe this is the root cause of #8823 and https://github.com/pypa/setuptools/issues/2353 - a PEP 517 build of a package is performed, its build requirements are installed into a directory that is added to `sys.path` for the subprocess that runs the build backend, but they are put _later_ in `sys.path` than an older version of `setuptools`. There's a `.pth` file in the latest `setuptools` that tries to do some magic, and the magic that it does is busted because the version of `setuptools` that's earlier on the module search path is an earlier version, incompatible with the `.pth` file.

A simpler reproducer of this than what I see above is:
```
python3.8 -m venv --system-site-packages venv38
./venv/bin/pip install --no-binary :all: --ignore-installed python-dateutil
```
which fails with:
```
Collecting python-dateutil
  Using cached python-dateutil-2.8.1.tar.gz (331 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 2:
   command: /home/matt/venv/bin/python /home/matt/venv/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7fsbsn8s/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools; python_version != '"'"'3.3'"'"'' 'setuptools<40.0; python_version == '"'"'3.3'"'"'' wheel setuptools_scm
       cwd: None
  Complete output (60 lines):
  Ignoring setuptools: markers 'python_version == "3.3"' don't match your environment
  Collecting setuptools
    Using cached setuptools-50.0.2.zip (2.2 MB)
  Collecting wheel
    Using cached wheel-0.35.1.tar.gz (59 kB)
  Collecting setuptools_scm
    Using cached setuptools_scm-4.1.2.tar.gz (48 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
  ERROR: Exception:
  Traceback (most recent call last):
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
      status = self.run(options, args)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
      return func(self, options, args)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 325, in run
      reqs, check_supported_wheels=not options.target_dir
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 340, in _get_abstract_dist_for
      abstract_dist = self.preparer.prepare_linked_requirement(req)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 483, in prepare_linked_requirement
      req, self.req_tracker, self.finder, self.build_isolation,
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 91, in _get_prepared_distribution
      abstract_dist.prepare_distribution_metadata(finder, build_isolation)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
      self._setup_isolation(finder)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/distributions/sdist.py", line 96, in _setup_isolation
      reqs = backend.get_requires_for_build_wheel()
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_vendor/pep517/wrappers.py", line 161, in get_requires_for_build_wheel
      'config_settings': config_settings
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_vendor/pep517/wrappers.py", line 265, in _call_hook
      raise BackendUnavailable(data.get('traceback', ''))
  pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 86, in _build_backend
      obj = import_module(mod_path)
    File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 994, in _gcd_import
    File "<frozen importlib._bootstrap>", line 971, in _find_and_load
    File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 994, in _gcd_import
    File "<frozen importlib._bootstrap>", line 971, in _find_and_load
    File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 678, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 5, in <module>
      import distutils.core
    File "/tmp/pip-build-env-zel77eqn/overlay/lib/python3.6/site-packages/_distutils_hack/__init__.py", line 83, in create_module
      return importlib.import_module('setuptools._distutils')
    File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  ModuleNotFoundError: No module named 'setuptools._distutils'

  ----------------------------------------
ERROR: Command errored out with exit status 2: /home/matt/venv/bin/python /home/matt/venv/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7fsbsn8s/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools; python_version != '"'"'3.3'"'"'' 'setuptools<40.0; python_version == '"'"'3.3'"'"'' wheel setuptools_scm Check the logs for full command output.
```

This failure is happening because there are two versions of `setuptools` on the search path, an older one (from the `site-packages` directory of the interpreter that created the virtualenv) and a newer one (installed as part of the build requirements for the PEP 517 build). The newer one is shipped along with a `.pth` file that tries to import `setuptools._distutils`. The `.pth` file runs even though another version of `setuptools` is before it in the module search path, but the import it performs fails because the old version, earlier on the search path, doesn't contain `setuptools._distutils`.
For completeness, when running from a virtualenv with `--system-site-packages`, the `sys.path` that I see in the subprocess when the `get_requires_for_build_wheel` hook is called is something like:
```py
['/home/matt/some_venv/lib/python3.8/site-packages/pip/_vendor/pep517',
 '/tmp/pip-build-env-5v90m1w9/site',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '/usr/local/lib/python3.8/dist-packages',
 '/usr/lib/python3/dist-packages',
 '/tmp/pip-build-env-5v90m1w9/overlay/lib/python3.8/site-packages',
 '/tmp/pip-build-env-5v90m1w9/normal/lib/python3.8/site-packages']
```
When the virtualenv wasn't created with `--system-site-packages`, I see:
```py
['/home/matt/venv2/lib/python3.8/site-packages/pip/_vendor/pep517',
 '/tmp/pip-build-env-lc6d9yps/site',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '/tmp/pip-build-env-lc6d9yps/overlay/lib/python3.8/site-packages',
 '/tmp/pip-build-env-lc6d9yps/normal/lib/python3.8/site-packages']
```
In https://github.com/pypa/setuptools/issues/2353#issuecomment-686844357, I've confirmed the issue does not require a virtualenv and can be replicated by simply ensuring that the package being installed has a dependency on a version of setuptools that's incompatible with the installed setuptools.
I wonder if python-build’s isolated environment implementation would have this problem. That project would be the best place for this functionality to live in IMO. cc @gaborbernat @FFY00
@uranusjr unless pip plans to use ``python-build`` as the wheel builder tool I don't see how ``python-build`` will help this issue 🤔 
That’s exactly my plan 😛 pip already uses pep517 under the hood, and the main reason it does not also use its isolated environment implementation is because pep517 maintainers don’t want anything outside of the core PEP 517 interface to be depended on. It would be best IMO if the isolated environment implementation is maintained outside of pip (so other projects can reuse it), and python-build would be in the perfect position since it does the exactly thing pip needs, and already plans to join PyPA.
> pep517 maintainers don’t want anything outside of the core PEP 517 interface to be depended on

Oh, yes -- we recently removed all of that code: https://github.com/pypa/pep517/pull/83, in preparation for python-build. :)

python-build is ready in my opinion, just waiting on some feedback and then we can make a new release.
Oh my independent how ready python build is we should offer a six month deprecation notice before we remove a core functionality... If we don't want to angry people using it. 
Can anyone confirm whether python-build have this particular problem though? All discussion would be moot (for this issue specifically) if the implementation is not better than pip’s.
It's a bit late and I am not on the computer right now, but it seems to me that python-build would not have this issue, since it does not put the site directories in `PYTHONPATH`, instead it symlinks them to the environment and sets `PYTHONHOME`. But please test this, I am not in a state to judge the issue properly right now, so I may have misinterpreted something 😝 .
We've been running into possibly-the-same-issue. There's a complete minimal repro repo at https://github.com/dHannasch/PyPy-CFFI, trying to pip-install on Travis and failing, if anyone's still lacking for that.

There's a lot of diagnostic output in the Travis log for anyone who wants it, but the key point is:
```
$ python -c "import setuptools; print(setuptools.__version__)"
41.0.1
The command "python -c "import setuptools; import setuptools.build_meta"" exited with 0.
The command "env-with-site-packages/bin/pip install . --no-build-isolation" exited with 0.
$ env-with-site-packages/bin/pip install .
Processing /home/travis/build/dHannasch/PyPy-CFFI
  Installing build dependencies ... |\/-done
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/home/travis/build/dHannasch/PyPy-CFFI/env-with-site-packages/site-packages/pip/_internal/distributions/sdist.py", line 96, in _setup_isolation
    reqs = backend.get_requires_for_build_wheel()
  File "/home/travis/build/dHannasch/PyPy-CFFI/env-with-site-packages/site-packages/pip/_vendor/pep517/wrappers.py", line 161, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/home/travis/build/dHannasch/PyPy-CFFI/env-with-site-packages/site-packages/pip/_vendor/pep517/wrappers.py", line 265, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/home/travis/build/dHannasch/PyPy-CFFI/env-with-site-packages/site-packages/pip/_vendor/pep517/_in_process.py", line 86, in _build_backend
    obj = import_module(mod_path)
  File "/home/travis/build/dHannasch/PyPy-CFFI/env-with-site-packages/lib-python/3/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'setuptools.build_meta'
```
I'm not 100% sure whether this should actually be a separate issue. It looks like this might be just the same issue, due to the Travis automatic virtualenv.
I get
```
Installing collected packages: brishgarden
  Running setup.py develop for brishgarden
    ERROR: Command errored out with exit status 1:
     command: /Users/evar/anaconda/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/evar/Base/_Code/python/BrishGarden/setup.py'"'"'; __file__='"'"'/Users/evar/Base/_Code/python/BrishGarden/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /Users/evar/Base/_Code/python/BrishGarden/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/evar/anaconda/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/evar/Base/_Code/python/BrishGarden/setup.py'"'"'; __file__='"'"'/Users/evar/Base/_Code/python/BrishGarden/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
```
with `pip 21.0.1` and `setuptools (54.2.0)`. This only happens if I use `pip install -e .`, not `pip install .`.
Also having this issue with `pip 21.0` and `setuptools 40.6.2`.

```
  Downloading django-modeltranslation-0.16.2.tar.gz (69 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /home/ec2-user/.virtualenvs/myvenv/bin/python3.6 /home/ec2-user/.virtualenvs/myvenv/lib64/python3.6/dist-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpl4vqe8cx
       cwd: /tmp/pip-install-liqkaggc/django-modeltranslation_8e2213e90aaf48b28e05683d729bab96
  Complete output (10 lines):
  Traceback (most recent call last):
    File "/home/ec2-user/.virtualenvs/myvenv/lib64/python3.6/dist-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/home/ec2-user/.virtualenvs/myvenv/lib64/python3.6/dist-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/ec2-user/.virtualenvs/myvenv/lib64/python3.6/dist-packages/pip/_vendor/pep517/_in_process.py", line 108, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/home/ec2-user/.virtualenvs/myvenv/lib64/python3.6/dist-packages/pip/_vendor/pep517/_in_process.py", line 99, in _build_backend
      obj = getattr(obj, path_part)
  AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'
  ```
Okay, yet another data point. I encountered this again for reasons that I'm not entirely clear about. However, I noticed that I'd done `pip install nvidia-pyindex` around the time things started going downhill, and this (apparently) created `~/.pip/pip.conf` and `~/.config/pip/pip.conf`. Those files referred to a temporary virtual-env I'd set up, although the lines mentioning it were comments. I removed those files, and my pip seems to be working again.

In the meantime I also did something like the following, admittedly in desperation:

```
$ cd /tmp
$ wget https://bootstrap.pypa.io/get-pip.py
$ deactivate
$ python3 /tmp/get-pip.py
Defaulting to user installation because normal site-packages is not writeable
Collecting pip
...elided for brevity...
$ which pip
/home/austinjp/.local/bin/pip
$ python3.6 /tmp/get-pip.py
$ python3.7 /tmp/get-pip.py
$ python3.8 /tmp/get-pip.py
$ sudo su -
# python3 /tmp/get-pip.py
Collecting pip
...elided for brevity...
# which pip
/usr/local/bin/pip
# python3.6 /tmp/get-pip.py
# python3.7 /tmp/get-pip.py
# python3.8 /tmp/get-pip.py
# ^D
$ 
```

Probably not my greatest moment. I also ran `apt list  | grep -i installed | grep -i python` and tried upgrading anything that looked in need of a poke, which I don't think actually necessitated any upgrades. Fun times :smiley: 

I was pulling my hair out over this yesterday and tried various things, so I can't easily replay the situation that led up to it. But for posterity, here's the error I was encountering when using a venv with **or without** `--system-site-packages`:

```
$ pip install -U pip
Exception:
Traceback (most recent call last):
  File "/home/austinjp/whatever/lib/python3.8/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/austinjp/whatever/lib/python3.8/site-packages/pip/commands/install.py", line 290, in run
    with self._build_session(options) as session:
  File "/home/austinjp/whatever/lib/python3.8/site-packages/pip/basecommand.py", line 68, in _build_session
    normalize_path(os.path.join(options.cache_dir, "http"))
  File "/usr/lib/python3.8/posixpath.py", line 76, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not int
```

And in case it's of any use, the contents of the `pip.conf` files that were created by NVIDIA's package:

```
$ cat ~/.pip/pip.conf 
# This file has been autogenerated or modified by NVIDIA PyIndex.
# In case you need to modify your PIP configuration, please be aware that
# some configuration files may have a priority order. Here are the following 
# files that may exists in your machine by order of priority:
#
# [Priority 1] Site level configuration files
#	1. `/tmp/8bCjAO1V/venv/pip.conf`
#
# [Priority 2] User level configuration files
#	1. `/home/austinjp/.config/pip/pip.conf`
#	2. `/home/austinjp/.pip/pip.conf`
#
# [Priority 3] Global level configuration files
#	1. `/etc/pip.conf`
#	2. `/etc/xdg/pip/pip.conf`

[global]
no-cache-dir = true
index-url = https://pypi.org/simple
extra-index-url =
                  https://pypi.ngc.nvidia.com
trusted-host =
               pypi.ngc.nvidia.com
```

I noticed this by running `pip config debug`:

```
$ pip config debug
env_var:
env:
global:
  /etc/xdg/pip/pip.conf, exists: False
  /etc/pip.conf, exists: False
site:
  /usr/pip.conf, exists: False
user:
  /home/austinjp/.pip/pip.conf, exists: True
    global.no-cache-dir: true
    global.index-url: https://pypi.org/simple
    global.extra-index-url: 
    https://pypi.ngc.nvidia.com
    global.trusted-host: 
    pypi.ngc.nvidia.com
  /home/austinjp/.config/pip/pip.conf, exists: True
    global.no-cache-dir: true
    global.index-url: https://pypi.org/simple
    global.extra-index-url: 
    https://pypi.ngc.nvidia.com
    global.trusted-host: 
    pypi.ngc.nvidia.com
```

I suspect the NVIDIA thing might be a red-herring, but I don't know for sure.

@uranusjr are you still interested in using pypa/build for this? Is there anything we can help with?
Re-reading comments above, I think we still lack concrete confirmation whether `build` would actually solve this?
Well, build doesn't expose the system site packages by virtue of creating a new virtual environment for every build. @FFY00's previous comment about `PYTHONHOME` is from when build wasn't using virtual environments.
Sounds like good news! Do you by chance know at hand how much slowdown is expected if pip uses virtual environments instead, and whether build currently uses venv or virtualenv for creating virtual environments? (Both have some issues IIRC. venv is pretty slow the last time I checked. virtualenv was difficult to vendor at least before 20.0, not sure what the status is right now.)

Also, how does build populate dependencies in the virtual environment right now? Pip's isolated environment injects the currently running pip into the environment for dependency population to cover some edge cases, and we need to keep that behaviour when switching to a "real" virtual environment.
build uses `virtualenv` if it's available and falls back to `venv`.  I don't have any numbers at hand, but `venv` creation takes a couple of seconds at a minimum. virtualenv uses a cache from what I know to speed the process up.

build shells out to `venv` and `virtualenv`, so the pip version that'll be installed will be either the pip version that comes with `ensurepip` or the pip version bundled in `virtualenv`.

> Do you by chance know at hand how much slowdown is expected if pip uses virtual environments instead, and whether build currently uses venv or virtualenv for creating virtual environments?

It can use both. It will prefer virtualenv if available. There may be some issues with vendoring because of the way we select venv or virtualenv, but we can work on that.

> Also, how does build populate dependencies in the virtual environment right now?

In venv, we run ensurepip, update pip if necessary, and remove setuptools. In virtualenv, it installs pip by default, I am not sure how it selects the version.
Hmm. Considering potential vendoring difficulties, I think we need to use venv. Without reading buid's implementation, pip will likely need to provide custom logic to

1. Override virtualenv detection (to avoid unexpected behavioural change when user has virtualenv installed)
2. Override venv creation to *not* call ensurepip
3. Implement post-environment creation logic to inject the correct pip version
I think the best way forward might be to bring your own `IsolatedEnv` implementation (see https://github.com/pypa/build/blob/cccaf931f88dff9e21ce89570e017278e7364dc6/src/build/env.py#L38) but that will require some refactoring on our part to subsume env creation under `IsolatedEnv`.  I expect you'll also want to customise the `install` method to suit your needs.
See the draft at https://github.com/pypa/build/pull/361.
I'm cautiously wary about the performance impact of using `venv` here -- part of why we ended up with the approach that we've got, is because generating a venv is fairly expensive. That said, we also zip up the running copy of pip; so... 🤷🏽 
Most of the time when using `venv` to create a virtual environment is spent on ensurepip, so if we are manually provisioning pip, as @uranusjr said we must, it could be fairly fast.
If you can't use virtualenv then I've done something wrong with it... So do
let me know why the solution isn't to just use virtualenv.

On Tue, 28 Sep 2021, 20:35 Pradyun Gedam, ***@***.***> wrote:

> I'm cautiously wary about the performance impact of using venv here --
> part of why we ended up with the approach that we've got, is because
> generating a venv is fairly expensive.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pypa/pip/issues/6264#issuecomment-929563474>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAFIQPU4O2JIFS2VMDBYGCDUEIKJJANCNFSM4GXGBDTQ>
> .
>

`venv --without-pip` installs `python.exe`, `pythonw.exe`, `pyvenv.cfg` and 4 activaion scripts on my Windows PC, and takes essentially no time. I think it's perfectly acceptable. It's not like the "old days" when creating a virtual environment installed a big chunk of the stdlib.

> If you can't use virtualenv then I've done something wrong with it

I don't think it's that we can't use virtualenv, but rather (if we're not installing pip etc) that we don't need it - `venv --without-pip` is perfectly sufficient, and comes with the stdlib, so there are no vendoring issues.
here's a hacky patch which fixes the issue by avoiding site in the isolated environment:

```diff
$ git show | cat
commit 4c10a5c1bdea62a506ace8e94e87eabc1fd45922
Author: Anthony Sottile <asottile@umich.edu>
Date:   Tue Dec 28 06:53:42 2021 -0800

    hack: workaround broken setuptools distutils pth

diff --git a/src/pip/_internal/build_env.py b/src/pip/_internal/build_env.py
index d326dc8cd..d9f97f52b 100644
--- a/src/pip/_internal/build_env.py
+++ b/src/pip/_internal/build_env.py
@@ -101,21 +101,6 @@ class BuildEnvironment:
                     """
                 import os, site, sys
 
-                # First, drop system-sites related paths.
-                original_sys_path = sys.path[:]
-                known_paths = set()
-                for path in {system_sites!r}:
-                    site.addsitedir(path, known_paths=known_paths)
-                system_paths = set(
-                    os.path.normcase(path)
-                    for path in sys.path[len(original_sys_path):]
-                )
-                original_sys_path = [
-                    path for path in original_sys_path
-                    if os.path.normcase(path) not in system_paths
-                ]
-                sys.path = original_sys_path
-
                 # Second, add lib directories.
                 # ensuring .pth file are processed.
                 for path in {lib_dirs!r}:
diff --git a/src/pip/_vendor/pep517/in_process/_in_process.py b/src/pip/_vendor/pep517/in_process/_in_process.py
index 954a4ab05..c3e5d5bcd 100644
--- a/src/pip/_vendor/pep517/in_process/_in_process.py
+++ b/src/pip/_vendor/pep517/in_process/_in_process.py
@@ -83,6 +83,7 @@ def _build_backend():
         extra_pathitems = backend_path.split(os.pathsep)
         sys.path[:0] = extra_pathitems
 
+    import sitecustomize
     ep = os.environ['PEP517_BUILD_BACKEND']
     mod_path, _, obj_path = ep.partition(':')
     try:
diff --git a/src/pip/_vendor/pep517/wrappers.py b/src/pip/_vendor/pep517/wrappers.py
index e031ed708..223e24b44 100644
--- a/src/pip/_vendor/pep517/wrappers.py
+++ b/src/pip/_vendor/pep517/wrappers.py
@@ -320,7 +320,7 @@ class Pep517HookCaller(object):
             with _in_proc_script_path() as script:
                 python = self.python_executable
                 self._subprocess_runner(
-                    [python, abspath(str(script)), hook_name, td],
+                    [python, '-S', abspath(str(script)), hook_name, td],
                     cwd=self.source_dir,
                     extra_environ=extra_environ
                 )
```

this appears to "fix" today's breakage with setuptools 61
In #11004 I created a reproduction for flit as well (not only setuptools). I don't know if it is useful but it can be found bellow:

```bash
> docker run --rm -it python:3.10-alpine3.14 /bin/ash


mkdir /tmp/proj
cd /tmp/proj
cat <<EOS > pyproject.toml
[build-system]
requires = ["flit_core >=3.2,<4"]
build-backend = "flit_core.buildapi"

[project]
name = "foobar"
version = "42"
description = "foobar"
EOS
touch foobar.py

# Create a virtual env with the right version of the build requirements
python3 -m venv --system-site-packages .venv
.venv/bin/python -m pip install -U pip "flit_core >=3.2,<4"
.venv/bin/python -m pip list
# Package    Version
# ---------- -------
# flit_core  3.7.1
# pip        22.0.4
# setuptools 58.1.0
# wheel      0.37.1

# Install an old version of flit to system's site-packages
python3 -m pip install "flit_core<3"
# Run pip install from inside the virtual environment
.venv/bin/python -m pip install .
```

Output:

```bash
Processing /tmp/proj
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      Traceback (most recent call last):
        File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/tmp/proj/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/usr/local/lib/python3.10/site-packages/flit_core/buildapi.py", line 19, in get_requires_for_build_wheel
          info = read_flit_config(pyproj_toml)
        File "/usr/local/lib/python3.10/site-packages/flit_core/inifile.py", line 60, in read_flit_config
          return prep_toml_config(d, path)
        File "/usr/local/lib/python3.10/site-packages/flit_core/inifile.py", line 84, in prep_toml_config
          raise ConfigError("TOML file missing [tool.flit] table.")
      flit_core.inifile.ConfigError: TOML file missing [tool.flit] table.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.


Hi guys, there seems to be other edge cases for which `pip` is loading a different version of the build-backend from the system's site/dist-packages, even when no virtual environment with `--system-site-packages` is envolved.
The following code snippet illustrates how to reach the edge case:

```bash
> docker run --rm -it debian:11 /bin/bash

apt update && apt install -y python3 python3-dev curl
cd /tmp
curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py --no-setuptools --no-wheel
python3 -m pip install -Uv setuptools==61.3.1
python3 -m pip list
# Package    Version
# ---------- -------
# pip        22.0.4
# setuptools 61.3.1

## Note that `setuptools` is installed in `/usr/local/lib/python3.9/dist-packages/setuptools`
python3 -c 'import setuptools; print(setuptools.__path__, setuptools.__version__)'
# ['/usr/local/lib/python3.9/dist-packages/setuptools'] 61.3.1

## Create a dummy package:
mkdir /tmp/proj
cd /tmp/proj
cat <<EOS > pyproject.toml
[build-system]
requires = ['setuptools==59.2.0']
build-backend = 'setuptools.build_meta'
EOS
cat <<EOS > setup.cfg
[metadata]
name = proj
version = 42
EOS
mkdir dir1 dir2
touch dir1/__init__.py
touch dir2/__init__.py


## Install
pip install --use-pep517 --isolated .
```

Output:

```bash
Processing /tmp/proj
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      error: Multiple top-level packages discovered in a flat-layout: ['dir2', 'dir1'].

      To avoid accidental inclusion of unwanted files or directories,
      setuptools will not proceed with this build.

      If you are trying to create a single distribution with multiple packages
      on purpose, you should not rely on automatic discovery.
      Instead, consider the following options:

      1. set up custom discovery (`find` directive with `include` or `exclude`)
      2. use a `src-layout`
      3. explicitly set `py_modules` or `packages` with a list of names

      To find more information, look for "package discovery" on setuptools docs.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
```

Note that `error: Multiple top-level packages discovered in a flat-layout` was introduced in version 61 of setuptools. Therefore the build backend is being loaded from the system/dist site-packages instead of the ephemeral isolated environment as expected from PEP 517[^1].

Also note that the same error happens if we remove any `.pth` file that could in theory be messing with the PYTONPATH:
```bash
rm -rf /usr/local/lib/python3.9/dist-packages/*.pth
rm -rf /usr/local/lib/python3.9/site-packages/*.pth
```

Is this behaviour expected/intentional?
Please let me know if you would like me to open a separated issue for that, since it is not related to `--system-site-packages`.

This is related to https://github.com/numpy/numpy/issues/21288


[^1]: According to [pip docs](https://pip.pypa.io/en/stable/cli/pip_install/):
`--use-pep517`: Use PEP 517 for building source distribution
> Hi guys, there seems to be other edge cases for which pip is loading a different version of the build-backend from the system's site/dist-packages, even when no virtual environment with --system-site-packages is envolved.

Can you reproduce this in a non-debian based environment? There's definitely some issues here, but it would be easier to determin what is a pip issue and what isn't if the reproducers don't involve debian's patches to Python and pip (specifically, the debian-specific `dist-packages` directory seems to be involved in your reproducer...)
Thank you very much @pfmoore. I am afraid I can't, I tried to reproduce it with `python:3.10-alpine3.14` but the build succeeds.

So I will assume that this particular problem is Debian specific and related to the patches. Please feel free to ignore my previous comment.
It's still worth having the reproducer, when we find out the cause of the main issue here we can check if it addresses the debian case as well.
Regarding Debian-specific, in my experiments I've only seen this on Debian 11, not on CentOS 7 and not on Ubuntu 18.04.
Note those 2 are pinned to setuptools 59 for python3.6.

(Ah...python dies before the carrying distro, let's another annoying story...)
@pradyunsg just to clarify a misconception that the new issue title might cause (as observed in https://github.com/pypa/pip/issues/11004#issuecomment-1087038461):

The issue occurs in 2 situations right?
a) when a virtual environment is created with --system-site-packages ***`OR`***
b) when using a Debian-based distribution without virtual environment.

So far https://github.com/pypa/pip/issues/6264#issuecomment-1086882745 cannot be reproduced outside Debian (at least I can't).

But https://github.com/pypa/pip/issues/6264#issuecomment-1086882620 can (the example shows it being built on Alpine).

I'd retitled this based on what you'd said in https://github.com/pypa/pip/issues/6264#issuecomment-1086884338. 

> I will assume that this particular problem is Debian specific and related to the patches.

If that's not the case, then, fair.
Oh sorry, that was a specific reply to Paul's comment on the last example, not meant for the entire discussion. Sorry for the confusion.

You can indeed reproduce the problem of `--system-site-packages` with other distributions, as shown in https://github.com/pypa/pip/issues/6264#issuecomment-1086882620.
> It's still worth having the reproducer, when we find out the cause of the main issue here we can check if it addresses the debian case as well.

I think the cause is clear. pip implements build isolation by removing the current platlib and purelib directories (and any directories from `.pth` files in those directories) from `sys.path` using the code [here](https://github.com/pypa/pip/blob/22.0.4/src/pip/_internal/build_env.py#L107). Therefore, if there are any system site directories that are not covered by that, they remain in `sys.path` in the build environment.

* In the case of a virtual environment with `--system-site-packages`, the current platlib/purelib directories are the ones inside the venv, so any _system_ site directories are not removed. I expect this to happen on any distribution.

* In the case of Debian, the system Python has an extra site directory, `/usr/local/lib/python3.6/dist-packages`, which is neither the platlib nor purelib directory, so it's not removed.

It seems to me that the simplest fix would be to remove all directories in `site.getsitepackages()` from `sys.path` and not just platlib/purelib.

I believe pypa/setuptools#3291 is another kind of this issue. Note that it neither involves virtualenv, nor Debian/Centos, but sympthoms are exactly the same: system-wide older version of setuptools has higher priority than one that is requested by `pypackage.toml`.
Inside a Debian GNU/Linux 10 (buster) Docker container we create a virtual-env with `--system-site-packages`
Inside we have `pip == 19.2.1`
We are using pyproject.toml, requirements.txt and setup.cfg and install the latest `setuptools == 63.2.0` using the `pyproject.toml`

We have the following code in the setup.cfg:
```
[options]
packages = find:
install_requires = file: requirements.txt
```

Now we hit the error where the `get_requires_for_build_wheel` fails with parsing error of the above setup.cfg.

```
File "/usr/lib/python3/dist-packages/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__
      requirement_string[e.loc:e.loc + 8]))
      pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "': requir'"
```

Upon inspection we found out that pip instead of loading the latest setuptools from the build environment loaded the setuptools == 40.8.0 from dist-packages and used the build_meta from 
`/usr/lib/python3/dist-packages/setuptools/build_meta.py`

For those struggling with same bug using pip and venv on Debian distros, the current workaround we are using is to either uninstall the `python3-setuptools` or try to use the `--no-build-isolation` flag with pip

Is there anyone working on a fix for this issue it seems to be open for awhile, is there any relevant PR to fix issues here, it doesn't look like easy to provide a minimal working example tho.

I'm on ArchLinux and have encountered this issue btw while installing/adding Pendulum to my dependencies using Poetry.
@AkechiShiro not that I know of. The [comment above](https://github.com/pypa/pip/issues/6264#issuecomment-1088660972) from @IRDonch suggests a possible fix, but we'd need a test, so working out how to reproduce this is still a blocker here (as is someone actually preparing a PR for it, I don't believe any of the pip developers are working on it).
Thanks @pfmoore, could anyone guide me where I could try and dev this fix, in which folders is the buggy code, in the coming months, I hope I can get to fix this, I'm just not very knowledgeable with the code base, it's my first time working on pip, I'll be looking at the CONTRIBUTING.md probably present in the repo.
> The [comment above](https://github.com/pypa/pip/issues/6264#issuecomment-1088660972) from @IRDonch suggests a possible fix

FYI, I tried to implement this fix, but I was blocked by #8273. virtualenv 19 includes a custom `site` module, which doesn't implement the `getsitepackages` function.

FWIW, this appears to affect mac system Python (macOS 12.6, Python 3.9.6) as well.

<details>

<summary>pyproject.toml</summary>

```toml
[build-system]
requires = [
    "setuptools>=60",
]
build-backend = "setuptools.build_meta"

[project]
version = "0.1.0"
requires-python = ">=3.7"
license = {text = "MIT License"}
name = "debug"
description = "..."
```
</details>
<details>

<summary>setup.py</summary>


```python
from setuptools import setup
import setuptools
import sys

print("\n\n")
print(f"in setup.py: {setuptools.__version__=}, {setuptools.__file__=}")
path_lines = '\n  '.join(sys.path)
print(f"sys.path=\n  {path_lines}")
print("\n\n")

setup()
```

</details>
<details>

<summary>pip list</summary>

```
$ /usr/bin/python3 -m pip install --upgrade --user setuptools pip
Requirement already satisfied: setuptools in /Users/minrk/Library/Python/3.9/lib/python/site-packages (65.3.0)
Requirement already satisfied: pip in /Users/minrk/Library/Python/3.9/lib/python/site-packages (22.2.2)
$ /usr/bin/python3 -m pip list
Package    Version
---------- -------
altgraph   0.17.2
future     0.18.2
macholib   1.15.2
pip        22.2.2
setuptools 65.3.0
six        1.15.0
UNKNOWN    0.0.0
wheel      0.37.0
```

</details>
<details>

<summary>Building wheel with pip:</summary>


```
$ /usr/bin/python3 -m pip wheel -v .
Processing /private/tmp/test-package
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Collecting setuptools>=60
    Using cached setuptools-65.3.0-py3-none-any.whl (1.2 MB)
  Installing collected packages: setuptools
  Successfully installed setuptools-65.3.0
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Running command Getting requirements to build wheel



  in setup.py: setuptools.__version__='58.0.4', setuptools.__file__='/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /Users/minrk/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/normal/lib/python3.9/site-packages



  running egg_info
  writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Running command pip subprocess to install backend dependencies
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Installing collected packages: wheel
  Successfully installed wheel-0.37.1
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)



  in setup.py: setuptools.__version__='58.0.4', setuptools.__file__='/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /Users/minrk/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/normal/lib/python3.9/site-packages



  running dist_info
  creating /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-emod82r0/UNKNOWN.egg-info
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-emod82r0/UNKNOWN.egg-info/SOURCES.txt'
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-emod82r0/UNKNOWN.egg-info/SOURCES.txt'
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: UNKNOWN
  Building wheel for UNKNOWN (pyproject.toml): started
  Running command Building wheel for UNKNOWN (pyproject.toml)



  in setup.py: setuptools.__version__='58.0.4', setuptools.__file__='/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /Users/minrk/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-y1d328ei/normal/lib/python3.9/site-packages



  running bdist_wheel
  running build
  running install
  running install_egg_info
  running egg_info
  writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
  Copying UNKNOWN.egg-info to build/bdist.macosx-10.9-universal2/wheel/UNKNOWN-0.0.0-py3.9.egg-info
  running install_scripts
  Building wheel for UNKNOWN (pyproject.toml): finished with status 'done'
  Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=964 sha256=75197df67b739145c46f3017b89c662f98118e90f27a93a29a356b76b7b10af0
  Stored in directory: /Users/minrk/Library/Caches/pip/wheels/63/83/5c/e98bf9706ef6a83be6f3ec105be9771fbd8f4d9d70f9257288
Successfully built UNKNOWN
```
</details>
<details>

<summary>And again without build isolation, fixes the issue:</summary>

```
$ /usr/bin/python3 -m pip wheel -v --no-build-isolation .
Processing /private/tmp/test-package
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)



  in setup.py: setuptools.__version__='65.3.0', setuptools.__file__='/Users/minrk/Library/Python/3.9/lib/python/site-packages/setuptools/__init__.py'
  sys.path=
    /Users/minrk/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /Users/minrk/Library/Python/3.9/lib/python/site-packages
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages



  running dist_info
  creating /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info
  writing /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/top_level.txt
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/SOURCES.txt'
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug.egg-info/SOURCES.txt'
  creating '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-vazupnbh/debug-0.1.0.dist-info'
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: debug
  Building wheel for debug (pyproject.toml): started
  Running command Building wheel for debug (pyproject.toml)



  in setup.py: setuptools.__version__='65.3.0', setuptools.__file__='/Users/minrk/Library/Python/3.9/lib/python/site-packages/setuptools/__init__.py'
  sys.path=
    /Users/minrk/Library/Python/3.9/lib/python/site-packages/pip/_vendor/pep517/in_process
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /Users/minrk/Library/Python/3.9/lib/python/site-packages
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages



  running bdist_wheel
  running build
  installing to build/bdist.macosx-10.9-universal2/wheel
  running install
  running install_egg_info
  running egg_info
  creating debug.egg-info
  writing debug.egg-info/PKG-INFO
  writing dependency_links to debug.egg-info/dependency_links.txt
  writing top-level names to debug.egg-info/top_level.txt
  writing manifest file 'debug.egg-info/SOURCES.txt'
  reading manifest file 'debug.egg-info/SOURCES.txt'
  writing manifest file 'debug.egg-info/SOURCES.txt'
  Copying debug.egg-info to build/bdist.macosx-10.9-universal2/wheel/debug-0.1.0-py3.9.egg-info
  running install_scripts
  creating build/bdist.macosx-10.9-universal2/wheel/debug-0.1.0.dist-info/WHEEL
  creating '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-wheel-buzyl83a/tmped7n05ks/debug-0.1.0-py3-none-any.whl' and adding 'build/bdist.macosx-10.9-universal2/wheel' to it
  adding 'debug-0.1.0.dist-info/METADATA'
  adding 'debug-0.1.0.dist-info/WHEEL'
  adding 'debug-0.1.0.dist-info/top_level.txt'
  adding 'debug-0.1.0.dist-info/RECORD'
  removing build/bdist.macosx-10.9-universal2/wheel
  Building wheel for debug (pyproject.toml): finished with status 'done'
  Created wheel for debug: filename=debug-0.1.0-py3-none-any.whl size=957 sha256=8c9826d6865bb428a68051a817b34b1f87844cf9fa367b8379674d1fca9c381e
  Stored in directory: /Users/minrk/Library/Caches/pip/wheels/63/83/5c/e98bf9706ef6a83be6f3ec105be9771fbd8f4d9d70f9257288
Successfully built debug
```
</details>

<details>

<summary>
EDIT: running in a venv _does_ help
</summary>


```
$  /usr/bin/python3 -m venv /tmp/debug-env
$ source /tmp/debug-env/bin/activate
(debug-env) $  python3 -m pip wheel -v .
Processing /Users/minrk/dev/py/6d73fc9159448f1f83f8a6d5c2b2dfac
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  Collecting setuptools>=60
    Using cached setuptools-65.3.0-py3-none-any.whl (1.2 MB)
  Installing collected packages: setuptools
  Successfully installed setuptools-65.3.0
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Running command Getting requirements to build wheel



  in setup.py: setuptools.__version__='65.3.0', setuptools.__file__='/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /private/tmp/debug-env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/normal/lib/python3.9/site-packages
  site.getsitepackages()=
    /private/tmp/debug-env/lib/python3.9/site-packages



  running egg_info
  writing debug.egg-info/PKG-INFO
  writing dependency_links to debug.egg-info/dependency_links.txt
  writing top-level names to debug.egg-info/top_level.txt
  reading manifest file 'debug.egg-info/SOURCES.txt'
  writing manifest file 'debug.egg-info/SOURCES.txt'
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Running command pip subprocess to install backend dependencies
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Installing collected packages: wheel
  Successfully installed wheel-0.37.1
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)



  in setup.py: setuptools.__version__='65.3.0', setuptools.__file__='/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /private/tmp/debug-env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/normal/lib/python3.9/site-packages
  site.getsitepackages()=
    /private/tmp/debug-env/lib/python3.9/site-packages



  running dist_info
  creating /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info
  writing /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/dependency_links.txt
  writing top-level names to /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/top_level.txt
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/SOURCES.txt'
  writing manifest file '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug.egg-info/SOURCES.txt'
  creating '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-modern-metadata-knoxvi11/debug-0.1.0.dist-info'
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: debug
  Building wheel for debug (pyproject.toml): started
  Running command Building wheel for debug (pyproject.toml)



  in setup.py: setuptools.__version__='65.3.0', setuptools.__file__='/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages/setuptools/__init__.py'
  sys.path=
    /private/tmp/debug-env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/site
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
    /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/overlay/lib/python3.9/site-packages
    /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-build-env-e83b76k7/normal/lib/python3.9/site-packages
  site.getsitepackages()=
    /private/tmp/debug-env/lib/python3.9/site-packages



  running bdist_wheel
  running build
  installing to build/bdist.macosx-10.9-universal2/wheel
  running install
  running install_egg_info
  running egg_info
  writing debug.egg-info/PKG-INFO
  writing dependency_links to debug.egg-info/dependency_links.txt
  writing top-level names to debug.egg-info/top_level.txt
  reading manifest file 'debug.egg-info/SOURCES.txt'
  writing manifest file 'debug.egg-info/SOURCES.txt'
  Copying debug.egg-info to build/bdist.macosx-10.9-universal2/wheel/debug-0.1.0-py3.9.egg-info
  running install_scripts
  creating build/bdist.macosx-10.9-universal2/wheel/debug-0.1.0.dist-info/WHEEL
  creating '/private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-wheel-9il5vyyn/tmp6zqsfanh/debug-0.1.0-py3-none-any.whl' and adding 'build/bdist.macosx-10.9-universal2/wheel' to it
  adding 'debug-0.1.0.dist-info/METADATA'
  adding 'debug-0.1.0.dist-info/WHEEL'
  adding 'debug-0.1.0.dist-info/top_level.txt'
  adding 'debug-0.1.0.dist-info/RECORD'
  removing build/bdist.macosx-10.9-universal2/wheel
  Building wheel for debug (pyproject.toml): finished with status 'done'
  Created wheel for debug: filename=debug-0.1.0-py3-none-any.whl size=959 sha256=b5f9cb097a864dd577a7cea3f7805b913d1968bbad235754d9d0293ded1e361c
  Stored in directory: /private/var/folders/qr/3vxfnp1x2t1fw55dr288mphc0000gn/T/pip-ephem-wheel-cache-qx6qpam5/wheels/0b/37/75/5000e2f11e9ea9460d2d869496734ee8dffb476f3090f04947
Successfully built debug
```
</details>
Minimal repro with docker on ubuntu:22.04: https://gist.github.com/minrk/6d73fc9159448f1f83f8a6d5c2b2dfac
I am seeing the system-wide paths have the wrong precedence: `/usr/lib/python3.6/site-packages/*` take precedence over the `/usr/local/lib/python3.6/*` for building my package for a tox environment in this case.  Does that fall in line with what others are seeing?

Resolving patch:
diff --git a/src/pip/_internal/build_env.py b/src/pip/_internal/build_env.py
--- a/src/pip/_internal/build_env.py
+++ b/src/pip/_internal/build_env.py
@@ -4,6 +4,7 @@
 import logging
 import os
 import pathlib
+import site
 import sys
 import textwrap
 from collections import OrderedDict
@@ -55,6 +56,26 @@ def get_runnable_pip() -> str:
     return os.fsdecode(source / "__pip-runner__.py")
 
 
+def _get_system_sitepackages() -> Set[str]:
+    """Get system site packages
+
+    Usually from site.getsitepackages,
+    but fallback on `get_purelib()/get_platlib()` if unavailable
+    (e.g. in a virtualenv created by virtualenv<20)
+
+    Returns normalized set of strings.
+    """
+    if hasattr(site, "getsitepackages"):
+        system_sites = site.getsitepackages()
+    else:
+        # virtualenv < 20 overwrites site.py without getsitepackages
+        # fallback on get_purelib/get_platlib.
+        # this is known to miss things, but shouldn't in the cases
+        # where getsitepackages() has been removed (inside a virtualenv)
+        system_sites = [get_purelib(), get_platlib()]
+    return {os.path.normcase(path) for path in system_sites}
+
+
 class BuildEnvironment:
     """Creates and manages an isolated environment to install build deps"""
 
@@ -75,9 +96,8 @@ def __init__(self) -> None:
         # Customize site to:
         # - ensure .pth files are honored
         # - prevent access to system site packages
-        system_sites = {
-            os.path.normcase(site) for site in (get_purelib(), get_platlib())
-        }
+        system_sites = _get_system_sitepackages()
+
         self._site_dir = os.path.join(temp_dir.path, "site")
         if not os.path.exists(self._site_dir):
             os.mkdir(self._site_dir)


Test patch:
diff --git a/tests/functional/test_build_env.py b/tests/functional/test_build_env.py
--- a/tests/functional/test_build_env.py
+++ b/tests/functional/test_build_env.py
@@ -4,7 +4,7 @@
 
 import pytest
 
-from pip._internal.build_env import BuildEnvironment
+from pip._internal.build_env import BuildEnvironment, _get_system_sitepackages
 from tests.lib import (
     PipTestEnvironment,
     TestPipResult,
@@ -226,6 +226,10 @@ def test_build_env_isolation(script: PipTestEnvironment) -> None:
     script.pip_install_local("-t", target, pkg_whl)
     script.environ["PYTHONPATH"] = target
 
+    system_sites = _get_system_sitepackages()
+    # there should always be something to exclude
+    assert system_sites
+
     run_with_build_env(
         script,
         "",
@@ -247,5 +251,14 @@ def test_build_env_isolation(script: PipTestEnvironment) -> None:
                     })), file=sys.stderr)
             print('sys.path:\n  ' + '\n  '.join(sys.path), file=sys.stderr)
             sys.exit(1)
+        """
+        f"""
+        # second check: direct check of exclusion of system site packages
+        import os
+
+        normalized_path = [os.path.normcase(path) for path in sys.path]
+        for system_path in {system_sites!r}:
+            assert system_path not in normalized_path, \
+            f"{{system_path}} found in {{normalized_path}}"
         """,
     )