Repository: pypa/pip (Python)

Issue:
22.1: Editable installs for packages with extras & flit causes `UndefinedEnvironmentName`
### Description

First of all, thanks for `pip` and for taking the time to read this issue!

We use editable installs on [sigstore-python](https://github.com/sigstore/sigstore-python) to provide a convenient development environment. For example, either with or without the `dev` extra:

```
pip install --editable .
pip install --editable .[dev]
```

Our dependencies are entirely managed within a `pyproject.toml`, with a small stub `setup.py` to help `pip` do an editable install.

Up until `pip==22.1`, these commands worked just fine. Upon upgrading to `22.1` (released today), we get this error:

```
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_vendor/packaging/markers.py", line 215, in _get_env
    raise UndefinedEnvironmentName(
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.
```

(I've truncated the traceback. I'll paste the full-length trace below.)

When I run with `--debug`, I see that the `dev` extra is being passed in as an environment marker, which then fails since `extra` is not included in the marker environment.

![image](https://user-images.githubusercontent.com/3059210/167964052-9a650ccc-8562-417e-bbe0-c9f5790738d2.png)

<img width="933" alt="Screen Shot 2022-05-11 at 7 34 06 PM" src="https://user-images.githubusercontent.com/3059210/167964111-4fef9399-51c8-4e82-9ed2-d1d31034dc46.png">



### Expected behavior

I expect `pip install --editable .` to behave the same as it did in the last stable release (22.0.4), i.e. install correctly with no errors.

### pip version

22.1

### Python version

3.10.0

### OS

macOS 12.3.1

### How to Reproduce

These steps should reproduce the failure:

```bash
git clone https://github.com/sigstore/sigstore-python && cd sigstore-python
make dev
```

If that fails to pull down the latest `pip` for whatever reason, you can do it explicitly:

```bash
git clone https://github.com/sigstore/sigstore-python && cd sigstore-python
python -m venv env && source env/bin/activate
pip install --upgrade pip
pip install --editable .
```

### Output

```sh-session
Obtaining file:///Users/william/devel/sigstore-python
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Getting requirements to build editable: started
  Getting requirements to build editable: finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 340, in run
    requirement_set = resolver.resolve(
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 75, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 506, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 188, in _make_candidate_from_link
    self._editable_candidate_cache[link] = EditableCandidate(
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 316, in __init__
    super().__init__(
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 161, in __init__
    self.dist = self._prepare()
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 230, in _prepare
    dist = self._prepare_distribution()
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 326, in _prepare_distribution
    return self._factory.preparer.prepare_editable_requirement(self._ireq)
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 543, in prepare_editable_requirement
    dist = _get_prepared_distribution(
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 57, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 45, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 114, in _install_build_reqs
    conflicting, missing = self.req.build_env.check_requirements(build_reqs)
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_internal/build_env.py", line 178, in check_requirements
    if req.marker is not None and not req.marker.evaluate():
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_vendor/packaging/markers.py", line 304, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_vendor/packaging/markers.py", line 234, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/Users/william/devel/sigstore-python/env/lib/python3.10/site-packages/pip/_vendor/packaging/markers.py", line 215, in _get_env
    raise UndefinedEnvironmentName(
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.
```


### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
OK, I can reproduce this by cloning sigstore and running `pip install -e ".[dev]"` in a clean venv. This is likely related to the fact that we're evaluating markers starting with this release, added in https://github.com/pypa/pip/pull/11030.

This seems to have broken editable installs for any package that has extras and uses setuptools. Also, TIL that setuptools throws all your runtime dependencies as build-time dependencies as well (at least for editable installs). That's... a very interesting choice.

I think the fix for this would be to not have any optional-dependencies installed in the build environment, by passing `{"extra": ""}` into the marker evaluation. I don't think we should be installing the optional-dependencies into the environment, especially since there's no mechanism to specify what extras should be selected for populating the build environment.
Not sure if this is helpful, but I am just checking Airflow and we have no problems with new `pip=22.1` with installing Airflow in editable mode with "[devel]" extras (and we use setuptools too - maybe not in the same way as `sigstore-python`.

1} Checkout https://github.com/apache/airflow 
2) Create & activate virtualenv
3) Upgrade pip to latest
4) ./scripts/tools/initialize_virtualenv.py

One of the things the venv intialization does is:

```
pip install -e '.[devel]' --constraint https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.9.txt
```

And it works flawlessly.
I also checked this scenario on one of my projects and could not reproduce.

Side note: since `sigstore-python` uses flit, which supports PEP 660 (as of flit_core>=3.4), the `setup.py` can be removed.
That does not resolve the issue, though.

Ah, indeed. I misunderstood that it was using setuptools' experimental support for pyproject.toml, and wasn't looking carefully. :)

This is indeed a flit-specific issue. When it can't figure out what the version of the package is (either by looking at `version` in pyproject.toml or looking for `__version__ = <str>` assignments in the top level module via AST), then it installs all the runtime dependencies to be able to import the package during the build: https://github.com/pypa/flit/blob/ec6e6b7c8994d5e79d0fd951d9e4c26a7c5d9977/flit_core/flit_core/buildapi.py#L35

This is happening because sigstore has an additional indirection for loading the version, for some reason, so a simple AST parse is not able to find the version.

So... the use case is significantly less disruptive -- for packages that use flit, define extras *and* don't want to maintain a static version in pyproject.toml or the `module.py`/`module/__init__.py` file.
This possibly could be fixed in sigstore then? It does not seem like worth fixing in `pip` ?
> This possibly could be fixed in sigstore then? It does not seem like worth fixing in pip ?

There are a number of options here:

1. sigstore could fix the issue by avoiding the extra indirection.
2. flit could improve their process for handling the case where they can't find the version statically.
3. pip could add a workaround as per #11112

It's worth noting that I agree with @pradyunsg's comment

> I don't think we should be installing the optional-dependencies into the environment, especially since there's no mechanism to specify what extras should be selected for populating the build environment.

I think it's worth us adding the workaround, so that our behaviour is explicit, and doesn't trigger an error. But flit needs to decide whether they want to support this usage, given we don't install extras, and possibly fix their approach. And honestly, IMO sigstore should probably just not try to be so clever 🙂 (or to put it another way, I'd be 100% fine with flit choosing to simply error in this case).
I’m running into this without editable mode and without specifying extras. There pure existence seems to be enough?

I have a flit project with:

```toml
[project.optional-dependencies]
dev = ['boto3-stubs[ecs,s3,sqs,ssm]', ...]
```

and running `pip install .` with pip 22.1 gives me

```
UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.
```

PS: Way to deflect blame, pip, you’re your own subprocess!

> note: This error originates from a subprocess, and is likely not a problem with pip.
I'm wondering - are we checking the build requirements are installed in *every* build environment, or just when `--no-build-isolation` is used? Because it seems to be that if we created the build environment, checking seems to be a bit over-cautious, as we should know we installed the right things?

Regardless, though, checking just non-optional dependencies should avoid the problem. We're not going to *install* build dependencies that are specified as optional, so a backend (or project) that needs them is probably just broken. But we should ignore them quietly, rather than just fail - being strict here doesn't serve any useful purpose.
I would appreciate a quick fix before I have to update all our CI pipelines with `pip install -U 'pip!=22.1'`
We currently do check requirements in all build environments. It’s probably a carry-over from the old resolver time, when the build environment can be populated incorrectly.
> And honestly, IMO sigstore should probably just not try to be so clever 🙂 (or to put it another way, I'd be 100% fine with flit choosing to simply error in this case).

I agree completely. I wasn't aware we were triggering such clever behavior! I'm going to make the necessary fixes on `sigstore`'s side, and IMO this should be a hard error in flit as well.
Flit supported it so far, so changing it will break a lot of packages, including basically all of mine (private, public, and commercial). Flit should definitely not suddenly stop supporting this.

In the future I’ll use hatch-vcs to make things like this more convenient, but until recently, flit (relying on this behavior) was the best option for me.
We also ran into this today. We dynamically generate the version via [flit-scm](https://pypi.org/project/flit-scm/) so flit getting the version by importing the package is a requirement for us.
At a minimum, flit could not include optional dependencies when generating the list of build dependencies, as they'll be ignored anyway. But this should be discussed on the flit tracker, not here.
I ran into this issue with a normal install from source (i.e. not an editable install). Regardless, #11112 fixed it for me.
Yeah this is not specific to editable installs, but any PEP 517 scenarios as well. This only happens to editables now because we recently added support to PEP 660, which shares the build environment population logic to PEP 517.



Patch:
diff --git a/news/11112.bugfix.rst b/news/11112.bugfix.rst
new file mode 100644
index 00000000000..87b4b56a4d5
--- /dev/null
+++ b/news/11112.bugfix.rst
@@ -0,0 +1,1 @@
+Properly filter out optional dependencies (i.e. extras) when checking build environment distributions.
diff --git a/src/pip/_internal/build_env.py b/src/pip/_internal/build_env.py
index 7783c9acda6..01165113b96 100644
--- a/src/pip/_internal/build_env.py
+++ b/src/pip/_internal/build_env.py
@@ -175,8 +175,10 @@ def check_requirements(
             )
             for req_str in reqs:
                 req = Requirement(req_str)
-                if req.marker is not None and not req.marker.evaluate():
-                    continue  # FIXME: Consider extras?
+                # We're explicitly evaluating with an empty extra value, since build
+                # environments are not provided any mechanism to select specific extras.
+                if req.marker is not None and not req.marker.evaluate({"extra": ""}):
+                    continue
                 dist = env.get_distribution(req.name)
                 if not dist:
                     missing.add(req_str)


Test patch:
diff --git a/tests/functional/test_build_env.py b/tests/functional/test_build_env.py
index a7f89687c08..1b9f42d94c0 100644
--- a/tests/functional/test_build_env.py
+++ b/tests/functional/test_build_env.py
@@ -178,6 +178,7 @@ def test_build_env_requirements_check(script: PipTestEnvironment) -> None:
             [
                 "bar==2.0; python_version < '3.0'",
                 "bar==3.0; python_version >= '3.0'",
+                "foo==4.0; extra == 'dev'",
             ],
         )
         assert r == (set(), set()), repr(r)