Repository: pypa/pip

Issue:
Package left un-upgraded with `pip install --upgrade`
### Description

When upgrading both `datasets` and `fsspec` in an environment, only `datasets` was upgraded to the latest version even though there was nothing constraining `fsspec` from being updated. `fsspec` is a dependency of `datasets` (`fsspec[http]>=2021.05.0` is specified in its [`setup.py`](https://github.com/huggingface/datasets/blob/1.14.0/setup.py)), and from looking at the resolver's debug output, it seems that pip is preferring to use the pre-installed version of `fsspec` instead of the newer one.

The documentation for `--upgrade` says, "Upgrade all specified packages to the newest available version." Unless I'm misunderstanding, all packages explicitly listed on the command line should have been updated if possible.

### Expected behavior

`fsspec` should be upgraded to `2021.10.1`.

### pip version

21.3.1

### Python version

3.7.12

### OS

macOS 10.15.7

### How to Reproduce

1. Create a file named `pip_fsspec_test.sh` with the following contents:

```bash
#!/usr/bin/env bash

set -euxo pipefail

# Prepare environment
python --version
pip install -U pip setuptools wheel
pip list

# Install outdated packages
pip install datasets==1.13.3 fsspec==2021.10.0
pip list --outdated

# Attempt to upgrade both packages
pip install -U 'datasets<=1.14.0' 'fsspec<=2021.10.1'
pip list --outdated
```

2. Run `bash pip_fsspec_test.sh &> pip_fsspec_log.txt` in a new environment to create [pip_fsspec_log.txt](https://github.com/pypa/pip/files/7402239/pip_fsspec_log.txt).
3. Change the penultimate line of the script to `PIP_RESOLVER_DEBUG=1 pip -vvv install -U 'datasets<=1.14.0' 'fsspec<=2021.10.1'`.
4. Run `bash pip_fsspec_test.sh &> pip_fsspec_verbose_log.txt` in a new environment to create [pip_fsspec_verbose_log.txt](https://github.com/pypa/pip/files/7402240/pip_fsspec_verbose_log.txt).

### Output

_See files under "How to Reproduce"._

### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
That said, looking at the report more closely, we're preferring the installed version of a package if it's specified with an extra, as a dependency of another package -- even when that dependency is explicitly specified on the CLI. That's... suboptimal.

Does `--upgrade-strategy=eager` do what you want?

Yeah it seems like we need the [`split("[")` treatment](https://github.com/pypa/pip/blob/main/src/pip/_internal/resolution/resolvelib/provider.py#L157-L164) in

https://github.com/pypa/pip/blob/b160e83aa967f89e8ca9ad28c23386681d9b50fc/src/pip/_internal/resolution/resolvelib/provider.py#L174-L189

It's quite interesting this bug took so long to surface.

Resolving patch:
diff --git a/src/pip/_internal/resolution/resolvelib/provider.py b/src/pip/_internal/resolution/resolvelib/provider.py
--- a/src/pip/_internal/resolution/resolvelib/provider.py
+++ b/src/pip/_internal/resolution/resolvelib/provider.py
@@ -1,6 +1,15 @@
 import collections
 import math
-from typing import TYPE_CHECKING, Dict, Iterable, Iterator, Mapping, Sequence, Union
+from typing import (
+    TYPE_CHECKING,
+    Dict,
+    Iterable,
+    Iterator,
+    Mapping,
+    Sequence,
+    TypeVar,
+    Union,
+)
 
 from pip._vendor.resolvelib.providers import AbstractProvider
 
@@ -37,6 +46,35 @@
 # services to those objects (access to pip's finder and preparer).
 
 
+D = TypeVar("D")
+V = TypeVar("V")
+
+
+def _get_with_identifier(
+    mapping: Mapping[str, V],
+    identifier: str,
+    default: D,
+) -> Union[D, V]:
+    """Get item from a package name lookup mapping with a resolver identifier.
+
+    This extra logic is needed when the target mapping is keyed by package
+    name, which cannot be directly looked up with an identifier (which may
+    contain requested extras). Additional logic is added to also look up a value
+    by "cleaning up" the extras from the identifier.
+    """
+    if identifier in mapping:
+        return mapping[identifier]
+    # HACK: Theoretically we should check whether this identifier is a valid
+    # "NAME[EXTRAS]" format, and parse out the name part with packaging or
+    # some regular expression. But since pip's resolver only spits out three
+    # kinds of identifiers: normalized PEP 503 names, normalized names plus
+    # extras, and Requires-Python, we can cheat a bit here.
+    name, open_bracket, _ = identifier.partition("[")
+    if open_bracket and name in mapping:
+        return mapping[name]
+    return default
+
+
 class PipProvider(_ProviderBase):
     """Pip's provider implementation for resolvelib.
 
@@ -150,28 +188,13 @@ def get_preference(  # type: ignore
             identifier,
         )
 
-    def _get_constraint(self, identifier: str) -> Constraint:
-        if identifier in self._constraints:
-            return self._constraints[identifier]
-
-        # HACK: Theoretically we should check whether this identifier is a valid
-        # "NAME[EXTRAS]" format, and parse out the name part with packaging or
-        # some regular expression. But since pip's resolver only spits out
-        # three kinds of identifiers: normalized PEP 503 names, normalized names
-        # plus extras, and Requires-Python, we can cheat a bit here.
-        name, open_bracket, _ = identifier.partition("[")
-        if open_bracket and name in self._constraints:
-            return self._constraints[name]
-
-        return Constraint.empty()
-
     def find_matches(
         self,
         identifier: str,
         requirements: Mapping[str, Iterator[Requirement]],
         incompatibilities: Mapping[str, Iterator[Candidate]],
     ) -> Iterable[Candidate]:
-        def _eligible_for_upgrade(name: str) -> bool:
+        def _eligible_for_upgrade(identifier: str) -> bool:
             """Are upgrades allowed for this project?
 
             This checks the upgrade strategy, and whether the project was one
@@ -185,13 +208,23 @@ def _eligible_for_upgrade(name: str) -> bool:
             if self._upgrade_strategy == "eager":
                 return True
             elif self._upgrade_strategy == "only-if-needed":
-                return name in self._user_requested
+                user_order = _get_with_identifier(
+                    self._user_requested,
+                    identifier,
+                    default=None,
+                )
+                return user_order is not None
             return False
 
+        constraint = _get_with_identifier(
+            self._constraints,
+            identifier,
+            default=Constraint.empty(),
+        )
         return self._factory.find_candidates(
             identifier=identifier,
             requirements=requirements,
-            constraint=self._get_constraint(identifier),
+            constraint=constraint,
             prefers_installed=(not _eligible_for_upgrade(identifier)),
             incompatibilities=incompatibilities,
         )


Test patch:
diff --git a/tests/functional/test_new_resolver.py b/tests/functional/test_new_resolver.py
--- a/tests/functional/test_new_resolver.py
+++ b/tests/functional/test_new_resolver.py
@@ -2180,3 +2180,36 @@ def test_new_resolver_dont_backtrack_on_extra_if_base_constrained(script):
     )
     assert "pkg-2.0" not in result.stdout, "Should not try 2.0 due to constraint"
     script.assert_installed(pkg="1.0", dep="1.0")
+
+
+def test_new_resolver_respect_user_requested_if_extra_is_installed(script):
+    create_basic_wheel_for_package(script, "pkg1", "1.0")
+    create_basic_wheel_for_package(script, "pkg2", "1.0", extras={"ext": ["pkg1"]})
+    create_basic_wheel_for_package(script, "pkg2", "2.0", extras={"ext": ["pkg1"]})
+    create_basic_wheel_for_package(script, "pkg3", "1.0", depends=["pkg2[ext]"])
+
+    # Install pkg3 with an older pkg2.
+    script.pip(
+        "install",
+        "--no-cache-dir",
+        "--no-index",
+        "--find-links",
+        script.scratch_path,
+        "pkg3",
+        "pkg2==1.0",
+    )
+    script.assert_installed(pkg3="1.0", pkg2="1.0", pkg1="1.0")
+
+    # Now upgrade both pkg3 and pkg2. pkg2 should be upgraded although pkg2[ext]
+    # is not requested by the user.
+    script.pip(
+        "install",
+        "--no-cache-dir",
+        "--no-index",
+        "--find-links",
+        script.scratch_path,
+        "--upgrade",
+        "pkg3",
+        "pkg2",
+    )
+    script.assert_installed(pkg3="1.0", pkg2="2.0", pkg1="1.0")