Repository: jazzband/pip-tools

Issue:
Resolver error on upgrade
When upgrading the (pinned) dependencies in requirements.in of a project with an existing requirements.txt, pip-compile fails with a resolver error. It claims that an indirect dependency (here: `cachecontrol[filecache]<0.14.0,>=0.13.0`) cannot be resolved even though a version (0.13.1) matching the constraint is listed among the considered versions.

pip-compile successfully creates a requirements.txt from the same requirements.in file (with the same pinned project dependencies), if I first delete the existing requirements.txt. However, I came across this error when [renovateBot](https://docs.renovatebot.com/) failed to update requirements.txt – that's a scenario where I cannot delete requirements.txt first.

#### Environment Versions

1. MacOS 13.5 (also reproduced under Ubuntu jammy)
1. Python version: 3.11.2 (also observed under Python 3.10 and 3.9)
1. pip version: 23.2.1
1. pip-tools version: 7.3.0

#### Steps to replicate

~~~shell
(pip-tools-venv) piptools  $ rm -f requirements.in requirements.txt
(pip-tools-venv) piptools  $ echo "poetry==1.5.1\n" > requirements.in
(pip-tools-venv) piptools  $ pip-compile --generate-hashes -q requirements.in
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
(pip-tools-venv) piptools  $ grep '^cachecontrol' requirements.txt
cachecontrol[filecache]==0.12.14 \
(pip-tools-venv) piptools  $ echo "poetry==1.6.1\n" > requirements.in
(pip-tools-venv) piptools  $ pip-compile --generate-hashes -q requirements.in
  ERROR: Could not find a version that satisfies the requirement cachecontrol[filecache]<0.14.0,>=0.13.0 (from poetry) (from versions: 0.5, 0.6, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.6, 0.10.7, 0.11.0, 0.11.1, 0.11.2, 0.11.3, 0.11.4, 0.11.5, 0.11.6, 0.11.7, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.12.4, 0.12.5, 0.12.6, 0.12.7, 0.12.8, 0.12.9, 0.12.10, 0.12.11, 0.12.12, 0.12.13, 0.12.14, 0.13.0, 0.13.1rc0, 0.13.1)
Traceback (most recent call last):
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 439, in resolve
    raise ResolutionImpossible(self.state.backtrack_causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('cachecontrol[filecache]<0.14.0,>=0.13.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/7d/25/f3bfda3c458d114005af99441d009936b85a34a730aeb9cf57fb2630d9f7/poetry-1.6.1-py3-none-any.whl (from https://pypi.org/simple/poetry/) (requires-python:>=3.8,<4.0)'))]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/piptools/scripts/compile.py", line 659, in cli
    results = resolver.resolve(max_rounds=max_rounds)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
                  ^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 101, in resolve
    raise error from e
pip._internal.exceptions.DistributionNotFound: No matching distribution found for cachecontrol[filecache]<0.14.0,>=0.13.0
(pip-tools-venv) piptools  $ rm requirements.txt
(pip-tools-venv) piptools  $ pip-compile --generate-hashes -q requirements.in
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
(pip-tools-venv) piptools  $ grep '^cachecontrol' requirements.txt
cachecontrol[filecache]==0.13.1 \
~~~

#### Expected result

requirements.txt is updated, no need to delete requirements.txt first

#### Actual result

The  resolver error shown if requirements.txt exists already generated for an older version of the project dependency.



Discussion:
I ran pip-compile in the debugger and saw (I think) what's the problem:
* In `piptools.resolver.BacktrackingResolver._do_resolve()`, the first resolution attempt fails because `compatible_existing_constraints` contains an incompatible version of `cachecontrol`. So far that's expected.
* In the exception handler, however, `cause_ireq_names` is assigned the value `{'cachecontrol[filecache]'}`, including the extra specifier. However, the corresponding key in `compatible_existing_constraints` is `'cachecontrol'` (without extra).

  Therefore, the handler cannot find and drop the incompatible dependency from `compatible_existing_constraints` (i.e., `cause_existing_ireq` is `None`) and re-raises the exception.

This happens with and without `--strip-extras`.
Does `pip install --no-deps cachecontrol==0.13.1 -vvvvv` succeed under the same OS+arch+Python? I think that the backtracking resolver rejects sdist-only releases that it can't build. Though, this one appears to have wheels so that's probably not it.

Have you tried without the hash generation? Also, maybe try with `--upgrade`.
* `pip install --no-deps cachecontrol==0.13.1 -vvvvv` does work. (Tested inside an Ubuntu Jammy container with Python 3.11.)
* Dropping `--generate-hashes` from the command line does not make any difference.
* With `--upgrade`, the error goes away – unfortunately, this does not help me in the RenovateBot scenario.

  RenovateBot _does_ use `--upgrade` if it creates a merge request for "lock file maintenance". (I.e., when it updates the indirect dependencies in requirements.txt.) It apparently does not use this option if it wants to update a pinned dependency in requirements.in. The only control that I have over the command line created by RenovateBot is that it [keeps the option listed in requirements.txt](https://docs.renovatebot.com/modules/manager/pip-compile/#pip-compile-arguments).


Of course, I am not familiar with the codebase of pip-tools. But what I observed in the debugger looks like a bug to me:
* The existing constraints read from requirements.txt are initially kept. I assume the intention is to upgrade as few (indirect) dependencies as possible (unless `--upgrade` is specified).
* If there are incompatible constraints, than the BacktrackingResolver apparently should drop the offending package(s) from the existing constraints and try again. But the dict with the existing constraints uses "plain" package names (without extras) as keys, whereas the name of the incompatible package listed in the exception does include the extras. Therefore, the code does not find the relevant entry in the existing constraints.

I will try whether I can come up with a PR.

Resolving patch:
diff --git a/piptools/resolver.py b/piptools/resolver.py
--- a/piptools/resolver.py
+++ b/piptools/resolver.py
@@ -648,7 +648,8 @@ def _do_resolve(
 
             # Collect all incompatible install requirement names
             cause_ireq_names = {
-                key_from_req(cause.requirement) for cause in cause_exc.causes
+                strip_extras(key_from_req(cause.requirement))
+                for cause in cause_exc.causes
             }
 
             # Looks like resolution is impossible, try to fix
diff --git a/piptools/utils.py b/piptools/utils.py
--- a/piptools/utils.py
+++ b/piptools/utils.py
@@ -23,6 +23,7 @@
 from click.utils import LazyFile
 from pip._internal.req import InstallRequirement
 from pip._internal.req.constructors import install_req_from_line, parse_req_from_line
+from pip._internal.resolution.resolvelib.base import Requirement as PipRequirement
 from pip._internal.utils.misc import redact_auth_from_url
 from pip._internal.vcs import is_url
 from pip._vendor.packaging.markers import Marker
@@ -72,8 +73,20 @@ def key_from_ireq(ireq: InstallRequirement) -> str:
         return key_from_req(ireq.req)
 
 
-def key_from_req(req: InstallRequirement | Requirement) -> str:
-    """Get an all-lowercase version of the requirement's name."""
+def key_from_req(req: InstallRequirement | Requirement | PipRequirement) -> str:
+    """
+    Get an all-lowercase version of the requirement's name.
+
+    **Note:** If the argument is an instance of
+    ``pip._internal.resolution.resolvelib.base.Requirement`` (like
+    ``pip._internal.resolution.resolvelib.requirements.SpecifierRequirement``),
+    then the name might include an extras specification.
+    Apply :py:func:`strip_extras` to the result of this function if you need
+    the package name only.
+
+    :param req: the requirement the key is computed for
+    :return: the canonical name of the requirement
+    """
     return str(canonicalize_name(req.name))
 
 


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -2779,6 +2779,139 @@ def test_cli_compile_strip_extras(runner, make_package, make_sdist, tmpdir):
     assert "[more]" not in out.stderr
 
 
+@pytest.mark.parametrize(
+    ("package_specs", "constraints", "existing_reqs", "expected_reqs"),
+    (
+        (
+            [
+                {
+                    "name": "test_package_1",
+                    "version": "1.1",
+                    "install_requires": ["test_package_2 ~= 1.1"],
+                },
+                {
+                    "name": "test_package_2",
+                    "version": "1.1",
+                    "extras_require": {"more": "test_package_3"},
+                },
+            ],
+            """
+            test_package_1 == 1.1
+            """,
+            """
+            test_package_1 == 1.0
+            test_package_2 == 1.0
+            """,
+            """
+            test-package-1==1.1
+            test-package-2==1.1
+            """,
+        ),
+        (
+            [
+                {
+                    "name": "test_package_1",
+                    "version": "1.1",
+                    "install_requires": ["test_package_2[more] ~= 1.1"],
+                },
+                {
+                    "name": "test_package_2",
+                    "version": "1.1",
+                    "extras_require": {"more": "test_package_3"},
+                },
+                {
+                    "name": "test_package_3",
+                    "version": "0.1",
+                },
+            ],
+            """
+            test_package_1 == 1.1
+            """,
+            """
+            test_package_1 == 1.0
+            test_package_2 == 1.0
+            test_package_3 == 0.1
+            """,
+            """
+            test-package-1==1.1
+            test-package-2==1.1
+            test-package-3==0.1
+            """,
+        ),
+        (
+            [
+                {
+                    "name": "test_package_1",
+                    "version": "1.1",
+                    "install_requires": ["test_package_2[more] ~= 1.1"],
+                },
+                {
+                    "name": "test_package_2",
+                    "version": "1.1",
+                    "extras_require": {"more": "test_package_3"},
+                },
+                {
+                    "name": "test_package_3",
+                    "version": "0.1",
+                },
+            ],
+            """
+            test_package_1 == 1.1
+            """,
+            """
+            test_package_1 == 1.0
+            test_package_2[more] == 1.0
+            test_package_3 == 0.1
+            """,
+            """
+            test-package-1==1.1
+            test-package-2==1.1
+            test-package-3==0.1
+            """,
+        ),
+    ),
+    ids=("no-extra", "extra-stripped-from-existing", "with-extra-in-existing"),
+)
+def test_resolver_drops_existing_conflicting_constraint(
+    runner,
+    make_package,
+    make_sdist,
+    tmpdir,
+    package_specs,
+    constraints,
+    existing_reqs,
+    expected_reqs,
+) -> None:
+    """
+    Test that the resolver will find a solution even if some of the existing
+    (indirect) requirements are incompatible with the new constraints.
+
+    This must succeed even if the conflicting requirement includes some extra,
+    no matter whether the extra is mentioned in the existing requirements
+    or not (cf. `issue #1977 <https://github.com/jazzband/pip-tools/issues/1977>`_).
+    """
+    expected_requirements = {line.strip() for line in expected_reqs.splitlines()}
+    dists_dir = tmpdir / "dists"
+
+    packages = [make_package(**spec) for spec in package_specs]
+    for pkg in packages:
+        make_sdist(pkg, dists_dir)
+
+    with open("requirements.txt", "w") as existing_reqs_out:
+        existing_reqs_out.write(dedent(existing_reqs))
+
+    with open("requirements.in", "w") as constraints_out:
+        constraints_out.write(dedent(constraints))
+
+    out = runner.invoke(cli, ["--strip-extras", "--find-links", str(dists_dir)])
+
+    assert out.exit_code == 0, out
+
+    with open("requirements.txt") as req_txt:
+        req_txt_content = req_txt.read()
+        assert expected_requirements.issubset(req_txt_content.splitlines())
+
+
 def test_resolution_failure(runner):
     """Test resolution impossible for unknown package."""
     with open("requirements.in", "w") as reqs_out:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -7,10 +7,13 @@
 import sys
 from pathlib import Path
 from textwrap import dedent
+from typing import Callable
 
 import pip
 import pytest
 from click import BadOptionUsage, Context, FileError
+from pip._internal.req import InstallRequirement
+from pip._internal.resolution.resolvelib.requirements import SpecifierRequirement
 from pip._vendor.packaging.version import Version
 
 from piptools.scripts.compile import cli as compile_cli
@@ -29,6 +32,7 @@
     is_pinned_requirement,
     is_url_requirement,
     key_from_ireq,
+    key_from_req,
     lookup_table,
     lookup_table_from_tuples,
     override_defaults_from_config_file,
@@ -285,6 +289,46 @@ def test_key_from_ireq_normalization(from_line):
     assert len(keys) == 1
 
 
+@pytest.mark.parametrize(
+    ("line", "expected"),
+    (
+        ("build", "build"),
+        ("cachecontrol[filecache]", "cachecontrol"),
+        ("some-package[a-b,c_d]", "some-package"),
+        ("other_package[a.b]", "other-package"),
+    ),
+)
+def test_key_from_req_on_install_requirement(
+    from_line: Callable[[str], InstallRequirement],
+    line: str,
+    expected: str,
+) -> None:
+    ireq = from_line(line)
+    result = key_from_req(ireq)
+
+    assert result == expected
+
+
+@pytest.mark.parametrize(
+    ("line", "expected"),
+    (
+        ("build", "build"),
+        ("cachecontrol[filecache]", "cachecontrol[filecache]"),
+        ("some-package[a-b,c_d]", "some-package[a-b,c-d]"),
+        ("other_package[a.b]", "other-package[a-b]"),
+    ),
+)
+def test_key_from_req_on_specifier_requirement(
+    from_line: Callable[[str], InstallRequirement],
+    line: str,
+    expected: str,
+) -> None:
+    req = SpecifierRequirement(from_line(line))
+    result = key_from_req(req)
+
+    assert result == expected
+
+
 @pytest.mark.parametrize(
     ("line", "expected"),
     (