Repository: pypa/pip

Issue:
Installing local packages with transitive dependencies and extras not working
Pip (version 20.3+) seems to have problems resolving dependencies when trying to install packages from the filesystem with transitive dependencies and `extras` defined if for these packages also a distribution package on `pypi` exists.  

Lets say there are the packages with dependencies defined by the following  `setup.py` files:
```python
import setuptools

setuptools.setup(
    name="opentelemetry-api",
    install_requires=[],
    extras_require={"test": []},
    version="0.17.dev0",
)
```

```python
import setuptools

setuptools.setup(
    name="opentelemetry-sdk",
    install_requires=["opentelemetry-api == 0.17.dev0"],
    extras_require={"test": []},
    version="0.17.dev0",
)
```

```python
import setuptools

setuptools.setup(
    name="opentelemetry-instrumentation",
    install_requires=[
        "opentelemetry-api == 0.17.dev0",
        "opentelemetry-sdk == 0.17.dev0",
    ],
    extras_require={"test": []},
    version="0.17.dev0",
)
```
These `setup.py` files refer to the local development version which does not exist on `pypi`. However, there exist release versions of these packages on `pypi`.

When now trying to install these packages locally from the filesystem with the extras specifier
```
 pip install -e './opentelemetry-api[test]' -e './opentelemetry-sdk[test]' -e './opentelemetry-instrumentation[test]'
```
pip (20.3) will fail with the following error message (pip 20.2.4 works however):
```
Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-api
Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-sdk
Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-instrumentation
INFO: pip is looking at multiple versions of opentelemetry-api[test] to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement opentelemetry-sdk==0.17.dev0 (from opentelemetry-instrumentation[test])
ERROR: No matching distribution found for opentelemetry-sdk==0.17.dev0
```
Runing pip with the `-v` switch it seems that pip tries to lookup the `opentelemetry-sdk` package from `pypi` even though it was specified in the command line as to be installed from the local filesystem.

After trying around a little bit the following scenarios seem to work.
* Installing only the first two packages: `pip install -e './opentelemetry-api[test]' -e './opentelemetry-sdk[test]'`.
* Installing without extras: `pip install -e './opentelemetry-api' -e './opentelemetry-sdk' -e './opentelemetry-instrumentation'`
* Rename the package names in the `setup.py` files (and also the dependencies) to something else that is not avialable on `pypi` (e.g. `opentelemetry-api` = `py1`, `opentelemetry-sdk` = `py2` and `opentelemetry-instrumentation` = `py3`).
* Modify the `install_requires` dependencies to include the `test` extra (e.g. `opentelemetry-api[test] == 0.17.dev0`)

So the problem seems to be a combination of transitive dependencies, extras and package lookup on `pypi` for packages installed from the local filesystem.


Discussion:
Hello and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us. 

While we're waiting for the resolver developers to look at your report and respond, I'll mention here one useful troubleshooting and workaround tip from [the documentation](https://pip.pypa.io/en/latest/user_guide/#how-to-upgrade-and-migrate):

If you run into resolution problems and need a workaround while you’re fixing their root causes, you can choose the old resolver behavior using the flag `--use-deprecated=legacy-resolver`. This will work until we release pip 21.0 (see [Deprecation timeline](https://pip.pypa.io/en/latest/user_guide/#deprecation-timeline-for-2020-resolver-changes)).

(If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the pip resolver beta period.)
Merging this into #8785 to avoid messages from scattering.
It appears that #8785 didn't fix this issue.

Running `pip install -e './opentelemetry-api[test]' -e './opentelemetry-sdk[test]' -e './opentelemetry-instrumentation[test]'` with the `setup.py` files above and the latest pip version 21.1.1 still fails with the following message:
```
Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-api
Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-sdk
Obtaining file:///home/mario/workspaces/playground/py-pip-test/opentelemetry-instrumentation
INFO: pip is looking at multiple versions of opentelemetry-api to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of opentelemetry-api[test] to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement opentelemetry-sdk==0.17.dev0 (from opentelemetry-instrumentation[test]) (from versions: 0.1a0, 0.2a0, 0.3a0, 0.4a0, 0.4a1, 0.5b0, 0.6b0, 0.7b1, 0.8b0, 0.9b0, 0.10b0, 0.11b0, 0.12b0, 0.13b0, 0.14b0, 0.15b0, 0.16b0, 0.16b1, 0.17b0, 1.0.0rc1, 1.0.0, 1.1.0, 1.2.0, 1.10a0)
ERROR: No matching distribution found for opentelemetry-sdk==0.17.dev0
```

I think putting `'./opentelemetry-sdk[test]'` first should work.
I tried that but it is still producing the same output. I thought that the order of packages should be irrelevant with the new resolver?

Anyway, in the install command above the packages were already listed according to their dependencies among each other: `api -> {}`, `sdk -> {api}` and `instrumentation -> {api, sdk}`. Also, install works fine with this order when the extras are omitted.

Ordering is still relevant, they are resolved in the same order as they are requested. But if installing works fine without extras, ordering is likely not the issue. I'll find some time to look into this.
I'm also hitting this for https://github.com/certbot/certbot where we also have multiple packages depending on each other like this.

In case it's useful, here's a stripped down example reproducing the problem with only two packages:
```
$ tree .             
.
├── certbot
│   └── setup.py
└── certbot-apache
    └── setup.py

2 directories, 2 files
```
`certbot/setup.py` is https://gist.github.com/bmw/89b369a00f53b66157850e8097973e1e and `certbot-apache/setup.py` is https://gist.github.com/bmw/214e15753d92342cb4c37c898eebb818

```
$ pip install -e 'certbot[docs]' -e certbot-apache
Obtaining file:///private/var/folders/3b/zg8fdh5j71x92yyzc1tyllfw0000gp/T/tmp.19lG8kl23R/certbot
Obtaining file:///private/var/folders/3b/zg8fdh5j71x92yyzc1tyllfw0000gp/T/tmp.19lG8kl23R/certbot-apache
ERROR: Could not find a version that satisfies the requirement certbot>=99.99.0.dev0 (from certbot-apache) (from versions: 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 0.11.1, 0.12.0, 0.13.0, 0.14.0, 0.14.1, 0.14.2, 0.15.0, 0.16.0, 0.17.0, 0.18.0, 0.18.1, 0.18.2, 0.19.0, 0.20.0, 0.21.0, 0.21.1, 0.22.0, 0.22.1, 0.22.2, 0.23.0, 0.24.0, 0.25.0, 0.25.1, 0.26.0, 0.26.1, 0.27.0, 0.27.1, 0.28.0, 0.29.0, 0.29.1, 0.30.0, 0.30.1, 0.30.2, 0.31.0, 0.32.0, 0.33.0, 0.33.1, 0.34.0, 0.34.1, 0.34.2, 0.35.0, 0.35.1, 0.36.0, 0.37.0, 0.37.1, 0.37.2, 0.38.0, 0.39.0, 0.40.0, 0.40.1, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0)
ERROR: No matching distribution found for certbot>=99.99.0.dev0
```
EDIT: Tweaked version numbers so the example still works as we continue to do releases.
Figured out the issue. The reason why #8785 did not fix this is because the requirements provided in this issue are all *unnamed* (i.e. a bare path without `#egg=pkg` or PEP 508 `pkg @`). This means that the resolver cannot correctly detect the user’s specified ordering from the command line. (So for a two-requirement set-up, the success rate is actually 50-50 depending on the requirement’s path/URL, although all the examples here are failures for obvious reasons.) I’ll work on a fix for this.

Edit: Editable-ness is not actually relevant here (although editable requirements hit this issue more since they tend to be unnamed), so I’m removing that label.

Resolving patch:
diff --git a/src/pip/_internal/resolution/resolvelib/factory.py b/src/pip/_internal/resolution/resolvelib/factory.py
--- a/src/pip/_internal/resolution/resolvelib/factory.py
+++ b/src/pip/_internal/resolution/resolvelib/factory.py
@@ -9,6 +9,7 @@
     Iterator,
     List,
     Mapping,
+    NamedTuple,
     Optional,
     Sequence,
     Set,
@@ -38,7 +39,10 @@
 from pip._internal.models.wheel import Wheel
 from pip._internal.operations.prepare import RequirementPreparer
 from pip._internal.req.constructors import install_req_from_link_and_ireq
-from pip._internal.req.req_install import InstallRequirement
+from pip._internal.req.req_install import (
+    InstallRequirement,
+    check_invalid_constraint_type,
+)
 from pip._internal.resolution.base import InstallRequirementProvider
 from pip._internal.utils.compatibility_tags import get_supported
 from pip._internal.utils.hashes import Hashes
@@ -81,6 +85,12 @@ class ConflictCause(Protocol):
 Cache = Dict[Link, C]
 
 
+class CollectedRootRequirements(NamedTuple):
+    requirements: List[Requirement]
+    constraints: Dict[str, Constraint]
+    user_requested: Dict[str, int]
+
+
 class Factory:
     def __init__(
         self,
@@ -408,7 +418,7 @@ def find_candidates(
             and all(req.is_satisfied_by(c) for req in requirements[identifier])
         )
 
-    def make_requirement_from_install_req(
+    def _make_requirement_from_install_req(
         self, ireq: InstallRequirement, requested_extras: Iterable[str]
     ) -> Optional[Requirement]:
         if not ireq.match_markers(requested_extras):
@@ -440,6 +450,36 @@ def make_requirement_from_install_req(
             return UnsatisfiableRequirement(canonicalize_name(ireq.name))
         return self.make_requirement_from_candidate(cand)
 
+    def collect_root_requirements(
+        self, root_ireqs: List[InstallRequirement]
+    ) -> CollectedRootRequirements:
+        collected = CollectedRootRequirements([], {}, {})
+        for i, ireq in enumerate(root_ireqs):
+            if ireq.constraint:
+                # Ensure we only accept valid constraints
+                problem = check_invalid_constraint_type(ireq)
+                if problem:
+                    raise InstallationError(problem)
+                if not ireq.match_markers():
+                    continue
+                assert ireq.name, "Constraint must be named"
+                name = canonicalize_name(ireq.name)
+                if name in collected.constraints:
+                    collected.constraints[name] &= ireq
+                else:
+                    collected.constraints[name] = Constraint.from_ireq(ireq)
+            else:
+                req = self._make_requirement_from_install_req(
+                    ireq,
+                    requested_extras=(),
+                )
+                if req is None:
+                    continue
+                if ireq.user_supplied and req.name not in collected.user_requested:
+                    collected.user_requested[req.name] = i
+                collected.requirements.append(req)
+        return collected
+
     def make_requirement_from_candidate(
         self, candidate: Candidate
     ) -> ExplicitRequirement:
@@ -452,7 +492,7 @@ def make_requirement_from_spec(
         requested_extras: Iterable[str] = (),
     ) -> Optional[Requirement]:
         ireq = self._make_install_req_from_spec(specifier, comes_from)
-        return self.make_requirement_from_install_req(ireq, requested_extras)
+        return self._make_requirement_from_install_req(ireq, requested_extras)
 
     def make_requires_python_requirement(
         self, specifier: Optional[SpecifierSet]
diff --git a/src/pip/_internal/resolution/resolvelib/resolver.py b/src/pip/_internal/resolution/resolvelib/resolver.py
--- a/src/pip/_internal/resolution/resolvelib/resolver.py
+++ b/src/pip/_internal/resolution/resolvelib/resolver.py
@@ -10,13 +10,9 @@
 from pip._vendor.resolvelib.structs import DirectedGraph
 
 from pip._internal.cache import WheelCache
-from pip._internal.exceptions import InstallationError
 from pip._internal.index.package_finder import PackageFinder
 from pip._internal.operations.prepare import RequirementPreparer
-from pip._internal.req.req_install import (
-    InstallRequirement,
-    check_invalid_constraint_type,
-)
+from pip._internal.req.req_install import InstallRequirement
 from pip._internal.req.req_set import RequirementSet
 from pip._internal.resolution.base import BaseResolver, InstallRequirementProvider
 from pip._internal.resolution.resolvelib.provider import PipProvider
@@ -28,7 +24,7 @@
 from pip._internal.utils.filetypes import is_archive_file
 from pip._internal.utils.misc import dist_is_editable
 
-from .base import Candidate, Constraint, Requirement
+from .base import Candidate, Requirement
 from .factory import Factory
 
 if TYPE_CHECKING:
@@ -78,41 +74,13 @@ def __init__(
     def resolve(
         self, root_reqs: List[InstallRequirement], check_supported_wheels: bool
     ) -> RequirementSet:
-
-        constraints: Dict[str, Constraint] = {}
-        user_requested: Dict[str, int] = {}
-        requirements = []
-        for i, req in enumerate(root_reqs):
-            if req.constraint:
-                # Ensure we only accept valid constraints
-                problem = check_invalid_constraint_type(req)
-                if problem:
-                    raise InstallationError(problem)
-                if not req.match_markers():
-                    continue
-                assert req.name, "Constraint must be named"
-                name = canonicalize_name(req.name)
-                if name in constraints:
-                    constraints[name] &= req
-                else:
-                    constraints[name] = Constraint.from_ireq(req)
-            else:
-                if req.user_supplied and req.name:
-                    canonical_name = canonicalize_name(req.name)
-                    if canonical_name not in user_requested:
-                        user_requested[canonical_name] = i
-                r = self.factory.make_requirement_from_install_req(
-                    req, requested_extras=()
-                )
-                if r is not None:
-                    requirements.append(r)
-
+        collected = self.factory.collect_root_requirements(root_reqs)
         provider = PipProvider(
             factory=self.factory,
-            constraints=constraints,
+            constraints=collected.constraints,
             ignore_dependencies=self.ignore_dependencies,
             upgrade_strategy=self.upgrade_strategy,
-            user_requested=user_requested,
+            user_requested=collected.user_requested,
         )
         if "PIP_RESOLVER_DEBUG" in os.environ:
             reporter: BaseReporter = PipDebuggingReporter()
@@ -126,13 +94,13 @@ def resolve(
         try:
             try_to_avoid_resolution_too_deep = 2000000
             result = self._result = resolver.resolve(
-                requirements, max_rounds=try_to_avoid_resolution_too_deep
+                collected.requirements, max_rounds=try_to_avoid_resolution_too_deep
             )
 
         except ResolutionImpossible as e:
             error = self.factory.get_installation_error(
                 cast("ResolutionImpossible[Requirement, Candidate]", e),
-                constraints,
+                collected.constraints,
             )
             raise error from e
 


Test patch:
diff --git a/tests/functional/test_new_resolver.py b/tests/functional/test_new_resolver.py
--- a/tests/functional/test_new_resolver.py
+++ b/tests/functional/test_new_resolver.py
@@ -1843,3 +1843,32 @@ def test_new_resolver_modifies_installed_incompatible(script):
         "d==1",
     )
     assert_installed(script, d="1", c="2", b="2", a="2")
+
+
+def test_new_resolver_transitively_depends_on_unnamed_local(script):
+    create_basic_wheel_for_package(script, name="certbot-docs", version="1")
+    certbot = create_test_package_with_setup(
+        script,
+        name="certbot",
+        version="99.99.0.dev0",
+        extras_require={"docs": ["certbot-docs"]}
+    )
+    certbot_apache = create_test_package_with_setup(
+        script,
+        name="certbot-apache",
+        version="99.99.0.dev0",
+        install_requires=["certbot>=99.99.0.dev0"],
+    )
+
+    script.pip(
+        "install",
+        "--no-cache-dir", "--no-index",
+        "--find-links", script.scratch_path,
+        f"{certbot}[docs]", certbot_apache,
+    )
+    assert_installed(
+        script,
+        certbot="99.99.0.dev0",
+        certbot_apache="99.99.0.dev0",
+        certbot_docs="1",
+    )