Repository: pypa/pip

Issue:
New resolver: Failure when package only specified with extras is not available from index
<!--
Please provide as much information as you can about your failure, so that we can understand the root cause.

Try if your issue has been fixed in the in-development version of pip. Use the following command to install pip from master:

    python -m pip install -U "pip @ https://github.com/pypa/pip/archive/master.zip"
-->

**What did you want to do?**
<!-- Include any inputs you gave to pip, for example:

* Package requirements: any CLI arguments and/or your requirements.txt file
* Already installed packages, outputted via `pip freeze`
-->

`pip install --use-feature=2020-resolver -r requirements.txt --no-deps`
`pip install --use-feature=2020-resolver -r requirements.txt`

requirements.txt only contains a list of packages to be installed in editable mode, with some depending on each other.

This is in a fresh miniconda environment and occurred on both 20.2.2 and master.

I ran pip using `--no-deps` first since in my experience, installing multiple editable mode packages with dependencies on each other fails otherwise. However, running just the normal install command directly in a fresh environment still fails with the new resolver, as below.

```
ERROR: Could not find a version that satisfies the requirement azureml-dataset-runtime[fuse]~=0.1.0.0 (from azureml-defaults)
ERROR: No matching distribution found for azureml-dataset-runtime[fuse]~=0.1.0.0
```

**Output**

This output is after running the 2nd pip install command, to actually install the package dependencies after installing the editable mode packages using `--no-deps`. Output has been slightly edited to remove full file paths.

```
ERROR: Cannot install azureml-dataset-runtime 0.1.0.0 (from src\azureml-dataset-runtime), -r requirements.txt (line 9), -r requirements.txt (line 16) and azureml-dataset-runtime[fuse] 0.1.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested azureml-dataset-runtime 0.1.0.0 (from src\azureml-dataset-runtime)
    azureml-automl-core 0.1.0.0 depends on azureml-dataset-runtime~=0.1.0.0
    azureml-train-automl-client 0.1.0.0 depends on azureml-dataset-runtime~=0.1.0.0
    azureml-dataset-runtime[fuse] 0.1.0.0 depends on azureml-dataset-runtime 0.1.0.0 (Installed)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
```

**Additional information**

<!--
It would be great if you could also include your dependency tree. For this you can use pipdeptree: https://pypi.org/project/pipdeptree/

For users installing packages from a private repository or local directory, please try your best to describe your setup. We'd like to understand how to reproduce the error locally, so would need (at a minimum) a description of the packages you are trying to install, and a list of dependencies for each package.
-->

The requirements file looks like this (all below packages should be available from pypi as well):

```
-e "src\azureml-core\."
-e "src\azureml-dataset-runtime\."
-e "src\azureml-defaults\."
-e "src\azureml-telemetry\."
-e "src\azureml-opendatasets\."
-e "src\azureml-pipeline\."
-e "src\azureml-pipeline-core\."
-e "src\azureml-pipeline-steps\."
-e "src\azureml-automl-core\."
-e "src\azureml-automl-runtime\."
-e "src\azureml-interpret\."
-e "src\azureml-explain-model\."
-e "src\azureml-train-restclients-hyperdrive\."
-e "src\azureml-train-core\."
-e "src\azureml-train\."
-e "src\azureml-train-automl-client\."
-e "src\azureml-train-automl-runtime\."
```


Discussion:
Can you provide either `setup.py` for all the listed projects (or at least `azureml-dataset-runtime` and `azureml-train-automl-client`), or a reduced setup that can reproduce the same error? It is impossible to tell what is going on without any context.
Sure. The below is a trimmed down version of setup.py for `azureml-dataset-runtime`.

`azureml-train-automl-client` takes a dependency on this via this entry for install_requires:
`'{}~={}'.format('azureml-dataset-runtime', VERSION)`

```
# ---------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# ---------------------------------------------------------

from setuptools import setup, find_packages
import os
import shutil

SELFVERSION = '0.1.0.0'
DATAPREP_VERSION = '>=2.0.1a,<2.1.0a'

REQUIRES = [
    'azureml-dataprep{}'.format(DATAPREP_VERSION),
    'pyarrow>=0.17.0,<2.0.0'
]

with open('README.md', 'r', encoding='utf-8') as f:
    long_description = f.read()
with open('../.inlinelicense', 'r', encoding='utf-8') as f:
    inline_license = f.read()

setup(
    name="azureml-dataset-runtime",
    version=SELFVERSION,
    description='',
    long_description=long_description,
    long_description_content_type='text/markdown',
    author='Microsoft Corp',
    license=inline_license,
    url='https://docs.microsoft.com/python/api/overview/azure/ml/?view=azure-ml-py',
    packages=find_packages(exclude=["*tests*"]),
    install_requires=REQUIRES,
    extras_require={
        'pandas': ['numpy>=1.14.0,<2.0.0', 'pandas>=0.23.4,<2.0.0'],
        'parquet': [],  # Keeping as no-op to avoid breaking scenarios where this extra is used.
        'pyspark': ['pyspark==2.3.0'],
        'fuse': ['fusepy>=3.0.1,<4.0.0'],
        'scipy': ['scipy>=1.1.0,<2.0.0']
    }
)

```
I was able to reproduce the error with the following:

```python
# a/setup.py
from setuptools import setup
setup(name="a", version="0.1.0.0", extras_require={"z": ["c"]})
```

```python
# b/setup.py
from setuptools import setup
setup(name="b", version="0.1.0.0", install_requires=["a[z]~=0.1.0.0"])
```

```python
# c/setup.py
from setuptools import setup
setup(name="c", version="0.1.0.0")
```

```console
$ pip install --use-feature=2020-resolver --no-deps ./a ./b
(snip, this works)

$ pip install --use-feature=2020-resolver ./a ./b
Processing file:///.../a
Processing file:///.../b
Requirement already satisfied: a[z]~=0.1.0.0 in .../a (from b==0.1.0.0) (0.1.0.0)
ERROR: Cannot install a 0.1.0.0 (from .../a) and a[z] 0.1.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested a 0.1.0.0 (from .../a)
    a[z] 0.1.0.0 depends on a 0.1.0.0 (Installed)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
```

----

A few observations. First, editable-ness does not matter.

Second, the extra indirection is needed to trigger this. `pip install ./a ./b` would work if `b` depends on `a` directly (instead of `a[z]`).

Third, the error if you don’t `install --no-deps` first is

```
ERROR: Could not find a version that satisfies the requirement a[z]~=0.1.0.0 (from b)
ERROR: No matching distribution found for a[z]~=0.1.0.0
```

Again, the extra indirection is needed to trigger the error. `pip install ./a ./b` works if `b` depends on `a` directly (instead of `a[z]`).

So the problem is that we’re doing something wrong when handling dependencies to an extra-ed requirement.
OK, I think I know what’s going on here. When a package requests dependencies, the resolver pass all the currently known specifiers for the provider to find candidates with. If `b` requests `a` (no extras), the resolver would pass `['a', 'a @ ./a']` since it knows that there is an additional specifier provided by the user.

When `b` requests `a[z]`, however, the resolver does not know that `a @ ./a` *also* provides `a[z]`, and thus only passes `['a[z]']` to the provider, resulting it failing to find a match.

The provider API will need to be tweaked slightly for it to be able to “tell” the resolver that `a` actually provides candidates for both `a` and `a[z]`. I will update resolvelib and come back to this afterwards.

---

The ResolutionImpossible exception triggered when `a` was first installed before hand is triggered by the same cause. But instead of not being able to find any candidates, the provider now is able to find exactly one match—the previously-installed distribution—but then rejects it since the installed distribution does not satisfy the direct link requirement. How this should be fixed (or not fixed) would depend on our resolution to #5780. `Factory.find_candidates()` will need to be updated to relect the decision and make an AlreadyInstalledCandidate be able to satisfy a SpecifierRequirement if the URL matches.
> I will update resolvelib and come back to this afterwards.

@uranusjr Any updates on this? I am not 100% sure how this would work and whether we *have* to make changes for this issue, prior to the new-resolver-is-default release.
I have a couple of plans to make this work (not yet decided which is best; the plans I came up with thus far are all a bit ugly). It’s not the end of the world if the release is made without fixing this, but we should try to get a fix in if possible.
I came up with a way to solve this without tweaking resolvelib. (See the PR linked above for the solution that does involve changing resolvelib.)

The main idea is, when the provider gets a dependency `a @ URL`, it should also emit a constraint on `a[x] @ URL` to the resolver. This makes the resolver aware of the URL spec when dealing with `a[x]`, but only if the user does supply `a[x]` somewhere as a concrete requirement. The problem then becomes making constraints flow through in the resolvelib resolver, instead of only dealt externally in pip code. The trick is to make constraints a subclass of `base.Requirement`, so they can be returned by `get_dependencies()`. When `find_matches()` receives only constraints and not concrete requirements, it returns a special “no-op” candidate that does not have any dependencies, satisfies anything, but ultimately installs nothing. (Regular candidates are returned when at least one concrete requirement is passed into `find_matches()`.) This should correctly represent the logical meaning of constraints in the resolver.

The problem to this solution, however, is that the provider will need to emit all combinations of extras, even if they may never be needed in the dependency graph. The object count would grow quite quickly (`factorial(len(extras) + 1) - 1`), potentially slowing down resolution further more. It would be best if we could generate the constraints lazily, but I haven’t thought of a way without duplicating a backtracking stack in the provider, which is probably a very bad idea.
Per [today's meeting](https://wiki.python.org/psf/PackagingWG/2020-10-06-pip-teamwidemeeting) it looks like the next step here is for @uranusjr and @pradyunsg to have a chat about what to do next. This is something we want to get taken care of before the 20.3 release in a few weeks.
Just been wondering if there's been any updates to this, since 20.3/new resolver will definitely break us in various ways unless this is fixed.
Per [notes from a meeting this week](https://wiki.python.org/psf/PackagingWG/2020-10-12-pip-teamwide-meeting) I believe Pradyun and Tzu-ping had a private conversation about this issue -- could one of you please share those notes here in the issue so we know what needs to happen next? Thanks!
Sorry for the delay. The issue happens when pip sees a direct URL requirement, and then a requirement of the same name with different extras. Since a requirment with different extras represent a different set of dependencies, the resolver treats these two as different requirements, and is not able to resolve the non-URL requirement into using that URL.

The workaround to this would be to install the editable/URL requirements first, like this:

```
$ pip install --use-feature=2020-resolver -r requirements.txt --no-deps
$ pip install --use-feature=2020-resolver -r requirements.txt
```

The first `--no-deps` installation would skip all the resolver stuff (thus avoid the error). Once all those packages are in site-packages, subsequent `install` calls would work correctly.

Some additional technical context that might make more sense to those into pip internals: Since `a[x]` provides a different set of packages from `a`, and that set may be different depending on the version of `a` chosen, the resolver internally creates a “fake” package that depends on the dependencies specified by extra `x`, and `a` that matches the same version of itself. This makes pip aware of the correct version of `a` when it sees `a[x]`. But this does not work the other way around—if you provide `a` first, the resolver does not know it needs `a[x]` when it sees `a` (it can’t just pull it in, otherwise the user would get unwanted packages when they don’t want `a[x]` but only `a`), and we don’t have a good way to inform the resolver about that direct URL can also be used to satisfy `a[x]` later on. This is why installing the packages without dependencies would work around the issue. The on-disk `.dist-info` can be used to represent that `a` package, so we don’t need to use the URL.

Ultimately, this is one of those issues that are probably fixable, but require too much resource that are more useful spent elsewhere. I would be happy to offer advices if anyone really wants to work on this, but am not personally very motivated to do the work myself.
Hmm, I just re-read the top post, and it seems like the `--no-deps` trick is not working for OP? @pradyunsg this seems to contradict with our findings yesterday; maybe something changed that makes things work now (the `get_preference` thing, maybe)?
Damn it, I was all over the place. To sum up things again: there are actually *two* issues here. The first is the direct-URL-with-different-extras thing I talked about in the previous comment, which is hit if you don’t run `--no-deps` first. We decided to not put too much resource on it, and recommend the `--no-deps` workaround instead.

The error you get if you `--no-deps` first is not related to the extras issue, but about the resolver’s upgrade strategy regarding direct URL requirements (a variant of #8711, blocked by #5780). We should (will?) fix that.
@pradyunsg could you dive into this before tomorrow's meeting, or before Wednesday's?
Per [today's meeting](https://wiki.python.org/psf/PackagingWG/2020-11-18-pip-team-meeting), Pradyun has decided that improving this behavior is not a blocker to the 20.3 release (in which the new resolver will be on by default); the new behavior does better than old resolver, though not as well as we want to. 
Apparently the the new `--use-feature=2020-resolver` is already activated in `20.3`? It broke docker my deployments.

Docker pipeline edits setup.py to use CI_JOB_TOKEN to clone deps:
```
ham/setup.py # gitlabci changes git+ssh to git+https
   -> git+https://gitlab.com/bar/foo.git@0.1
   -> git+https://gitlab.com/bar/lau.git@0.1
foo/setup.py # points still to repo via ssh
   -> git+ssh://gitlab.com/bar/lau.git@0.1
```
Under 20.2.4:

It clones `foo` and `lau` via https git and installs it

Under 20.3:

It clones `foo` and `lau` via https git and installs it. It also tries to install `lau` again via ssh

Under 20.2.4 + `--use-feature=2020-resolver`:

Same error as 20.3.

@delijati The reason to your error is not the same as described in this issue. The two `lau` URLs are different. Yes, the two URLs ultimately download the same code, but pip does not have any special knowledge about GitHub, and therefore (correctly) treated as different packages.
@delijati Yes, the new resolver is activated by default in pip 20.3. [Here's more info on that, including how to specify the old resolver temporarily](https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020) as a workaround.
@uranusjr jup the new dependecy resolver made just transparent that there was an error all along. Thanks i hope i can finally convince the team to use an artifactory store ;) or i have to stick with `--use-deprecated=legacy-resolver`
I have merged #8939, #9143, and #9204 here, which all have the same root cause, described above (https://github.com/pypa/pip/issues/8785#issuecomment-678885871).
I've attempted to use the workaround described in #9143 (use `legacy-resolver`). However, this approach has another downside. In python/importlib_metadata@c769ba8fae245265bdb3f173a41e0e8c4f2a2d4b, I implemented the workaround, but now when I run `tox` on my local workstation, it doesn't work at all:

```
importlib_metadata main $ tox
python create: /Users/jaraco/code/public/importlib_metadata/.tox/python
python develop-inst: /Users/jaraco/code/public/importlib_metadata
ERROR: invocation failed (exit code 3), logfile: /Users/jaraco/code/public/importlib_metadata/.tox/python/log/python-1.log
=============================================================== log start ===============================================================
An error occurred during configuration: option use-deprecated: invalid choice: 'legacy-resolver' (choose from )

================================================================ log end ================================================================
________________________________________________________________ summary ________________________________________________________________
ERROR:   python: InvocationError for command /Users/jaraco/code/public/importlib_metadata/.tox/python/bin/python -m pip install --exists-action w -e '/Users/jaraco/code/public/importlib_metadata[testing]' (exited with code 3)
```

I have the latest tox installed with the latest virtualenv:

```
importlib_metadata main $ which tox
/Users/jaraco/.local/bin/tox
importlib_metadata main $ ~/.local/pipx/venvs/tox/bin/python -m pip list
Package         Version
--------------- -------
appdirs         1.4.4
distlib         0.3.1
filelock        3.0.12
packaging       20.8
pip             20.3.3
pluggy          0.13.1
py              1.10.0
pyparsing       2.4.7
setuptools      51.1.2
six             1.15.0
toml            0.10.2
tox             3.21.0
tox-pip-version 0.0.7
virtualenv      20.3.0
wheel           0.36.2
```

Yet, somehow, pip 20.2.4 is getting used.

I tried forcing a later pip with `tox-pip-version`:

```diff
diff --git a/tox.ini b/tox.ini
index 11f52d7..46676f3 100644
--- a/tox.ini
+++ b/tox.ini
@@ -4,6 +4,8 @@ minversion = 3.2
 # https://github.com/jaraco/skeleton/issues/6
 tox_pip_extensions_ext_venv_update = true
 toxworkdir={env:TOX_WORK_DIR:.tox}
+requires =
+	tox-pip-version
 
 
 [testenv]
@@ -15,6 +17,8 @@ extras = testing
 setenv =
 	# workaround pypa/pip#9143
 	PIP_USE_DEPRECATED=legacy-resolver
+pip_version =
+	pip>=20.3.1
 
 
 [testenv:docs]
```

But even with that, pip fails to upgrade itself with the same error. It's proving difficult to implement the workaround in a reliable way. I somehow need a way for tox to invoke pip first without the workaround to upgrade pip, then invoke it with the workaround to install the the project and its dependencies.

I probably could spend some more time researching how tox and virtualenv work together to install different pip versions in different environments and maybe come up with a workaround, but for now what I'm doing is manually removing the workaround on my local environments when creating tox environments, then undoing that change.

I'm open to suggestions, but unblocked and done for the day, so no reply is needed. I mainly just wanted to capture this secondary issue.

Resolving patch:
diff --git a/src/pip/_internal/resolution/resolvelib/candidates.py b/src/pip/_internal/resolution/resolvelib/candidates.py
--- a/src/pip/_internal/resolution/resolvelib/candidates.py
+++ b/src/pip/_internal/resolution/resolvelib/candidates.py
@@ -33,6 +33,18 @@
 ]
 
 
+def as_base_candidate(candidate: Candidate) -> Optional[BaseCandidate]:
+    """The runtime version of BaseCandidate."""
+    base_candidate_classes = (
+        AlreadyInstalledCandidate,
+        EditableCandidate,
+        LinkCandidate,
+    )
+    if isinstance(candidate, base_candidate_classes):
+        return candidate
+    return None
+
+
 def make_install_req_from_link(link, template):
     # type: (Link, InstallRequirement) -> InstallRequirement
     assert not template.editable, "template is editable"
diff --git a/src/pip/_internal/resolution/resolvelib/factory.py b/src/pip/_internal/resolution/resolvelib/factory.py
--- a/src/pip/_internal/resolution/resolvelib/factory.py
+++ b/src/pip/_internal/resolution/resolvelib/factory.py
@@ -1,3 +1,4 @@
+import contextlib
 import functools
 import logging
 from typing import (
@@ -16,6 +17,8 @@
     cast,
 )
 
+from pip._vendor.packaging.requirements import InvalidRequirement
+from pip._vendor.packaging.requirements import Requirement as PackagingRequirement
 from pip._vendor.packaging.specifiers import SpecifierSet
 from pip._vendor.packaging.utils import NormalizedName, canonicalize_name
 from pip._vendor.pkg_resources import Distribution
@@ -54,6 +57,7 @@
     ExtrasCandidate,
     LinkCandidate,
     RequiresPythonCandidate,
+    as_base_candidate,
 )
 from .found_candidates import FoundCandidates, IndexCandidateInfo
 from .requirements import (
@@ -123,6 +127,15 @@ def force_reinstall(self):
         # type: () -> bool
         return self._force_reinstall
 
+    def _fail_if_link_is_unsupported_wheel(self, link: Link) -> None:
+        if not link.is_wheel:
+            return
+        wheel = Wheel(link.filename)
+        if wheel.supported(self._finder.target_python.get_tags()):
+            return
+        msg = f"{link.filename} is not a supported wheel on this platform."
+        raise UnsupportedWheel(msg)
+
     def _make_extras_candidate(self, base, extras):
         # type: (BaseCandidate, FrozenSet[str]) -> ExtrasCandidate
         cache_key = (id(base), extras)
@@ -275,6 +288,51 @@ def iter_index_candidate_infos():
             incompatible_ids,
         )
 
+    def _iter_explicit_candidates_from_base(
+        self,
+        base_requirements: Iterable[Requirement],
+        extras: FrozenSet[str],
+    ) -> Iterator[Candidate]:
+        """Produce explicit candidates from the base given an extra-ed package.
+
+        :param base_requirements: Requirements known to the resolver. The
+            requirements are guaranteed to not have extras.
+        :param extras: The extras to inject into the explicit requirements'
+            candidates.
+        """
+        for req in base_requirements:
+            lookup_cand, _ = req.get_candidate_lookup()
+            if lookup_cand is None:  # Not explicit.
+                continue
+            # We've stripped extras from the identifier, and should always
+            # get a BaseCandidate here, unless there's a bug elsewhere.
+            base_cand = as_base_candidate(lookup_cand)
+            assert base_cand is not None, "no extras here"
+            yield self._make_extras_candidate(base_cand, extras)
+
+    def _iter_candidates_from_constraints(
+        self,
+        identifier: str,
+        constraint: Constraint,
+        template: InstallRequirement,
+    ) -> Iterator[Candidate]:
+        """Produce explicit candidates from constraints.
+
+        This creates "fake" InstallRequirement objects that are basically clones
+        of what "should" be the template, but with original_link set to link.
+        """
+        for link in constraint.links:
+            self._fail_if_link_is_unsupported_wheel(link)
+            candidate = self._make_candidate_from_link(
+                link,
+                extras=frozenset(),
+                template=install_req_from_link_and_ireq(link, template),
+                name=canonicalize_name(identifier),
+                version=None,
+            )
+            if candidate:
+                yield candidate
+
     def find_candidates(
         self,
         identifier: str,
@@ -283,59 +341,48 @@ def find_candidates(
         constraint: Constraint,
         prefers_installed: bool,
     ) -> Iterable[Candidate]:
-
-        # Since we cache all the candidates, incompatibility identification
-        # can be made quicker by comparing only the id() values.
-        incompat_ids = {id(c) for c in incompatibilities.get(identifier, ())}
-
+        # Collect basic lookup information from the requirements.
         explicit_candidates = set()  # type: Set[Candidate]
         ireqs = []  # type: List[InstallRequirement]
         for req in requirements[identifier]:
             cand, ireq = req.get_candidate_lookup()
-            if cand is not None and id(cand) not in incompat_ids:
+            if cand is not None:
                 explicit_candidates.add(cand)
             if ireq is not None:
                 ireqs.append(ireq)
 
-        for link in constraint.links:
-            if not ireqs:
-                # If we hit this condition, then we cannot construct a candidate.
-                # However, if we hit this condition, then none of the requirements
-                # provided an ireq, so they must have provided an explicit candidate.
-                # In that case, either the candidate matches, in which case this loop
-                # doesn't need to do anything, or it doesn't, in which case there's
-                # nothing this loop can do to recover.
-                break
-            if link.is_wheel:
-                wheel = Wheel(link.filename)
-                # Check whether the provided wheel is compatible with the target
-                # platform.
-                if not wheel.supported(self._finder.target_python.get_tags()):
-                    # We are constrained to install a wheel that is incompatible with
-                    # the target architecture, so there are no valid candidates.
-                    # Return early, with no candidates.
-                    return ()
-            # Create a "fake" InstallRequirement that's basically a clone of
-            # what "should" be the template, but with original_link set to link.
-            # Using the given requirement is necessary for preserving hash
-            # requirements, but without the original_link, direct_url.json
-            # won't be created.
-            ireq = install_req_from_link_and_ireq(link, ireqs[0])
-            candidate = self._make_candidate_from_link(
-                link,
-                extras=frozenset(),
-                template=ireq,
-                name=canonicalize_name(ireq.name) if ireq.name else None,
-                version=None,
+        # If the current identifier contains extras, add explicit candidates
+        # from entries from extra-less identifier.
+        with contextlib.suppress(InvalidRequirement):
+            parsed_requirement = PackagingRequirement(identifier)
+            explicit_candidates.update(
+                self._iter_explicit_candidates_from_base(
+                    requirements.get(parsed_requirement.name, ()),
+                    frozenset(parsed_requirement.extras),
+                ),
             )
-            if candidate is None:
-                # _make_candidate_from_link returns None if the wheel fails to build.
-                # We are constrained to install this wheel, so there are no valid
-                # candidates.
-                # Return early, with no candidates.
+
+        # Add explicit candidates from constraints. We only do this if there are
+        # kown ireqs, which represent requirements not already explicit. If
+        # there are no ireqs, we're constraining already-explicit requirements,
+        # which is handled later when we return the explicit candidates.
+        if ireqs:
+            try:
+                explicit_candidates.update(
+                    self._iter_candidates_from_constraints(
+                        identifier,
+                        constraint,
+                        template=ireqs[0],
+                    ),
+                )
+            except UnsupportedWheel:
+                # If we're constrained to install a wheel incompatible with the
+                # target architecture, no candidates will ever be valid.
                 return ()
 
-            explicit_candidates.add(candidate)
+        # Since we cache all the candidates, incompatibility identification
+        # can be made quicker by comparing only the id() values.
+        incompat_ids = {id(c) for c in incompatibilities.get(identifier, ())}
 
         # If none of the requirements want an explicit candidate, we can ask
         # the finder for candidates.
@@ -351,7 +398,8 @@ def find_candidates(
         return (
             c
             for c in explicit_candidates
-            if constraint.is_satisfied_by(c)
+            if id(c) not in incompat_ids
+            and constraint.is_satisfied_by(c)
             and all(req.is_satisfied_by(c) for req in requirements[identifier])
         )
 
@@ -366,13 +414,7 @@ def make_requirement_from_install_req(self, ireq, requested_extras):
             return None
         if not ireq.link:
             return SpecifierRequirement(ireq)
-        if ireq.link.is_wheel:
-            wheel = Wheel(ireq.link.filename)
-            if not wheel.supported(self._finder.target_python.get_tags()):
-                msg = "{} is not a supported wheel on this platform.".format(
-                    wheel.filename,
-                )
-                raise UnsupportedWheel(msg)
+        self._fail_if_link_is_unsupported_wheel(ireq.link)
         cand = self._make_candidate_from_link(
             ireq.link,
             extras=frozenset(ireq.extras),


Test patch:
diff --git a/tests/functional/test_install_direct_url.py b/tests/functional/test_install_direct_url.py
--- a/tests/functional/test_install_direct_url.py
+++ b/tests/functional/test_install_direct_url.py
@@ -1,26 +1,12 @@
-import re
-
 import pytest
 
-from pip._internal.models.direct_url import DIRECT_URL_METADATA_NAME, DirectUrl
 from tests.lib import _create_test_package, path_to_url
-
-
-def _get_created_direct_url(result, pkg):
-    direct_url_metadata_re = re.compile(
-        pkg + r"-[\d\.]+\.dist-info." + DIRECT_URL_METADATA_NAME + r"$"
-    )
-    for filename in result.files_created:
-        if direct_url_metadata_re.search(filename):
-            direct_url_path = result.test_env.base_path / filename
-            with open(direct_url_path) as f:
-                return DirectUrl.from_json(f.read())
-    return None
+from tests.lib.direct_url import get_created_direct_url
 
 
 def test_install_find_links_no_direct_url(script, with_wheel):
     result = script.pip_install_local("simple")
-    assert not _get_created_direct_url(result, "simple")
+    assert not get_created_direct_url(result, "simple")
 
 
 def test_install_vcs_editable_no_direct_url(script, with_wheel):
@@ -29,7 +15,7 @@ def test_install_vcs_editable_no_direct_url(script, with_wheel):
     result = script.pip(*args)
     # legacy editable installs do not generate .dist-info,
     # hence no direct_url.json
-    assert not _get_created_direct_url(result, "testpkg")
+    assert not get_created_direct_url(result, "testpkg")
 
 
 def test_install_vcs_non_editable_direct_url(script, with_wheel):
@@ -37,7 +23,7 @@ def test_install_vcs_non_editable_direct_url(script, with_wheel):
     url = path_to_url(pkg_path)
     args = ["install", f"git+{url}#egg=testpkg"]
     result = script.pip(*args)
-    direct_url = _get_created_direct_url(result, "testpkg")
+    direct_url = get_created_direct_url(result, "testpkg")
     assert direct_url
     assert direct_url.url == url
     assert direct_url.info.vcs == "git"
@@ -47,7 +33,7 @@ def test_install_archive_direct_url(script, data, with_wheel):
     req = "simple @ " + path_to_url(data.packages / "simple-2.0.tar.gz")
     assert req.startswith("simple @ file://")
     result = script.pip("install", req)
-    assert _get_created_direct_url(result, "simple")
+    assert get_created_direct_url(result, "simple")
 
 
 @pytest.mark.network
@@ -59,7 +45,7 @@ def test_install_vcs_constraint_direct_url(script, with_wheel):
         "#egg=pip-test-package"
     )
     result = script.pip("install", "pip-test-package", "-c", constraints_file)
-    assert _get_created_direct_url(result, "pip_test_package")
+    assert get_created_direct_url(result, "pip_test_package")
 
 
 def test_install_vcs_constraint_direct_file_url(script, with_wheel):
@@ -68,4 +54,4 @@ def test_install_vcs_constraint_direct_file_url(script, with_wheel):
     constraints_file = script.scratch_path / "constraints.txt"
     constraints_file.write_text(f"git+{url}#egg=testpkg")
     result = script.pip("install", "testpkg", "-c", constraints_file)
-    assert _get_created_direct_url(result, "testpkg")
+    assert get_created_direct_url(result, "testpkg")
diff --git a/tests/functional/test_new_resolver.py b/tests/functional/test_new_resolver.py
--- a/tests/functional/test_new_resolver.py
+++ b/tests/functional/test_new_resolver.py
@@ -12,6 +12,7 @@
     create_test_package_with_setup,
     path_to_url,
 )
+from tests.lib.direct_url import get_created_direct_url
 from tests.lib.path import Path
 from tests.lib.wheel import make_wheel
 
@@ -1788,3 +1789,41 @@ def test_new_resolver_avoids_incompatible_wheel_tags_in_constraint_url(
 
     assert_installed(script, base="0.1.0")
     assert_not_installed(script, "dep")
+
+
+def test_new_resolver_direct_url_with_extras(tmp_path, script):
+    pkg1 = create_basic_wheel_for_package(script, name="pkg1", version="1")
+    pkg2 = create_basic_wheel_for_package(
+        script,
+        name="pkg2",
+        version="1",
+        extras={"ext": ["pkg1"]},
+    )
+    pkg3 = create_basic_wheel_for_package(
+        script,
+        name="pkg3",
+        version="1",
+        depends=["pkg2[ext]"],
+    )
+
+    # Make pkg1 and pkg3 visible via --find-links, but not pkg2.
+    find_links = tmp_path.joinpath("find_links")
+    find_links.mkdir()
+    with open(pkg1, "rb") as f:
+        find_links.joinpath(pkg1.name).write_bytes(f.read())
+    with open(pkg3, "rb") as f:
+        find_links.joinpath(pkg3.name).write_bytes(f.read())
+
+    # Install with pkg2 only available with direct URL. The extra-ed direct
+    # URL pkg2 should be able to provide pkg2[ext] required by pkg3.
+    result = script.pip(
+        "install",
+        "--no-cache-dir", "--no-index",
+        "--find-links", str(find_links),
+        pkg2, "pkg3",
+    )
+
+    assert_installed(script, pkg1="1", pkg2="1", pkg3="1")
+    assert not get_created_direct_url(result, "pkg1")
+    assert get_created_direct_url(result, "pkg2")
+    assert not get_created_direct_url(result, "pkg3")
diff --git a/tests/lib/direct_url.py b/tests/lib/direct_url.py
new file mode 100644
--- /dev/null
+++ b/tests/lib/direct_url.py
@@ -0,0 +1,15 @@
+import re
+
+from pip._internal.models.direct_url import DIRECT_URL_METADATA_NAME, DirectUrl
+
+
+def get_created_direct_url(result, pkg):
+    direct_url_metadata_re = re.compile(
+        pkg + r"-[\d\.]+\.dist-info." + DIRECT_URL_METADATA_NAME + r"$"
+    )
+    for filename in result.files_created:
+        if direct_url_metadata_re.search(filename):
+            direct_url_path = result.test_env.base_path / filename
+            with open(direct_url_path) as f:
+                return DirectUrl.from_json(f.read())
+    return None