Repository: pypa/pip

Issue:
pip resolver installing incompatible versions
### Description

See reproduce steps for setup.

It installed `certifi==2021.5.30`, and then reported an error for a version mismatch. Instead, pip should have looked at the requirements and installed `2020.12.5` to work with all the other dependencies.

It seems that passing `--upgrade` makes it work.

Ref: https://github.com/linkedin/datahub/issues/2717

### Expected behavior

It should have installed `certifi==2020.12.5`, since the main requirements are `certifi<2021.0.0` and greater than a version from 2017. 

### pip version

21.1.2

### Python version

3.8.6

### OS

MacOS

### How to Reproduce

The following steps:
```sh
python3.8 -m venv venv
source venv/bin/activate
pip install --upgrade pip wheel setuptools  # sanity check
pip freeze  # empty, since I'm in a fresh venv
PIP_RESOLVER_DEBUG=1 pip install 'acryl-datahub[snowflake]==0.8.1.2'
```

Yields:
```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
snowflake-connector-python 2.4.5 requires certifi<2021.0.0, but you have certifi 2021.5.30 which is incompatible.
```



### Output

Edited by @uranusjr: I put this into a Gist instead since the log is way too long to reasonably read and reference in comments.

https://gist.github.com/uranusjr/74ec344ce72f9c9cbe91b262bcd49dd5


### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).
pip resolver installing incompatible versions
### Description

See reproduce steps for setup.

It installed `certifi==2021.5.30`, and then reported an error for a version mismatch. Instead, pip should have looked at the requirements and installed `2020.12.5` to work with all the other dependencies.

It seems that passing `--upgrade` makes it work.

Ref: https://github.com/linkedin/datahub/issues/2717

### Expected behavior

It should have installed `certifi==2020.12.5`, since the main requirements are `certifi<2021.0.0` and greater than a version from 2017. 

### pip version

21.1.2

### Python version

3.8.6

### OS

MacOS

### How to Reproduce

The following steps:
```sh
python3.8 -m venv venv
source venv/bin/activate
pip install --upgrade pip wheel setuptools  # sanity check
pip freeze  # empty, since I'm in a fresh venv
PIP_RESOLVER_DEBUG=1 pip install 'acryl-datahub[snowflake]==0.8.1.2'
```

Yields:
```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
snowflake-connector-python 2.4.5 requires certifi<2021.0.0, but you have certifi 2021.5.30 which is incompatible.
```



### Output

Edited by @uranusjr: I put this into a Gist instead since the log is way too long to reasonably read and reference in comments.

https://gist.github.com/uranusjr/74ec344ce72f9c9cbe91b262bcd49dd5


### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
The error message explains _why_ that occurs. Changing this behaviour is tracked in #9094. Consolidating this issue into that.
@pradyunsg I think this one might be a bit different, since there’s only one pip install command and nothing was installed previously (fresh venv). 
I can reproduce this in a fresh virtual environment. This is very weird. I suspect there is a package doing funky things to bypass the resolver and install its own things.
Hmm. So the resolver has no trouble understanding certifi needs to be pinned.

It first fetches 2021.5.30 because requests needs `certifi>=2017.4.17` (line 161):

```
Reporter.adding_requirement(SpecifierRequirement('certifi>=2017.4.17'), LinkCandidate('https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl#sha256=c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e (from https://pypi.org/simple/requests/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*)'))
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
```

And then correctly triggers a downgrade when snowflake-connector-python needs `certifi<2021.0.0` (line 245):

```
Reporter.adding_requirement(SpecifierRequirement('certifi<2021.0.0'), LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
```

Nothing afterwards invalidated certifi 2020.12.5, so the resolver runs to the end…

and proceeds to pick the wrong version of certifi!? (line 384)

```
('certifi', LinkCandidate('https://files.pythonhosted.org/packages/05/1b/0a0dece0e8aa492a6ec9e4ad2fe366b511558cdc73fd3abc82ba7348e875/certifi-2021.5.30-py2.py3-none-any.whl#sha256=50b1e4f8446b06f41be7dd6338db18e0990601dce795c2b1686458aa7e8fa7d8 (from https://pypi.org/simple/certifi/)'))
```

So this is probably a bug in the resolver when it collects the final results. Still need to investigate more. Unfortunately(?) `pip install requests snowflake-connector-python` gives me the correct pin (even though it went through basically all the same steps regarding certifi).
AHA. Apologies for my confusion here!

---

> and proceeds to pick the wrong version of certifi!? (line 384)

Yea, the resolver is marking the wrong requirement against the `snowflake-connector-python` candidate for some reason.

    (SpecifierRequirement('certifi>=2017.4.17'), via=LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))

From line 245:

    Reporter.adding_requirement(SpecifierRequirement('certifi<2021.0.0'), LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))

From line 249:

    Reporter.adding_requirement(SpecifierRequirement('certifi>=2017.4.17'), LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))

That's... suboptimal at best. :/

I dug into for like two hours and finally think I know what’s going on.

The trigger to this issue is that snowflake-connector-python specifies certifi *twice*. From [the wheel’s metadata](https://pypi-filebrowser.herokuapp.com/snowflake-connector-python/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl/METADATA):

```
Metadata-Version: 2.1
Name: snowflake-connector-python
Version: 2.4.5
...
Requires-Dist: certifi (<2021.0.0)
...
Requires-Dist: certifi (>=2017.4.17)
...
```

A wrikle to this is that there’s a reproducibility issue, because the issue only happen when `certifi<2021.0.0` is fed into the resolver *before* `certifi>=2017.4.17`. But `pkg_resources` shuffles the `Requires-Dist` values on calculation, so to reproduce this deterministically, you need to change line 3041 of [`pkg_resources.DistInfoDistribution._compute_dependencies()`](https://github.com/pypa/pip/blob/7c3abccd22c4bbd6c1da2ae41845ffbd26c1bb17/src/pip/_vendor/pkg_resources/__init__.py#L3026-L3047) to something like:

```python
dm[None].extend(reqs_for_extra(None))
```

so the order of requirements don’t change between runs.

Now onto the actual resolver stuff. When the resolver pins through a candidate, it converts all of the candidate’s dependencies into criteria so they can be merged into existing criteria:

https://github.com/pypa/pip/blob/7c3abccd22c4bbd6c1da2ae41845ffbd26c1bb17/src/pip/_vendor/resolvelib/resolvers.py#L200-L205

But it uses a dict for the job. So when a candidate specifies a package multiple times (like snowflake-connector-python), the latter specification would override the previous one. Since `certifi<2021.0.0` is specified first (after we patch `pkg_resources`; without the patch the resolver has 50% chance to get the “right” one), it got lost during resolution.

I have to say, I haven’t had such an exercise debugging an issue like this for quite a while. This kind of things is why I’m doing open source work, so thanks for that, in a way 😆 I’m unfortunately quite busy tomorrow (it’s late Saturday here), but will try to find some time to fix asap.
The error message explains _why_ that occurs. Changing this behaviour is tracked in #9094. Consolidating this issue into that.
@pradyunsg I think this one might be a bit different, since there’s only one pip install command and nothing was installed previously (fresh venv). 
I can reproduce this in a fresh virtual environment. This is very weird. I suspect there is a package doing funky things to bypass the resolver and install its own things.
Hmm. So the resolver has no trouble understanding certifi needs to be pinned.

It first fetches 2021.5.30 because requests needs `certifi>=2017.4.17` (line 161):

```
Reporter.adding_requirement(SpecifierRequirement('certifi>=2017.4.17'), LinkCandidate('https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl#sha256=c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e (from https://pypi.org/simple/requests/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*)'))
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
```

And then correctly triggers a downgrade when snowflake-connector-python needs `certifi<2021.0.0` (line 245):

```
Reporter.adding_requirement(SpecifierRequirement('certifi<2021.0.0'), LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
```

Nothing afterwards invalidated certifi 2020.12.5, so the resolver runs to the end…

and proceeds to pick the wrong version of certifi!? (line 384)

```
('certifi', LinkCandidate('https://files.pythonhosted.org/packages/05/1b/0a0dece0e8aa492a6ec9e4ad2fe366b511558cdc73fd3abc82ba7348e875/certifi-2021.5.30-py2.py3-none-any.whl#sha256=50b1e4f8446b06f41be7dd6338db18e0990601dce795c2b1686458aa7e8fa7d8 (from https://pypi.org/simple/certifi/)'))
```

So this is probably a bug in the resolver when it collects the final results. Still need to investigate more. Unfortunately(?) `pip install requests snowflake-connector-python` gives me the correct pin (even though it went through basically all the same steps regarding certifi).
AHA. Apologies for my confusion here!

---

> and proceeds to pick the wrong version of certifi!? (line 384)

Yea, the resolver is marking the wrong requirement against the `snowflake-connector-python` candidate for some reason.

    (SpecifierRequirement('certifi>=2017.4.17'), via=LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))

From line 245:

    Reporter.adding_requirement(SpecifierRequirement('certifi<2021.0.0'), LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))

From line 249:

    Reporter.adding_requirement(SpecifierRequirement('certifi>=2017.4.17'), LinkCandidate('https://files.pythonhosted.org/packages/d7/1e/84c75b4bf224ec81db26e49a13f961d3c25d60571e3d42556392cae718fd/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl#sha256=131fb0dc7a11878287ab19d89bfd5c956fd8f11d1a2e07dd5c343a5e3eacfe1e (from https://pypi.org/simple/snowflake-connector-python/) (requires-python:>=3.6)'))

That's... suboptimal at best. :/

I dug into for like two hours and finally think I know what’s going on.

The trigger to this issue is that snowflake-connector-python specifies certifi *twice*. From [the wheel’s metadata](https://pypi-filebrowser.herokuapp.com/snowflake-connector-python/snowflake_connector_python-2.4.5-cp38-cp38-macosx_10_13_x86_64.whl/METADATA):

```
Metadata-Version: 2.1
Name: snowflake-connector-python
Version: 2.4.5
...
Requires-Dist: certifi (<2021.0.0)
...
Requires-Dist: certifi (>=2017.4.17)
...
```

A wrikle to this is that there’s a reproducibility issue, because the issue only happen when `certifi<2021.0.0` is fed into the resolver *before* `certifi>=2017.4.17`. But `pkg_resources` shuffles the `Requires-Dist` values on calculation, so to reproduce this deterministically, you need to change line 3041 of [`pkg_resources.DistInfoDistribution._compute_dependencies()`](https://github.com/pypa/pip/blob/7c3abccd22c4bbd6c1da2ae41845ffbd26c1bb17/src/pip/_vendor/pkg_resources/__init__.py#L3026-L3047) to something like:

```python
dm[None].extend(reqs_for_extra(None))
```

so the order of requirements don’t change between runs.

Now onto the actual resolver stuff. When the resolver pins through a candidate, it converts all of the candidate’s dependencies into criteria so they can be merged into existing criteria:

https://github.com/pypa/pip/blob/7c3abccd22c4bbd6c1da2ae41845ffbd26c1bb17/src/pip/_vendor/resolvelib/resolvers.py#L200-L205

But it uses a dict for the job. So when a candidate specifies a package multiple times (like snowflake-connector-python), the latter specification would override the previous one. Since `certifi<2021.0.0` is specified first (after we patch `pkg_resources`; without the patch the resolver has 50% chance to get the “right” one), it got lost during resolution.

I have to say, I haven’t had such an exercise debugging an issue like this for quite a while. This kind of things is why I’m doing open source work, so thanks for that, in a way 😆 I’m unfortunately quite busy tomorrow (it’s late Saturday here), but will try to find some time to fix asap.

Resolving patch:
diff --git a/src/pip/_vendor/resolvelib/__init__.py b/src/pip/_vendor/resolvelib/__init__.py
--- a/src/pip/_vendor/resolvelib/__init__.py
+++ b/src/pip/_vendor/resolvelib/__init__.py
@@ -11,7 +11,7 @@
     "ResolutionTooDeep",
 ]
 
-__version__ = "0.7.0"
+__version__ = "0.7.1"
 
 
 from .providers import AbstractProvider, AbstractResolver
diff --git a/src/pip/_vendor/resolvelib/resolvers.py b/src/pip/_vendor/resolvelib/resolvers.py
--- a/src/pip/_vendor/resolvelib/resolvers.py
+++ b/src/pip/_vendor/resolvelib/resolvers.py
@@ -134,11 +134,11 @@ def _push_new_state(self):
         )
         self._states.append(state)
 
-    def _merge_into_criterion(self, requirement, parent):
+    def _add_to_criteria(self, criteria, requirement, parent):
         self._r.adding_requirement(requirement=requirement, parent=parent)
 
         identifier = self._p.identify(requirement_or_candidate=requirement)
-        criterion = self.state.criteria.get(identifier)
+        criterion = criteria.get(identifier)
         if criterion:
             incompatibilities = list(criterion.incompatibilities)
         else:
@@ -147,12 +147,12 @@ def _merge_into_criterion(self, requirement, parent):
         matches = self._p.find_matches(
             identifier=identifier,
             requirements=IteratorMapping(
-                self.state.criteria,
+                criteria,
                 operator.methodcaller("iter_requirement"),
                 {identifier: [requirement]},
             ),
             incompatibilities=IteratorMapping(
-                self.state.criteria,
+                criteria,
                 operator.attrgetter("incompatibilities"),
                 {identifier: incompatibilities},
             ),
@@ -171,7 +171,7 @@ def _merge_into_criterion(self, requirement, parent):
         )
         if not criterion.candidates:
             raise RequirementsConflicted(criterion)
-        return identifier, criterion
+        criteria[identifier] = criterion
 
     def _get_preference(self, name):
         return self._p.get_preference(
@@ -197,11 +197,10 @@ def _is_current_pin_satisfying(self, name, criterion):
             for r in criterion.iter_requirement()
         )
 
-    def _get_criteria_to_update(self, candidate):
-        criteria = {}
-        for r in self._p.get_dependencies(candidate=candidate):
-            name, crit = self._merge_into_criterion(r, parent=candidate)
-            criteria[name] = crit
+    def _get_updated_criteria(self, candidate):
+        criteria = self.state.criteria.copy()
+        for requirement in self._p.get_dependencies(candidate=candidate):
+            self._add_to_criteria(criteria, requirement, parent=candidate)
         return criteria
 
     def _attempt_to_pin_criterion(self, name):
@@ -210,7 +209,7 @@ def _attempt_to_pin_criterion(self, name):
         causes = []
         for candidate in criterion.candidates:
             try:
-                criteria = self._get_criteria_to_update(candidate)
+                criteria = self._get_updated_criteria(candidate)
             except RequirementsConflicted as e:
                 causes.append(e.criterion)
                 continue
@@ -226,12 +225,13 @@ def _attempt_to_pin_criterion(self, name):
             if not satisfied:
                 raise InconsistentCandidate(candidate, criterion)
 
+            self._r.pinning(candidate=candidate)
+            self.state.criteria.update(criteria)
+
             # Put newly-pinned candidate at the end. This is essential because
             # backtracking looks at this mapping to get the last pin.
-            self._r.pinning(candidate=candidate)
             self.state.mapping.pop(name, None)
             self.state.mapping[name] = candidate
-            self.state.criteria.update(criteria)
 
             return []
 
@@ -338,10 +338,9 @@ def resolve(self, requirements, max_rounds):
         self._states = [State(mapping=collections.OrderedDict(), criteria={})]
         for r in requirements:
             try:
-                name, crit = self._merge_into_criterion(r, parent=None)
+                self._add_to_criteria(self.state.criteria, r, parent=None)
             except RequirementsConflicted as e:
                 raise ResolutionImpossible(e.criterion.information)
-            self.state.criteria[name] = crit
 
         # The root state is saved as a sentinel so the first ever pin can have
         # something to backtrack to if it fails. The root state is basically
diff --git a/src/pip/_vendor/resolvelib/structs.py b/src/pip/_vendor/resolvelib/structs.py
--- a/src/pip/_vendor/resolvelib/structs.py
+++ b/src/pip/_vendor/resolvelib/structs.py
@@ -75,6 +75,18 @@ def __init__(self, mapping, accessor, appends=None):
         self._accessor = accessor
         self._appends = appends or {}
 
+    def __repr__(self):
+        return "IteratorMapping({!r}, {!r}, {!r})".format(
+            self._mapping,
+            self._accessor,
+            self._appends,
+        )
+
+    def __bool__(self):
+        return bool(self._mapping or self._appends)
+
+    __nonzero__ = __bool__  # XXX: Python 2.
+
     def __contains__(self, key):
         return key in self._mapping or key in self._appends
 
@@ -90,7 +102,7 @@ def __iter__(self):
         return itertools.chain(self._mapping, more)
 
     def __len__(self):
-        more = len(k for k in self._appends if k not in self._mapping)
+        more = sum(1 for k in self._appends if k not in self._mapping)
         return len(self._mapping) + more
 
 


Test patch: