Repository: pypa/pip

Issue:
Error checking if package with dotted name is installed  (pip==20.2)
Hi,

I've got an issue while installing namespace distribution dependencies. While it worked fine with `pip==20.1.1` resolving fails with `pip==20.2`.

* **Python**: 3.8.2
* **Pip**: 20.2
* **Platform**:
```
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04 LTS
Release:	20.04
Codename:	focal
```

My namespace distributions follow [PEP420](https://www.python.org/dev/peps/pep-0420/). However, I use a slightly different naming scheme than proposed in https://github.com/pypa/sample-namespace-packages/tree/master/native.

To reproduce the issue the following steps need to be executed:

1. Clone https://github.com/pypa/sample-namespace-packages

```
$ git clone https://github.com/pypa/sample-namespace-packages.git && cd sample-namespace-packages
```

2. Apply the following diff:
```
$ git diff
diff --git a/native/pkg_a/setup.py b/native/pkg_a/setup.py
index 8021a7a..1238c0c 100644
--- a/native/pkg_a/setup.py
+++ b/native/pkg_a/setup.py
@@ -16,7 +16,7 @@ from setuptools import setup
 
 
 setup(
-    name='example_pkg_a',
+    name='example_pkg.a',
 
     version='1',
 
diff --git a/native/pkg_b/setup.py b/native/pkg_b/setup.py
index 872122a..1830dcc 100644
--- a/native/pkg_b/setup.py
+++ b/native/pkg_b/setup.py
@@ -16,7 +16,7 @@ from setuptools import setup
 
 
 setup(
-    name='example_pkg_b',
+    name='example_pkg.b',
 
     version='1',
 
@@ -27,6 +27,7 @@ setup(
     author_email='jonwayne@google.com',
 
     license='Apache Software License',
+    install_requires=['example_pkg.a==1'],
```

3. Check `pip` version:
```
$ pip --version
pip 20.2 from /home/ubuntu/work/projects/sample-namespace-packages/venv/lib/python3.8/site-packages/pip (python 3.8)
```

4. Install `pkg_a`:
```
$ pip install native/pkg_a/
Processing ./native/pkg_a
Using legacy 'setup.py install' for example-pkg.a, since package 'wheel' is not installed.
Installing collected packages: example-pkg.a
    Running setup.py install for example-pkg.a ... done
Successfully installed example-pkg.a-1
$ pip freeze
example-pkg.a==1
pkg-resources==0.0.0
```

5. Try to install `pkg_b` which should fail:
```
$ pip install native/pkg_b/
Processing ./native/pkg_b
ERROR: Could not find a version that satisfies the requirement example_pkg.a==1 (from example-pkg.b==1) (from versions: none)
ERROR: No matching distribution found for example_pkg.a==1 (from example-pkg.b==1)
```

Though, when naming the namespace distributions as described in https://github.com/pypa/sample-namespace-packages/tree/master/native i.e. with a `_` instead of a `.`, the distributions install correctly.

It took me a while until I noted that the issue is the naming convention, I didn't follow thoroughly.

**Question**: Is the change from `pip==20.1.1` to `pip==20.2` on purpose?

If you need any further details, let me know.


Discussion:
This seems like a normalisation issue. Dot, underscore, and dash are actually treated as equivalents when comparing package names, so pip is probably not doing that comparison correctly somewhere.
You refer to the snippet below? 

https://github.com/pypa/pip/blob/31299ee37058fafc84535ee3a47f0468433aeb20/src/pip/_vendor/packaging/utils.py#L16-L23
@damb that `canonicalize_name` should be fine. It's the way pip uses it that has a bug since 20.2.
Here is a reproducer for a variant of what is probably the same bug:

```bash
#!/bin/bash
set -ex
cd $(mktemp -d)
python3 -m venv venv
. venv/bin/activate
pip install -U "pip>=20.2" wheel py3o.formats
echo "from setuptools import setup; setup(name='myproject', install_requires=['py3o.formats'])" > setup.py
pip install --no-index -e .
```

So the bug is in `check_if_exists` and has been introduced in https://github.com/pypa/pip/pull/8054.

Apparently, passing a canonical name with dots replaced by dashes is not compatible with `pkg_resources.get_distribution()`

Example:

```console
$ pip install zope.inferface
$ python -c "import pkg_resources as pr; print(pr.get_distribution('zope.interface'))"
zope.interface 5.1.0
$ python -c "import pkg_resources as pr; print(pr.get_distribution('zope-interface'))"
...
pkg_resources.DistributionNotFound: The 'zope-interface' distribution was not found and is required by the application
```

You beat me to it 😛 Relevent code:

https://github.com/pypa/pip/blob/31299ee37058fafc84535ee3a47f0468433aeb20/src/pip/_vendor/pkg_resources/__init__.py#L1317-L1322

We’ll probably need to ask setuptools maintainers whether this is an oversight or a delibrate decision, and whether we can patch this logic ourselves.
Or alternatively, we can rollback #8054 (the root issue it tries to resolve is exactly `pkg_resources` does not handle dot in names well), and migrate wholesale to `importlib.metadata`.
Looks like we'd need a bugfix release.

> Or alternatively, we can rollback #8054

I like this plan. Anyone wanna file a PR for this?

I wanted to verify why `check_if_exists` still needs to call `pkg_resources.get_distribution` while we have our own `misc.get_distribution` which seems to do the right thing. I will probably not have time for that until the week-end or early next week though.

Oh, good point! IIRC that was planned as a follow-up PR (since the refactoring would be difficult to review if committed together with the `get_distribution()` change), but it never happened. I’ll try rewriting that and see if that’s enough to get this issue resolved.
I had filed an issue to remove the last remaining usage of `pkg_resources.get_distribution` (https://github.com/pypa/pip/issues/8550) as a follow-up of https://github.com/pypa/pip/pull/8054, but I was waiting on https://github.com/pypa/pip/pull/8114 to get merged first before proceeding on it further.


I experiemented a bit, and it seems like switching from calling `pkg_resources.get_distribution()` to pip’s own would indeed solve this particular issue.

Since this is a quite significant issue (there are many popular packages preferring to be referred by a dot in the name, e.g. coverage.py), I’m going to do the one required replacement in a separate PR, so we can get a bugfix version out as soon as possible. #8114 and the other replacements can happen afterwards.

Resolving patch:
diff --git a/src/pip/_internal/req/req_install.py b/src/pip/_internal/req/req_install.py
--- a/src/pip/_internal/req/req_install.py
+++ b/src/pip/_internal/req/req_install.py
@@ -429,25 +429,13 @@ def check_if_exists(self, use_user_site):
         """
         if self.req is None:
             return
-        # get_distribution() will resolve the entire list of requirements
-        # anyway, and we've already determined that we need the requirement
-        # in question, so strip the marker so that we don't try to
-        # evaluate it.
-        no_marker = Requirement(str(self.req))
-        no_marker.marker = None
-
-        # pkg_resources uses the canonical name to look up packages, but
-        # the name passed passed to get_distribution is not canonicalized
-        # so we have to explicitly convert it to a canonical name
-        no_marker.name = canonicalize_name(no_marker.name)
-        try:
-            self.satisfied_by = pkg_resources.get_distribution(str(no_marker))
-        except pkg_resources.DistributionNotFound:
+        existing_dist = get_distribution(self.req.name)
+        if not existing_dist:
             return
-        except pkg_resources.VersionConflict:
-            existing_dist = get_distribution(
-                self.req.name
-            )
+
+        existing_version = existing_dist.parsed_version
+        if not self.req.specifier.contains(existing_version, prereleases=True):
+            self.satisfied_by = None
             if use_user_site:
                 if dist_in_usersite(existing_dist):
                     self.should_reinstall = True
@@ -461,11 +449,13 @@ def check_if_exists(self, use_user_site):
             else:
                 self.should_reinstall = True
         else:
-            if self.editable and self.satisfied_by:
+            if self.editable:
                 self.should_reinstall = True
                 # when installing editables, nothing pre-existing should ever
                 # satisfy
                 self.satisfied_by = None
+            else:
+                self.satisfied_by = existing_dist
 
     # Things valid for wheels
     @property


Test patch:
diff --git a/tests/functional/test_install_upgrade.py b/tests/functional/test_install_upgrade.py
--- a/tests/functional/test_install_upgrade.py
+++ b/tests/functional/test_install_upgrade.py
@@ -1,3 +1,4 @@
+import itertools
 import os
 import sys
 import textwrap
@@ -7,6 +8,7 @@
 from tests.lib import pyversion  # noqa: F401
 from tests.lib import assert_all_changes
 from tests.lib.local_repos import local_checkout
+from tests.lib.wheel import make_wheel
 
 
 @pytest.mark.network
@@ -439,3 +441,34 @@ def prep_ve(self, script, version, pip_src, distribute=False):
             cwd=pip_src,
             expect_stderr=True,
         )
+
+
+@pytest.mark.parametrize("req1, req2", list(itertools.product(
+    ["foo.bar", "foo_bar", "foo-bar"], ["foo.bar", "foo_bar", "foo-bar"],
+)))
+def test_install_find_existing_package_canonicalize(script, req1, req2):
+    """Ensure an already-installed dist is found no matter how the dist name
+    was normalized on installation. (pypa/pip#8645)
+    """
+    # Create and install a package that's not available in the later stage.
+    req_container = script.scratch_path.joinpath("foo-bar")
+    req_container.mkdir()
+    req_path = make_wheel("foo_bar", "1.0").save_to_dir(req_container)
+    script.pip("install", "--no-index", req_path)
+
+    # Depend on the previously installed, but now unavailable package.
+    pkg_container = script.scratch_path.joinpath("pkg")
+    pkg_container.mkdir()
+    make_wheel(
+        "pkg",
+        "1.0",
+        metadata_updates={"Requires-Dist": req2},
+    ).save_to_dir(pkg_container)
+
+    # Ensure the previously installed package can be correctly used to match
+    # the dependency.
+    result = script.pip(
+        "install", "--no-index", "--find-links", pkg_container, "pkg",
+    )
+    satisfied_message = "Requirement already satisfied: {}".format(req2)
+    assert satisfied_message in result.stdout, str(result)