Repository: pypa/pip

Issue:
pip>=20.2 needlessly re-installs dependencies available on the system in system-site-packages venv
**What did you want to do?**

We are using a venv that explicitly does use system-site-packages, in order to allow users in a container, which does not allow write access to /, to install their own packages while still using all the (partially very complex to install/build) packages provided by the container image.
This used to work fine until pip>=20.2, and is still broken in latest pip dev build as of today.

**Output**

```
Singularity> python3 -m venv --system-site-packages test_venv
Singularity> source ./test_venv/bin/activate
(test_venv) Singularity> pip list | grep numpy
numpy              1.19.1
WARNING: You are using pip version 20.1.1; however, version 20.2.1 is available.
You should consider upgrading via the '/home/users/fmaussion/test_venv/bin/python3 -m pip install --upgrade pip' command.
(test_venv) Singularity> pip install numpy
Requirement already satisfied: numpy in /usr/local/pyenv/versions/3.7.8/lib/python3.7/site-packages (1.19.1)
WARNING: You are using pip version 20.1.1; however, version 20.2.1 is available.
You should consider upgrading via the '/home/users/fmaussion/test_venv/bin/python3 -m pip install --upgrade pip' command.
(test_venv) Singularity> pip install --upgrade pip setuptools
Collecting pip
  Using cached pip-20.2.1-py2.py3-none-any.whl (1.5 MB)
Collecting setuptools
  Using cached setuptools-49.2.1-py3-none-any.whl (789 kB)
Installing collected packages: pip, setuptools
  Attempting uninstall: pip
    Found existing installation: pip 20.1.1
    Uninstalling pip-20.1.1:
      Successfully uninstalled pip-20.1.1
  Attempting uninstall: setuptools
    Found existing installation: setuptools 47.1.0
    Uninstalling setuptools-47.1.0:
      Successfully uninstalled setuptools-47.1.0
Successfully installed pip-20.2.1 setuptools-49.2.1
(test_venv) Singularity> pip install numpy
Collecting numpy
  Using cached numpy-1.19.1-cp37-cp37m-manylinux2010_x86_64.whl (14.5 MB)
Installing collected packages: numpy
Successfully installed numpy-1.19.1
```

As you can see, before updating pip to 20.2, it correctly notices that numpy is already installed and does not need to do anything.
After the update, it goes and installs a copy of the same numpy version in the venv.
The same happens when installing a package that depends on numpy. Also the same goes for any other dependency that's already provided by the system site-packages.
"pip list" correctly lists numpy as installed in either case.

**Additional information**

The reason we are doing it this way is because building some of those dependencies, which are binary and annoying to deal with (most prominently gdal and fiona). The singularity container image has all of them pre-built, with optimizations specific for the host systems CPU.

For now I've instructed users to either use --no-deps when installing things, or to explicitly pin pip to <20.2.


Discussion:
I can reproduce. It might be a side effect of changes made to `check_if_exists`.
This is due to the `local_only` argument of `search_distributions` that is True by default when called from `utils.misc.get_distribution()`.

Resolving patch:
diff --git a/src/pip/_internal/commands/search.py b/src/pip/_internal/commands/search.py
--- a/src/pip/_internal/commands/search.py
+++ b/src/pip/_internal/commands/search.py
@@ -140,6 +140,7 @@ def print_results(hits, name_column_width=None, terminal_width=None):
             write_output(line)
             if name in installed_packages:
                 dist = get_distribution(name)
+                assert dist is not None
                 with indent_log():
                     if dist.version == latest:
                         write_output('INSTALLED: %s (latest)', dist.version)
diff --git a/src/pip/_internal/utils/misc.py b/src/pip/_internal/utils/misc.py
--- a/src/pip/_internal/utils/misc.py
+++ b/src/pip/_internal/utils/misc.py
@@ -483,22 +483,39 @@ def user_test(d):
             ]
 
 
-def search_distribution(req_name):
+def _search_distribution(req_name):
+    # type: (str) -> Optional[Distribution]
+    """Find a distribution matching the ``req_name`` in the environment.
 
+    This searches from *all* distributions available in the environment, to
+    match the behavior of ``pkg_resources.get_distribution()``.
+    """
     # Canonicalize the name before searching in the list of
     # installed distributions and also while creating the package
     # dictionary to get the Distribution object
     req_name = canonicalize_name(req_name)
-    packages = get_installed_distributions(skip=())
+    packages = get_installed_distributions(
+        local_only=False,
+        skip=(),
+        include_editables=True,
+        editables_only=False,
+        user_only=False,
+        paths=None,
+    )
     pkg_dict = {canonicalize_name(p.key): p for p in packages}
     return pkg_dict.get(req_name)
 
 
 def get_distribution(req_name):
-    """Given a requirement name, return the installed Distribution object"""
+    # type: (str) -> Optional[Distribution]
+    """Given a requirement name, return the installed Distribution object.
+
+    This searches from *all* distributions available in the environment, to
+    match the behavior of ``pkg_resources.get_distribution()``.
+    """
 
     # Search the distribution by looking through the working set
-    dist = search_distribution(req_name)
+    dist = _search_distribution(req_name)
 
     # If distribution could not be found, call working_set.require
     # to update the working set, and try to find the distribution
@@ -514,7 +531,7 @@ def get_distribution(req_name):
             pkg_resources.working_set.require(req_name)
         except pkg_resources.DistributionNotFound:
             return None
-    return search_distribution(req_name)
+    return _search_distribution(req_name)
 
 
 def egg_link_path(dist):


Test patch:
diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py
--- a/tests/unit/test_utils.py
+++ b/tests/unit/test_utils.py
@@ -5,6 +5,7 @@
 
 """
 import codecs
+import itertools
 import os
 import shutil
 import stat
@@ -34,6 +35,7 @@
     build_url_from_netloc,
     egg_link_path,
     format_size,
+    get_distribution,
     get_installed_distributions,
     get_prog,
     hide_url,
@@ -192,26 +194,30 @@ def test_noegglink_in_sitepkgs_venv_global(self):
 @patch('pip._internal.utils.misc.dist_in_usersite')
 @patch('pip._internal.utils.misc.dist_is_local')
 @patch('pip._internal.utils.misc.dist_is_editable')
-class Tests_get_installed_distributions:
-    """test util.get_installed_distributions"""
-
-    workingset = [
-        Mock(test_name="global"),
-        Mock(test_name="editable"),
-        Mock(test_name="normal"),
-        Mock(test_name="user"),
-    ]
-
-    workingset_stdlib = [
+class TestsGetDistributions(object):
+    """Test get_installed_distributions() and get_distribution().
+    """
+    class MockWorkingSet(list):
+        def require(self, name):
+            pass
+
+    workingset = MockWorkingSet((
+        Mock(test_name="global", key="global"),
+        Mock(test_name="editable", key="editable"),
+        Mock(test_name="normal", key="normal"),
+        Mock(test_name="user", key="user"),
+    ))
+
+    workingset_stdlib = MockWorkingSet((
         Mock(test_name='normal', key='argparse'),
         Mock(test_name='normal', key='wsgiref')
-    ]
+    ))
 
-    workingset_freeze = [
+    workingset_freeze = MockWorkingSet((
         Mock(test_name='normal', key='pip'),
         Mock(test_name='normal', key='setuptools'),
         Mock(test_name='normal', key='distribute')
-    ]
+    ))
 
     def dist_is_editable(self, dist):
         return dist.test_name == "editable"
@@ -287,6 +293,46 @@ def test_freeze_excludes(self, mock_dist_is_editable,
             skip=('setuptools', 'pip', 'distribute'))
         assert len(dists) == 0
 
+    @pytest.mark.parametrize(
+        "working_set, req_name",
+        itertools.chain(
+            itertools.product([workingset], (d.key for d in workingset)),
+            itertools.product(
+                [workingset_stdlib], (d.key for d in workingset_stdlib),
+            ),
+        ),
+    )
+    def test_get_distribution(
+        self,
+        mock_dist_is_editable,
+        mock_dist_is_local,
+        mock_dist_in_usersite,
+        working_set,
+        req_name,
+    ):
+        """Ensure get_distribution() finds all kinds of distributions.
+        """
+        mock_dist_is_editable.side_effect = self.dist_is_editable
+        mock_dist_is_local.side_effect = self.dist_is_local
+        mock_dist_in_usersite.side_effect = self.dist_in_usersite
+        with patch("pip._vendor.pkg_resources.working_set", working_set):
+            dist = get_distribution(req_name)
+        assert dist is not None
+        assert dist.key == req_name
+
+    @patch('pip._vendor.pkg_resources.working_set', workingset)
+    def test_get_distribution_nonexist(
+        self,
+        mock_dist_is_editable,
+        mock_dist_is_local,
+        mock_dist_in_usersite,
+    ):
+        mock_dist_is_editable.side_effect = self.dist_is_editable
+        mock_dist_is_local.side_effect = self.dist_is_local
+        mock_dist_in_usersite.side_effect = self.dist_in_usersite
+        dist = get_distribution("non-exist")
+        assert dist is None
+
 
 def test_rmtree_errorhandler_nonexistent_directory(tmpdir):
     """