Repository: pypa/pip

Issue:
pip sometimes prohibits 2 .dist-info directories in wheels
**Environment**

* pip version: 19.3.1
* Python version: 3.8.0
* OS: Linux

**Description**

Currently, pip rejects wheel files containing multiple `.dist-info` directories, but only if they start with the name of the package being installed.

**Expected behavior**

pip should prohibit more than 1 top-level `.dist-info` directory, without regard to the name of the directories (besides ending with `.dist-info`).

From [PEP 427](https://www.python.org/dev/peps/pep-0427/#abstract):

> A wheel is a ZIP-format archive with a specially formatted file name and the `.whl` extension. It contains a **single** distribution

And from [PEP 376](https://www.python.org/dev/peps/pep-0376/#one-dist-info-directory-per-installed-distribution):

> One .dist-info directory per installed distribution


**How to Reproduce**

Save `t.sh` below, make it executable, and execute it.

<details>
<summary>t.sh</summary>

```
#!/bin/sh
cd "$(mktemp -d)"
python -m venv env
. env/bin/activate
pip install --upgrade pip wheel

echo "= Versions ====================================================================="
python -V
pip -V
wheel version

echo "= Setup ========================================================================"
echo "from setuptools import setup; setup(name='hello')" > setup.py

echo "= (1) Base case ================================================================"
echo "= (1) Build ===================================================================="
python setup.py bdist_wheel

echo "= (1) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl

echo "= (2) 2 .dist-info dirs (expected) ============================================="
echo "= (2) Build ===================================================================="
python setup.py bdist_wheel

echo "= (2) Add second .dist-info (shared prefix) ===================================="
mkdir hello-example-0.0.0.dist-info
zip dist/hello-0.0.0-py3-none-any.whl hello-example-0.0.0.dist-info

echo "= (2) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl

echo "= (3) 2 .dist-info dirs (NOT expected) ========================================="
echo "= (3) Build ===================================================================="
python setup.py bdist_wheel

echo "= (3) Add second .dist-info (no shared prefix) ================================="
mkdir goodbye-example-0.0.0.dist-info
zip dist/hello-0.0.0-py3-none-any.whl goodbye-example-0.0.0.dist-info

echo "= (3) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl
```

</details>

**Output**

<details>
<summary>Output</summary>

```
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
Installing collected packages: pip, wheel
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3.1 wheel-0.33.6
= Versions =====================================================================
Python 3.8.0
pip 19.3.1 from /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip (python 3.8)
wheel 0.33.6
= Setup ========================================================================
= (1) Base case ================================================================
= (1) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
creating hello.egg-info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
writing manifest file 'hello.egg-info/SOURCES.txt'
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (1) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
Successfully installed hello-0.0.0
= (2) 2 .dist-info dirs (expected) =============================================
= (2) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (2) Add second .dist-info (shared prefix) ====================================
  adding: hello-example-0.0.0.dist-info/ (stored 0%)
= (2) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 446, in run
    installed = install_given_reqs(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 58, in install_given_reqs
    requirement.install(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 858, in install
    self.move_wheel_files(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 487, in move_wheel_files
    wheel.move_wheel_files(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/wheel.py", line 461, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/wheel.py", line 408, in clobber
    assert not info_dir, ('Multiple .dist-info directories: ' +
AssertionError: Multiple .dist-info directories: /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/hello-example-0.0.0.dist-info, /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/hello-0.0.0.dist-info
= (3) 2 .dist-info dirs (NOT expected) =========================================
= (3) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (3) Add second .dist-info (no shared prefix) =================================
  adding: goodbye-example-0.0.0.dist-info/ (stored 0%)
= (3) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
Successfully installed hello-0.0.0
```

</details>

The unexpected case is `(3) Install` - the installation should not go through since there are multiple `.dist-info` directories.


Discussion:
Does pip allow multiple dist-info dirs (in general) to handle installing from flat directory? (It’s the case IIRC.) I kind of feel the responsibility to produce a valid wheel should fall on to the wheel *builder* (so build backend).

pip can be free to interpret and invalid wheel and do what’s most comfortable (e.g. choose one by whatever logic, as the current behaviour) since the input already breaks the contract. It is also a valid behaviour to error out as well, but I wouldn’t say it’s necessary.
I think erroring out results in the best end-user experience here -- you gave pip something invalid, so it did the wrong thing vs you gave pip something invalid and it refused to use it.
> Does pip allow multiple dist-info dirs (in general) to handle installing from flat directory?

Good question. The current behavior is [here](https://github.com/pypa/pip/blob/81805a5776767b17533afc934121968f3c03a4d9/src/pip/_internal/operations/build/metadata_legacy.py#L61-L78) - we essentially guess. I think this use case is different from installing a wheel because the wheel format has a spec and we control everything about the directory it is unpacked into.

> I kind of feel the responsibility to produce a valid wheel should fall on to the wheel _builder_ (so build backend).

Agreed.

> pip can be free to interpret and invalid wheel and do what’s most comfortable

I am trying to refactor some of our wheel installation logic. The purpose of calling out the expected behavior here is to make it easier to make things more comfortable. :)

Our current validation approach has lead to situations like this one, but extracting it while preserving the exact same behavior would result in code that is harder to follow and doesn't really map to the spec. I expect taking the stricter approach will lead to fewer "print better errors" issues, since we can print a nice error upfront, at least in this case.
I’m totally on board if the behaviour change would improve the code quality.

Resolving patch:
diff --git a/src/pip/_internal/operations/install/wheel.py b/src/pip/_internal/operations/install/wheel.py
--- a/src/pip/_internal/operations/install/wheel.py
+++ b/src/pip/_internal/operations/install/wheel.py
@@ -315,8 +315,10 @@ def install_unpacked_wheel(
     :param pycompile: Whether to byte-compile installed Python files
     :param warn_script_location: Whether to check that scripts are installed
         into a directory on PATH
-    :raises UnsupportedWheel: when the directory holds an unpacked wheel with
-        incompatible Wheel-Version
+    :raises UnsupportedWheel:
+        * when the directory holds an unpacked wheel with incompatible
+          Wheel-Version
+        * when the .dist-info dir does not match the wheel
     """
     # TODO: Investigate and break this up.
     # TODO: Look into moving this into a dedicated class for representing an
@@ -381,8 +383,7 @@ def clobber(
                     continue
                 elif (
                     is_base and
-                    s.endswith('.dist-info') and
-                    canonicalize_name(s).startswith(canonicalize_name(name))
+                    s.endswith('.dist-info')
                 ):
                     assert not info_dir, (
                         'Multiple .dist-info directories: {}, '.format(
@@ -445,6 +446,15 @@ def clobber(
         req_description
     )
 
+    info_dir_name = canonicalize_name(os.path.basename(info_dir[0]))
+    canonical_name = canonicalize_name(name)
+    if not info_dir_name.startswith(canonical_name):
+        raise UnsupportedWheel(
+            "{} .dist-info directory {!r} does not start with {!r}".format(
+                req_description, os.path.basename(info_dir[0]), canonical_name
+            )
+        )
+
     # Get the defined entry points
     ep_file = os.path.join(info_dir[0], 'entry_points.txt')
     console, gui = get_entrypoints(ep_file)


Test patch:
diff --git a/tests/functional/test_install_wheel.py b/tests/functional/test_install_wheel.py
--- a/tests/functional/test_install_wheel.py
+++ b/tests/functional/test_install_wheel.py
@@ -1,9 +1,11 @@
 import distutils
 import glob
 import os
+import shutil
 
 import pytest
 
+from tests.lib import create_basic_wheel_for_package
 from tests.lib.path import Path
 
 
@@ -432,3 +434,41 @@ def test_wheel_install_with_no_cache_dir(script, tmpdir, data):
     package = data.packages.joinpath("simple.dist-0.1-py2.py3-none-any.whl")
     result = script.pip('install', '--no-cache-dir', '--no-index', package)
     result.assert_installed('simpledist', editable=False)
+
+
+def test_wheel_install_fails_with_extra_dist_info(script):
+    package = create_basic_wheel_for_package(
+        script,
+        "simple",
+        "0.1.0",
+        extra_files={
+            "unrelated-2.0.0.dist-info/WHEEL": "Wheel-Version: 1.0",
+            "unrelated-2.0.0.dist-info/METADATA": (
+                "Name: unrelated\nVersion: 2.0.0\n"
+            ),
+        },
+    )
+    result = script.pip(
+        "install", "--no-cache-dir", "--no-index", package, expect_error=True
+    )
+    assert "Multiple .dist-info directories" in result.stderr
+
+
+def test_wheel_install_fails_with_unrelated_dist_info(script):
+    package = create_basic_wheel_for_package(script, "simple", "0.1.0")
+    new_name = "unrelated-2.0.0-py2.py3-none-any.whl"
+    new_package = os.path.join(os.path.dirname(package), new_name)
+    shutil.move(package, new_package)
+
+    result = script.pip(
+        "install",
+        "--no-cache-dir",
+        "--no-index",
+        new_package,
+        expect_error=True,
+    )
+
+    assert (
+        "'simple-0.1.0.dist-info' does not start with 'unrelated'"
+        in result.stderr
+    )
diff --git a/tests/lib/__init__.py b/tests/lib/__init__.py
--- a/tests/lib/__init__.py
+++ b/tests/lib/__init__.py
@@ -921,8 +921,9 @@ def create_test_package_with_setup(script, **setup_kwargs):
     return pkg_path
 
 
-def create_basic_wheel_for_package(script, name, version,
-                                   depends=None, extras=None):
+def create_basic_wheel_for_package(
+    script, name, version, depends=None, extras=None, extra_files=None
+):
     if depends is None:
         depends = []
     if extras is None:
@@ -966,6 +967,9 @@ def hello():
         "{dist_info}/RECORD": ""
     }
 
+    if extra_files:
+        files.update(extra_files)
+
     # Some useful shorthands
     archive_name = "{name}-{version}-py2.py3-none-any.whl".format(
         name=name, version=version