Repository: pypa/pip

Issue:
File encoding issue in _install_wheel (pip 20.2, python 2)
**Environment**

* pip version: 20.2
* Python version: 2.7
* OS: linux

**Description**

`pip install mr.bob` fails with pip 20.2:

```
Processing /home/sbi-local/.cache/pip/wheels/d3/64/58/cd7b9ff05f450397981fa37a0a279ed1d12cf763af16d8d7d7/mr.bob-0.1.2-py2-none-any.whl
Requirement already satisfied: Jinja2>=2.5.0 in /home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages (from mr.bob) (2.11.2)
Requirement already satisfied: six>=1.2.0 in /home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages (from mr.bob) (1.15.0)
Requirement already satisfied: setuptools in /home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages (from mr.bob) (44.1.1)
Requirement already satisfied: MarkupSafe>=0.23 in /home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages (from Jinja2>=2.5.0->mr.bob) (1.1.1)
Installing collected packages: mr.bob
ERROR: Exception:
Traceback (most recent call last):
  File "/home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
    status = self.run(options, args)
  File "/home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 421, in run
    pycompile=options.compile,
  File "/home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages/pip/_internal/req/__init__.py", line 90, in install_given_reqs
    pycompile=pycompile,
  File "/home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 831, in install
    requested=self.user_supplied,
  File "/home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages/pip/_internal/operations/install/wheel.py", line 829, in install_wheel
    requested=requested,
  File "/home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages/pip/_internal/operations/install/wheel.py", line 658, in _install_wheel
    file.save()
  File "/home/sbi-local/.virtualenvs/tempenv-7d78147760d35/lib/python2.7/site-packages/pip/_internal/operations/install/wheel.py", line 442, in save
    with self._zip_file.open(self.src_record_path) as f:
  File "/usr/lib/python2.7/zipfile.py", line 984, in open
    zinfo = self.getinfo(name)
  File "/usr/lib/python2.7/zipfile.py", line 932, in getinfo
    'There is no item named %r in the archive' % name)
KeyError: "There is no item named u'mrbob/tests/templates/encodingc\\u030c/mapc\\u030ca/c\\u0301a.bob' in the archive"
```

I've not had time to investigate if that error comes from a bug in that old package or is a bug in the new `_install_wheel`. It works fine with pip 20.1 though, as well as with pip 20.2 on python 3.


Discussion:
cc/ @chrahunt 
This looks like a regression in 20.2, so I'm tentatively adding it to the 20.2.1 milestone.

Encoding `src_record_path` in utf-8 before calling zipfile [open](https://github.com/pypa/pip/blob/e51a027964b524cccd7b3fe530dd59f87edc7d8f/src/pip/_internal/operations/install/wheel.py#L442) and [getinfo](https://github.com/pypa/pip/blob/e51a027964b524cccd7b3fe530dd59f87edc7d8f/src/pip/_internal/operations/install/wheel.py#L446) seems to fix the issue, although I'm not sure it's the right approach.

@chrahunt @uranusjr do you have an advice about this?
This sounds like a hairy topic. Quoting Python 2 documentation on `zipfile`:

> There is no official file name encoding for ZIP files. If you have unicode file names, you must convert them to byte strings in your desired encoding before passing them to `write()`. WinZip interprets all file names as encoded in CP437, also known as DOS Latin.

So I guess there is no “correct” way here. Python 3 [detects UTF-8 support with a header flag](https://github.com/python/cpython/blob/3.8/Lib/zipfile.py#L1549-L1553). I’m not familiar with the ZIP spec to comment on the approach, and would assume it’s the best we can do.
I posted #8684 for this.

Resolving patch:
diff --git a/src/pip/_internal/operations/install/wheel.py b/src/pip/_internal/operations/install/wheel.py
--- a/src/pip/_internal/operations/install/wheel.py
+++ b/src/pip/_internal/operations/install/wheel.py
@@ -78,6 +78,7 @@
         Union,
         cast,
     )
+    from zipfile import ZipInfo
 
     from pip._vendor.pkg_resources import Distribution
 
@@ -420,6 +421,15 @@ def __init__(self, src_record_path, dest_path, zip_file):
         self._zip_file = zip_file
         self.changed = False
 
+    def _getinfo(self):
+        # type: () -> ZipInfo
+        if not PY2:
+            return self._zip_file.getinfo(self.src_record_path)
+        # Python 2 does not expose a way to detect a ZIP's encoding, but the
+        # wheel specification (PEP 427) explicitly mandates that paths should
+        # use UTF-8, so we assume it is true.
+        return self._zip_file.getinfo(self.src_record_path.encode("utf-8"))
+
     def save(self):
         # type: () -> None
         # directory creation is lazy and after file filtering
@@ -439,11 +449,12 @@ def save(self):
         if os.path.exists(self.dest_path):
             os.unlink(self.dest_path)
 
-        with self._zip_file.open(self.src_record_path) as f:
+        zipinfo = self._getinfo()
+
+        with self._zip_file.open(zipinfo) as f:
             with open(self.dest_path, "wb") as dest:
                 shutil.copyfileobj(f, dest)
 
-        zipinfo = self._zip_file.getinfo(self.src_record_path)
         if zip_item_is_executable(zipinfo):
             set_extracted_file_to_default_mode_plus_executable(self.dest_path)
 


Test patch: