Repository: pypa/pip

Issue:
Set encoding for tar file and use unicode path for unpacking
When tarfile.TarFile decodes filenames in Python 2.7 by default it uses
sys.getfilesystemencoding. On Windows this returns "mbcs", which is
lossy when converting from proper utf-8 to bytes (results in '?' for out
of range characters).

We now pass an encoding to tarfile.open which will be used instead.
Since the encoding argument is only ever used for the PAX format, and
since the PAX format guarantees utf-8 encoded information, this should
work in all circumstances.

For filesystem APIs in Python 2, the type of the path object passed
dictates the underlying Windows API that is called. For `str` it is the
`*A` (for ANSI) APIs. For `unicode` it is the `*W` (for Wide character)
APIs. To use the second set of APIs, which properly handles unicode
filenames, we convert the byte path to utf-8.

Fixes #7667.
Filename encoding error in some environments with PAX sdist
**Environment**

* pip version: any
* Python version: 2.7
* OS: Windows, non-Windows in C locale

(pip Windows CI hits this)

**Description**
The PAX format wheel 0.34.1 sdists fail to install on Python 2.7 on Windows with a UnicodeEncodeError, or on non-Windows systems in a non-utf-8 locale: https://github.com/pypa/wheel/issues/331

**Expected behavior**
Unicode filename from the PAX tarball is correctly encoded for the local filesystem.

**How to Reproduce**
Attempt to install a PAX formatted tarball containing a file name that cannot be encoded to the default code page (Windows) or the default locale encoding (non-Windows).

In GNU tar, the affected paths are pre-mangled to something ASCII compatible, but PAX tar preserves them correctly, so the installer needs to handle them itself.

**Output**

See 
https://dev.azure.com/pypa/pip/_build/results?buildId=18040&view=logs&j=404e6841-f5ba-57d9-f2c8-8c5322057572&t=0219f6bf-240d-5b08-c877-377b12af5079&l=309 for a Windows example in the pip test suite.

The wheel issue linked above has some Linux examples.


Discussion:
This should resolve the issue on Windows, but non-Windows systems have the opposite problem: if the locale uses a non-universal encoding (e.g. ascii), then they'll trigger an encoding error when attempting to open the Unicode path.

There's no obviously correct answer for what to do in that case. Failing loudly at install time at least highlights the potential filename corruption issue, but another reasonable alternative would be to try encoding with the locale encoding first, and fall back to utf-8 if that fails (emitting a warning that the filename may not be correctly encoded due to locale issues).
I suggest that we ignore this because it's a Python 2 specific issue, and there's are more impactful tasks for volunteers to work on, than this problem that's only occurring on EoL Python versions.

It's our documented policy that pip's maintainers won't necessarily be solving problems that are Python 2 only.
According to #7667 this also affects Python < 3.7 on systems using less capable encodings (e.g. `C`)
(Back on a real computer rather than my phone)

The Windows aspect is genuinely fixed at the interpreter level, as of CPython 3.6 (the filesystem encoding is always UTF-8 instead of mbcs). The problem still exists in Python 3.5 as well as in 2.7 (Chris's patch fixes that, but at the risk of causing new problems on non-Windows systems).

For non-Windows systems, the problem exists by default in 3.5 and 3.6, and can still be induced in 3.7 or later by setting "LC_ALL=C" (since that will not only set a bad filesystem encoding, it will also turn off locale coercion). (I'm not actually sure what Python 2.7 will do, but I suspect it will just unconditionally pass the UTF-8 bytes from the PAX file to the local filesystem)

On non-Windows systems, the problem is amenable to an environment fix, which is "Ensure your installs run with a proper locale set". At the CPython level, we officially gave up on ever getting the C locale to actually work properly in a Unicode-centric world: https://www.python.org/dev/peps/pep-0011/#legacy-c-locale

So while I do think it would be nice to actually handle this at the pip level, it's also reasonable to tell people that hit these kinds of encoding issues to make sure that they at least set "LANG=C.UTF-8" in their build environments.
Closing since this is a Python-2 only fix, and, well... it's not been updated in quite a while!
I’d say this affects Python 3 as well (3.6 is supported for at least another year) since more and more people are running pip in containers, where `LANG` is usually set to `C`. Fixing this would save us from answering to support tickets asking this question, which IMO is worth it given the small changeset.
Reopening so I don’t forget to do something. I’ll probably file another PR against master later today to replace this.
Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the ``master`` branch into this pull request or rebase this pull request against ``master`` then it will be eligible for code review and hopefully merging!
@ncoghlan Just an FYI, the issue I noted on https://github.com/pypa/wheel/issues/331 was using Python 3.6 (in case that has any bearing here).
In the process of justifying not fixing this, I figured out enough to fix it. :( See #7668.
@johnthagen Yeah, the non-universal locale encoding problem I mention in https://github.com/pypa/pip/pull/7668#issuecomment-579706165 will apply Python 3 as well.

However 3.7+ mitigate it significantly, as they don't believe the OS when it claims to be using ASCII, and automatically switch to using UTF-8 instead.

Resolving patch:
diff --git a/src/pip/_internal/utils/unpacking.py b/src/pip/_internal/utils/unpacking.py
--- a/src/pip/_internal/utils/unpacking.py
+++ b/src/pip/_internal/utils/unpacking.py
@@ -178,7 +178,7 @@ def untar_file(filename, location):
             filename,
         )
         mode = "r:*"
-    tar = tarfile.open(filename, mode)
+    tar = tarfile.open(filename, mode, encoding="utf-8")
     try:
         leading = has_leading_dir([member.name for member in tar.getmembers()])
         for member in tar.getmembers():


Test patch:
diff --git a/tests/unit/test_utils_unpacking.py b/tests/unit/test_utils_unpacking.py
--- a/tests/unit/test_utils_unpacking.py
+++ b/tests/unit/test_utils_unpacking.py
@@ -168,6 +168,25 @@ def test_unpack_tar_success(self):
         untar_file(test_tar, self.tempdir)
 
 
+def test_unpack_tar_unicode(tmpdir):
+    test_tar = tmpdir / "test.tar"
+    # tarfile tries to decode incoming
+    with tarfile.open(
+        test_tar, "w", format=tarfile.PAX_FORMAT, encoding="utf-8"
+    ) as f:
+        metadata = tarfile.TarInfo("dir/åäö_日本語.py")
+        f.addfile(metadata, "hello world")
+
+    output_dir = tmpdir / "output"
+    output_dir.mkdir()
+
+    untar_file(test_tar, str(output_dir))
+
+    output_dir_name = str(output_dir)
+    contents = os.listdir(output_dir_name)
+    assert u"åäö_日本語.py" in contents
+
+
 @pytest.mark.parametrize('args, expected', [
     # Test the second containing the first.
     (('parent/sub', 'parent/'), False),