Repository: pypa/pip

Issue:
Pip 20.2 causes UnicodeDecodeError
<!--
If you're reporting an issue for `--unstable-feature=resolver`, use the "Dependency resolver failures / errors" template instead.
-->

**Environment**

* pip version: 20.2
* Python version: 2.7.5
* OS: CentOS 7.8.2003

<!-- Feel free to add more information about your environment here -->

**Description**

Installing some(?) packages fails with a `UnicodeDecodeError` exception.

**Expected behavior**

I expect the packages to be installed successfully like it does with 20.1

**How to Reproduce**

Build the following Dockerfile:

```dockerfile
FROM centos:7

RUN yum -y install python-virtualenv
RUN virtualenv --python=python2 env
RUN ./env/bin/pip install -U pip==20.2 setuptools==44.1.1 wheel==0.34.2
RUN ./env/bin/pip install ansible==2.7.9
```

```console
$ docker build -t pip-bug .
```

**Output**

```console
$ docker build -t pip-bug .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM centos:7
 ---> b5b4d78bc90c
Step 2/5 : RUN yum -y install python-virtualenv
 ---> Using cache
 ---> 809b2152d1a1
Step 3/5 : RUN virtualenv --python=python2 env
 ---> Using cache
 ---> 69dd1f9b3061
Step 4/5 : RUN ./env/bin/pip install -U pip==20.2 setuptools==44.1.1 wheel==0.34.2
 ---> Using cache
 ---> 4cfb8184c74b
Step 5/5 : RUN ./env/bin/pip install ansible==2.7.9
 ---> Running in bc33d8f35d95
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting ansible==2.7.9
  Downloading ansible-2.7.9.tar.gz (11.8 MB)
Collecting jinja2
  Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting PyYAML
  Downloading PyYAML-5.3.1.tar.gz (269 kB)
Collecting paramiko
  Downloading paramiko-2.7.1-py2.py3-none-any.whl (206 kB)
Collecting cryptography
  Downloading cryptography-3.0-cp27-cp27mu-manylinux2010_x86_64.whl (2.7 MB)
Requirement already satisfied: setuptools in /env/lib/python2.7/site-packages (from ansible==2.7.9) (44.1.1)
Collecting MarkupSafe>=0.23
  Downloading MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl (24 kB)
Collecting pynacl>=1.0.1
  Downloading PyNaCl-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl (964 kB)
Collecting bcrypt>=3.1.3
  Downloading bcrypt-3.1.7-cp27-cp27mu-manylinux1_x86_64.whl (59 kB)
Collecting six>=1.4.1
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting cffi!=1.11.3,>=1.8
  Downloading cffi-1.14.1-cp27-cp27mu-manylinux1_x86_64.whl (388 kB)
Collecting ipaddress; python_version < "3"
  Downloading ipaddress-1.0.23-py2.py3-none-any.whl (18 kB)
Collecting enum34; python_version < "3"
  Downloading enum34-1.1.10-py2-none-any.whl (11 kB)
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: ansible, PyYAML
  Building wheel for ansible (setup.py): started
  Building wheel for ansible (setup.py): finished with status 'done'
  Created wheel for ansible: filename=ansible-2.7.9-py2-none-any.whl size=9428759 sha256=9eb07c04d0a932220c25ea55fa20f5d52c10f8fd96914493bc0df59676c21891
  Stored in directory: /root/.cache/pip/wheels/bf/17/e3/c01fddfaa1e22530097c1b1933eecd8d1ddf5477f9e588006a
  Building wheel for PyYAML (setup.py): started
  Building wheel for PyYAML (setup.py): finished with status 'done'
  Created wheel for PyYAML: filename=PyYAML-5.3.1-cp27-cp27mu-linux_x86_64.whl size=45644 sha256=c50f35b17810fd226067a47388cdf0ec2decc00389c7cf39f5bbef02755857da
  Stored in directory: /root/.cache/pip/wheels/d1/d5/a0/3c27cdc8b0209c5fc1385afeee936cf8a71e13d885388b4be2
Successfully built ansible PyYAML
Installing collected packages: MarkupSafe, jinja2, PyYAML, six, pycparser, cffi, ipaddress, enum34, cryptography, pynacl, bcrypt, paramiko, ansible
Successfully installed MarkupSafe-1.1.1 PyYAML-5.3.1 ansible-2.7.9 bcrypt-3.1.7 cffi-1.14.1 cryptography-3.0 enum34-1.1.10 ipaddress-1.0.23 jinja2-2.11.2 paramiko-2.7.1 pycparser-2.20 pynacl-1.4.0 six-1.15.0
Traceback (most recent call last):
  File "./env/bin/pip", line 11, in <module>
    sys.exit(main())
  File "/env/lib/python2.7/site-packages/pip/_internal/cli/main.py", line 75, in main
    return command.main(cmd_args)
  File "/env/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 121, in main
    return self._main(args)
  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/env/lib/python2.7/site-packages/pip/_internal/cli/command_context.py", line 28, in main_context
    yield
  File "/env/lib/python2.7/site-packages/pip/_vendor/contextlib2.py", line 479, in __exit__
    _reraise_with_existing_context(exc_details)
  File "/env/lib/python2.7/site-packages/pip/_vendor/contextlib2.py", line 353, in _reraise_with_existing_context
    exec("raise exc_type, exc_value, exc_tb")
  File "/env/lib/python2.7/site-packages/pip/_vendor/contextlib2.py", line 468, in __exit__
    if cb(*exc_details):
  File "/env/lib/python2.7/site-packages/pip/_vendor/contextlib2.py", line 396, in _exit_wrapper
    return cm_exit(cm, *exc_details)
  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/env/lib/python2.7/site-packages/pip/_internal/utils/temp_dir.py", line 46, in global_tempdir_manager
    _tempdir_manager = old_tempdir_manager
  File "/env/lib/python2.7/site-packages/pip/_vendor/contextlib2.py", line 479, in __exit__
    _reraise_with_existing_context(exc_details)
  File "/env/lib/python2.7/site-packages/pip/_vendor/contextlib2.py", line 353, in _reraise_with_existing_context
    exec("raise exc_type, exc_value, exc_tb")
  File "/env/lib/python2.7/site-packages/pip/_vendor/contextlib2.py", line 468, in __exit__
    if cb(*exc_details):
  File "/env/lib/python2.7/site-packages/pip/_vendor/contextlib2.py", line 396, in _exit_wrapper
    return cm_exit(cm, *exc_details)
  File "/env/lib/python2.7/site-packages/pip/_internal/utils/temp_dir.py", line 175, in __exit__
    self.cleanup()
  File "/env/lib/python2.7/site-packages/pip/_internal/utils/temp_dir.py", line 199, in cleanup
    rmtree(ensure_text(self._path))
  File "/env/lib/python2.7/site-packages/pip/_vendor/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/env/lib/python2.7/site-packages/pip/_vendor/retrying.py", line 212, in call
    raise attempt.get()
  File "/env/lib/python2.7/site-packages/pip/_vendor/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/env/lib/python2.7/site-packages/pip/_vendor/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/env/lib/python2.7/site-packages/pip/_internal/utils/misc.py", line 139, in rmtree
    onerror=rmtree_errorhandler)
  File "/usr/lib64/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib64/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib64/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib64/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib64/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib64/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib64/python2.7/shutil.py", line 241, in rmtree
    fullname = os.path.join(path, name)
  File "/env/lib64/python2.7/posixpath.py", line 80, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128)
The command '/bin/sh -c ./env/bin/pip install ansible==2.7.9' returned a non-zero code: 1
```



Discussion:
What is the `LANG` environment variable in the container? I believe this is related to a change we made to always use Unicode for paths for Windows compatibility.
> What is the `LANG` environment variable in the container? I believe this is related to a change we made to always use Unicode for paths for Windows compatibility.

```console
$ locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
```
After making the following changes to the Dockerfile, it works.

```diff
 FROM centos:7

+ENV LANG en_US.utf8
+ENV LC_ALL en_US.utf8

 RUN yum -y install python-virtualenv
 RUN virtualenv --python=python2 env
 RUN ./env/bin/pip install -U pip==20.2 setuptools==44.1.1 wheel==0.34.2
 RUN ./env/bin/pip install ansible==2.7.9
```
If I understand correctly, this only affects Python 2. If this not the case, please do provide a reproduction case for a newer Python version. :)
> If I understand correctly, this only affects Python 2.

That's correct. I couldn't reproduce the issue using Python 3.6:

```diff
 FROM centos:7

-RUN yum -y install python-virtualenv
+RUN yum -y install python-virtualenv python3
-RUN virtualenv --python=python2 env
+RUN virtualenv --python=python3 env
 RUN ./env/bin/pip install -U pip==20.2 setuptools==44.1.1 wheel==0.34.2
 RUN ./env/bin/pip install ansible==2.7.9
```
Python 3 always use the text type for paths and handle the encoding correctly. I think the correct fix would be to only call `ensure_text` on Windows.

Resolving patch:
diff --git a/src/pip/_internal/utils/misc.py b/src/pip/_internal/utils/misc.py
--- a/src/pip/_internal/utils/misc.py
+++ b/src/pip/_internal/utils/misc.py
@@ -137,7 +137,7 @@ def get_prog():
 # Retry every half second for up to 3 seconds
 @retry(stop_max_delay=3000, wait_fixed=500)
 def rmtree(dir, ignore_errors=False):
-    # type: (Text, bool) -> None
+    # type: (AnyStr, bool) -> None
     shutil.rmtree(dir, ignore_errors=ignore_errors,
                   onerror=rmtree_errorhandler)
 
diff --git a/src/pip/_internal/utils/temp_dir.py b/src/pip/_internal/utils/temp_dir.py
--- a/src/pip/_internal/utils/temp_dir.py
+++ b/src/pip/_internal/utils/temp_dir.py
@@ -10,6 +10,7 @@
 from pip._vendor.contextlib2 import ExitStack
 from pip._vendor.six import ensure_text
 
+from pip._internal.utils.compat import WINDOWS
 from pip._internal.utils.misc import enum, rmtree
 from pip._internal.utils.typing import MYPY_CHECK_RUNNING
 
@@ -193,10 +194,17 @@ def cleanup(self):
         """Remove the temporary directory created and reset state
         """
         self._deleted = True
-        if os.path.exists(self._path):
-            # Make sure to pass unicode on Python 2 to make the contents also
-            # use unicode, ensuring non-ASCII names and can be represented.
+        if not os.path.exists(self._path):
+            return
+        # Make sure to pass unicode on Python 2 to make the contents also
+        # use unicode, ensuring non-ASCII names and can be represented.
+        # This is only done on Windows because POSIX platforms use bytes
+        # natively for paths, and the bytes-text conversion omission avoids
+        # errors caused by the environment configuring encodings incorrectly.
+        if WINDOWS:
             rmtree(ensure_text(self._path))
+        else:
+            rmtree(self._path)
 
 
 class AdjacentTempDirectory(TempDirectory):


Test patch: