Repository: pypa/pip

Issue:
Pip 20.2.1 causes UnicodeDecodeError
**Environment**

* pip version: 20.2.1
* Python version: Python 3.7.3
* OS: Windows 10 Pro 1909

I using [venv](https://docs.python.org/3/tutorial/venv.html) environment.

My venv path has multi byte character, hangul, i think it may cause error.

**Description**

Pip 20.2.1 causes `UnicodeDecodeError` when install some packages. In my case, `rope`.

But I could downgrade pip to 20.1 using pip. 20.1 has no issue.

There are similar issue. (#8658) But my enviroment is different OS (Win10 vs CentOS) and Python. (3.7 vs 2.7) And traceback is different, too. I think both issues are not same.

**Expected behavior**


**How to Reproduce**

1. Install PIP 20.2.1
2. Try to install `rope`

**Output**

```
PS C:\Users\정한솔\source\repos\retinanet> & c:/Users/정한솔/source/repos/retinanet/venv/Scripts/Activate.ps1
(venv) PS C:\Users\정한솔\source\repos\retinanet> pip install rope
ERROR: Exception:
Traceback (most recent call last):
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\base_command.py", line 216, in _main
    status = self.run(options, args)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\commands\install.py", line 325, in run
    reqs, check_supported_wheels=not options.target_dir
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 236, in _check_skip_installed 
    req_to_install.check_if_exists(self.use_user_site)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\req\req_install.py", line 432, in check_if_exists
    existing_dist = get_distribution(self.req.name)
    dist = search_distribution(req_name)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 492, in search_distribution
    packages = get_installed_distributions(skip=())
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 477, in get_installed_distributions
    return [d for d in working_set
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 478, in <listcomp>
    if local_test(d) and
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 386, in dist_is_local
    return is_local(dist_location(dist))
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 566, in dist_location
    egg_link = egg_link_path(dist)
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 542, in egg_link_path
    if not virtualenv_no_global() and user_site:
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 111, in virtualenv_no_global
    return _no_global_under_venv()
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 72, in _no_global_under_venv
    cfg_lines = _get_pyvenv_cfg_lines()
  File "c:\users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 55, in _get_pyvenv_cfg_lines
    return f.read().splitlines()  # avoids trailing newlines
UnicodeDecodeError: 'cp949' codec can't decode byte 0xec in position 16: illegal multibyte sequence
(venv) PS C:\Users\정한솔\source\repos\retinanet> python -m pip install rope
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\base_command.py", line 216, in _main
    status = self.run(options, args)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\cli\req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\commands\install.py", line 325, in run
    reqs, check_supported_wheels=not options.target_dir
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 331, in _get_abstract_dist_for
    skip_reason = self._check_skip_installed(req)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 236, in _check_skip_installed
    req_to_install.check_if_exists(self.use_user_site)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\req\req_install.py", line 432, in check_if_exists
    existing_dist = get_distribution(self.req.name)
    dist = search_distribution(req_name)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 492, in search_distribution
    packages = get_installed_distributions(skip=())
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 477, in get_installed_distributions
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 478, in <listcomp>
    if local_test(d) and
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 386, in dist_is_local
    return is_local(dist_location(dist))
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 566, in dist_location
    egg_link = egg_link_path(dist)
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\misc.py", line 542, in egg_link_path
    if not virtualenv_no_global() and user_site:
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 111, in virtualenv_no_global
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 72, in _no_global_under_venv
    cfg_lines = _get_pyvenv_cfg_lines()
  File "C:\Users\정한솔\source\repos\retinanet\venv\lib\site-packages\pip\_internal\utils\virtualenv.py", line 55, in _get_pyvenv_cfg_lines
    return f.read().splitlines()  # avoids trailing newlines
UnicodeDecodeError: 'cp949' codec can't decode byte 0xec in position 16: illegal multibyte sequence
(venv) PS C:\Users\정한솔\source\repos\retinanet> python -m pip install pip==20.1 --ignore-installed
Collecting pip==20.1
  Using cached pip-20.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Successfully installed pip-20.2.1
(venv) PS C:\Users\정한솔\source\repos\retinanet> python -m pip install rope
Collecting rope
  Using cached rope-0.17.0.tar.gz (248 kB)
Building wheels for collected packages: rope
  Building wheel for rope (setup.py) ... done
  Created wheel for rope: filename=rope-0.17.0-py3-none-any.whl size=180521 sha256=bfb5eb13225f3cd93ec36b87982e799b8ee1b50c0d7a00cfde2864fff2160805
  Stored in directory: c:\users\정한솔\appdata\local\pip\cache\wheels\fc\68\52\627ca0d67f266c203ff5ef7e441036cf2049cdbb3e030c9e0a
Successfully built rope
Installing collected packages: rope
Successfully installed rope-0.17.0
(venv) PS C:\Users\정한솔\source\repos\retinanet> python --version
Python 3.7.3
```



Discussion:
The ticket you linked is a different issue. ~Is this specific to 20.2.1 (and presumably 20.2.0)? Or does the same happen on 20.1 and earlier?~ Edit: I missed the line you said 20.1 does not have this issue, sorry.
OK, I got this one sorted out. The root cause to the error is that `pyvenv.cfg` is opened with a different encoding from what it is written by venv.

`pyvenv.cfg` exists before 20.2, however, and is not the reason this pops up now. The trigger is #8695; this code path is not run prior to 20.2, but the redundant `dist_is_local` check causes `pyvenv.cfg` to be read, triggering this error when the file has incompatible encoding.

So this exact error should no longer be a problem once #8702 is merged. *But* the root exception cause remains, and could still be hit in the future in other situations. @hansoli68 would you be able to check what encoding your `pyvenv.cfg` is written with? [PEP 405](https://www.python.org/dev/peps/pep-0405/) does not seem to specify what encoding `pyvenv.cfg` should be written in, and/or upload the file somewhere?

We’ll need to have a conversation with CPython maintainers (and perhaps also virtualenv maintainers) to clarify this.

Resolving patch:
diff --git a/src/pip/_internal/utils/virtualenv.py b/src/pip/_internal/utils/virtualenv.py
--- a/src/pip/_internal/utils/virtualenv.py
+++ b/src/pip/_internal/utils/virtualenv.py
@@ -1,5 +1,6 @@
 from __future__ import absolute_import
 
+import io
 import logging
 import os
 import re
@@ -51,7 +52,9 @@ def _get_pyvenv_cfg_lines():
     """
     pyvenv_cfg_file = os.path.join(sys.prefix, 'pyvenv.cfg')
     try:
-        with open(pyvenv_cfg_file) as f:
+        # Although PEP 405 does not specify, the built-in venv module always
+        # writes with UTF-8. (pypa/pip#8717)
+        with io.open(pyvenv_cfg_file, encoding='utf-8') as f:
             return f.read().splitlines()  # avoids trailing newlines
     except IOError:
         return None


Test patch: