Repository: pypa/setuptools

Issue:
setup.cfg should standardize on UTF-8 for encoding
In jaraco/configparser#34, I learned that although setuptools v40.7.0 presumably added support for non-ASCII, there are still environments where loading non-ASCII is failing.

```
configparser # easy_install --version
setuptools 40.8.0 from c:\python37\lib\site-packages (Python 3.7)
configparser 3.7.2 # python setup.py egg_info
Traceback (most recent call last):
  File "setup.py", line 5, in <module>
    package_dir={'': 'src'},  
  File "C:\Python37\lib\site-packages\setuptools\__init__.py", line 144, in setup
    _install_setup_requires(attrs)  
  File "C:\Python37\lib\site-packages\setuptools\__init__.py", line 137, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "C:\Python37\lib\site-packages\setuptools\dist.py", line 702, in parse_config_files
    self._parse_config_files(filenames=filenames)
  File "C:\Python37\lib\site-packages\setuptools\dist.py", line 599, in _parse_config_files
    (parser.read_file if six.PY3 else parser.readfp)(reader)
  File "C:\Python37\lib\configparser.py", line 717, in read_file
    self._read(f, source)
  File "C:\Python37\lib\configparser.py", line 1014, in _read
    for lineno, line in enumerate(fp, start=1):
  File "C:\Python37\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 103: character maps to <undefined>
```
setup.cfg should standardize on UTF-8 for encoding
In jaraco/configparser#34, I learned that although setuptools v40.7.0 presumably added support for non-ASCII, there are still environments where loading non-ASCII is failing.

```
configparser # easy_install --version
setuptools 40.8.0 from c:\python37\lib\site-packages (Python 3.7)
configparser 3.7.2 # python setup.py egg_info
Traceback (most recent call last):
  File "setup.py", line 5, in <module>
    package_dir={'': 'src'},  
  File "C:\Python37\lib\site-packages\setuptools\__init__.py", line 144, in setup
    _install_setup_requires(attrs)  
  File "C:\Python37\lib\site-packages\setuptools\__init__.py", line 137, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "C:\Python37\lib\site-packages\setuptools\dist.py", line 702, in parse_config_files
    self._parse_config_files(filenames=filenames)
  File "C:\Python37\lib\site-packages\setuptools\dist.py", line 599, in _parse_config_files
    (parser.read_file if six.PY3 else parser.readfp)(reader)
  File "C:\Python37\lib\configparser.py", line 717, in read_file
    self._read(f, source)
  File "C:\Python37\lib\configparser.py", line 1014, in _read
    for lineno, line in enumerate(fp, start=1):
  File "C:\Python37\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 103: character maps to <undefined>
```


Discussion:
Hmm. On further investigation, I discovered that [encoding detection is employed](https://github.com/pypa/setuptools/blob/6575418283ec605f6bca82c525f6680d897ce282/setuptools/dist.py#L593), meaning that adding `# coding: utf-8` to the file corrected the issue.
In [this comment](https://github.com/pypa/setuptools/issues/1661#issuecomment-465088372), the user suggests the default encoding should be UTF-8. I agree and in fact I'd like to work toward the only encoding to be UTF-8 (drop support for encoding detection).
I see that the encoding detection was added in #1180. @benoit-pierre, do you recall why you chose to have the encoding declared rather than demanding and relying on UTF-8?
Backward compatibility: to no break some existing workflows (using an encoding other than UTF-8 with a corresponding locale).
My suggestion was to at least reduce the prevalence of this problem, but detecting *intentionally* non-user locales like "POSIX" or "C", and upgrade to UTF-8 for those.

Requiring people to add a `coding` declaration to non-coding files in every PyPI package containing a non-ascii property, such as author name, is horribly unnecessary churn.

If a user has a specific locale set, that can be a problem solved another day.  But again, if the setup.cfg cant be decoded with the user locale, the fallback should be to attempt to decode it with the best guess of the authors locale - utf-8 , especially for any package which has been downloaded from pypi which strongly implies the users locale is irrelevant as the file is not written by the user, but by an author on the other side of the world.
Also related, I believe pytest was reading it was ascii, but now defaults to utf-8.  It is causing breakages in pytest 3.3.2, at least, but not in current pytest.
I am strongly inclined to _assume_ UTF-8 (which also supports ASCII). I'm also inclined to remove support for the `coding` unless there's a strong use-case for it.
@jayvdb Would you be willing to put together a PR?
Hi @jaraco , I have a related issue.
setup.cfg contains Unicode symbols and set explicit UTF-8 encoding:
```
"# -*- coding: utf-8 -*-                                                         
[metadata]
...
```
When I run tox to test itself under **Python2**:
```
Processing ./.tox/.tmp/package/2/tox-3.8.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/src/tmp/pip-req-build-jUmuTe/setup.py", line 18, in <module>
        package_dir={"": "src"},
      File "/usr/src/RPM/BUILD/python-module-tox-3.8.0/.tox/py27/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/usr/src/RPM/BUILD/python-module-tox-3.8.0/.tox/py27/lib/python2.7/site-packages/setuptools/__init__.py", line 137, in _install_setup_requires
        dist.parse_config_files(ignore_option_errors=True)
      File "/usr/src/RPM/BUILD/python-module-tox-3.8.0/.tox/py27/lib/python2.7/site-packages/setuptools/dist.py", line 702, in parse_config_files
        self._parse_config_files(filenames=filenames)
      File "/usr/src/RPM/BUILD/python-module-tox-3.8.0/.tox/py27/lib/python2.7/site-packages/setuptools/dist.py", line 599, in _parse_config_files
        (parser.read_file if six.PY3 else parser.readfp)(reader)
      File "/usr/lib64/python2.7/ConfigParser.py", line 324, in readfp
        self._read(fp, filename)
      File "/usr/lib64/python2.7/ConfigParser.py", line 479, in _read
        line = fp.readline()
      File "/usr/src/RPM/BUILD/python-module-tox-3.8.0/.tox/py27/lib64/python2.7/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 345: ordinal not in range(128)
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /usr/src/tmp/pip-req-build-jUmuTe/
```

This is because "edit_config" doesn't pass down the original encoding:
```
(Pdb) bt
  /usr/src/RPM/BUILD/python-module-tox-3.8.0/setup.py(18)<module>()
-> package_dir={"": "src"},
  /usr/lib/python2.7/site-packages/setuptools/__init__.py(145)setup()
-> return distutils.core.setup(**attrs)
  /usr/lib64/python2.7/distutils/core.py(151)setup()
-> dist.run_commands()
  /usr/lib64/python2.7/distutils/dist.py(953)run_commands()
-> self.run_command(cmd)
  /usr/lib64/python2.7/distutils/dist.py(972)run_command()
-> cmd_obj.run()
  /usr/lib/python2.7/site-packages/setuptools/command/sdist.py(54)run()
-> self.make_distribution()
  /usr/lib/python2.7/site-packages/setuptools/command/sdist.py(78)make_distribution()
-> orig.sdist.make_distribution(self)
  /usr/lib64/python2.7/distutils/command/sdist.py(456)make_distribution()
-> self.make_release_tree(base_dir, self.filelist.files)
  /usr/lib/python2.7/site-packages/setuptools/command/sdist.py(168)make_release_tree()
-> self.get_finalized_command('egg_info').save_version_info(dest)
  /usr/lib/python2.7/site-packages/setuptools/command/egg_info.py(191)save_version_info()
-> edit_config(filename, dict(egg_info=egg_info))
> /usr/lib/python2.7/site-packages/setuptools/command/setopt.py(74)edit_config()
-> opts.write(f)
(Pdb)
```
The output is something like:
```
[metadata]
name = tox
```

```
locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
```
Given that `setopt` and its `edit_config` function need to write to the config file, I'm even more strongly inclined now to remove support for specifying an encoding in setup.cfg files and instead insist on UTF-8, especially since commands like `bdist_rpm` invoke `egg_info` which in turn rewrites the config file.
Hmm. On further investigation, I discovered that [encoding detection is employed](https://github.com/pypa/setuptools/blob/6575418283ec605f6bca82c525f6680d897ce282/setuptools/dist.py#L593), meaning that adding `# coding: utf-8` to the file corrected the issue.
In [this comment](https://github.com/pypa/setuptools/issues/1661#issuecomment-465088372), the user suggests the default encoding should be UTF-8. I agree and in fact I'd like to work toward the only encoding to be UTF-8 (drop support for encoding detection).
I see that the encoding detection was added in #1180. @benoit-pierre, do you recall why you chose to have the encoding declared rather than demanding and relying on UTF-8?
Backward compatibility: to no break some existing workflows (using an encoding other than UTF-8 with a corresponding locale).
My suggestion was to at least reduce the prevalence of this problem, but detecting *intentionally* non-user locales like "POSIX" or "C", and upgrade to UTF-8 for those.

Requiring people to add a `coding` declaration to non-coding files in every PyPI package containing a non-ascii property, such as author name, is horribly unnecessary churn.

If a user has a specific locale set, that can be a problem solved another day.  But again, if the setup.cfg cant be decoded with the user locale, the fallback should be to attempt to decode it with the best guess of the authors locale - utf-8 , especially for any package which has been downloaded from pypi which strongly implies the users locale is irrelevant as the file is not written by the user, but by an author on the other side of the world.
Also related, I believe pytest was reading it was ascii, but now defaults to utf-8.  It is causing breakages in pytest 3.3.2, at least, but not in current pytest.
I am strongly inclined to _assume_ UTF-8 (which also supports ASCII). I'm also inclined to remove support for the `coding` unless there's a strong use-case for it.
@jayvdb Would you be willing to put together a PR?
Hi @jaraco , I have a related issue.
setup.cfg contains Unicode symbols and set explicit UTF-8 encoding:
```
"# -*- coding: utf-8 -*-                                                         
[metadata]
...
```
When I run tox to test itself under **Python2**:
```
Processing ./.tox/.tmp/package/2/tox-3.8.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/src/tmp/pip-req-build-jUmuTe/setup.py", line 18, in <module>
        package_dir={"": "src"},
      File "/usr/src/RPM/BUILD/python-module-tox-3.8.0/.tox/py27/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/usr/src/RPM/BUILD/python-module-tox-3.8.0/.tox/py27/lib/python2.7/site-packages/setuptools/__init__.py", line 137, in _install_setup_requires
        dist.parse_config_files(ignore_option_errors=True)
      File "/usr/src/RPM/BUILD/python-module-tox-3.8.0/.tox/py27/lib/python2.7/site-packages/setuptools/dist.py", line 702, in parse_config_files
        self._parse_config_files(filenames=filenames)
      File "/usr/src/RPM/BUILD/python-module-tox-3.8.0/.tox/py27/lib/python2.7/site-packages/setuptools/dist.py", line 599, in _parse_config_files
        (parser.read_file if six.PY3 else parser.readfp)(reader)
      File "/usr/lib64/python2.7/ConfigParser.py", line 324, in readfp
        self._read(fp, filename)
      File "/usr/lib64/python2.7/ConfigParser.py", line 479, in _read
        line = fp.readline()
      File "/usr/src/RPM/BUILD/python-module-tox-3.8.0/.tox/py27/lib64/python2.7/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 345: ordinal not in range(128)
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /usr/src/tmp/pip-req-build-jUmuTe/
```

This is because "edit_config" doesn't pass down the original encoding:
```
(Pdb) bt
  /usr/src/RPM/BUILD/python-module-tox-3.8.0/setup.py(18)<module>()
-> package_dir={"": "src"},
  /usr/lib/python2.7/site-packages/setuptools/__init__.py(145)setup()
-> return distutils.core.setup(**attrs)
  /usr/lib64/python2.7/distutils/core.py(151)setup()
-> dist.run_commands()
  /usr/lib64/python2.7/distutils/dist.py(953)run_commands()
-> self.run_command(cmd)
  /usr/lib64/python2.7/distutils/dist.py(972)run_command()
-> cmd_obj.run()
  /usr/lib/python2.7/site-packages/setuptools/command/sdist.py(54)run()
-> self.make_distribution()
  /usr/lib/python2.7/site-packages/setuptools/command/sdist.py(78)make_distribution()
-> orig.sdist.make_distribution(self)
  /usr/lib64/python2.7/distutils/command/sdist.py(456)make_distribution()
-> self.make_release_tree(base_dir, self.filelist.files)
  /usr/lib/python2.7/site-packages/setuptools/command/sdist.py(168)make_release_tree()
-> self.get_finalized_command('egg_info').save_version_info(dest)
  /usr/lib/python2.7/site-packages/setuptools/command/egg_info.py(191)save_version_info()
-> edit_config(filename, dict(egg_info=egg_info))
> /usr/lib/python2.7/site-packages/setuptools/command/setopt.py(74)edit_config()
-> opts.write(f)
(Pdb)
```
The output is something like:
```
[metadata]
name = tox
```

```
locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
```
Given that `setopt` and its `edit_config` function need to write to the config file, I'm even more strongly inclined now to remove support for specifying an encoding in setup.cfg files and instead insist on UTF-8, especially since commands like `bdist_rpm` invoke `egg_info` which in turn rewrites the config file.

Resolving patch:
diff --git a/setuptools/dist.py b/setuptools/dist.py
--- a/setuptools/dist.py
+++ b/setuptools/dist.py
@@ -35,7 +35,6 @@
 from setuptools import windows_support
 from setuptools.monkey import get_unpatched
 from setuptools.config import parse_configuration
-from .unicode_utils import detect_encoding
 import pkg_resources
 
 __import__('setuptools.extern.packaging.specifiers')
@@ -587,13 +586,9 @@ def _parse_config_files(self, filenames=None):
 
         parser = ConfigParser()
         for filename in filenames:
-            with io.open(filename, 'rb') as fp:
-                encoding = detect_encoding(fp)
+            with io.open(filename, encoding='utf-8') as reader:
                 if DEBUG:
-                    self.announce("  reading %s [%s]" % (
-                        filename, encoding or 'locale')
-                    )
-                reader = io.TextIOWrapper(fp, encoding=encoding)
+                    self.announce("  reading {filename}".format(**locals()))
                 (parser.read_file if six.PY3 else parser.readfp)(reader)
             for section in parser.sections():
                 options = parser.options(section)
diff --git a/setuptools/unicode_utils.py b/setuptools/unicode_utils.py
--- a/setuptools/unicode_utils.py
+++ b/setuptools/unicode_utils.py
@@ -1,6 +1,5 @@
 import unicodedata
 import sys
-import re
 
 from setuptools.extern import six
 
@@ -43,15 +42,3 @@ def try_encode(string, enc):
         return string.encode(enc)
     except UnicodeEncodeError:
         return None
-
-
-CODING_RE = re.compile(br'^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)')
-
-
-def detect_encoding(fp):
-    first_line = fp.readline()
-    fp.seek(0)
-    m = CODING_RE.match(first_line)
-    if m is None:
-        return None
-    return m.group(1).decode('ascii')


Test patch:
diff --git a/setuptools/tests/test_config.py b/setuptools/tests/test_config.py
--- a/setuptools/tests/test_config.py
+++ b/setuptools/tests/test_config.py
@@ -9,7 +9,6 @@
 from setuptools.dist import Distribution, _Distribution
 from setuptools.config import ConfigHandler, read_configuration
 from setuptools.extern.six.moves import configparser
-from setuptools.tests import is_ascii
 from . import py2_only, py3_only
 from .textwrap import DALS
 
@@ -446,10 +445,6 @@ def test_interpolation(self, tmpdir):
             with get_dist(tmpdir):
                 pass
 
-    skip_if_not_ascii = pytest.mark.skipif(
-        not is_ascii, reason='Test not supported with this locale')
-
-    @skip_if_not_ascii
     def test_non_ascii_1(self, tmpdir):
         fake_env(
             tmpdir,
@@ -457,18 +452,8 @@ def test_non_ascii_1(self, tmpdir):
             'description = éàïôñ\n',
             encoding='utf-8'
         )
-        with pytest.raises(UnicodeDecodeError):
-            with get_dist(tmpdir):
-                pass
-
-    def test_non_ascii_2(self, tmpdir):
-        fake_env(
-            tmpdir,
-            '# -*- coding: invalid\n'
-        )
-        with pytest.raises(LookupError):
-            with get_dist(tmpdir):
-                pass
+        with get_dist(tmpdir):
+            pass
 
     def test_non_ascii_3(self, tmpdir):
         fake_env(
@@ -479,7 +464,6 @@ def test_non_ascii_3(self, tmpdir):
         with get_dist(tmpdir):
             pass
 
-    @skip_if_not_ascii
     def test_non_ascii_4(self, tmpdir):
         fake_env(
             tmpdir,
@@ -491,8 +475,10 @@ def test_non_ascii_4(self, tmpdir):
         with get_dist(tmpdir) as dist:
             assert dist.metadata.description == 'éàïôñ'
 
-    @skip_if_not_ascii
-    def test_non_ascii_5(self, tmpdir):
+    def test_not_utf8(self, tmpdir):
+        """
+        Config files encoded not in UTF-8 will fail
+        """
         fake_env(
             tmpdir,
             '# vim: set fileencoding=iso-8859-15 :\n'
@@ -500,8 +486,9 @@ def test_non_ascii_5(self, tmpdir):
             'description = éàïôñ\n',
             encoding='iso-8859-15'
         )
-        with get_dist(tmpdir) as dist:
-            assert dist.metadata.description == 'éàïôñ'
+        with pytest.raises(UnicodeDecodeError):
+            with get_dist(tmpdir):
+                pass
 
 
 class TestOptions:
diff --git a/setuptools/tests/test_setopt.py b/setuptools/tests/test_setopt.py
new file mode 100644
--- /dev/null
+++ b/setuptools/tests/test_setopt.py
@@ -0,0 +1,36 @@
+# coding: utf-8
+
+from __future__ import unicode_literals
+
+import io
+
+import six
+
+from setuptools.command import setopt
+from setuptools.extern.six.moves import configparser
+
+
+class TestEdit:
+    @staticmethod
+    def parse_config(filename):
+        parser = configparser.ConfigParser()
+        with io.open(filename, encoding='utf-8') as reader:
+            (parser.read_file if six.PY3 else parser.readfp)(reader)
+        return parser
+
+    @staticmethod
+    def write_text(file, content):
+        with io.open(file, 'wb') as strm:
+            strm.write(content.encode('utf-8'))
+
+    def test_utf8_encoding_retained(self, tmpdir):
+        """
+        When editing a file, non-ASCII characters encoded in
+        UTF-8 should be retained.
+        """
+        config = tmpdir.join('setup.cfg')
+        self.write_text(str(config), '[names]\njaraco=джарако')
+        setopt.edit_config(str(config), dict(names=dict(other='yes')))
+        parser = self.parse_config(str(config))
+        assert parser.get('names', 'jaraco') == 'джарако'
+        assert parser.get('names', 'other') == 'yes'