Repository: pypa/setuptools

Issue:
Regression 38.7.0: ascii decode error when installing package from source
When installing https://github.com/file/file/blob/master/python/ from source
python 2.7.14 
setuptools 38.7.0
fails:
```
Traceback (most recent call last):
  File "setup.py", line 21, in <module>
    'Topic :: Software Development :: Libraries :: Python Modules',
  File "/home/asavah/kross/host/lib/python2.7/site-packages/setuptools-38.7.0-py2.7.egg/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/home/asavah/kross/host/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/home/asavah/kross/host/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/home/asavah/kross/host/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/asavah/kross/host/lib/python2.7/site-packages/setuptools-38.7.0-py2.7.egg/setuptools/command/install.py", line 69, in run
    self.do_egg_install()
  File "/home/asavah/kross/host/lib/python2.7/site-packages/setuptools-38.7.0-py2.7.egg/setuptools/command/install.py", line 111, in do_egg_install
    self.run_command('bdist_egg')
  File "/home/asavah/kross/host/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/home/asavah/kross/host/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/asavah/kross/host/lib/python2.7/site-packages/setuptools-38.7.0-py2.7.egg/setuptools/command/bdist_egg.py", line 163, in run
    self.run_command("egg_info")
  File "/home/asavah/kross/host/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/home/asavah/kross/host/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/asavah/kross/host/lib/python2.7/site-packages/setuptools-38.7.0-py2.7.egg/setuptools/command/egg_info.py", line 271, in run
    writer(self, ep.name, os.path.join(self.egg_info, ep.name))
  File "/home/asavah/kross/host/lib/python2.7/site-packages/setuptools-38.7.0-py2.7.egg/setuptools/command/egg_info.py", line 604, in write_pkg_info
    metadata.write_pkg_info(cmd.egg_info)
  File "/home/asavah/kross/host/lib/python2.7/distutils/dist.py", line 1106, in write_pkg_info
    self.write_pkg_file(pkg_info)
  File "/home/asavah/kross/host/lib/python2.7/site-packages/setuptools-38.7.0-py2.7.egg/setuptools/dist.py", line 76, in write_pkg_file
    file.write('%s: %s\n' % (field, attr_val))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xc1' in position 23: ordinal not in range(128)
```

Setuptools 38.6.0 works fine.

Edit: python 3.6.4 + setuptools 38.7.0 also works fine.


Discussion:
[Changelog indicates](http://setuptools.readthedocs.io/en/latest/history.html#v38-7-0) that change was to add the maintainer to the package metadata (#1288). My guess is these packages had non-ascii values in these fields and simply including them means we need to deal with the non-ascii values.

@pganssle Can you investigate?
Indeed the package has A with acute in author's name, https://github.com/file/file/blob/master/python/setup.py#L10
Very strange because I actually [test for this](https://github.com/pypa/setuptools/blob/master/setuptools/tests/test_dist.py#L87), not sure why my test didn't catch this.
> not sure why my test didn't catch this.

Probably because of https://github.com/pypa/setuptools/blob/master/setuptools/tests/test_dist.py#L1
which is not present in dist.py

This does actually look like the errors I was getting on Python 2.7 when I was using `io.StringIO` as the file I was writing, rather than `StringIO.StringIO`. I switched it over because I checked that "Author" was written out correctly on Python 2.7, so assumed it was an artifact of my choice of stream emulator.

I'll change the test over to use `io.StringIO` again, as that seems to better emulate the real situation. 

@asavah Good call on the encoding - I was assuming that did not matter because the strings to encode were not *in* that file, but I think the fact that the strings being formatted are being taken as ASCII is what's throwing this off.

I'll test this out and send in a new PR. Sorry about the fuss.

As an aside, I actually met the author whose name is causing the problem (@turicas) after [this talk at PyCon](https://www.youtube.com/watch?v=bx3NOoroV-M), about how dealing with text/unicode is difficult, coincidentally enough.
Changing the encoding on the `dist.py` file didn't help. I'll dig into this a bit more. It's obviously something different about how I'm handling the encoding, but the `author` field is not actually new, it's just being loaded in a slightly different way, and it *used* to work.
Found the problem. On Python 2.7 there is an `_encode_field` function that is called in `get_contact()`, but I was looking at the current master branch's implementation and couldn't understand what was going on. PR incoming.

Resolving patch:
diff --git a/setuptools/dist.py b/setuptools/dist.py
--- a/setuptools/dist.py
+++ b/setuptools/dist.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 __all__ = ['Distribution']
 
 import re
@@ -35,10 +36,13 @@ def _get_unpatched(cls):
     warnings.warn("Do not call this function", DeprecationWarning)
     return get_unpatched(cls)
 
+
 def get_metadata_version(dist_md):
     if dist_md.long_description_content_type or dist_md.provides_extras:
         return StrictVersion('2.1')
-    elif getattr(dist_md, 'python_requires', None) is not None:
+    elif (dist_md.maintainer is not None or
+          dist_md.maintainer_email is not None or
+          getattr(dist_md, 'python_requires', None) is not None):
         return StrictVersion('1.2')
     elif (dist_md.provides or dist_md.requires or dist_md.obsoletes or
             dist_md.classifiers or dist_md.download_url):
@@ -59,7 +63,7 @@ def write_pkg_file(self, file):
     file.write('Summary: %s\n' % self.get_description())
     file.write('Home-page: %s\n' % self.get_url())
 
-    if version == '1.2':
+    if version < StrictVersion('1.2'):
         file.write('Author: %s\n' % self.get_contact())
         file.write('Author-email: %s\n' % self.get_contact_email())
     else:
@@ -72,6 +76,9 @@ def write_pkg_file(self, file):
 
         for field, attr in optional_fields:
             attr_val = getattr(self, attr)
+            if six.PY2:
+                attr_val = self._encode_field(attr_val)
+
             if attr_val is not None:
                 file.write('%s: %s\n' % (field, attr_val))
 
@@ -88,7 +95,7 @@ def write_pkg_file(self, file):
     if keywords:
         file.write('Keywords: %s\n' % keywords)
 
-    if version == '1.2':
+    if version >= StrictVersion('1.2'):
         for platform in self.get_platforms():
             file.write('Platform: %s\n' % platform)
     else:
@@ -556,7 +563,7 @@ def fetch_build_egg(self, req):
                 # don't use any other settings
                 'find_links', 'site_dirs', 'index_url',
                 'optimize', 'site_dirs', 'allow_hosts',
-        ))
+            ))
         if self.dependency_links:
             links = self.dependency_links[:]
             if 'find_links' in opts:


Test patch:
diff --git a/setuptools/tests/test_dist.py b/setuptools/tests/test_dist.py
--- a/setuptools/tests/test_dist.py
+++ b/setuptools/tests/test_dist.py
@@ -1,20 +1,26 @@
 # -*- coding: utf-8 -*-
+
+from __future__ import unicode_literals
+
+import io
+
 from setuptools import Distribution
 from setuptools.extern.six.moves.urllib.request import pathname2url
 from setuptools.extern.six.moves.urllib_parse import urljoin
-from setuptools.extern.six import StringIO
 
 from .textwrap import DALS
 from .test_easy_install import make_nspkg_sdist
 
 import pytest
 
+
 def test_dist_fetch_build_egg(tmpdir):
     """
     Check multiple calls to `Distribution.fetch_build_egg` work as expected.
     """
     index = tmpdir.mkdir('index')
     index_url = urljoin('file://', pathname2url(str(index)))
+
     def sdist_with_index(distname, version):
         dist_dir = index.mkdir(distname)
         dist_sdist = '%s-%s.tar.gz' % (distname, version)
@@ -63,37 +69,47 @@ def merge_dicts(d1, d2):
 
     test_cases = [
         ('No author, no maintainer', attrs.copy()),
-        ('Author (no e-mail), no maintainer', merge_dicts(attrs,
+        ('Author (no e-mail), no maintainer', merge_dicts(
+            attrs,
             {'author': 'Author Name'})),
-        ('Author (e-mail), no maintainer', merge_dicts(attrs,
+        ('Author (e-mail), no maintainer', merge_dicts(
+            attrs,
             {'author': 'Author Name',
              'author_email': 'author@name.com'})),
-        ('No author, maintainer (no e-mail)', merge_dicts(attrs,
+        ('No author, maintainer (no e-mail)', merge_dicts(
+            attrs,
             {'maintainer': 'Maintainer Name'})),
-        ('No author, maintainer (e-mail)', merge_dicts(attrs,
+        ('No author, maintainer (e-mail)', merge_dicts(
+            attrs,
             {'maintainer': 'Maintainer Name',
              'maintainer_email': 'maintainer@name.com'})),
-        ('Author (no e-mail), Maintainer (no-email)', merge_dicts(attrs,
+        ('Author (no e-mail), Maintainer (no-email)', merge_dicts(
+            attrs,
             {'author': 'Author Name',
              'maintainer': 'Maintainer Name'})),
-        ('Author (e-mail), Maintainer (e-mail)', merge_dicts(attrs,
+        ('Author (e-mail), Maintainer (e-mail)', merge_dicts(
+            attrs,
             {'author': 'Author Name',
              'author_email': 'author@name.com',
              'maintainer': 'Maintainer Name',
              'maintainer_email': 'maintainer@name.com'})),
-        ('No author (e-mail), no maintainer (e-mail)', merge_dicts(attrs,
+        ('No author (e-mail), no maintainer (e-mail)', merge_dicts(
+            attrs,
             {'author_email': 'author@name.com',
              'maintainer_email': 'maintainer@name.com'})),
-        ('Author unicode', merge_dicts(attrs,
+        ('Author unicode', merge_dicts(
+            attrs,
             {'author': '鉄沢寛'})),
-        ('Maintainer unicode', merge_dicts(attrs,
+        ('Maintainer unicode', merge_dicts(
+            attrs,
             {'maintainer': 'Jan Łukasiewicz'})),
     ]
 
     return test_cases
 
+
 @pytest.mark.parametrize('name,attrs', __maintainer_test_cases())
-def test_maintainer_author(name, attrs):
+def test_maintainer_author(name, attrs, tmpdir):
     tested_keys = {
         'author': 'Author',
         'author_email': 'Author-email',
@@ -103,12 +119,17 @@ def test_maintainer_author(name, attrs):
 
     # Generate a PKG-INFO file
     dist = Distribution(attrs)
-    PKG_INFO = StringIO()
-    dist.metadata.write_pkg_file(PKG_INFO)
-    PKG_INFO.seek(0)
+    fn = tmpdir.mkdir('pkg_info')
+    fn_s = str(fn)
+
+    dist.metadata.write_pkg_info(fn_s)
+
+    with io.open(str(fn.join('PKG-INFO')), 'r', encoding='utf-8') as f:
+        raw_pkg_lines = f.readlines()
+
+    # Drop blank lines
+    pkg_lines = list(filter(None, raw_pkg_lines))
 
-    pkg_lines = PKG_INFO.readlines()
-    pkg_lines = [_ for _ in pkg_lines if _]   # Drop blank lines
     pkg_lines_set = set(pkg_lines)
 
     # Duplicate lines should not be generated
@@ -122,4 +143,3 @@ def test_maintainer_author(name, attrs):
         else:
             line = '%s: %s' % (fkey, val)
             assert line in pkg_lines_set
-