Repository: pex-tool/pex

Issue:
Python 2: SyntaxError when installing distributions with coding hints in setup.py
I think this is a regression introduced in #231, which was released in pex 1.1.7.  I cannot reproduce this problem on pex 1.1.6.  Affects Python 2 only.

Reproduce with:

```
wget -q 'https://github.com/pantsbuild/pex/releases/download/v1.1.7/pex27'
chmod a+x pex27
./pex27 backoff
```

Stack trace:

```
**** Failed to install backoff-1.1.0. stdout:

**** Failed to install backoff-1.1.0. stderr:
Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "setup.py", line 0
SyntaxError: encoding declaration in Unicode string

Traceback (most recent call last):
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 328, in execute
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 260, in _wrap_coverage
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 292, in _wrap_profiling
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 371, in _execute
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 429, in execute_entry
  File "/private/tmp/pex/pex27/.bootstrap/_pex/pex.py", line 447, in execute_pkg_resources
  File "/Users/saj/.pex/install/pex-1.1.7-py2.py3-none-any.whl.1089a1bbad10037ad29421de8a162a77c90c8133/pex-1.1.7-py2.py3-none-any.whl/pex/bin/pex.py", line 538, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/Users/saj/.pex/install/pex-1.1.7-py2.py3-none-any.whl.1089a1bbad10037ad29421de8a162a77c90c8133/pex-1.1.7-py2.py3-none-any.whl/pex/bin/pex.py", line 489, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/Users/saj/.pex/install/pex-1.1.7-py2.py3-none-any.whl.1089a1bbad10037ad29421de8a162a77c90c8133/pex-1.1.7-py2.py3-none-any.whl/pex/resolver.py", line 200, in resolve
    dist = self.build(package, resolvable.options)
  File "/Users/saj/.pex/install/pex-1.1.7-py2.py3-none-any.whl.1089a1bbad10037ad29421de8a162a77c90c8133/pex-1.1.7-py2.py3-none-any.whl/pex/resolver.py", line 257, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/Users/saj/.pex/install/pex-1.1.7-py2.py3-none-any.whl.1089a1bbad10037ad29421de8a162a77c90c8133/pex-1.1.7-py2.py3-none-any.whl/pex/resolver.py", line 168, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage(u'file:///Users/saj/.pex/build/backoff-1.1.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
```

Even if we were to ignore the problem above, I don't think it is right to assume a UTF-8 encoding on all files.

From [PEP 263](https://www.python.org/dev/peps/pep-0263/):

> If a Unicode string with a coding declaration is passed to compile(), a SyntaxError will be raised.

From the same PEP:

> The builtin compile() API will be enhanced to accept Unicode as input. 8-bit string input is subject to the standard procedure for encoding detection as described above.

Instead of opening `setup.py` as a text file and decoding it ourselves, it sounds like it should be possible to hand a binary stream directly to the `compile()` builtin and let Python deal with this muck.



Discussion:

Resolving patch:
diff --git a/pex/installer.py b/pex/installer.py
--- a/pex/installer.py
+++ b/pex/installer.py
@@ -32,12 +32,12 @@ def function_wrapper(self, *args, **kw):
 
 
 class InstallerBase(object):
-  SETUP_BOOTSTRAP_HEADER = "import io, sys"
+  SETUP_BOOTSTRAP_HEADER = "import sys"
   SETUP_BOOTSTRAP_MODULE = "sys.path.insert(0, %(path)r); import %(module)s"
   SETUP_BOOTSTRAP_FOOTER = """
 __file__ = 'setup.py'
 sys.argv[0] = 'setup.py'
-exec(compile(io.open(__file__, encoding='utf-8').read().replace('\\r\\n', '\\n'), __file__, 'exec'))
+exec(compile(open(__file__, 'rb').read(), __file__, 'exec'))
 """
 
   class Error(Exception): pass


Test patch: