hylang/hy — pull request #710 by unmerged (2014-12-10 03:17:50+00:00)
Title: Fixed some unicode errors in python2

# What

Fixed UnicodeEncodeError when compiler is reporting
HyMacroExpansionError with macro containing unicode.

Added decoding of command line arguments passed with "-c" to fix
UnicodeDecodeError caused by args containing unicode.

# Why

## Macroexpansion error reporting error
### Expected:
$ ./hy
hy 0.10.1 using CPython(default) 3.3.5 on Linux
=> (defmacro bug❤ (x)
... (throw Exception))
=>
=> (bug❤ foo)
File "", line 1, column 1

(bug❤ foo)
^--------^
HyMacroExpansionError: expanding `bug❤': Exception()

### Got:
$ ./hy
hy 0.10.1 using CPython(default) 2.7.7 on Linux
=> (defmacro bug❤ (x)
... (throw Exception))
=>
=> (bug❤ foo)
Traceback (most recent call last):
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/compiler.py", line 2211, in hy_compile
result = compiler.compile(tree)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/compiler.py", line 420, in compile
ret = self.compile_atom(_type, tree)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/compiler.py", line 412, in compile_atom
ret = _compile_table[atom_type](self, atom)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/compiler.py", line 564, in compile_raw_list
ret = self._compile_branch(entries)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/compiler.py", line 451, in _compile_branch
return _branch(self.compile(expr) for expr in exprs)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/compiler.py", line 305, in _branch
results = list(results)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/compiler.py", line 451, in
return _branch(self.compile(expr) for expr in exprs)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/compiler.py", line 432, in compile
raise_empty(HyCompileError, e, sys.exc_info()[2])
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/_compat.py", line 60, in raise_empty
raise t(*args)
HyCompileError: Internal Compiler Bug 😱
⤷ UnicodeEncodeError: 'ascii' codec can't encode character u'\u2764' in position 3: ordinal not in range(128)
Compilation traceback:
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/compiler.py", line 420, in compile
ret = self.compile_atom(_type, tree)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/compiler.py", line 412, in compile_atom
ret = _compile_table[atom_type](self, atom)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/compiler.py", line 1724, in compile_expression
expression = macroexpand(expression, self.module_name)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/macros.py", line 174, in macroexpand
tree = macroexpand_1(tree, module_name)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/macros.py", line 203, in macroexpand_1
msg = "expanding `" + str(tree[0]) + "': " + repr(e)
=>

## Command line args error

$ ./hy -c '(defn bug❤ (x) 0)' # all unicode stuff is borked
Traceback (most recent call last):
File "./hy", line 9, in
load_entry_point('hy==0.10.1', 'console_scripts', 'hy')()
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/cmdline.py", line 334, in hy_main
sys.exit(cmdline_handler("hy", sys.argv))
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/cmdline.py", line 304, in cmdline_handler
return run_command(options.command)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/cmdline.py", line 180, in run_command
import_buffer_to_module("__main__", source)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/importer.py", line 89, in import_buffer_to_module
_ast = import_buffer_to_ast(buf, module_name)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/importer.py", line 58, in import_buffer_to_ast
return hy_compile(import_buffer_to_hst(buf), module_name)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/importer.py", line 47, in import_buffer_to_hst
return tokenize(buf + "\n")
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/lex/__init__.py", line 33, in tokenize
return parser.parse(lexer.lex(buf))
File "/home/user/.local/lib64/python2.7/site-packages/rply/parser.py", line 22, in parse
current_state = self._reduce_production(t, symstack, statestack, state)
File "/home/user/.local/lib64/python2.7/site-packages/rply/parser.py", line 76, in _reduce_production
value = p.func(targ)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/lex/parser.py", line 52, in wrapped
ret = fun(p)
File "/home/user/.local/lib64/python2.7/site-packages/hy-0.10.1-py2.7.egg/hy/lex/parser.py", line 287, in t_identifier
return HySymbol(obj)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3: ordinal not in range(128)