dropbox/pyston — pull request #246 by jvkersch (2014-12-18 23:41:25+00:00)
Title: Enable unicode type (WIP)

This is a very preliminary PR to provide some support for unicode (#130), opened to solicit some feedback. Basically, this PR tries to do the bare minimum needed to get the unicode type working, together with its repr. On the command line, things like the following snippet work:
```
Pyston v0.2 (rev 42c3ecea3443), targeting Python 2.7.6
>> x = u'소주 한잔'
>> x
u'\uc18c\uc8fc \ud55c\uc794'
```
Since Pyston seems to be configured as a wide unicode build, the code just does some tricks to get unicode strings into UTF-32, and then wraps those into a `Unicode` instance. Calling into the existing CPython unicode implementation is then just a matter of passing in the underlying `const PyUnicode *` to the appropriate routine.

Does this sound like a viable approach? Should we instead try to support the Python 3 unicode implementation?

Things that I haven't addressed yet:
- [ ] Parsing via Pypa;
- [ ] When creating a unicode instance, `boxUnicode` iterates over the parsed string to convert it into a wide string. This can easily be avoided by reading in a wide string in the first place.
- [ ] There's no support for supplying an encoding string for a source file (and `compile` in parse_ast.py will complain if you do provide one), so that we can't run meaningful comparisons with Python.

I'm also not sure as to how much of `Objects/unicodeobject.c` I should try to include. I could definitely add that file in its entirety (surrounded by `#if 0 ... #endif`) but would rather do that as a separate PR, to avoid drowning out changes to the Pyston source tree.