micropython/micropython — pull request #671 by Rosuav (2014-06-07 02:00:36+00:00)
Title: First shot at Unicode support

This may well not yet be ready for merging. Request for comments, mainly!

I've added a test, and it and all the rest of the tests pass. There are quite a few places in the code where things assume correct UTF-8 encoding, and things may crash and burn if given malformed byte streams.

In theory, a bytes() is allowed to have anything in it, and will always have its charlen and its len the same; a str() must be correctly-formed UTF-8, and its charlen is less than its len by the number of continuation bytes.

The flags byte is currently unused and always 1. Later on, this could get a marker specifying that the string is ASCII-only, which could simplify some code paths for the very common case; alternatively, that could be dropped.

```
Previously:
text	data	bss	dec	hex	filename
275857	1352	1216	278425	43f99	micropython
Now:
text	data	bss	dec	hex	filename
281009	1352	1216	283577	453b9	micropython
```
If I'm reading this correctly, Unicode support has cost you 5KB.