brianmario/utf8 — pull request #3 by tanoku (2012-03-14 03:06:00+00:00)
Title: Cleanup & sanitize

From the commit notes:

~~~~~~
- The old UTF8.c code has been removed
- Core UTF8 functions have been borrowed from the utf8proc library
- buffer.c has been updated from Sundown
- In several methods in String::UTF8 have been rewritten
~~~~~~

Turns out that the 5th and 6th bytes have been deprecated since 2003 in order to keep UTF-8 and UTF-16 compatible. The more you know.

I'm 99% sure that the cleanup bug is fixed. The code in `utf8.c` wasn't really wrong, I double checked for the 3rd time -- I think the problem was in the actual cleanup function. It was doing some very sketchy logic. Regardless, the utf8 handling code now comes from a well-tested library, and the cleanup logic has been rewritten.

In fact, I've rewritten everything in `String::UTF8` except `slice`, because the code in slice is scaaaary. That could use a cleanup if you are bored. ;d

The other rewritten functions have very minor changes. Just using the `utf8proc` API in a smarter way.

Love and ponies!