pkulchenko/ZeroBraneStudio — pull request #211 by kikito (2013-10-01 11:09:34+00:00)
Title: fix the fixutf8 function

This function marked some valid 4-bytes utf-8 characters such as `U+24B62` as invalid (it matched the first 3 bytes, and then discarded the 4rth).

This was happening because the code was not conforming to the standard, which is the following:

| Code Points | First | Second | Third | Fourth |
|=========================================================|
| U+0000..U+007F | 00..7F | | | |
| U+0080..U+07FF | C2..DF | 80..BF | | |
| U+0800..U+0FFF | E0 | A0..BF | 80..BF | |
| U+1000..U+CFFF | E1..EC | 80..BF | 80..BF | |
| U+D000..U+D7FF | ED | 80..9F | 80..BF | |
| U+E000..U+FFFF | EE..EF | 80..BF | 80..BF | |
| U+10000..U+3FFFF | F0 | 90..BF | 80..BF | 80..BF |
| U+40000..U+FFFFF | F1..F3 | 80..BF | 80..BF | 80..BF |
| U+100000..U+10FFFF | F4 | 80..8F | 80..BF | 80..BF |

This is table 3-7 ("Well-Formed UTF-8 Byte Sequences") from page 95 of [The Unicode Standard 6.2.0](http://www.unicode.org/versions/Unicode6.2.0/UnicodeStandard-6.2.pdf)

Notice how there are 3 rows for 4-byte characters instead of 2. In particular, the rule that begins with `F0` (240) should match 4 bytes, not 3.

This might indicate that there was also a bug on the original Perl code from which this function was taken. But I don't feel like reading regular expressions in Perl ATM :)