dscape/clarinet — pull request #29 by davidgwking (2014-12-10 02:15:38+00:00)
Title:  critical fixes for multibyte characters in stream implementation

* There was some logic to handle multibyte characters when split across multiple chunks, but it was lacking.
* Fixed an issue where if a chunk starts with a multibyte character, the rest of the chunk is discarded.
* Fixed an issue where if a chunk starts with a single byte character and ends with a partial multibyte character, that character will now be sequenced on the subsequent chunk.
* Parser now only combines single byte characters and applies multibyte character logic to remaining bytes.
* Extended the generic test engine a bit so that it allows for custom chunked test cases.
* Extended utf8 chunk test so that it covers for scenarios.

Apologies in advance for all of the whitespace changes. I realized that I hadn't switched my editor until it was far too late. If this is problematic, I can circle back.