twitter/twitter-text-conformance — Issue #28: Behavior of characters outside of the Basic Multilingual Plane should be tested
Author: j3h
Created: 2012-01-20 01:31:11+00:00
State: open

There are currently no conformance tests for characters outside of the Basic Multilingual Plane. These characters are significant because they are the only characters whose UTF-16 representation is more than one code unit. Many languages use UTF-16 as their native character encoding (including Java and JavaScript).

Anyway, this problem manifested itself as a difference between the output of the twitter-text-rb and twitter-text-java libraries. It affects all parts of the libraries that depend on counting characters. According to [the Twitter documentation on counting characters](https://dev.twitter.com/docs/counting-characters), this codepoint should only count as one "character," but some libraries count it as two. This is particularly noticeable when looking at extracted entities from [tweets that contain these characters](https://twitter.com/#!/j3h/status/160167729096167424).

This issue would include a pull request, but I haven't been able to figure out a way to encode one of those characters into YAML in a way that is recognized by common YAML libraries. As far as I understand the spec, directly embedding the character as UTF-8 should be allowed, but at least some Java YAML parsers won't accept that (valid) YAML document. In addition, YAML 32-bit Unicode escapes are not supported by most YAML libraries.