judofyr/duktape.rb — pull request #9 by josh (2014-12-09 23:34:40+00:00)
Title: Encodings

We need to decide on an encoding policy. Right now all the returned strings are untagged.

> In Duktape, Ecmascript strings are encoded with CESU-8 encoding. CESU-8 matches UTF-8 except that it allows codepoints in the surrogate pair range (U+D800 to U+DFFF) to be encoded directly; these are prohibited in UTF-8. CESU-8, like UTF-8, encodes all 7-bit ASCII characters as-is which is convenient for C code.

I can't say I fully understand CESU-8, but it sounds we should always be transcoding Ruby Strings to UTF-8 before handing them off to Duktape. As for the returned object, they should either be tagged as UTF-8 or transcoded back to default internal/external (whatever one). I'd be fine with always having UTF-8 strings.

/cc @judofyr @brianmario