fsharp/FSharp.Data — pull request #624 by mrange (2014-05-07 22:01:14+00:00)
Title: JSON: bug fix for embedded unicode characters

While preparing to build up a performance test suite for the JSON parser (in order to instrument some optimizations I am planning) I discovered a potential bug related to embedded unicode characters:

JSON objects like these didn't give the correct result (TwitterStream.json contains a value like this):
```
["\ud83d\udc36"]
```

jsonlint.org and my reference parser thinks the result should look like this:
```
["🐶"]
```

With FSharp.Data I get:
```
["��"]
```

Experimenting a bit in FSI:
```
open System.Text

let printAsHex (ch : char) =
printfn "%x" <| int ch

printfn "Test oracle"
let x = "\ud83d\udc36"
printAsHex x.[0]
printAsHex x.[1]

printfn "Using UnicodingEncoding.Encoding.GetChars (JsonParser uses this)"
let y = UnicodeEncoding.Unicode.GetChars [|byte 0xd8; byte 0x3d; byte 0xdc; byte 0x36|]
printAsHex y.[0]
printAsHex y.[1]
```
It prints
```
Test oracle
d83d
dc36
Using UnicodingEncoding.Encoding.GetChars (JsonParser uses this)
3dd8
36dc
```

This makes me believe we are seeing an endian issue here.

Updated the parser code accordingly and added a test case. Note for self; in the case of embedded unicode characters string.Substring is invoked which isn't good for performance in case of many embedded unicode characters.

Regards,
Mårten

PS. As I use sublime to edit the code there's unfortunately some trailing whitespace removals as well.