Repository: ipython/ipython

Issue:
IPython mangles unicode characters 
This occurs in IJulia, but I've checked the ZMQ message boundary so I think it probably happens somewhere between the browser and before it hits the kernel. When entering a high unicode character (such as U+1D712) it arrives at the kernel as 

```
6-element Array{Uint8,1}:
 0xed
 0xa0
 0xb5
 0xed
 0xbc
 0x92
```

rather than the expected 

```
4-element Array{Uint8,1}:
 0xf0
 0x9d
 0x9c
 0x92
```

.
Now, in julia we do our best to retain invalid unicode data which gives the effect that if you print the above again it prints correctly in the browser. I assume somewhere in the software stack there's an encoding that's not quite UTF8, though it does convert back, so that's nice. I also checked that printing the proper UTF8 character shows up correctly in the browser. Now, this might be a browser issue, but I checked that it is set to UTF8 and it happens in Firefox and Safari as well and when I paste it out into a texteditor I get the proper UTF8 character. 

Let me know if there's anything I can do to help investigate.



Discussion:
Python suggests that those two are both valid utf-8 represenatations of the same code point:

```
In [1]: b'\xed\xa0\xb5\xed\xbc\x92'.decode('utf-8') == b'\xf0\x9d\x9c\x92'.decode('utf-8')
Out[1]: True
In [2]: b'\xed\xa0\xb5\xed\xbc\x92'.decode('utf-8') == u'\U0001d712'
Out[2]: True
```

This is beginning to looks suspiciously like CESU-8. It's nice that it's being decoded to the right unicode character in python and maybe we should do that too in julia, but I think it would still be nice if we got proper utf8 over the wire. 

EDIT: I was thinking CESU8 because the hex of the two characters in the six-byte sequence is (UTF8 encoded) d835 and df12 which are the UTF16 surrogates here. 

Update: I use wireshark to look at the websocket traffic and the UTF8 is transmitted properly as

```
7b 22 63 6f 64 65 22 3a 22 5c 22 *f0 9d 9c 92* 5c  {"code":"\"....\
```

(emphasis mine)

Looks like it's the Python stdlib `ensure_ascii` flag, which defaults to True:

``` python
In [23]: json.dumps(u'\U0001d712', ensure_ascii=True)
Out[23]: '"\\ud835\\udf12"'

In [24]: json.dumps(u'\U0001d712', ensure_ascii=False)
Out[24]: u'"\U0001d712"'
```

Awesome detective work. Any problem with just turning that off?

It should be okay to do that, but I don't think anything incorrect is currently happening. The JSON spec says any character may be escaped, and that unicode escapes must use four hex characters, so high code points must be split. Both byte sequences are valid utf8 representations of the same unicode code point.


Resolving patch:
diff --git a/IPython/kernel/zmq/session.py b/IPython/kernel/zmq/session.py
--- a/IPython/kernel/zmq/session.py
+++ b/IPython/kernel/zmq/session.py
@@ -6,23 +6,10 @@
 * A SessionFactory to be used as a base class for configurables that work with
 Sessions.
 * A Message object for convenience that allows attribute-access to the msg dict.
-
-Authors:
-
-* Min RK
-* Brian Granger
-* Fernando Perez
 """
-#-----------------------------------------------------------------------------
-#  Copyright (C) 2010-2011  The IPython Development Team
-#
-#  Distributed under the terms of the BSD License.  The full license is in
-#  the file COPYING, distributed as part of this software.
-#-----------------------------------------------------------------------------
 
-#-----------------------------------------------------------------------------
-# Imports
-#-----------------------------------------------------------------------------
+# Copyright (c) IPython Development Team.
+# Distributed under the terms of the Modified BSD License.
 
 import hashlib
 import hmac
@@ -80,7 +67,7 @@ def squash_unicode(obj):
 #-----------------------------------------------------------------------------
 
 # ISO8601-ify datetime objects
-json_packer = lambda obj: jsonapi.dumps(obj, default=date_default)
+json_packer = lambda obj: jsonapi.dumps(obj, default=date_default, ensure_ascii=False)
 json_unpacker = lambda s: jsonapi.loads(s)
 
 pickle_packer = lambda o: pickle.dumps(squash_dates(o),-1)


Test patch: