Repository: ipython/ipython

Issue:
nbconvert can't handle Heading with Chinese characters on Japanese Windows OS.
Convert following notebook by `ipython nbconvert test.ipynb` will raise Exception:

```
  File "C:\Python27\lib\site-packages\ipython-1.0.0_dev-py2.7.egg\IPython\nbconv
ert\filters\strings.py", line 83, in add_anchor
    h = ElementTree.fromstring(py3compat.cast_bytes_py2(html))
  File "C:\Python27\lib\xml\etree\ElementTree.py", line 1301, in XML
    parser.feed(text)
  File "C:\Python27\lib\xml\etree\ElementTree.py", line 1643, in feed
    self._raiseerror(v)
  File "C:\Python27\lib\xml\etree\ElementTree.py", line 1507, in _raiseerror
    raise err
ParseError: not well-formed (invalid token): line 1, column 9
```

Here is the content of the notebook. I am using a Japanese Windows, the default encoding is:

```
In [1]: from IPython.utils import encoding

In [2]: encoding.DEFAULT_ENCODING
Out[2]: 'cp932'
```

When call `py3compat.cast_bytes_py2(html)` it can't convert the Chinese characters correctly.

```
{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "\u6269\u5c55\u7c7b\u578b(cdef\u7c7b)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}
```



Discussion:
`py3compat.cast_bytes_py2(u"\u6269\u5c55\u7c7b\u578b(cdef\u7c7b)")` works fine for me Window 7 x64, Python 3.3.  What version of python are you using?  I'll test some more and report back.

`nbconvert --to html` worked, I'll try py2.7

Problem only seems to exist on py2.x

@jdfreder : `cast_bytes_py2` is a no-op on Py 3, IIRC, so you wouldn't see any problem there.

Ah thanks @takluyver .  I'm walking through it with winpdb right now, it's either Jinja or the FilesWriter (probably the later)

**Finally** found the problem, the  `add_anchor` filter doesn't support Unicode on Win7&py2.x .  I'm going to look for a solution a little bit later tonight.

Edit: Should have looked there in the first place, since it was in your trace.  The thing is, I can't duplicate the error on my machine because I'm running ascii default encoding.  So for me, when I try to convert that nb you posted, I end up with a bunch of question marks.  I've found that removing the `| add_anchor` from the html_basic template is a quick fix.  However, I'd rather just fix the add_anchor function.  I'll look under the file's history and see what I can find about why it was added.


Resolving patch:
diff --git a/IPython/nbconvert/filters/strings.py b/IPython/nbconvert/filters/strings.py
--- a/IPython/nbconvert/filters/strings.py
+++ b/IPython/nbconvert/filters/strings.py
@@ -80,7 +80,7 @@ def add_anchor(html):
     
     For use in heading cells
     """
-    h = ElementTree.fromstring(py3compat.cast_bytes_py2(html))
+    h = ElementTree.fromstring(py3compat.cast_bytes_py2(html, encoding='utf-8'))
     link = html2text(h).replace(' ', '-')
     h.set('id', link)
     a = ElementTree.Element("a", {"class" : "anchor-link", "href" : "#" + link})


Test patch: