BLD: ipython_directive, handle non-ascii execution results
State: closed

@jorisvandenbossche I think this is a clean solution to the non-utf8
output problem we discussed, specifically the example in io.rst. 
This PR should produce the right output on windows and be more amenable
to py3 compat work too. If you confirm that it's good I feel this is clean enough
to try and get upstream, completing the upstreaming effort.

related #5530, https://github.com/pydata/pandas/pull/5925#issuecomment-32407859


Labels:

Comments:
@y-p This looks nice! And in any case, the docs build on Windows.

But yet some possible quirks, I commented inline.

For the faster doc building, will try out later (no time anymore today).

@y-p Seems you did some of the wishlist items in https://github.com/pydata/pandas/pull/5530#issuecomment-33500566 yourself, nice!

@jorisvandenbossche asked me in https://github.com/pydata/pandas/pull/5530#issuecomment-33677242 to try on py3. Tried it on win+py3, I get an instant fail. Since this PR is about sphinx speedup, section building and non-ascii handling on platforms such as Windows, I think the py3 compat should be done in a new PR after this one is merged.

The bad output in the cell is the result of pprint_thing used by the repr, and it''s output
matches that of the console encoding. on most linux that's utf8, if you try to join that up to
a unicode string:

```
In [10]: u''+u'Träumen'.encode('utf8')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-10-181286dcc4b0> in <module>()
----> 1 u''+u'Träumen'.encode('utf8')

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128)
```

That's the problem. I've changed things accordingly.
On windows, the encoding might be different so the output might be differently encoded.
So I changed conf.py to set pandas' output encoding to utf8 regardless before processing
the docs. Should work.

Hmm, it's still the same. But the docs build, so that's the most important for me if the output looks nice on linux.

And I think you forgot to put in the commit that changes io.rst (the `:output_econding:`).

That's not required. So this is nicer code that's no worse then the previous attempt?
If so, I'll merge, and leave that windows thing to when I can setup a windows box to 
comfortably test things myself.

Indeed, I can confirm this is nicer code for the same (incorrect on windows but building) result :-)

What is the addition of `:output_encoding:` to ipython directive for? If it is not needed in this case?

BTW, I forgot almost I tested it, and it does matter (as is what I would understand from the code). Without specifying `:output_encoding: 'latin_1'` I get question marks like `0  Tr�umen`. If I specify it I get like `Trumen` as I mentioned before. But in both cases the it builds.

I inserted a decoding step on all write to stdout from code executing
in the embedded ipython shell. That option sets the endocing to use for it.
It may actually be better to just always use utf8, I'll have to consider before 
submitting upstream.

You may get `Trumen` but that's wrong as well. I'll stop using you as a CI
server, and get to it when I can. Thanks for the feedback. appreciated.

@y-p Do you find this good enough to push upstream? If you don't have time, I can try to do this.