Unicode : change df.to_string() and friends to always return unicode objects
State: closed

closes #2225

**Note**: Although all the tests pass with minor fixes, this PR has an above-average chance of
breaking things for people who have relied on broken behaviour thus far.

`df.tidy_repr` combines several strings to produce a result. when one component is unicode
and other other is a non-ascii bytestring, it tries to convert the latter back to a unicode string
using the 'ascii' codec and fails.

I suggest that `_get_repr` -> `to_string` should always return unicode, as implemented by this PR,
and that the `force_unicode` argument be deprecated everyhwere. 

The `force_unicode` argument in `to_string` conflates two things:
- which codec to use to decode the string (which can only be a hopeful guess)
- whether to return a unicode() object or str() object, 

The first is now no longer necessary since `pprint_thing` already resorts to the same hack 
of using utf-8  (with errors='replace') as a fallback.
I believe making the latter optional is wrong, precisely because it brings about situations 
like the test case above.
`to_string`, like all internal functions , should utilize unicode objects, whenever feasible.


Labels:

Comments:
This seems pretty reasonable. Should I take a chance merging this for 0.9.1? I've encountered the bug you fixed here before

I would at least wait a few days before merging this (perhaps @jseabold or someone else would like 
to argue their use-case ). 

I guess the question is what code will break because the string is coming back as unicode. Obviously if you had `df.to_string(force_unicode=True).decode('utf-8')` that is going to break. Maybe this should be held off until 0.10 series

it depends whether you consider this a bug fix or a breaking change. I'm fine with 0.10 though.

Let wait 'til 0.10. Let's merge it into master as soon as the release is out though.

Agreed...

This would be great. As of right now, you have to do something dirty (at least that's the only way I found it works) like `DataFrame(series).to_string(force_unicode=True, header=False)` to correctly print a Series object with unicode characters to a utf-8 console.

I took this a step further, Realizing that the unicode issue really matters only
when we want to get a string representation of an object.

So:
- I Converted more related functions to work exclusively with unicode. 
- Since everything should taper down to pprint_thing at the bottom, any utf-8 bytestrings
  should get silently decoded into unicode.
- If your data is not unicode and not utf-8, it's unreasonable to expect str(df) to do
  the right thing, and so you'll get �  (the unicode replacement character), but not exceptions
  (hopefully).
- fixing a couple of corner cases along the way, I added all the boilerplae so that
  str(x)/unicode(x)/bytes(x) work on py2 and py3 for series/df/panel.

Yell if something broke.

@Aldanor I see you deleted your comment but I checked that your example works now, at least on my environment...

@wesm Thanks, sounds good. I just didn't want to confuse everyone cause I wasn't sure this wasn't something specific to my environment. I will try and test it again soon as I can.

Added str/unicode/bytes support for `Index`,`MultiIndex`.

takeback