pydata/pandas — pull request #2104 by y-p (2012-10-22 18:27:53+00:00)
Title: Unicode III : revenge of the character planes

This is the 3rd installment in the unicode saga, and once again,
there's a lot of issues to point out. fair warning given.

#1994 cleaned up things to use basestring and unicode() over str() where needed
#2005 consolidated and implemented unicode-friendly pretty-printing

The attempt here is to ensure that the internal representation of strings is
always a unicode object, as is the case by default in python3.

**It's not finished**, but I'd like to get feedback.

A.Implementation
---
1. core.encoding implements the functionality for traversing sequences and decoding
strings into unicode objects inplace. this includes key/values of dicts (one
of the many supported input structures for constructing pandas data objects).
2. pure ascii strings are left untouched.
3. For strings containing unicode characters, we do actually change the input,
but since getters are symmetricly handled, the result should be seamless (if you
set a key using an encoded bytestrings,invoking a getter with the same key should
get you back the data, even though internally, the data is stored under the equiv.
unicode key).
4. this decoding step is placed in "choke-points" such as constructors, getters and
setters. I've probably missed a few at this point.
5. the encoding to be used for decoding is specified in a configurable (which will
be settable by the user, possibly via the mechanism in #2097.
6. if decoding fails, the user gets a message explaining what happend. if the `chardet
package is available, the code tries to suggest an encoding to the user based
on the data. `chardet` has enough false-positives to make automatic-detection
a bad idea.
7. as part of the unicode-or-bust theme, the csv reader now uses utf-8 by default,
which forces the user of UnicodeReader, so you get either unicode or a decode error,
forcing the user to specify an encoding.

B. Benefits
---
1. Making internal representations unicode should help reduce the whack-a-mole
which has been going on.
2. Once things are guranteed unicode, lib code can stop checking for and handling
corner-cases.
3. Mixing unicode and encoded byte-strings, or bytestrings encoded with
different encodings can create insoluable situations.
A related example is `fmt._put_lines()`, which was altered to fix #891 so
it would handle the case of mixing pure ascii with unicode.However, it now fails
when unicode is mixed with encoded byte-strings (which makes `repr_html()` fail,
yielding sometimes-html-sometimes-text repr confusion). This PR fixes that problem
amongst other things.
4. since python encodes input from the console using the console encoding, we can
convert things to unicode transparently for the user, so the behaviour becomes
closer to that of python3 - enter a string at the console, and you get unicode by
default. (no need for unicode literal).

C. Disadvantages
---

1. It's hackish.
2. There's a performance hit (@wesm, what benchmarks would you like to see?)
3. Immutable sequences are currently not handled. not just because of copying, but
also because an experiment converting tuples raised all sorts of mysterious
problems, in cython code amongst other things.
4. Probably difficult to cover every possible entry-point for bytes-strings.
5. Not sure about c-parser compat. issues.

D. Points to consider (feedback welcome)
---

1. there will be a configurable to turn all of this on or off. Not sure what the
default should be though.
2. Should the encoding used for decoding be specifiable per-object? or just
a global configurable avaialable to the user.
3. what's the best way to deal with Immutable sequences?

The configurables will be sorted out later (Pending #2097?).