astropy/astropy — pull request #1974 by taldcroft (2014-01-17 16:53:45+00:00)
Title: Add funcs to convert table cols to/from unicode/bytestring

This PR provides a conversion layer to make it easy to implement the "unicode-sandwich" advocated by Ned Batchelder [in this talk](http://nedbatchelder.com/text/unipain.html). The idea is that (for Python 3) files and network data are always bytes and internal manipulation is always unicode.

Before going ahead with tests / docs I'm interested in feedback if this is going in the right direction.

Motivation for this follows from what I wrote on numpy-dev today, which is copied below:

I've been playing around with porting a stack of analysis libraries to Python 3 and this is a very timely thread and comment. What I discovered right away is that all the string data coming from binary HDF5 files show up (as expected) as 'S' type,, but that trying to make everything actually work in Python 3 without converting to 'U' is a big mess of whack-a-mole.

Yes, it's possible to change my libraries to use bytestring literals everywhere, but the Python 3 user experience becomes horrible because to interact with the data all downstream applications need to use bytestring literals everywhere. E.g. doing a simple filter like `string_array == 'foo'` doesn't work, and this will break all existing code when trying to run in Python 3. And every time you try to print something it has this horrible "b" in front. Ugly, and it just won't work well in the end.

Following the excellent advice at http://nedbatchelder.com/text/unipain.html, I've come to the conclusion that the best way to support Python 3 is to bite the bullet and do the "unicode sandwich". That is to say convert all external bytestring values to 'U' arrays for internal (and user) manipulation, and back to 'S' for delivery to files / network etc. This is a pain and very inefficient, but at least the the Python 3 user experience is natural and pleasant. I figure if you are manipulating anything less than ~Gb of text data then it won't be a disaster.