pydata/pandas — pull request #8793 by jreback (2014-11-12 01:55:30+00:00)
Title: ENH: serialization of categorical to HDF5 (GH7621)

This is implemented by storing the codes directly in the table. And a metadata VLArray of the categories.
Query and appending work as expected. The only quirk is that I don't allow you to append to a table unless the new data has exactly the same categories. Otherwise the codes become meaningless.

This has the nice property of drastically shrinking the storage cost compared to regular strings (which are stored as fixed width of the maximum for that particular column).

```
In [14]: df = DataFrame({'a' : Series(list('abccdef')).astype('category'), 'b' : np.random.randn(7)})

In [15]: df
Out[15]:
a b
0 a -0.094609
1 b -1.814638
2 c 0.214974
3 c -0.195395
4 d 0.206022
5 e 1.130589
6 f -0.832810

In [19]: store = pd.HDFStore('test.h5',mode='w')

In [20]: store.append('df',df,data_columns=['a'])

In [21]: store.select('df',where=["a in ['b','d']"])
Out[21]:
a b
1 b -1.814638
4 d 0.206022

In [22]: store.select('df',where=["a in ['b','d']"]).dtypes
Out[22]:
a category
b float64
dtype: object

In [25]: store.get_storer('df').group
Out[25]:
/df (Group) u''
children := ['table' (Table), 'meta' (Group)]

In [26]: store.get_storer('df').group.table
Out[26]:
/df/table (Table(7,)) ''
description := {
"index": Int64Col(shape=(), dflt=0, pos=0),
"values_block_0": Float64Col(shape=(1,), dflt=0.0, pos=1),
"a": Int8Col(shape=(), dflt=0, pos=2)}
byteorder := 'little'
chunkshape := (3855,)
autoindex := True
colindexes := {
"a": Index(6, medium, shuffle, zlib(1)).is_csi=False,
"index": Index(6, medium, shuffle, zlib(1)).is_csi=False}

In [27]: store.get_storer('df').group.meta
Out[27]:
/df/meta (Group) u''
children := ['a' (VLArray)]
```