Pytablesv4
State: closed

Refactor of PyTables support to allow multiple table types.

This commit allows for support of multiple table types in a pytables hdf file,
      supporting the existing infrastructure in a backwards compatible manner (LegacyTable)
      while extending to a slightly modified format to support AppendableTables and future support of WORMTables

AppendableTables are implementations of the current table format with two enhancements:
         - mixed dtype support
         - writing routines in cython for enhanced performance

WORMTables (not implemented - but pretty straightforward)
        these tables can support a fixed 'table' (meaning not-appendable), that is searchable via queries
        this would have greatly enhanced write performance compared with AppendableTables, and a similar read performance profile

In addition, the tables allow for arbitrary axes to be indexed (e.g. you could save a panel that allows indexing on major_axis,minor_axis AND items),
        so all dimensions are queryable (currently only major/minor axes allow this query)

all tests pass (with 1 exception)
        a frame table round-trip - will fail on a comparison of a sorted index of the frame vs the index of the table (which is as written), not sure why this should be the case?


Labels:

Comments:
Awesome. Will plan to merge this for 0.10...

one more small doc update to this (coming later today) then hopefully ready to merge - in theory should cleanly merge

only thing I couldn't figure out was how to get my cython module picked up in the pathing  (I had to explicitly path it) 

On Nov 27, 2012, at 4:53 PM, Wes McKinney notifications@github.com wrote:

> Awesome. Will plan to merge this for 0.10...
> 
> —
> Reply to this email directly or view it on GitHub.

ok....ready to merge....I think i accidently pushed some merges off master: a4191c62ac14903f7bcb3705bb95d234c36964ef and 64c2eb60d2eecf0d17a45dd743f5493baa83b43f
but they are current so prob shouldn't matter

pls check the link in the whatsnew docs (to the full docs) - not sure it's pointed at the right thing

awesome - any issues? 

I had to fix an issue resulting from my laptop having pytables 2.3b1 (int('3b1') failed in the version check), and there was one test failure that might also be related to < 2.3. OK now

great! pushed a small doc update in: 4dcd24a1e8dce5967801793798c3830db2ea17a8 not sure if this got linked, so did it as another PR.

On a related issue. In test_table_mixed_dtypes in test_pytables. I have tried adding Timestamp objects to the store. After .consolidate() it seems that the df still has the datetime cols as dtype = 'object'; it required a .convert_objects() to fix...is this a bug?

```
import pandas
import pandas.util.testing as tm
from pandas import Timestamp
df = tm.makeDataFrame()
df['obj1'] = 'foo'
df['obj2'] = 'bar'
df['bool1'] = df['A'] > 0
df['bool2'] = df['B'] > 0
df['bool3'] = True
df['int1'] = 1
df['int2'] = 2
df['datetime'] = Timestamp('20010101')
df = df.consolidate()

In [16]: df.get_dtype_counts()
Out[16]: 
bool       3
float64    4
int64      2
object     3

In [18]: df.convert_objects().get_dtype_counts()
Out[18]: 
bool              3
datetime64[ns]    1
float64           4
int64             2
object            2
```

I am going to prob create an issue of a couple of things that will prob add (as an Issue?)...
e.g. support datetimes in mixed-type HDFStore tables...