Pytables selection enhancement & docs update for HDF5 tables
State: closed

1. added **str** (to do **repr**)
2. row removal in tables is much faster if rows are consecutive
3. added Term class, refactored Selection (this is backwards compatible)
    Term is a concise way of specifying conditions for queries, e.g.
   
   ```
   Term(dict(field = 'index', op = '>', value = '20121114'))
   Term('index', '20121114')
   Term('index', '>', '20121114')
   Term('index', ['20121114','20121114'])
   Term('index', datetime(2012,11,14))
   Term('index>20121114')
   ```
   
    updated tests for same
   
     this should close GH #1996
4. added docs for HDF5 table in io.html
5. append on a table that didn't exist was failing (because of testing of the index_kind attribute first - which may not exist)
    fixed & added test
6. added create_table_index method to create indicies on tables (which, btw now works quite well as Int64 indicies are used as opposed to the Time64Col which has a bug); includes a check on the pytables version requirement
   
   this should close GH #698
7. added min_itemsize as a paremeter to append; allows bigger default indexer columns upon table creation (even if you don't append something that big - but might later, avoid the truncation issue)
8. incorporated 0.9.1 whatsnew docs for where & mask into Indexing Section of main docs


Labels:

Comments:
@thisch ..thanks...that is fixed.....let me know if anything else is unclear.....this functionality has mostly been in pandas for a while, but undoced....so you can try it out 

@wesm I have also some preliminary work on speeding up table writes (with panels), currently takes about 9.5s for 1M rows (e.g. a 6 x 1000 x 1000 panel); made code a lot simpler and using cython..down to about 6s; about 1/2 of overhead is from pytables actually writing it, other from creating a list of tuples (which is then turned into a recarray by pytables) - prob will be able to PR this next week

I think you can write whole blocks of data instead of going row by row and go much faster? @John-Colvin has worked on this I think

I had some discussions with John (waiting for some sample timings)

but I think there r really 2 cases here

1) if u want a searchable table u could write it in one shot (say blocks of columns) - selecting is a bit tricky as u have to compute the values indices (eg select from major and minor then compute the offsets for the values) - could be done - but this becomes a fixed table - u cannot easily append (and have the ability to search) -

2) the current approach, while slower in writing allows searching and appending

so I suppose some use cases might prefer having faster writing and still have a searching ability 

I prefer to write my tables in small batches, but need to preserve searching (and reading is quite fast anyhow)

I suppose If there is enough interest could support both approaches

On Nov 16, 2012, at 10:16 PM, Wes McKinney notifications@github.com wrote:

> I think you can write whole blocks of data instead of going row by row and go much faster? @John-Colvin has worked on this I think
> 
> —
> Reply to this email directly or view it on GitHub.

closing this - going to put in a new PR soon that is a bit cleaner