soft-matter/trackpy — pull request #144 by danielballan (2014-08-14 19:56:14+00:00)
Title: PERF: In PandasHDFStore, store data in tabular format.

This decreases the file size of framewise-data h5 files by about 40X. The reading and writing times are roughly the same as before: I measured 7% slower.

Pandas stores fixed-format DataFrames very inefficiently. It is better -- and arguably more idiomatically anyway -- to store them in tabular format. Note that each frame is its own table and tables are never actually appended.

h5 files generated before this PR need to remain readable by trackpy, and they are. This -- and the reduced file size -- are demonstrated below.

On current `master`:

In [2]: v = tp.Video(os.path.join(os.path.dirname(tp.__file__), 'tests', 'water', 'bulk-water.mov'), as_grey=True)

In [3]: !rm -f store-1.h5

In [4]: with tp.PandasHDFStore('store-1.h5') as s:
...: for frame in v[:8]:
...: f = tp.locate(frame, 9, invert=True)
...: s.put(f)
...:
/home/dallan/pandas-danielballan/pandas/io/pytables.py:2464: PerformanceWarning:
your performance may suffer as PyTables will pickle object types that it cannot
map directly to c-types [inferred_type->unicode,key->axis0] [items->None]

In [5]: !du -h store-1.h5
25M	store-1.h5

Note large file size. Now on this PR:

In [2]: v = tp.Video(os.path.join(os.path.dirname(tp.__file__), 'tests', 'water', 'bulk-water.mov'), as_grey=True)

In [3]: !rm -f store-2.h5

In [4]: with tp.PandasHDFStore('store-2.h5') as s:
for frame in v[:8]:
f = tp.locate(frame, 9, invert=True)
s.put(f)
...:

The time to locate and write a frame is the same before and after this PR (with 5%).
In [5]: !du -h store-2.h5
656K	store-2.h5

We can verify that the first file, the one generated on trackpy `master`, is still readable.

In [6]: with tp.PandasHDFStore('store-1.h5') as s:
print pd.concat(iter(s)).head()
...:
x y mass size ecc signal ep \
0 278.507438 5.390083 605 3.198657 0.756301 -125.520116 -0.010121
1 552.711169 11.264398 3438 2.182217 0.109113 66.479884 0.034783
2 309.115707 6.178312 1789 2.771330 0.151396 -99.520116 -0.015572
3 115.329098 5.970775 787 2.507929 0.249666 -107.520116 -0.016876
4 294.554176 5.738678 2981 2.146512 0.155674 35.479884 0.067173

frame
0 0
1 0
2 0
3 0
4 0

This fix propagates to `PandasHDFStoreBig`. The frames cache is still a fixed-format node, but that is not a major cost. The other implementation, `PandasHDFStoreSingleNode`, already uses tabular mode.