pydata/pandas — pull request #4698 by adgaudio (2013-08-27 23:52:04+00:00)
Title: HDFStore.append_to_multiple doesn't write rows that are all np.nan

Hi,

Using HDFStore.append_to_multiple, if an entire row written to any one table consists entirely of np.nan, the row is not written to the table, but is written to the other tables. The following code reproduces and fixes the issue.

I would prefer that append_to_multiple maintain synchronized rows across tables, and to my knowledge, the best way to do that is drop that row from the other tables. We would probably need a fix that looks something like this PR.

I'm not sure if this is the best way to do this, and would love some feedback!

Thanks,
Alex

The Fix:

# setup
d = {'table1': ['col1', 'col2'], 'table2': ['col3', 'col4']}
value = pandas.DataFrame([[1,2,3,4], [5,6,None, None]], columns=['col1', 'col2', 'col3', 'col4'])

# fix (as in PR)
dfs = (value[cols] for cols in d.values())
valid_index = reduce(
lambda index1, index2: index2.intersection(index1),
[df.dropna(how='all').index for df in dfs])
value = value.ix[valid_index]

To reproduce the error:

In [1]: from pandas import DataFrame, HDFStore

In [2]: store = HDFStore('tmp1.h5')

In [3]: df = DataFrame([[1,2], [3,None]], columns=['a', 'b'])

In [4]: store.append_to_multiple({'table1': ['a'], 'table2': ['b']}, df, 'table1')

In [5]: store
Out[5]:

File path: tmp1.h5
/table1 frame_table (typ->appendable,nrows->2,ncols->1,indexers->[index],dc->[a])
/table2 frame_table (typ->appendable,nrows->1,ncols->1,indexers->[index])
store.select_as_multiple(['table1', 'table2'])

In [6]: store.select_as_multiple(['table1', 'table2'])
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
in ()
----> 1 store.select_as_multiple(['table1', 'table2'])

/Users/alexgaudio/.virtualenvs/ds/lib/python2.7/site-packages/pandas/io/pytables.pyc in select_as_multiple(self, keys, where, selector, columns, start, stop, iterator, chunksize, auto_close, **kwargs)
540 nrows = t.nrows
541 elif t.nrows != nrows:
--> 542 raise ValueError("all tables must have exactly the same nrows!")
543
544 # select coordinates from the selector table

ValueError: all tables must have exactly the same nrows!