Repository: pandas-dev/pandas

Issue:
COMPAT: reading generic PyTables Table format fails with sub-selection
I created a file using pytables, and now I would like to read it into pandas. My (naive) expectation was that these two tools were compatible. But I am getting an error

``` python
pd.read_hdf(output_fname, '/floats/trajectories', start=0, stop=10)
```

gives

```
ValueError: Shape of passed values is (1, 10), indices imply (1, 753664023)
```

The file is 44 GB, so I can't really post it. I would be happy to post the `h5dump --head` metadata if that would help.

(Cross post with blaze/dask#747.)

COMPAT: reading generic PyTables Table format fails with sub-selection
I created a file using pytables, and now I would like to read it into pandas. My (naive) expectation was that these two tools were compatible. But I am getting an error

``` python
pd.read_hdf(output_fname, '/floats/trajectories', start=0, stop=10)
```

gives

```
ValueError: Shape of passed values is (1, 10), indices imply (1, 753664023)
```

The file is 44 GB, so I can't really post it. I would be happy to post the `h5dump --head` metadata if that would help.

(Cross post with blaze/dask#747.)



Discussion:
@rabernat 

pandas can read a `Table` from `PyTables`. Their is simply not enough meta data to generally read other HDF5 constructs. Sure it could be done in some way, but is not a very easy/nice use case. The pandas written formats _can_ be read by `PyTables`, but might not be semantically readable.

> pandas can read a Table from PyTables

My file was created by pytables. That's why I was surprised it didn't work.

well you would have to show a specific case then

The example below reproduces the error. The problem is with the `start` and `stop` kwargs. Without those, it works fine. But they are necessary if dask is going to be able to chunk the file.

``` python
import tables
import pandas as pd
import numpy as np

output_fname = 'test.h5'
class LFloat(tables.IsDescription):
    npart   = tables.Int32Col(pos=1)   # float id number, starts at 1
    time    = tables.Float32Col(pos=2)  # time of the datapoint
    x       = tables.Float32Col(pos=3)  # x position
    y       = tables.Float32Col(pos=4)  # y position
    z       = tables.Float32Col(pos=5)  # z position

dtype = tables.description.dtype_from_descr(LFloat)

nrecs = 10
with tables.openFile(output_fname, mode='w', title='Float Data') as h5file:
    group = h5file.createGroup("/", 'floats', 'Float Data')
    table = h5file.createTable(group, 'trajectories', LFloat,
                                "Float Trajectories", expectedrows=nrecs)
    for n in range(nrecs):
        d = np.empty(1, dtype)
        d['npart'] = n
        table.append(d)

    table.cols.npart.createIndex()
    table.flush()

df = pd.read_hdf('test.h5', '/floats/trajectories', start=0, stop=5)
```

ok, this is a bug in the reading this tables. never had a need to do this, so its untested. So if you want to submit a pull-request fix would be great. The issue is [here](https://github.com/pydata/pandas/blob/master/pandas/io/pytables.py#L1660) I think.

Note that with simple floats,ints,strings you would be ok. But any other data types will simply fail (e.g. datetimes). You are almost certainly better off saving this using pandas format (which is readable by PyTables).

Thanks @jreback! I'll see what I can do.

I am not married to pytables for my application. I chose it because it allows me to write hdf files incrementally and thus scales to out-of-core file sizes. If pandas can do that directly, then I would rather use pure pandas. But my impression was that you have to first create the whole pandas dataframe in memory and then serialize to hdf. Correct?

certainly not. You can create want you need then stream to hdf very much like `PyTables`. This in fact was the reason it was built. You can chunk write/read.

Just create what you need, serialize, repeat. This is the `format='table'` which allows query and append.  pandas also supports `format='fixed'` which are fixed sized `PyTables` structures that are much faster for IO (almost like pure numpy arrays), but don't allow appending nor querying.

See docs [here](http://pandas.pydata.org/pandas-docs/stable/io.html#hdf5-pytables).

Just tried your suggestion. I ended up with segfaults when trying to append to my HDFStore. I think this is related to #10672. I guess I need to update pytables to 3.2.1.

3.2 is buggy yep

@jreback Could you point me again to the source of the issue? I think [link](https://github.com/pydata/pandas/blob/master/pandas/io/pytables.py#L1660) you provided earlier is not valid anymore. 

I'm also encountering this issue and looking for the solution. In my use case I'm using `PyTables` to collect measurement data to hdf5 file in the `table` format. I'm easily able to collect 20GB of (blosc compressed) file and currently are not able to process it.

I'm also encountering MemoryError if the index is bigger than RAM, but will prepare new bug report for it.
https://github.com/pandas-dev/pandas/pull/13267 fixed this for ``format='fixed'``. 
> #13267 fixed this for format='fixed'.

I do not quite understand what this means. @rabernat's [comment above](https://github.com/pandas-dev/pandas/issues/11188#issuecomment-143436365) shows a nice minimal working example demonstrating the problem that he found, and that also @mchwalisz has found (and that I have right now). Can we modify the example (ideally just the _reading_ part, and not the data _generation_ part) so that it works?


> @jreback Could you point me again to the source of the issue? I think link you provided earlier is not valid anymore.

0.17.0 was released shortly after @jreback's comment (Sep 2015) and I think this is the line @jreback was pointing us to: https://github.com/pandas-dev/pandas/blob/v0.17.0/pandas/io/pytables.py#L1660

Now here: https://github.com/pandas-dev/pandas/blob/429078b01af47b356084c1da270b5b275fbf50f4/pandas/io/pytables.py#L1822
@rabernat 

pandas can read a `Table` from `PyTables`. Their is simply not enough meta data to generally read other HDF5 constructs. Sure it could be done in some way, but is not a very easy/nice use case. The pandas written formats _can_ be read by `PyTables`, but might not be semantically readable.

> pandas can read a Table from PyTables

My file was created by pytables. That's why I was surprised it didn't work.

well you would have to show a specific case then

The example below reproduces the error. The problem is with the `start` and `stop` kwargs. Without those, it works fine. But they are necessary if dask is going to be able to chunk the file.

``` python
import tables
import pandas as pd
import numpy as np

output_fname = 'test.h5'
class LFloat(tables.IsDescription):
    npart   = tables.Int32Col(pos=1)   # float id number, starts at 1
    time    = tables.Float32Col(pos=2)  # time of the datapoint
    x       = tables.Float32Col(pos=3)  # x position
    y       = tables.Float32Col(pos=4)  # y position
    z       = tables.Float32Col(pos=5)  # z position

dtype = tables.description.dtype_from_descr(LFloat)

nrecs = 10
with tables.openFile(output_fname, mode='w', title='Float Data') as h5file:
    group = h5file.createGroup("/", 'floats', 'Float Data')
    table = h5file.createTable(group, 'trajectories', LFloat,
                                "Float Trajectories", expectedrows=nrecs)
    for n in range(nrecs):
        d = np.empty(1, dtype)
        d['npart'] = n
        table.append(d)

    table.cols.npart.createIndex()
    table.flush()

df = pd.read_hdf('test.h5', '/floats/trajectories', start=0, stop=5)
```

ok, this is a bug in the reading this tables. never had a need to do this, so its untested. So if you want to submit a pull-request fix would be great. The issue is [here](https://github.com/pydata/pandas/blob/master/pandas/io/pytables.py#L1660) I think.

Note that with simple floats,ints,strings you would be ok. But any other data types will simply fail (e.g. datetimes). You are almost certainly better off saving this using pandas format (which is readable by PyTables).

Thanks @jreback! I'll see what I can do.

I am not married to pytables for my application. I chose it because it allows me to write hdf files incrementally and thus scales to out-of-core file sizes. If pandas can do that directly, then I would rather use pure pandas. But my impression was that you have to first create the whole pandas dataframe in memory and then serialize to hdf. Correct?

certainly not. You can create want you need then stream to hdf very much like `PyTables`. This in fact was the reason it was built. You can chunk write/read.

Just create what you need, serialize, repeat. This is the `format='table'` which allows query and append.  pandas also supports `format='fixed'` which are fixed sized `PyTables` structures that are much faster for IO (almost like pure numpy arrays), but don't allow appending nor querying.

See docs [here](http://pandas.pydata.org/pandas-docs/stable/io.html#hdf5-pytables).

Just tried your suggestion. I ended up with segfaults when trying to append to my HDFStore. I think this is related to #10672. I guess I need to update pytables to 3.2.1.

3.2 is buggy yep

@jreback Could you point me again to the source of the issue? I think [link](https://github.com/pydata/pandas/blob/master/pandas/io/pytables.py#L1660) you provided earlier is not valid anymore. 

I'm also encountering this issue and looking for the solution. In my use case I'm using `PyTables` to collect measurement data to hdf5 file in the `table` format. I'm easily able to collect 20GB of (blosc compressed) file and currently are not able to process it.

I'm also encountering MemoryError if the index is bigger than RAM, but will prepare new bug report for it.
https://github.com/pandas-dev/pandas/pull/13267 fixed this for ``format='fixed'``. 
> #13267 fixed this for format='fixed'.

I do not quite understand what this means. @rabernat's [comment above](https://github.com/pandas-dev/pandas/issues/11188#issuecomment-143436365) shows a nice minimal working example demonstrating the problem that he found, and that also @mchwalisz has found (and that I have right now). Can we modify the example (ideally just the _reading_ part, and not the data _generation_ part) so that it works?


> @jreback Could you point me again to the source of the issue? I think link you provided earlier is not valid anymore.

0.17.0 was released shortly after @jreback's comment (Sep 2015) and I think this is the line @jreback was pointing us to: https://github.com/pandas-dev/pandas/blob/v0.17.0/pandas/io/pytables.py#L1660

Now here: https://github.com/pandas-dev/pandas/blob/429078b01af47b356084c1da270b5b275fbf50f4/pandas/io/pytables.py#L1822

Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -1624,7 +1624,8 @@ def infer(self, handler):
         new_self.read_metadata(handler)
         return new_self
 
-    def convert(self, values, nan_rep, encoding, errors):
+    def convert(self, values, nan_rep, encoding, errors, start=None,
+                stop=None):
         """ set the values from this selection: take = take ownership """
 
         # values is a recarray
@@ -1813,10 +1814,29 @@ class GenericIndexCol(IndexCol):
     def is_indexed(self):
         return False
 
-    def convert(self, values, nan_rep, encoding, errors):
-        """ set the values from this selection: take = take ownership """
+    def convert(self, values, nan_rep, encoding, errors, start=None,
+                stop=None):
+        """ set the values from this selection: take = take ownership
+
+        Parameters
+        ----------
+
+        values : np.ndarray
+        nan_rep : str
+        encoding : str
+        errors : str
+        start : int, optional
+            Table row number: the start of the sub-selection.
+        stop : int, optional
+            Table row number: the end of the sub-selection. Values larger than
+            the underlying table's row count are normalized to that.
+        """
+
+        start = start if start is not None else 0
+        stop = (min(stop, self.table.nrows)
+                if stop is not None else self.table.nrows)
+        self.values = Int64Index(np.arange(stop - start))
 
-        self.values = Int64Index(np.arange(self.table.nrows))
         return self
 
     def get_attr(self):
@@ -2159,7 +2179,8 @@ def validate_attr(self, append):
                 raise ValueError("appended items dtype do not match existing "
                                  "items dtype in table!")
 
-    def convert(self, values, nan_rep, encoding, errors):
+    def convert(self, values, nan_rep, encoding, errors, start=None,
+                stop=None):
         """set the data from this selection (and convert to the correct dtype
         if we can)
         """
@@ -3431,8 +3452,11 @@ def read_axes(self, where, **kwargs):
         # convert the data
         for a in self.axes:
             a.set_info(self.info)
+            # `kwargs` may contain `start` and `stop` arguments if passed to
+            # `store.select()`. If set they determine the index size.
             a.convert(values, nan_rep=self.nan_rep, encoding=self.encoding,
-                      errors=self.errors)
+                      errors=self.errors, start=kwargs.get('start'),
+                      stop=kwargs.get('stop'))
 
         return True
 


Test patch:
diff --git a/pandas/tests/io/pytables/test_compat.py b/pandas/tests/io/pytables/test_compat.py
new file mode 100644
--- /dev/null
+++ b/pandas/tests/io/pytables/test_compat.py
@@ -0,0 +1,76 @@
+import pytest
+
+import pandas as pd
+from pandas.tests.io.test_pytables import ensure_clean_path
+from pandas.util.testing import assert_frame_equal
+
+tables = pytest.importorskip('tables')
+
+
+@pytest.fixture
+def pytables_hdf5_file():
+    """Use PyTables to create a simple HDF5 file."""
+
+    table_schema = {
+        'c0': tables.Time64Col(pos=0),
+        'c1': tables.StringCol(5, pos=1),
+        'c2': tables.Int64Col(pos=2),
+    }
+
+    t0 = 1561105000.0
+
+    testsamples = [
+        {'c0': t0, 'c1': 'aaaaa', 'c2': 1},
+        {'c0': t0 + 1, 'c1': 'bbbbb', 'c2': 2},
+        {'c0': t0 + 2, 'c1': 'ccccc', 'c2': 10**5},
+        {'c0': t0 + 3, 'c1': 'ddddd', 'c2': 4294967295},
+    ]
+
+    objname = 'pandas_test_timeseries'
+
+    with ensure_clean_path('written_with_pytables.h5') as path:
+        # The `ensure_clean_path` context mgr removes the temp file upon exit.
+        with tables.open_file(path, mode='w') as f:
+            t = f.create_table('/', name=objname, description=table_schema)
+            for sample in testsamples:
+                for key, value in sample.items():
+                    t.row[key] = value
+                t.row.append()
+
+        yield path, objname, pd.DataFrame(testsamples)
+
+
+class TestReadPyTablesHDF5:
+    """
+    A group of tests which covers reading HDF5 files written by plain PyTables
+    (not written by pandas).
+
+    Was introduced for regression-testing issue 11188.
+    """
+
+    def test_read_complete(self, pytables_hdf5_file):
+        path, objname, df = pytables_hdf5_file
+        result = pd.read_hdf(path, key=objname)
+        expected = df
+        assert_frame_equal(result, expected)
+
+    def test_read_with_start(self, pytables_hdf5_file):
+        path, objname, df = pytables_hdf5_file
+        # This is a regression test for pandas-dev/pandas/issues/11188
+        result = pd.read_hdf(path, key=objname, start=1)
+        expected = df[1:].reset_index(drop=True)
+        assert_frame_equal(result, expected)
+
+    def test_read_with_stop(self, pytables_hdf5_file):
+        path, objname, df = pytables_hdf5_file
+        # This is a regression test for pandas-dev/pandas/issues/11188
+        result = pd.read_hdf(path, key=objname, stop=1)
+        expected = df[:1].reset_index(drop=True)
+        assert_frame_equal(result, expected)
+
+    def test_read_with_startstop(self, pytables_hdf5_file):
+        path, objname, df = pytables_hdf5_file
+        # This is a regression test for pandas-dev/pandas/issues/11188
+        result = pd.read_hdf(path, key=objname, start=1, stop=2)
+        expected = df[1:2].reset_index(drop=True)
+        assert_frame_equal(result, expected)
diff --git a/pandas/tests/io/test_pytables.py b/pandas/tests/io/test_pytables.py
--- a/pandas/tests/io/test_pytables.py
+++ b/pandas/tests/io/test_pytables.py
@@ -105,7 +105,7 @@ def ensure_clean_store(path, mode='a', complevel=None, complib=None,
 def ensure_clean_path(path):
     """
     return essentially a named temporary file that is not opened
-    and deleted on existing; if path is a list, then create and
+    and deleted on exiting; if path is a list, then create and
     return list of filenames
     """
     try: