Repository: pandas-dev/pandas

Issue:
allow HDFStore to remain open when TableIterator is returned from read_hdf
Hi,
I'm using a TableIterator from pandas.read_hdf function (with the keyword argument iterator=True), I am unable to retrieve any data due to the error "ClosedNodeError: the node object is closed".

For instance:

```
pandas.DataFrame({'a':[1,2,3], 'b':[4,5,6]}).to_hdf("test.h5", "test", append=True)
it = pandas.read_hdf("test.h5","test",iterator=True)
iter(it).next()

Traceback (most recent call last):
  File "<ipython-input-22-5634d86698ab>", line 1, in <module>
    iter(it).next()
  File "/usr/local/lib/python2.7/site-packages/pandas/io/pytables.py", line 912, in __iter__
    v = self.func(current, stop)
  ...
  File "/usr/local/lib/python2.7/site-packages/tables/node.py", line 355, in _g_check_open
    raise ClosedNodeError("the node object is closed")
ClosedNodeError: the node object is closed
```

I looked through source code of panda.io.pytables and found that in the
get_store function, store.close() is always run when read_hdf returns, even if
it returns an TableIterator. My assumption is that store should remain open in
order for TableIterator to work. Can you please let me know if this fix is
acceptable, or is there an easier way to do this?

Thanks,
Sean



Discussion:
its only open/closed automatically when you use `read_hdf`, otherwise use store as normal. The example uses the more verbose syntax. http://pandas.pydata.org/pandas-docs/dev/io.html#iterator

@seanyeh anything further? otherwise pls close 

I do find it odd that it allows an option that doesn't work. Thanks anyway

where does the option not work? it works exactly how its supposed to in `read_hdf`, which provides a context to open/close the file. When used with an already open store it doesn't close it.

How else would you expect it to work?

@jreback what's the point of passing iterator=True if you can't iterate over the result?

It seems intuitive that this should work, right?

``` python
for x in pandas.read_hdf("test.h5", "test", iterator=True):
    print x
```

but according to the example above, that would raise the closed node error.

Maybe it would make more sense to have TableIterator handle the cleanup if `read_hdf` is passed a path/string instead of an open store? (so, in `__iter__`, after control passes out of the while loop, do the cleanup that is necessary to close it up).

@jtratner  @seanyeh 
originally you always had to open/close stores yourself, `read_hdf` does this for you; I suppose it could be enabled with iterator/chunksize support like above (in which the context manager knowns to close it, but only after iter is done). 

I guess if the context manager is passed an open handle then it shouldn't close it...

+1 for enabling read_hdf(..., iterator=True).  

Arguably, since pytables automatically closes the h5 file via atexit.register(close_open_files), we don't need to explicitly close it.  

@adgaudio I disagree.

 These files need explicit open/close; or use with a context manager. ]

 This is straightforward to address and will be fixed soon.

 Relying on the system to close files is not good from a safety perspective nor good programming practice. 

Yea, true.  Thanks everyone - I'll look forward to using the patch :)

Agree with @jreback. atexit is fragile and there's no reason not to handle
this explicitly.

On a separate note - @is it problematic to pass (or set) a reference to the
store to the TableIterator? Makes it much cleaner to handle that way...

@adgaudio https://github.com/adgaudio I disagree.

These files need explicit open/close; or use with a context manager. ]

This is straightforward to address and will be fixed soon.

Relying on the system to close files is not good from a safety perspective
nor good programming practice.

—
Reply to this email directly or view it on
GitHubhttps://github.com/pydata/pandas/pull/3937#issuecomment-19635061
.


Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -196,12 +196,27 @@ def to_hdf(path_or_buf, key, value, mode=None, complevel=None, complib=None, app
 
 def read_hdf(path_or_buf, key, **kwargs):
     """ read from the store, closeit if we opened it """
-    f = lambda store: store.select(key, **kwargs)
+    f = lambda store, auto_close: store.select(key, auto_close=auto_close, **kwargs)
 
     if isinstance(path_or_buf, basestring):
-        with get_store(path_or_buf) as store:
-            return f(store)
-    f(path_or_buf)
+
+        # can't auto open/close if we are using an iterator
+        # so delegate to the iterator
+        store = HDFStore(path_or_buf)
+        try:
+            return f(store, True)
+        except:
+
+            # if there is an error, close the store
+            try:
+                store.close()
+            except:
+                pass
+
+            raise
+
+    # a passed store; user controls open/close
+    f(path_or_buf, False)
     
 class HDFStore(object):
     """
@@ -405,7 +420,7 @@ def get(self, key):
             raise KeyError('No object named %s in the file' % key)
         return self._read_group(group)
 
-    def select(self, key, where=None, start=None, stop=None, columns=None, iterator=False, chunksize=None, **kwargs):
+    def select(self, key, where=None, start=None, stop=None, columns=None, iterator=False, chunksize=None, auto_close=False, **kwargs):
         """
         Retrieve pandas object stored in file, optionally based on where
         criteria
@@ -419,6 +434,7 @@ def select(self, key, where=None, start=None, stop=None, columns=None, iterator=
         columns : a list of columns that if not None, will limit the return columns
         iterator : boolean, return an iterator, default False
         chunksize : nrows to include in iteration, return an iterator
+        auto_close : boolean, should automatically close the store when finished, default is False
 
         """
         group = self.get_node(key)
@@ -434,9 +450,11 @@ def func(_start, _stop):
             return s.read(where=where, start=_start, stop=_stop, columns=columns, **kwargs)
 
         if iterator or chunksize is not None:
-            return TableIterator(func, nrows=s.nrows, start=start, stop=stop, chunksize=chunksize)
+            if not s.is_table:
+                raise TypeError("can only use an iterator or chunksize on a table")
+            return TableIterator(self, func, nrows=s.nrows, start=start, stop=stop, chunksize=chunksize, auto_close=auto_close)
 
-        return TableIterator(func, nrows=s.nrows, start=start, stop=stop).get_values()
+        return TableIterator(self, func, nrows=s.nrows, start=start, stop=stop, auto_close=auto_close).get_values()
 
     def select_as_coordinates(self, key, where=None, start=None, stop=None, **kwargs):
         """
@@ -473,7 +491,7 @@ def select_column(self, key, column, **kwargs):
         """
         return self.get_storer(key).read_column(column = column, **kwargs)
 
-    def select_as_multiple(self, keys, where=None, selector=None, columns=None, start=None, stop=None, iterator=False, chunksize=None, **kwargs):
+    def select_as_multiple(self, keys, where=None, selector=None, columns=None, start=None, stop=None, iterator=False, chunksize=None, auto_close=False, **kwargs):
         """ Retrieve pandas objects from multiple tables
 
         Parameters
@@ -541,9 +559,9 @@ def func(_start, _stop):
             return concat(objs, axis=axis, verify_integrity=True)
 
         if iterator or chunksize is not None:
-            return TableIterator(func, nrows=nrows, start=start, stop=stop, chunksize=chunksize)
+            return TableIterator(self, func, nrows=nrows, start=start, stop=stop, chunksize=chunksize, auto_close=auto_close)
 
-        return TableIterator(func, nrows=nrows, start=start, stop=stop).get_values()
+        return TableIterator(self, func, nrows=nrows, start=start, stop=stop, auto_close=auto_close).get_values()
 
 
     def put(self, key, value, table=None, append=False, **kwargs):
@@ -916,16 +934,20 @@ class TableIterator(object):
         Parameters
         ----------
 
-        func   : the function to get results
+        store : the reference store
+        func  : the function to get results
         nrows : the rows to iterate on
         start : the passed start value (default is None)
-        stop : the passed stop value (default is None)
+        stop  : the passed stop value (default is None)
         chunksize : the passed chunking valeu (default is 50000)
+        auto_close : boolean, automatically close the store at the end of iteration,
+            default is False
         kwargs : the passed kwargs
         """
 
-    def __init__(self, func, nrows, start=None, stop=None, chunksize=None):
-        self.func   = func
+    def __init__(self, store, func, nrows, start=None, stop=None, chunksize=None, auto_close=False):
+        self.store = store
+        self.func  = func
         self.nrows = nrows or 0
         self.start = start or 0
 
@@ -937,6 +959,7 @@ def __init__(self, func, nrows, start=None, stop=None, chunksize=None):
             chunksize = 100000
 
         self.chunksize = chunksize
+        self.auto_close = auto_close
 
     def __iter__(self):
         current = self.start
@@ -950,9 +973,16 @@ def __iter__(self):
 
             yield v
 
+        self.close()
+            
+    def close(self):
+        if self.auto_close:
+            self.store.close()
+
     def get_values(self):
-        return self.func(self.start, self.stop)
-        
+        results = self.func(self.start, self.stop)
+        self.close()
+        return results
 
 class IndexCol(object):
     """ an index column description class


Test patch:
diff --git a/pandas/io/tests/test_pytables.py b/pandas/io/tests/test_pytables.py
--- a/pandas/io/tests/test_pytables.py
+++ b/pandas/io/tests/test_pytables.py
@@ -2078,6 +2078,7 @@ def test_select_iterator(self):
             results = []
             for s in store.select('df',chunksize=100):
                 results.append(s)
+            self.assert_(len(results) == 5)
             result = concat(results)
             tm.assert_frame_equal(expected, result)
 
@@ -2085,7 +2086,28 @@ def test_select_iterator(self):
             for s in store.select('df',chunksize=150):
                 results.append(s)
             result = concat(results)
-            tm.assert_frame_equal(expected, result)
+            tm.assert_frame_equal(result, expected)
+
+        with tm.ensure_clean(self.path) as path:
+
+            df = tm.makeTimeDataFrame(500)
+            df.to_hdf(path,'df_non_table')
+            self.assertRaises(TypeError, read_hdf, path,'df_non_table',chunksize=100)
+            self.assertRaises(TypeError, read_hdf, path,'df_non_table',iterator=True)
+
+        with tm.ensure_clean(self.path) as path:
+
+            df = tm.makeTimeDataFrame(500)
+            df.to_hdf(path,'df',table=True)
+
+            results = []
+            for x in read_hdf(path,'df',chunksize=100):
+                results.append(x)
+
+            self.assert_(len(results) == 5)
+            result = concat(results)
+            tm.assert_frame_equal(result, df)
+            tm.assert_frame_equal(result, read_hdf(path,'df'))
 
         # multiple