ENH: enable support for iterator with read_hdf in HDFStore (GH3937)
State: closed

closes #3937

DOC: update v0.11.1.rst


Labels:

Comments:
@jreback I had a slightly different idea about how to do this that I actually think simplifies everything (turn store into a context manager itself).  That lets you do something like:

``` python
with get_store(path) as store:
     store['foo'] = bar
```

Check it out here: https://github.com/jtratner/pandas/tree/change-store-to-contextmanager-type

You could actually remove `get_store` entirely if you didn't default HDFStore to call open in its constructor, but for right now I added a classmethod to do that.

aside from the ability to directly use `HDFStore('file')` instead of `get_store`, is this any different?

Major difference is that it lets you use `with store:` anywhere, even if
passed a store, so there's no differentiation in functionality based on
where you use it. (e.g., with a file handle, you can always use it in a
with statement after it's created). e.g.

``` python
store = get_store('asdfa.h5')
# do some things
with store:
    # do some stuff where it's important to close it up
```

Whereas right now, you have to use `get_store` in a with statement.

It also would let you use the iterator in a with statement directly - I was
thinking about my suggestion to put it at the end of the yield statement
and I think I was wrong about it (or at least not covering all the bases):
if an error gets raised during the iteration, the store will never be
closed.

On Wed, Jun 19, 2013 at 7:34 AM, jreback notifications@github.com wrote:

> aside from the ability to directly use HDFStore('file') instead of
> get_store, is this any different?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/3949#issuecomment-19678133
> .

I think this will close it _always_ though....e.g

```
store = get_store(...)
with store:
   ......
store._handle.isopen == False
```

as an aside, prob should add `is_open` property to store anyhow

?

I don't think the code you have there will work as written. Get store returns a context manager not a HDFStore. You only get the store if you call  `__enter__` on it. 

Another side note would it make more sense to have open raise an error or warning instead of raw input? Causes the test suite to unexpectedly hang when you try to use it. 

Maybe instead the check could be (pseudocode)

``` python
if is open and mode==w: 
    if not force: raise some exception   
```

Because I think that the larger issue is that you can call open repeatedly and have it overwrite itself in w mode

my example was using your code, the question is how does the context manager know **not** to `auto_close`?

the whole bizness of `_quiet` and that type of input was some _original_ code.....needs to be changed

It doesn't, if you use store as a context manager, the expectation should be that it closes on leaving the with context, just like other file objects. Only table iterator needs to know about auto closing. 

ok then.....if you want to do a PR, ok....only thing is need a test for it

(also pls add `is_open` property to top-level store)

```
@property
def is_open(self):
    try:
         return self._handle.isopen
     except:
         return False
```

And actually given the way that you can reopen it, you could repeatedly use it as a context manager and have it open and close each time, so that it's only open when you are specifically choosing to use it (eg if it's important for more than one program to access it) . If we could get rid of 'w' the mode it'd be a lot better, because then it's never problematic to reopen. (no chance of  overwriting anything). 

fyi, I _always_ open in `mode='a'`, even when I am reading, unfort have to preserve `mode='r'` just to prevent accidental writing issues

@jreback can we deprecate support for 'w' mode?

i think u need it
how else to overwrite a file?
( of course can remove it)
but file interfaces allow to overwrite