Repository: pandas-dev/pandas

Issue:
API: Should this raise an error or a warning in HDFStore?
Currently in 0.12.

```
In [3]: DataFrame(randn(10,2)).to_hdf('test_storer.h5','df')

In [4]: DataFrame(randn(10,2)).to_hdf('test_table.h5','df',table=True)
```

Selecting from a table is good with or without a where

```
In [6]: pd.read_hdf('test_table.h5','df',where='index>5')
Out[6]: 
          0         1
6  0.296947 -1.876055
7 -0.200516 -0.641670
8 -0.177828  0.877286
9  0.836903 -1.626247
```

This raises if where is not None, the theory being that passing a non-None where was a mistake.

However, then you have to a-prior know if you are dealing with a `Storer` or a `Table`

```
In [7]: pd.read_hdf('test_storer.h5','df',where='index>5')
TypeError: cannot pass a where specification when reading a Storer
```

make this a warning instead?



Discussion:
This is definitely a confusing error and error message.  Is there a reason that someone would want to have a "Storer" vs. a table? (i.e., why is the default to create something that can't be queried later on...speed?) Why is the default not to store it in PyTables format?

You definitely can't just emit a warning, because ignoring the where clause potentially means (unintentionally) loading a big database into memory.

I'm not convinced this is a `TypeError` either, but I think it needs to emit an error message that not only clearly states what's going on in terms of the file format, but also specifically direct how to make it possible to work that way.

``` python
In [7]: pd.read_hdf('test_storer.h5','df',where='index>5')
TypeError: Basic HDF5 tables cannot be queried with a 'where' clause. Either do not supply a where clause or load from a 'PyTable' instead (e.g., HDF5 data created using 'to_hdf(...table=True)).
```

You probably also need to document this better, given that the only time `table=True` is mentioned in the dev docs on IO is in the Notes & Caveats section:

```
You can not append/select/delete to a non-table (table creation is determined on the first append, or by passing `table=True` in a put operation)
```

I never actually use `put` and `table=True`...you are prob more familiar with `append` which always creates a table, while `put` always creates a `storer`

These are both `PyTable` formats of course.

there is a speed difference (mainly in writing) and they are written in different formats.
and you cannot `append` nor `select` from a storer,  (you can only `get` it in it entirety)

actually this is documented pretty well in the doc strings.  what exactly do you find confusing about it?

@jreback now that you've explained, it's a bit clearer. 

I went to the IOTools section of the docs to try to understand what the difference is between the two of them. So, are all the examples with the actual HDFStore assuming that you are working with a `PyTable`?

You write:

> These are both `PyTable` formats of course.

Then why can you query one but not the other?

All this said, maybe this is something that becomes really clear when you start to use the functionality. I have a really far removed view of it, since I haven't had the cause to use this (yet).

yes...all are `PyTables`. There is a distinction between the `storer` format, which stores `PyTables` arrays. (I could never thing of a good name, and didn't to use basic, maybe array is better). and `table` which store in the `PyTables` table format

You can actually index into a `storer` format, you just can't query (that is you can so things like: `array[0:100]`, kind of like a numpy array on disk).

With a `table` you can `select`, that is do something like `index>0 & index<100`. The real difference is that `tables` can be appended.

The reason I brought this up is that I happen to sometimes write `storer's` but mostly write tables. However, I always query them (so in my own code I basically catch the exception and act on it). Just wanted to know what other people do.

leaving it as an exception....maybe i'll update the error message a bit (and I think `TypeError` is correct, as you are basically doing a query on the wrong `type` here)

Agreed on raising the `TypeError`, but I only ever use `table`s.

Yeah, I've got no insight into what other people do with PyTables :P


Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -1746,9 +1746,11 @@ def f(values, freq=None, tz=None):
 
     def validate_read(self, kwargs):
         if kwargs.get('columns') is not None:
-            raise TypeError("cannot pass a column specification when reading a Storer")
+            raise TypeError("cannot pass a column specification when reading a non-table "
+                            "this store must be selected in its entirety")
         if kwargs.get('where') is not None:
-            raise TypeError("cannot pass a where specification when reading a Storer")
+            raise TypeError("cannot pass a where specification when reading from a non-table "
+                            "this store must be selected in its entirety")
 
     @property
     def is_exists(self):


Test patch: