HDFStore: Fix empty result of keys() method on non-pandas hdf5 file
State: closed

First try to get pandas style tables, if there are none, return the list of non-pandas (hdf5 native) tables in the file (if any)

- [x] closes #29916
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: API Design
name: IO HDF5
description: read_hdf, HDFStore

Comments:
is this better handled upstream in dask? I am ok with a keyword to .keys but cannot change the behavior like this.
I have created a new PR #32723 which has the extra 'include' keyword.
I haven't added the 'all' value as mixing pandas tables and native tables in one file is an anti-pattern (@jreback 's words), which should not be encouraged.
> is this better handled upstream in dask?

I'll try to find out, there was not a lot of response so far, so I'll ping and ask.

> I am ok with a keyword to .keys but cannot change the behavior like this.

I understand and I am with you wrt not changing the API behavoiur.
I am not sure it is worthile to change anything here. The fact that HDFStore doesn't recognize non-pandas tables is a non-goal here.
> I am not sure it is worthile to change anything here. The fact that HDFStore doesn't recognize non-pandas tables is a non-goal here.

This is a disappointing attitude, because this used to work. It's not an enhancement, but a regression that was broken by an 'optimisation'

Groups and keys aren't a feature of pandas, but a feature of the HDF5 standard. If pandas is going to claim support for the HDF5 file format, then it should support the whole format. This isn't even restricted to the dataframe/tables level. You're basically stating the equivalent of 'supporting files stored in a directory not created by pandas is a non-goal'. 

At the dataframe level, I don't understand why you'd not want the ability to load data from a non-pandas table and create a dataframe from it - pandas isn't the only library out there for working with HDF5, and it's safe to assume not all users will be working with data that came out of pandas in the first place (got my hand up here.)

I do, however understand why you'd not want to be converting back from a dataframe to a 'non-pandas table' - being able to read data from a non-pandas HDF5 but only write pandas tables makes some kind of sense.

If pandas is insistent on following this line of thought, then maybe you could at least update the documentation to declare that HDF5 files written not written with pandas won't work with pandas (or declare a derivative file format, called PHDF5 or something and very specifically define what's supported and what's not)

Just one person's opinion.

EDIT: I've just noticed that the docs for read_hdf does say 'retrieve a **pandas** object stored in a file' so my comment about the docs above is only half-relevant
> > I am not sure it is worthile to change anything here. The fact that HDFStore doesn't recognize non-pandas tables is a non-goal here.
> 
> This is a disappointing attitude, because this used to work. It's not an enhancement, but a regression that was broken by an 'optimisation'
> 
> Groups and keys aren't a feature of pandas, but a feature of the HDF5 standard. If pandas is going to claim support for the HDF5 file format, then it should support the whole format. This isn't even restricted to the dataframe/tables level. You're basically stating the equivalent of 'supporting files stored in a directory not created by pandas is a non-goal'.
> 
> At the dataframe level, I don't understand why you'd not want the ability to load data from a non-pandas table and create a dataframe from it - pandas isn't the only library out there for working with HDF5, and it's safe to assume not all users will be working with data that came out of pandas in the first place (got my hand up here.)

Sorry to jump in here, but **importing** non-pandas files should work just fine and was not the issue here. It is just not possible to list all the keys/groups with standard pandas anymore which lead to downstream problems, most notable with dask. You can have a look at the PR here or at the linked dask issue on how to list all of them with pytables.

If you have trouble importing hdf5 files in general, then that may be a separate issue.

[...]
> If pandas is insistent on following this line of thought, then maybe you could at least update the documentation to declare that HDF5 files written not written with pandas won't work with pandas (or declare a derivative file format, called PHDF5 or something and very specifically define what's supported and what's not)

Yeah I cannot understand that either, but it is the developers' decision to cripple hdf5 support to pandas specific files.

> Just one person's opinion.
> 
> EDIT: I've just noticed that the docs for read_hdf does say 'retrieve a **pandas** object stored in a file' so my comment about the docs above is only half-relevant

That would be a really bad decision indeed.
> Sorry to jump in here, 

No need to apologise. It is a discussion, after all

> but **importing** non-pandas files should work just fine and was not the issue here. It is just not possible to list all the keys/groups with standard pandas anymore which lead to downstream problems, most notable with dask. You can have a look at the PR here or at the linked dask issue on how to list all of them with pytables.

The problem isn't just with dask (I'm not using dask). I am trying to import data from non-pandas HDF5 files, to be put into a dataframe, which would then be exported to pandas HDF5 files. My issue was with not being able to list the keys/groups - I have no control of the format of the data coming in, and I need to 'walk' the groups to figure out what's in the file (the file format has a versioning system in it, so I need to walk the groups to figure out how many there are, and where to pull my specific dataset from) - so I guess you could say I have the same issue as dask. They've implemented a workaround which doesn't use the public API of pandas (by drilling down directly into pytables)

> 
> If you have trouble importing hdf5 files in general, then that may be a separate issue.
> 

No, importing the data from a pandas file works fine, my issue is with groups/keys also.

> [...]
> 
> > If pandas is insistent on following this line of thought, then maybe you could at least update the documentation to declare that HDF5 files written not written with pandas won't work with pandas (or declare a derivative file format, called PHDF5 or something and very specifically define what's supported and what's not)
> 
> Yeah I cannot understand that either, but it is the developers' decision to cripple hdf5 support to pandas specific files.

The biggest issue I have is the 'we are not adding new methods' statement. I understand pandas wants a clean API, and certainly doesn't want to roll back a change made so long ago. What I can't fathom is why pandas couldn't have a seperate raw_keys() and raw_groups() method in addition to the keys() and groups() methods. This wouldn't break the existing API, and would allow us to still walk these groups while still keeping in place this 'optimisation'

> 
> > Just one person's opinion.
> > EDIT: I've just noticed that the docs for read_hdf does say 'retrieve a **pandas** object stored in a file' so my comment about the docs above is only half-relevant
> 
> That would be a really bad decision indeed.

What would be a bad decision? I don't understand this last statement.

Thanks for your insight
> > Sorry to jump in here,
> 
> No need to apologise. It is a discussion, after all

I believe that the discussion should take place at the issue, not here.

> > but **importing** non-pandas files should work just fine and was not the issue here. It is just not possible to list all the keys/groups with standard pandas anymore which lead to downstream problems, most notable with dask. You can have a look at the PR here or at the linked dask issue on how to list all of them with pytables.
> 
> The problem isn't just with dask (I'm not using dask). I am trying to import data from non-pandas HDF5 files, to be put into a dataframe, which would then be exported to pandas HDF5 files. My issue was with not being able to list the keys/groups - I have no control of the format of the data coming in, and I need to 'walk' the groups to figure out what's in the file (the file format has a versioning system in it, so I need to walk the groups to figure out how many there are, and where to pull my specific dataset from) - so I guess you could say I have the same issue as dask. They've implemented a workaround which doesn't use the public API of pandas (by drilling down directly into pytables)

Probably then dask may be actually the right tool, it allows wildcards in the key argument. Otherwise you could probably use plain pytables in the first round to get the keys, and then pandas to read them. I don't think that the issue will be fixed in pandas.

> > [...]
> > > If pandas is insistent on following this line of thought, then maybe you could at least update the documentation to declare that HDF5 files written not written with pandas won't work with pandas (or declare a derivative file format, called PHDF5 or something and very specifically define what's supported and what's not)
> > 
> > 
> > Yeah I cannot understand that either, but it is the developers' decision to cripple hdf5 support to pandas specific files.
> 
> The biggest issue I have is the 'we are not adding new methods' statement. I understand pandas wants a clean API, and certainly doesn't want to roll back a change made so long ago. What I can't fathom is why pandas couldn't have a seperate raw_keys() and raw_groups() method in addition to the keys() and groups() methods. This wouldn't break the existing API, and would allow us to still walk these groups while still keeping in place this 'optimisation'

I understand the intention to not change the API. Hdf5 is apparently a very minor part such that nobody really bothers. The best would have been to revert the change and admit that it was screwed up. Would have saved a lot of developer resources.

> > > Just one person's opinion.
> > > EDIT: I've just noticed that the docs for read_hdf does say 'retrieve a **pandas** object stored in a file' so my comment about the docs above is only half-relevant
> > 
> > 
> > That would be a really bad decision indeed.
> 
> What would be a bad decision? I don't understand this last statement.

Sorry, my comment was about `read_hdf` only supporting pandas objects, which would be a really bad choice imo.

I think it is best to continue at the issue, and not fill this pull request further.
ok I think I commented else where that If you want to support `.keys(nodes='pandas|all')` I think would be ok (and if .groups needs this keyword then ok too).
prefer  soln in #32723