Fixed HDFSTore.groups() performance.  
State: closed

No longer walks every node, but rather every group.

- [x] closes #21372
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Labels:
name: Performance
description: Memory or execution speed performance
name: IO HDF5
description: read_hdf, HDFStore

Comments:
I'm not sure if this deserves it's own 'whatsnew' entry... it is pretty simple.
I believe we have an asv for this, can you run. and yes this *does* need a whatsnew, you can put in 0.23.2 perf.
This is my first pull request for pandas.  What is "asv"?

I've updated the whatsnew.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21543?src=pr&el=h1) Report
> Merging [#21543](https://codecov.io/gh/pandas-dev/pandas/pull/21543?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/506935c3482cb678b75dc7a70f48707c70b4ae79?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21543/graphs/tree.svg?width=650&src=pr&token=eZ4WkYLtcO&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/21543?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #21543   +/-   ##
=======================================
  Coverage   91.92%   91.92%           
=======================================
  Files         153      153           
  Lines       49563    49563           
=======================================
  Hits        45559    45559           
  Misses       4004     4004
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.32% <ø> (ø)` | :arrow_up: |
| #single | `41.8% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21543?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/21543/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `92.43% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21543?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21543?src=pr&el=footer). Last update [506935c...e9a02df](https://codecov.io/gh/pandas-dev/pandas/pull/21543?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

http://pandas.pydata.org/pandas-docs/stable/contributing.html#running-the-performance-test-suite
Thanks.

I ran the `io.hdf.*` benchmarks and got the following:
```
       before           after         ratio
     [49188296]       [38b70fcf]
+     7.02±0.07ms         25.1±1ms     3.57  io.hdf.HDFStoreDataFrame.time_query_store_table
+      28.9±0.4ms           50.7ms     1.75  io.hdf.HDFStoreDataFrame.time_read_store_mixed
+      16.3±0.4ms           28.3ms     1.74  io.hdf.HDFStoreDataFrame.time_query_store_table_wide
+     25.5±0.01ms           38.0ms     1.49  io.hdf.HDFStoreDataFrame.time_read_store
+        68.1±2ms           98.1ms     1.44  io.hdf.HDFStoreDataFrame.time_read_store_table_mixed
+        41.8±1ms         57.5±7ms     1.37  io.hdf.HDFStoreDataFrame.time_write_store_mixed
-          22.5ms           19.5ms     0.86  io.hdf.HDFStoreDataFrame.time_read_store_table_wide
-          29.8ms         25.1±1ms     0.84  io.hdf.HDFStoreDataFrame.time_write_store
-          10.4μs       8.63±0.9μs     0.83  io.hdf.HDFStoreDataFrame.time_store_repr
-      6.04±0.1ms       4.61±0.2ms     0.76  io.hdf.HDFStoreDataFrame.time_store_info
```

Which is obviously an issue.

Because this doesn't make sense (some of those tests don't even appear to be touching the part of the code that I changed...), I ran it again:

```
       before           after         ratio
     [49188296]       [38b70fcf]
+      28.2±0.3ms         41.0±1ms     1.45  io.hdf.HDFStoreDataFrame.time_write_store_mixed
+         200±3ms            284ms     1.42  io.hdf.HDFStoreDataFrame.time_write_store_table_dc
+        44.5±2ms         60.7±2ms     1.36  io.hdf.HDFStorePanel.time_read_store_table_panel
+     5.84±0.03μs       7.64±0.3μs     1.31  io.hdf.HDFStoreDataFrame.time_store_repr
+        49.3±3ms           58.0ms     1.18  io.hdf.HDFStoreDataFrame.time_write_store_table
+      70.6±0.4ms         80.8±2ms     1.15  io.hdf.HDFStoreDataFrame.time_write_store_table_mixed
-      28.4±0.2ms       24.4±0.2ms     0.86  io.hdf.HDFStoreDataFrame.time_read_store
-         159±2ms         136±10ms     0.85  io.hdf.HDF.time_read_hdf('table')
-         152±4ms          128±5ms     0.84  io.hdf.HDF.time_write_hdf('table')
-      16.6±0.9ms       13.1±0.2ms     0.79  io.hdf.HDFStoreDataFrame.time_query_store_table_wide
-      5.10±0.4ms      3.04±0.08ms     0.60  io.hdf.HDFStoreDataFrame.time_store_info
```

And I get what looks to be a completely different set of tests that are different.

A third time gives me:

```
       before           after         ratio
     [49188296]       [38b70fcf]
+      27.6±0.1ms       36.1±0.7ms     1.31  io.hdf.HDFStoreDataFrame.time_read_store_mixed
+      19.1±0.3ms       22.5±0.7ms     1.18  io.hdf.HDFStoreDataFrame.time_read_store_table_wide
-      71.7±0.7ms       56.6±0.1ms     0.79  io.hdf.HDFStoreDataFrame.time_write_store_table_mixed
-     26.0±0.07ms       19.0±0.2ms     0.73  io.hdf.HDFStoreDataFrame.time_write_store
-        60.6±3ms         41.4±1ms     0.68  io.hdf.HDFStorePanel.time_read_store_table_panel
-        42.7±2ms       26.9±0.1ms     0.63  io.hdf.HDFStoreDataFrame.time_write_store_mixed
```

Which is at least a little better.  I think this test is fairly sensitive to what else is going on on my computer.  In order to get a more accurate result, I think I'll need to clean boot and run the tests without anything else running.
yeah there don't appear to be any benchmarks for this (except for ``.info()``) which is related. can you see if it makes sense to add any benchmarks here (i don't remember exactly what .info hits)
`.info()` hits `.keys()`, so there is some benchmarking supported here.

This particular performance issue wasn't really a problem till I hit a couple hundred (thousand?) keys in a hdf file, so I don't know if we want to create a dataset to test this.
I rebased, so ping on green.
thanks @spott