ENH: allow saving wide dataframes to hdf with format table
State: closed

based on #11788
closes #6245

This PR allows to save wide dataframes to hdf. It will break forward compatibility, old versions of pandas will not be able to read hdfs from new versions.
The column is saved to a vlarray with object atomic. This is rather slow for very wide dfs but does not require any type checks.
One test fails because the saved column is not compressed but the compression level is checked for all nodes. I think this is fine but i wanted to consult with you before rewriting the test.

Two tests were added to check if a wide df can be saved and if it is appendable.

Please let me know what you think and if there is additional need for test coverage.

Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=h1) Report
> Merging [#26135](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/872a23bca9f74816aaf525d1b00d790c3407cc08?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `97.82%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26135/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26135      +/-   ##
==========================================
+ Coverage   91.99%      92%   +0.01%     
==========================================
  Files         175      175              
  Lines       52387    52414      +27     
==========================================
+ Hits        48191    48222      +31     
+ Misses       4196     4192       -4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.55% <89.13%> (+0.01%)` | :arrow_up: |
| #single | `40.79% <97.82%> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `90.36% <97.82%> (+0.13%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `75% <0%> (-12.5%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.9% <0%> (-0.12%)` | :arrow_down: |
| [pandas/io/excel/\_base.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsL19iYXNlLnB5) | `92.82% <0%> (-0.07%)` | :arrow_down: |
| [pandas/tseries/holiday.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvaG9saWRheS5weQ==) | `93.17% <0%> (-0.04%)` | :arrow_down: |
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `98.2% <0%> (ø)` | :arrow_up: |
| [pandas/core/reshape/tile.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS90aWxlLnB5) | `97.67% <0%> (ø)` | :arrow_up: |
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `94.08% <0%> (ø)` | :arrow_up: |
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `91.5% <0%> (+0.13%)` | :arrow_up: |
| [pandas/core/groupby/ops.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9vcHMucHk=) | `95.97% <0%> (+1.73%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=footer). Last update [872a23b...05aac5b](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=h1) Report
> Merging [#26135](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/ebcfee442b04c083b054461bb421edc031b0cb40?src=pr&el=desc) will **decrease** coverage by `1.26%`.
> The diff coverage is `97.82%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26135/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26135      +/-   ##
==========================================
- Coverage      93%   91.74%   -1.27%     
==========================================
  Files         182      174       -8     
  Lines       50311    50808     +497     
==========================================
- Hits        46793    46615     -178     
- Misses       3518     4193     +675
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.25% <89.13%> (-1.42%)` | :arrow_down: |
| #single | `41.75% <97.82%> (-0.74%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `90.36% <97.82%> (+0.01%)` | :arrow_up: |
| [pandas/plotting/\_misc.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19taXNjLnB5) | `38.23% <0%> (-26.63%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-21.06%)` | :arrow_down: |
| [pandas/io/gcs.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2djcy5weQ==) | `80% <0%> (-20%)` | :arrow_down: |
| [pandas/io/s3.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3MzLnB5) | `89.47% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/groupby/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9iYXNlLnB5) | `91.83% <0%> (-8.17%)` | :arrow_down: |
| [pandas/io/excel/\_xlrd.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsL194bHJkLnB5) | `94.54% <0%> (-5.46%)` | :arrow_down: |
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `90.53% <0%> (-4.53%)` | :arrow_down: |
| [pandas/util/\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX2RlY29yYXRvcnMucHk=) | `91.34% <0%> (-4.01%)` | :arrow_down: |
| [pandas/plotting/\_core.py](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb3JlLnB5) | `83.89% <0%> (-3.76%)` | :arrow_down: |
| ... and [171 more](https://codecov.io/gh/pandas-dev/pandas/pull/26135/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=footer). Last update [ebcfee4...f725d20](https://codecov.io/gh/pandas-dev/pandas/pull/26135?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback 
I included tests for several kinds of different column types that were stored with the current release version of pandas.
But I think i need some help with the failing tests. There are two tests failing:

1. "Docstring validation" and "Testing docstring validaton script" are failing. code_checks.sh runs fine locally. I don't understand how my changes are effecting these.

2. In one of the travis enviroments most pytables test fail with "ValueError: cannot set WRITEABLE flag to True of this array" This seems to be related to #24839. I can reproduce it locally by creating an environment with the corresponding environment.yml The tests are passed fine if I manually upgrade pytables to 3.5.1 (which was fixed to work with numpy 1.6).
If i understand correctly xfail_non_writeable was introduced in #25517 to prevent code that triggers this bug to fail the tests. By using vlarrays for all columns when format='table' this is now triggered for most tests. Does it makes sense to mask all the failing tests with xfail_non_writeable? Or is it more reasonable to not merge this PR until pytables 3.5.1 is the standard in all travis environments?
can you merge master and update
@jreback Updated.
1. "Docstring validation" and "Testing docstring validaton script" is ok now. 
2. "ValueError: cannot set WRITEABLE flag to True of this array" still leads to lots of fails in one of the test environments.

something is not right with your patch; merge upstream/master
Hi Jeff,
thanks for your feedback. I need some help understanding what the issue is. I just pulled upstream master and it merged without conflict. I don't have any local changes. Can you specify were the problem is?
maybe u didn’t push
the PR has 377 changed files
 I see, thanks for the help. You were right, i didn't push.
can you merge master; note we moved the test_pytables to a subdirectory
@P-Tillmann can you rebase
@jreback @jbrockmendel I updated to current master. But travis stil has an environment with pytables "cannot set WRITEABLE flag" bug for vlarrays. And since this PR uses a lot of vlarrays it fails almost all tests.
@P-Tillmann is there a min version for pytables where that was fixed?
@P-Tillmann is this still active?
Closing as stale but if this is still relevant please ping and can reopen