GH15943 Fixed defaults for compression in HDF5
State: closed

 - [x] closes #15943
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master --name-only -- '*.py' | flake8 --diff``
 - [x] whatsnew entry


Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
no problem fixing the bug,, but why are you changing that default? 
I was not trying to change any defaults, it was more a case of unclear descriptions? I asked what was the desired outcome of a fix, but no clear answer was given. If faulty assumptions were made, tell me what the default behavior should be.

The default was listed as None in the help text (which does not really say a lot about what it should do). In pytable terms None means it defaults to zlib, but it is also not unreasonable to assume it means disable all compression. Looking at the existing code was not an option either, since this was the broken part the pull-request was meant to fix. 

complevel defaulted to 9 in the code, but the helpt-text said it defaulted to 0. These are also not compatible.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16355?src=pr&el=h1) Report
> Merging [#16355](https://codecov.io/gh/pandas-dev/pandas/pull/16355?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d298414d2724fed40254350117adcabc4e546e48?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16355/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/16355?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16355      +/-   ##
==========================================
- Coverage   90.95%   90.91%   -0.04%     
==========================================
  Files         161      161              
  Lines       49276    49276              
==========================================
- Hits        44817    44800      -17     
- Misses       4459     4476      +17
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.68% <60%> (-0.03%)` | :arrow_down: |
| #single | `40.17% <100%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16355?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/16355?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `92.27% <ø> (ø)` | :arrow_up: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/16355?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `92.87% <100%> (-0.17%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/16355?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `63.23% <0%> (-1.82%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/16355?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.23% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16355?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16355?src=pr&el=footer). Last update [d298414...af7ecbf](https://codecov.io/gh/pandas-dev/pandas/pull/16355?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> It seems when complib is None, pd.HDFStore does not use any compression even though a user supplies complevel > 0. If there is no good default option for complib, then in my opinion, HDFStore should raise an exception when (complib is None) and (complevel > 0) instead. This is because clearly the user wants compression but it cannot be done - I think explicitly raising an exception is better than silently ignoring the option.

so this seems a clear bug.

independently what happens if ``complib=None`` (which is the default) -> no compression. Are the docs not clear?

so would take the bug fix and doc updates / clarifications.

But switching the default is not likely to happen (where is the default ``zlib`` in pytables anyhow)?

Admittedly I only read pytables documentation and haven't checked the code, but the default complib is zlib and the default complevel is 0. The complib option is not used for turning compression on and off, this is what the complevel option is for. It is all described in the link you provided to the relevant [pytables docs](http://www.pytables.org/usersguide/libref/helper_classes.html#filtersclassdescr).

The bug fix you want for this is that a valueerror exception is raised when complib is none and complevel > 0? 

Do I also put the complevel back to defaulting to 9 if complib is set and correct the docs instead?  

again not sure where you are actually looking in pytables. We don't pass filters, nor do we pass the complib/complevel unless its defined. If you can show something different, all ears. In any event pls just do the actual bug fix here. If you think that the default should change, then by all means open an issue.
This is starting to feel like beating a dead horse, but I would really like to know where I went wrong.

The filters are set [here](https://github.com/pandas-dev/pandas/blob/master/pandas/io/pytables.py#L586), when complib is defined. This is also where complevel is set to 9 if undefined. The options for the filters are described [here](http://www.pytables.org/usersguide/libref/helper_classes.html#filtersclassdescr). I find it confusing having identically named options that are passed more or less directly to the underlying pytables have slightly different semantics. 

I agree with the decision on maintaining previous and expected behavior. I was just unsure what that was. 
@linebp Can you tell me if this summary is correct?

On master:

```python
In [4]: df = pd.DataFrame({"A": [1]})

In [6]: df.to_hdf("/tmp/foo.h5", "df", complevel=9)
```

doesn't use any compression, since `complib=None`. With your PR, the file would be compressed, since `self._complib = complib if complib else tables.filters.default_complib` will evaluate to `zlib`.

So on master, having either `compblib=None` or `complevel=None` means no compression. With your PR, having `complevel = 0` is no compression. So the defaults haven't changed, but the behavior of `complevel > 0` has changed.

---

If that summary is correct, then I'm 100% on board to just change the behavior as you have here. I would move the whatsnew to the bug fixes section (or maybe backwards incompatible changes?).
Can you also update this section in the rst docs? http://pandas-docs.github.io/pandas-docs-travis/io.html#compression
> doesn't use any compression, since complib=None. With your PR, the file would be compressed, since self._complib = complib if complib else tables.filters.default_complib will evaluate to zlib.

> So on master, having either compblib=None or complevel=None means no compression. With your PR, having complevel = 0 is no compression. So the defaults haven't changed, but the behavior of complevel > 0 has changed.

this would be fine. the default (if nothing is passed) needs to be NO compression. If a user passes complib and/or complevel then they are explicit about compression.
@TomAugspurger your summary is almost correct. Having compblib='blosc' and complevel=None results in compression, in this case complevel defaults to 9. The summary looks like:
**Master:**
- complib defined + complevel missing: compression with level set to 9
- complib defined + complevel defined: compression with level set to level
- complib missing + complevel missing: no compression 
- complib missing + complevel defined: no compression

**PR:**
- complib defined + complevel missing: No compression
- complib defined + complevel defined: compression with level set to level
- complib missing + complevel missing: no compression 
- complib missing + complevel defined: compression with zlib and level set to level

I pasted the results of my experiments on master below if anyone is interested, the relevant info is in the FILTERS tag. I added tests to cover the various combinations of missing and defined complevel and complib in my PR.

```
$ python
Python 3.6.0 |Anaconda 4.3.0 (64-bit)| (default, Dec 23 2016, 12:22:00) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.__version__
'0.19.0+515.gd0a281f'
>>> df = pd.DataFrame({"A": [1]})
>>> df.to_hdf("/tmp/foo.h5", "df", complevel=9)
>>> 

$ h5dump -H -p --onlyattr=0 /tmp/foo.h5
HDF5 "/tmp/foo.h5" {
GROUP "/" {
   GROUP "df" {
      DATASET "axis0" {
         DATATYPE  H5T_STRING {
            STRSIZE 1;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
         STORAGE_LAYOUT {
            CONTIGUOUS
            SIZE 1
            OFFSET 3096
         }
         FILTERS {
            NONE
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_IFSET
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_LATE
         }
      }
      DATASET "axis1" {
         DATATYPE  H5T_STD_I64LE
         DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
         STORAGE_LAYOUT {
            CONTIGUOUS
            SIZE 8
            OFFSET 3097
         }
         FILTERS {
            NONE
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_IFSET
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_LATE
         }
      }
      DATASET "block0_items" {
         DATATYPE  H5T_STRING {
            STRSIZE 1;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
         STORAGE_LAYOUT {
            CONTIGUOUS
            SIZE 1
            OFFSET 3113
         }
         FILTERS {
            NONE
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_IFSET
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_LATE
         }
      }
      DATASET "block0_values" {
         DATATYPE  H5T_STD_I64LE
         DATASPACE  SIMPLE { ( 1, 1 ) / ( 1, 1 ) }
         STORAGE_LAYOUT {
            CONTIGUOUS
            SIZE 8
            OFFSET 3105
         }
         FILTERS {
            NONE
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_IFSET
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_LATE
         }
      }
   }
}
}

$ python                  
Python 3.6.0 |Anaconda 4.3.0 (64-bit)| (default, Dec 23 2016, 12:22:00) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> df = pd.DataFrame({"A": [1]})
>>> df.to_hdf("/tmp/foo.h5", "df", complib='blosc')
>>> 

$ h5dump -H -p --onlyattr=0 /tmp/foo.h5
HDF5 "/tmp/foo.h5" {
GROUP "/" {
   GROUP "df" {
      DATASET "axis0" {
         DATATYPE  H5T_STRING {
            STRSIZE 1;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SIMPLE { ( 1 ) / ( 65536 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 65536 )
            SIZE 291 (0.003:1 COMPRESSION)
         }
         FILTERS {
            USER_DEFINED_FILTER {
               FILTER_ID 32001
               COMMENT blosc
               PARAMS { 2 2 1 65536 9 1 }
            }
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_ALLOC
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_INCR
         }
      }
      DATASET "axis1" {
         DATATYPE  H5T_STD_I64LE
         DATASPACE  SIMPLE { ( 1 ) / ( 8192 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 8192 )
            SIZE 396 (0.020:1 COMPRESSION)
         }
         FILTERS {
            USER_DEFINED_FILTER {
               FILTER_ID 32001
               COMMENT blosc
               PARAMS { 2 2 8 65536 9 1 }
            }
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_ALLOC
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_INCR
         }
      }
      DATASET "block0_items" {
         DATATYPE  H5T_STRING {
            STRSIZE 1;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SIMPLE { ( 1 ) / ( 65536 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 65536 )
            SIZE 291 (0.003:1 COMPRESSION)
         }
         FILTERS {
            USER_DEFINED_FILTER {
               FILTER_ID 32001
               COMMENT blosc
               PARAMS { 2 2 1 65536 9 1 }
            }
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_ALLOC
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_INCR
         }
      }
      DATASET "block0_values" {
         DATATYPE  H5T_STD_I64LE
         DATASPACE  SIMPLE { ( 1, 1 ) / ( 8192, 1 ) }
         STORAGE_LAYOUT {
            CHUNKED ( 8192, 1 )
            SIZE 396 (0.020:1 COMPRESSION)
         }
         FILTERS {
            USER_DEFINED_FILTER {
               FILTER_ID 32001
               COMMENT blosc
               PARAMS { 2 2 8 65536 9 1 }
            }
         }
         FILLVALUE {
            FILL_TIME H5D_FILL_TIME_ALLOC
            VALUE  H5D_FILL_VALUE_DEFAULT
         }
         ALLOCATION_TIME {
            H5D_ALLOC_TIME_INCR
         }
      }
   }
}
}

``` 
@linebp ok, your PR chart looks good. even though this is a bug fix, let's put it in 0.21.0. 
Great! I'll edit the docs and whatsnew and update the PR.
I made some changes as requested and its all checking out green.
thanks @linebp 

nice fixes & docs! 
have a look at the built docs: http://pandas-docs.github.io/pandas-docs-travis/ (will take a bit to generate them). and if needed pls issue a followup PR.