BUG: GH29310 HDF file compression not working
State: closed

- [x] closes #29310 
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Re #29310, the complib and complevel parameters were not being passed down all the way previously, hence HDF compression not working.  

I noticed that the implementation of to_hdf() specifies that compression is not allowed for fixed formats:  
` if not s.is_table and complib:
            raise ValueError("Compression not supported on Fixed format stores")`

I'm guessing that means the performance comparison section for https://github.com/pandas-dev/pandas/pull/28890/files will also need to be updated to remove the test_fixed_compress test @WuraolaOyewusi? 

Also, after the update, the following test is currently failing: 
![image](https://user-images.githubusercontent.com/13011161/68165200-a4736c00-ff13-11e9-964f-3ebc105e7e46.png)

due to a ValueError for using compression with a fixed format, and I'm not sure as to why the expected behaviour for this test is what it is? Why should setting complib disable compression? I would appreciate any further info on that.






Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Hello @yp1996! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









































* In the file [`pandas/tests/arrays/sparse/test_array.py`](https://github.com/pandas-dev/pandas/blob/49641f1a51b3a8ae48484fe52d308e9cbf423207/pandas/tests/arrays/sparse/test_array.py):

> [Line 663:24](https://github.com/pandas-dev/pandas/blob/49641f1a51b3a8ae48484fe52d308e9cbf423207/pandas/tests/arrays/sparse/test_array.py#L663): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 668:24](https://github.com/pandas-dev/pandas/blob/49641f1a51b3a8ae48484fe52d308e9cbf423207/pandas/tests/arrays/sparse/test_array.py#L668): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','

















* In the file [`pandas/tests/indexing/test_callable.py`](https://github.com/pandas-dev/pandas/blob/49641f1a51b3a8ae48484fe52d308e9cbf423207/pandas/tests/indexing/test_callable.py):

> [Line 20:39](https://github.com/pandas-dev/pandas/blob/49641f1a51b3a8ae48484fe52d308e9cbf423207/pandas/tests/indexing/test_callable.py#L20): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 23:39](https://github.com/pandas-dev/pandas/blob/49641f1a51b3a8ae48484fe52d308e9cbf423207/pandas/tests/indexing/test_callable.py#L23): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 93:42](https://github.com/pandas-dev/pandas/blob/49641f1a51b3a8ae48484fe52d308e9cbf423207/pandas/tests/indexing/test_callable.py#L93): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','



























##### Comment last updated at 2020-01-22 06:08:45 UTC
> * [x]  closes #29310
> * [x]  passes `black pandas`
> * [x]  passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
> 
> Re #29310, the complib and complevel parameters were not being passed down all the way previously, hence HDF compression not working.
> 
> I noticed that the implementation of to_hdf() specifies that compression is not allowed for fixed formats:
> ` if not s.is_table and complib: raise ValueError("Compression not supported on Fixed format stores")`
> 
> I'm guessing that means the performance comparison section for https://github.com/pandas-dev/pandas/pull/28890/files will also need to be updated to remove the test_fixed_compress test @WuraolaOyewusi?
> 
> Also, after the update, the following test is currently failing:
> ![image](https://user-images.githubusercontent.com/13011161/68165200-a4736c00-ff13-11e9-964f-3ebc105e7e46.png)
> 
> due to a ValueError for using compression with a fixed format, and I'm not sure as to why the expected behaviour for this test is what it is? Why should setting complib disable compression? I would appreciate any further info on that.

I think rather than silently refuse compression of fixed format, It should raise an error instead.
pls rebase
rebased, fixing a couple of annotations `foo: str = None` --> `foo: Optional[str] = None`
@yp1996 can you merge master
HI @yp1996  - sorry to chase you up, just wanted to ask whether you're still working on this :)
Seems that there's some test failures, but I can't check them as it says "build not found"
Hey @jreback @MarcoGorell, sorry i'd gone mia - i've merged with master but the build is still failing, please feel free to look into it!
> the build is still failing, please feel free to look into it!

For a start, there's some linting issues - did you run `black` against the `.py` files you changed?

Closing as it looks like this has gone stale and @joybhallaa has expressed interest in working on it