add compression support for 'read_pickle' and 'to_pickle'
State: closed

closes #11666 
My code is not pythonic enough, maybe need some refactor. Any comment is welcome.


Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
tests!

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/13317?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@a1d3ff3`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `100%`.


```diff
@@            Coverage Diff            @@
##             master   #13317   +/-   ##
=========================================
  Coverage          ?      91%           
=========================================
  Files             ?      143           
  Lines             ?    49312           
  Branches          ?        0           
=========================================
  Hits              ?    44874           
  Misses            ?     4438           
  Partials          ?        0
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/13317?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/pickle.py](https://codecov.io/gh/pandas-dev/pandas/compare/a1d3ff3e3ec407915adb9d37107cd64a2028dd76...332b462a36b1babd25917eea2324e7942166d75c?src=pr&el=tree#diff-cGFuZGFzL2lvL3BpY2tsZS5weQ==) | `79.54% <100%> (ø)` | |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/compare/a1d3ff3e3ec407915adb9d37107cd64a2028dd76...332b462a36b1babd25917eea2324e7942166d75c?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.25% <100%> (ø)` | |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/compare/a1d3ff3e3ec407915adb9d37107cd64a2028dd76...332b462a36b1babd25917eea2324e7942166d75c?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `70.25% <100%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/13317?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/13317?src=pr&el=footer). Last update [a1d3ff3...e9c5fd2](https://codecov.io/gh/pandas-dev/pandas/compare/a1d3ff3e3ec407915adb9d37107cd64a2028dd76...e9c5fd251331107f61d8e53119874151dd28f8f2?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
code looks ok, needs quite a few tests though. see [here](https://github.com/pydata/pandas/blob/master/pandas/io/tests/parser/compression.py) for what to test (and you can model them pretty much like this).

OK, I added the test code in test_pickle.py. I will move them to compression.py tomorrow.

@goldenbull no, this should be _separate_ from the other `compression.py`, it just needs to be a test class in `test_pickle.py`. Just model after that. Its too complicated / not necessary to combine the tests.

tests added.

can you rebase / update

Rebased onto master branch. It's been a long time since my last commit, my memory is fading 🤕 

@goldenbull something went wrong with your rebase. Normally doing:

```
git fetch upstream
git checkout pickle_io_compression
git rebase upstream/master
```

should solve this

Actually I didn't use `rebase` because there are a lot of conflicts during rebasing. Instead I switched to origin/master and `git merge pickle_io_compression`, and then used `push -f` to overwrite my remote branch by force. I know `push -f` is a discouraged command but I think it works for this case.
So what is the issue you find? Maybe it's the problem about `push -f`?

There is nothing perse wrong with `push -f` (you would have to do that as well after the rebase I showed), but the problem is that there is now a huge diff here in this PR (and many included commits), which makes it impossible to review (just take a look at the "Files changed" tab on github). 
If the rebase is too painful, and you have only a few commits, another possibility is to cherry-pick the few commits on a clean branch from current master, so you only have to solve the merge conflicts once.

o yes, I understand now. Rebase is necessary for code review 😄 

@goldenbull Can you update based on the review comments?
@jorisvandenbossche sorry, so busy these days. Let me update this week.
This task becomes easier because `_get_handle` is already provided 😄 
Please comment on my new commit
For the failing test, you probably have to add `tm._skip_if_no_lzma()` for that extension that lzma has to be used. See eg https://github.com/pandas-dev/pandas/pull/14576/files#diff-af728bb1f226f19b5aff0eca51e9b57aR51
@dhimmel as you looked recently to the compression code, always welcome to review this PR!
https://github.com/pandas-dev/pandas/blob/master/pandas/io/tests/generate_legacy_storage_files.py

generates pickle files - we already have a ton that we test

can certainly add compressed ones

can you rebase / update
@goldenbull just want to encourage you on this pull request. Just realized the current limits that there is no way to do compressed pickle IO in pandas.

This would be a huge enhancement for https://github.com/cognoma/machine-learning where reading TSVs into pandas is slow, causing difficulty for our contributors. Pickles load almost instantaneously, but are too big uncompressed.
@dhimmel btw, nothing stopping anyone from picking this up! its actually almost all of the way there!
@jreback just for clarification, are there still comments that have to be addressed, or is it just rebasing? (I see commits after the last round of comment, but long time ago that I looked at the PR). 
yeah i don't remember 
i think it was pretty close just needed some more testing / doc updates 
I see that all tests are now moved to `pandas\tests` folder and test scripts are changed a lot. How shall I write the tests? Is there anything I should pay attention?
Btw, I got two errors when running nosetests on `pandas\tests\io\test_pickle.py` :
```
======================================================================
ERROR: pandas.tests.io.test_pickle.test_pickles
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
TypeError: test_pickles() missing 2 required positional arguments: 'current_pickle_data' and 'version'

======================================================================
ERROR: pandas.tests.io.test_pickle.test_round_trip_current
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
TypeError: test_round_trip_current() missing 1 required positional argument: 'current_pickle_data'

```
@goldenbull

- make sure you are rebased on master. Some things have moved around
- install and use ``pytest`` which we switched
- you'll want to mimic tests like this: https://github.com/pandas-dev/pandas/blob/master/pandas/tests/io/parser/compression.py; these are for the parser but you are using the same underlying routines; these should be separate and in ``tests\io\test_pickle.py``
merged the latest master branch.
parameterized style is easier to read. 👍 
@goldenbull looks good!

can you add a whatsnew sub-section (pickle has gained compression support or something like this).

show an example of writing / reading (you will need to use ``:suppress:`` as this will create a file which you need to remove at the end.

IOW, something like

```
.. ipython:: python
   
   fn = 'compressed.pkl'

   df = ....
   df.to_pickle(fn, compression='gzip')
   pd.read_pickle(fn)

.. ipython:: python
   :suppress:
   import os
   os.remove(fn)
```
also have a look at the io.rst docs on pickle. maybe worth adding a small section on using compression (see how we did this for read_csv, and copy the same formatting)
thanks @goldenbull !

note that I changed the tests (put in to a class, which you can do with pytest as long as it inherits from ``object``, NOT ``tm.TestCase``). and the docs a bit: https://github.com/pandas-dev/pandas/commit/5667a3ad0489815c1239cba785300952c9799000