Write pickle to file-like without intermediate in-memory buffer
State: closed

Before this change, calling `pickle.dumps()` created an in-memory byte buffer, negating the advantage
of zero-copy pickle protocol 5. After this change, `pickle.dump` writes directly to open file(-like),
cutting peak memory in half in most cases.

Profiling was done with pandas@master and python 3.8.5

Related issues:
- https://github.com/pandas-dev/pandas/issues/34244

![image](https://user-images.githubusercontent.com/25141514/95683850-9929c580-0be5-11eb-8ab1-b9e43407c5bb.png)

## Update: ASV results
`$ asv continuous -f 1.1 origin/master HEAD -b pickle` yields:
```
       before           after         ratio
     [58f74686]       [3c1ee270]
                      <zero-copy-pickle>
-           91.9M            80.1M     0.87  io.pickle.Pickle.peakmem_write_pickle

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```


- [ ] closes #34244
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: IO Pickle
description: read_pickle, to_pickle

Comments:
also to actually close this issue i think would be good to add some tests that explicity set the pickle protocal to 5
> also to actually close this issue i think would be good to add some tests that explicity set the pickle protocal to 5

Isn't that covered by py38 tests where pickle.HIGHEST_PROTOCOL (==5) is used by default? 
> > also to actually close this issue i think would be good to add some tests that explicity set the pickle protocal to 5
> 
> Isn't that covered by py38 tests where pickle.HIGHEST_PROTOCOL (==5) is used by default?

probably but let's make it explict as well (e.g. parameterize on 4,5,highest when PY38 or greater)
Added ASV peakmem_ benchmark. The benchmark currently uses an ~11MB dataframe - I have no experience with ASV to tell how reliable memory benchmarking is for small-ish objects, any advice appreciated.
> Added ASV peakmem_ benchmark. The benchmark currently uses an ~11MB dataframe - I have no experience with ASV to tell how reliable memory benchmarking is for small-ish objects, any advice appreciated.

it *should* show a diff here in any event. what kind of results are you seeing?
> > Added ASV peakmem_ benchmark. The benchmark currently uses an ~11MB dataframe - I have no experience with ASV to tell how reliable memory benchmarking is for small-ish objects, any advice appreciated.
> 
> it _should_ show a diff here in any event. what kind of results are you seeing?

Yep, added results above - `peakmem_write_pickle` shows the 11MB drop nicely - in practice the relative gain is obv much more significant than shown for larger dataframes.
Hello @ig248! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-12 22:51:15 UTC
lgtm @ig248 just resolved the conflict and can merge on green.
thanks @ig248 very nice!