ENH: Allow compression in NDFrame.to_csv to be a dict with optional arguments (#26023)
State: closed

- [x] closes #26023 
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv

Comments:
I am -1 on adding another parameter here. What makes Zip extraction different than the other compressions methods?
Other compression methods apply to a single file and add an extension to the name (i.e. `data.csv.gz` extracted results in `data.csv`), but ZIP archives are meant to have each file within have a name. As is, calling `df.to_csv('data.zip')` creates an archive containing a CSV file named `data.zip` as well.  

An alternative would be to infer the file name, but that would be a breaking change.
So I understand the use-case.  However, I do agree that adding another parameter is not necessarily the best way to do this.  This parameter is really related to our `compression` parameter.

This leads me to think that a `dict` might make more sense.  It allows us to configure the compression without the bloat.  What do you think?

cc @jreback (related to https://github.com/pandas-dev/pandas/pull/25990#issuecomment-480669922)
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=h1) Report
> Merging [#26024](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b90f9db6d23f64b0ce9ff469445e1c31013a07c0?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `92.59%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26024/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26024      +/-   ##
==========================================
- Coverage   91.96%   91.95%   -0.01%     
==========================================
  Files         175      175              
  Lines       52405    52425      +20     
==========================================
+ Hits        48193    48207      +14     
- Misses       4212     4218       +6
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.51% <92.59%> (-0.01%)` | :arrow_down: |
| #single | `40.72% <37.03%> (-0.14%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `93.54% <100%> (ø)` | :arrow_up: |
| [pandas/io/formats/csvs.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvY3N2cy5weQ==) | `98.23% <100%> (+0.03%)` | :arrow_up: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `91.5% <90.9%> (-0.33%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `75% <0%> (-12.5%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.9% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=footer). Last update [b90f9db...5853a28](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@gfyoung I agree that this approach makes more sense - I'll modify the functions to optionally take a `dict` for `compression`, i.e. `{'compression': 'zip', 'arcname': 'data.csv'}`. Thanks for the feedback!
@drew-heenan : Sounds good!  One minor thing: let's use `method` as the key instead of `compression`.  Otherwise, we'll be extracting `compression["compression"]`, which is a little redundant.
@WillAyd I've added type annotations to parameters which I've changed, but it seems that doing so caused the typing validation check to fail on other parts of the files which I have not modified. 
Yea I think a general rename would make things clearer

Sent from my iPhone

> On Apr 13, 2019, at 6:24 PM, Drew Heenan <notifications@github.com> wrote:
> 
> @drew-heenan commented on this pull request.
> 
> In pandas/io/common.py:
> 
> >              if is_path:
>                  f = bz2.BZ2File(path_or_buf, mode)
>              else:
>                  f = bz2.BZ2File(path_or_buf)
>  
>          # ZIP Compression
> -        elif compression == 'zip':
> -            zf = BytesZipFile(path_or_buf, mode)
> +        elif compression_method == 'zip':
> +            arcname = None
> @WillAyd The specific instance of arcname above is no longer there; Though, do you mean change arcname to archive_name in general, including the dict key?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=h1) Report
> Merging [#26024](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c104a0c9ce4c1dd0d63df121f0fa50657b6d6447?src=pr&el=desc) will **decrease** coverage by `0.91%`.
> The diff coverage is `96.29%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26024/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26024      +/-   ##
==========================================
- Coverage   92.91%   91.99%   -0.92%     
==========================================
  Files         184      180       -4     
  Lines       50385    50793     +408     
==========================================
- Hits        46813    46726      -87     
- Misses       3572     4067     +495
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.63% <96.29%> (-0.95%)` | :arrow_down: |
| #single | `41.83% <48.14%> (-0.59%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `94.2% <100%> (-0.07%)` | :arrow_down: |
| [pandas/io/formats/csvs.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvY3N2cy5weQ==) | `98.22% <100%> (+0.04%)` | :arrow_up: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `94.52% <95.45%> (+0.47%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/io/excel/\_openpyxl.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsL19vcGVucHl4bC5weQ==) | `84.71% <0%> (-3.23%)` | :arrow_down: |
| [pandas/util/\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX2RlY29yYXRvcnMucHk=) | `92.13% <0%> (-3.22%)` | :arrow_down: |
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `93.48% <0%> (-1.14%)` | :arrow_down: |
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `94.38% <0%> (-0.93%)` | :arrow_down: |
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `95.95% <0%> (-0.84%)` | :arrow_down: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `85.05% <0%> (-0.74%)` | :arrow_down: |
| ... and [164 more](https://codecov.io/gh/pandas-dev/pandas/pull/26024/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=footer). Last update [c104a0c...779511e](https://codecov.io/gh/pandas-dev/pandas/pull/26024?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

can u merge master
can you merge master and update to comments
@drew-heenan can you update

@pandas-dev/pandas-core if someone can merge master
Merged master to keep updated. @drew-heenan let us know if can continue with comments!
@jreback comments addressed
@jreback @WillAyd do you recall how close this was to being finished? Probably just need to fix merge conflicts from black?
@TomAugspurger : This was pretty close (especially since the comments later reduced to those regarding annotations).  I think if we fix merge conflicts and blacken this, we should be good.
I can fix up today
Hello @drew-heenan! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-08-25 20:17:59 UTC
@WillAyd : Do you plan on annotating on affected methods in this PR?  I see that some of the inputs to some of the affected methods are annotated, but not all of them.
I can take a look
OK @gfyoung added annotations where I think feasible. Two blockers prevented full annotations of modified funcs, particularly _get_handle:

  1. An actual bug in type shed (see https://github.com/python/typeshed/issues/3124)
  2. Variable reuse (see `f` in `_get_handle`); this requires a refactor that I think expands the diff a little much so better done as follow up


Merged again to keep fresh. There's a Mypy failure I'll have to look at later.

@TomAugspurger @gfyoung would you still have time to review this one coming up? Have a lot in the queue so just want to prioritize workflow with you
It looks pretty good overall!
Thanks @drew-heenan