support binary file handles in to_csv
State: closed

- [x] fixes #19827, fixes #35058, fixes #23854 *, fixes #13068 *, and fixes #22555
- [x] 3 tests added
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The first commit addresses https://github.com/pandas-dev/pandas/issues/35058#issuecomment-653844610: python's `open` cannot take an `encoding` argument when `mode` contains a `'b'` (opened in binary mode). This avoids an error when executing `df.to_csv("output.csv", mode="w+b")`.

The second commit fixes #35058, #19827,  #23854 *, and #13068 *: `to_csv` supports file handles in binary mode if `mode` contains a `b` and it honors `encoding`. `to_csv` re-invented a lot that was already done in `get_handle`. Let `get_handle` do the heavy lifting and remove all special cases from `to_csv`.

The third commit fixes #22555: some compression algorithms did not set the mode to be writeable for file handles. Together with the re-factoring in the second commit, it is now possible to write to binary file handles with compression!

*requesting an encoding for a non-binary file handles through `to_csv` still doesn't work but imho also doesn't make sense: specify the encoding yourself when opening the file or use a binary file handle

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO CSV
description: read_csv, to_csv

Comments:
thank you @jreback! The first commit is technically a bug fix as it prevents an error from occurring (`df.to_csv("output.csv", mode="w+b")` - to be fair there is no good reason for using binary mode when a filename is passed) but the second commit is a new feature as it allows to use binary file handles with `to_csv`. I will add two entries to `whatsnew`.

I think that this PR should also fix #23854 @eode, #19827 @colobas, and #13068 @graingert (as long as the user puts a `'b'` in `mode`).
> thank you @jreback! The first commit is technically a bug fix as it prevents an error from occurring (`df.to_csv("output.csv", mode="w+b")` - to be fair there is no good reason for using binary mode when a filename is passed) but the second commit is a new feature as it allows to use binary file handles with `to_csv`. I will add two entries to `whatsnew`.
> 
> I think that this PR should also fix #23854 @eode, #19827 @colobas, and #13068 @graingert (as long as the user puts a `'b'` in `mode`).

awesome! can you have tests for each of these issues, edit the top of the PR to say you close, and add to the whatsnew?
This PR looks good!  Let's add the tests that @jreback requested, and I think we should be good.
#19827: my issue #35058 is actually a duplicate of #19827 (I had a different use case but it is the same underlying technical issue). His code example is already covered in the test. I referenced his issue in the test.

#23854 and #13068: this PR will fix 'half' of both issues. If the file handle is in binary mode it will fix their issue (honor the encoding) but for non-binary file handles the issues are still not addressed. I added a test-case containing sample code from these two issues to validate the encoding for binary file handles.

~~I could try to extend this PR (or open a new one?) to cover the encoding of non-binary file handles as well (using the StringIO buffer approach for all file handles should fix this).~~ I don't think this is possible (`string.encode()` returns bytes but a non-binary file handle wants a string). People who provide a file handle and want their encoding to be honored should provide a binary file handle (or set the encoding themselves in the `open` call when creating a non-binary file handle). I changed the `to_csv` documentation to note that `encoding` isn't supported for non-binary file handles. I wouldn't mind to also mark #23854 and #13068 as fixed (one half is fixed and the other half is 'not supported').
Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-07 00:24:21 UTC
I overhauled a part of `to_csv` that re-invented parts of `get_handle` (my initial PR made use of these re-inventions). The code is smaller and simpler :)
~~There seems to be an unclosed file handle on [windows causing a test failure](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=39443&view=logs&j=ea1e4a48-f1c1-5933-c4f7-7e56de389fce&t=8514fc12-a1cb-5592-2cd7-a350b714e992&l=84). I cannot reproduce this error on my linux machine. @jreback @gfyoung do you have an idea how to debug that? edit: it seems that at least one merged PR triggers the [same failure](https://travis-ci.org/github/pandas-dev/pandas/jobs/709289923).~~ edit: seemed to be an CI hiccup, it passes now :)

[This test](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=39443&view=logs&j=3a03f79d-0b41-5610-1aa4-b4a014d0bc70&t=4d05ed0e-1ed3-5bff-dd63-1e957f2766a9&l=27) has some unclosed file warnings, but other PRs have the same warnings (so hopefully not related to this PR).

~~Except the above test failure,~~ I think/hope this PR is done :)
rebased and 1.1 -> 1.2

@jreback @gfyoung
@twoertwein : It looks like there are test failures unrelated to your PR.  You may need to rebase or merge `master` again.
@gfyoung rebased. Master seems to have the same issue, all other tests pass :)
@twoertwein : The changes look okay to me right now.  Unfortunately, we will need to wait until the `master` failure is fixed.
@gfyoung if any other comments.
@jreback ping
thanks @twoertwein very nice!
@jreback  and @gfyoung  Thank you very much for your help getting this PR merged :)