ENH: added compression kw to to_csv GH7615
State: closed

This closes #7615 and represents work that started in #2636.


Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv

Comments:
I need help from pandas people (@jreback ?)
I get this single error on Travis-CI:

```
Traceback (most recent call last):

  File "/home/travis/build/pydata/pandas/pandas/tests/test_frame.py", line 7364, in test_to_csv_compression

    df.to_csv(filename, compression="gzip")

  File "/home/travis/build/pydata/pandas/pandas/core/frame.py", line 1294, in to_csv

    formatter.save()

  File "/home/travis/build/pydata/pandas/pandas/core/format.py", line 1460, in save

    self._save()

  File "/home/travis/build/pydata/pandas/pandas/core/format.py", line 1546, in _save

    self._save_header()

  File "/home/travis/build/pydata/pandas/pandas/core/format.py", line 1542, in _save_header

    writer.writerow(encoded_labels)

io.UnsupportedOperation: not writable
```

I assume "not writable" means that the file I'm trying to write to, which I got using `ensure_clean()`, is not writable. I don't understand why, I copied the test code from `test_to_csv_float_format`.

and does it work locally for you?

I couldn't build (`python setup.py develop`) pandas on my windows pc or on a digitalocean ubuntu droplet - does pandas have a docker image you can recommend?
I'll try again, but please have a look at [the travis build](https://travis-ci.org/pydata/pandas/builds/83293956) - some builds passed, does that make sense?

pls read the contributing docs [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

creation of the proper build env is pretty easy using conda (and even local tools on linux, but windows we recommend conda as its dead simple). docker is much too heavyweight for most things iMHO.

pls build and debug locally first.

I had problems building pandas (`python setup.py develop`), but I managed to locally build pandas, see below.

**Test problem and solution**
The test error was caused because the compressed file was opened as read-only. I had to change the `mode` argument used to create compressed file handles in `pandas.core.common._get_handle()`. The function has a mode arg, but it was overriding it with `rb` for compressed files. 
Anyway, thanks for helping me out with this!!

**Build problem and solution**
Building on my Windows laptop, I had other versions of `gcc.exe` installed. Calling `which gcc` should return `C:\Anaconda\envs\pandas_dev\MinGW\bin\gcc.exe` (depending on the path to Anaconda). 
On my system I had to manually add `C:\Anaconda\envs\pandas_dev\MinGW\bin` to the begining of `PATH`:

```
set PATH=C:\Anaconda\envs\pandas_dev\MinGW\bin;%PATH%
```

and if you already tried `python setup.py develop` once, then you probably need to do `python setup.py clean` before trying again.

pls add a release note in 0.17.1 (enhancements section), and squash.

Sure thing. I've never 'squashed' before - any comments/instructions beyond the `contributing.html` doc?

nope, that says it all

Fixed following code review (d3f63d8).
Also added some test code to explicitly make sure that the file was compressed, rather than just using the `read_csv`, in case `to_csv` and `read_csv` both "cheat" but in the same way (such as ignoring the `compression` flag).

OK this was fun, I learned a lot. Thanks.

looks good. pls add a whatsnew note in enhancements for 0.17.1

Right. I did it but it got lost when I squashed. I'll do it again and squash properly.

Anything else comes to mind?

thank you sir!

Thank you, I've enjoyed the process and learned much.