Repository: pandas-dev/pandas

Issue:
ENH: DataFrame.to_csv support for "compression='gzip'"
the DataFrame.to_csv method seems to accept a "compression" named parameter:

```
import numpy as np,pandas as pd
data=np.arange(10).reshape(5,2)
df=pd.DataFrame(data,columns=['a','b'])
df.to_csv('test.csv.gz',compression='gzip')
```

However, the file it creates is not compressed at all:
francesco@i3 ~/Desktop $ cat test.csv.gz 
,a,b
0,0,1
1,2,3
2,4,5
3,6,7
4,8,9

How about either (i) actually implementing compression, or at least (ii) raise an error? The current behavior is confusing...



Discussion:
`to_csv` allows `**kwds` so arbitrary additional arguments are 'accepted' (this is mainly for compatibility IIRC with some of the other `to_*` functions which allow this), but ignored. I suppose that could be removed (not sure why it was their in the first place). That said, only arguments in the doc-string are public.

Would accept  a pull-request to limit this.

I think this out-of-scope for Pandas--just use this: https://docs.python.org/2/library/gzip.html

please close

But compression='gzip' is accepted (and enacted) in pd.read_csv, which is why I was assuming to_csv behaves the same.

The way you initially phrased the issue suggested that you were just guessing at keyword arguments--'compression' isn't a [documented](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html?highlight=to_csv#pandas.DataFrame.to_csv) argument so I don't think your confusion is shared by many. You're welcome to submit a pull-request, I don't feel religious about this at all

Sorry, what I meant is:
1. compression is documented and working for read_csv:
   http://pandas.pydata.org/pandas-docs/stable/generated/pandas.io.parsers.read_csv.html
2. compression is not documented and not working for to_csv.

There is no error in the documentation and both (1) and (2) make sense to me.
It's (1) and (2) together, i.e., the fact that to_csv behaves differently from read_csv without telling the user, that seemed a bit inconsistent to me.

Closing on the grounds that I won't be fixing it myself, and probably it's not a proper bug.

Thanks! I definitely didn't mean to antagonize you--agreed that it's an unfortunate inconsistency

Would we want this feature, if someone would implement it? If so, we can leave it open marked as an enhancement proposal?

I would also like to_csv to have the same functionality of from_csv.

+1, a compression argument for `DataFrame.to_csv` would spare many user headaches.

In Python 3.4, I use the following workaround:

``` python
with gzip.open('path_to_file', 'wt') as write_file:
    data_frame.to_csv(write_file)
```

@dhimmel If you're interested in putting in the work, I think we're still open to a PR to add this feature.

@shoyer, okay I will keep this in mind. I have a [bit to learn](https://github.com/pydata/pandas/blob/master/CONTRIBUTING.md) first.


Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -2846,11 +2846,10 @@ def _get_handle(path, mode, encoding=None, compression=None):
 
         if compression == 'gzip':
             import gzip
-            f = gzip.GzipFile(path, 'rb')
+            f = gzip.GzipFile(path, mode)
         elif compression == 'bz2':
             import bz2
-
-            f = bz2.BZ2File(path, 'rb')
+            f = bz2.BZ2File(path, mode)
         else:
             raise ValueError('Unrecognized compression type: %s' %
                              compression)
diff --git a/pandas/core/format.py b/pandas/core/format.py
--- a/pandas/core/format.py
+++ b/pandas/core/format.py
@@ -1259,7 +1259,7 @@ class CSVFormatter(object):
 
     def __init__(self, obj, path_or_buf=None, sep=",", na_rep='', float_format=None,
                  cols=None, header=True, index=True, index_label=None,
-                 mode='w', nanRep=None, encoding=None, quoting=None,
+                 mode='w', nanRep=None, encoding=None, compression=None, quoting=None,
                  line_terminator='\n', chunksize=None, engine=None,
                  tupleize_cols=False, quotechar='"', date_format=None,
                  doublequote=True, escapechar=None, decimal='.'):
@@ -1281,6 +1281,7 @@ def __init__(self, obj, path_or_buf=None, sep=",", na_rep='', float_format=None,
         self.index_label = index_label
         self.mode = mode
         self.encoding = encoding
+        self.compression = compression
 
         if quoting is None:
             quoting = csv.QUOTE_MINIMAL
@@ -1470,7 +1471,8 @@ def save(self):
             close = False
         else:
             f = com._get_handle(self.path_or_buf, self.mode,
-                                encoding=self.encoding)
+                                encoding=self.encoding, 
+                                compression=self.compression)
             close = True
 
         try:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -1210,7 +1210,7 @@ def to_panel(self):
 
     def to_csv(self, path_or_buf=None, sep=",", na_rep='', float_format=None,
                columns=None, header=True, index=True, index_label=None,
-               mode='w', encoding=None, quoting=None,
+               mode='w', encoding=None, compression=None, quoting=None, 
                quotechar='"', line_terminator='\n', chunksize=None,
                tupleize_cols=False, date_format=None, doublequote=True,
                escapechar=None, decimal='.', **kwds):
@@ -1247,6 +1247,10 @@ def to_csv(self, path_or_buf=None, sep=",", na_rep='', float_format=None,
         encoding : string, optional
             A string representing the encoding to use in the output file,
             defaults to 'ascii' on Python 2 and 'utf-8' on Python 3.
+        compression : string, optional
+            a string representing the compression to use in the output file, 
+            allowed values are 'gzip', 'bz2',
+            only used when the first argument is a filename
         line_terminator : string, default '\\n'
             The newline character or character sequence to use in the output
             file
@@ -1275,6 +1279,7 @@ def to_csv(self, path_or_buf=None, sep=",", na_rep='', float_format=None,
         formatter = fmt.CSVFormatter(self, path_or_buf,
                                      line_terminator=line_terminator,
                                      sep=sep, encoding=encoding,
+                                     compression=compression,
                                      quoting=quoting, na_rep=na_rep,
                                      float_format=float_format, cols=columns,
                                      header=header, index=index,


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -7328,6 +7328,63 @@ def test_to_csv_path_is_none(self):
         recons = pd.read_csv(StringIO(csv_str), index_col=0)
         assert_frame_equal(self.frame, recons)
 
+    def test_to_csv_compression_gzip(self):
+        ## GH7615
+        ## use the compression kw in to_csv
+        df = DataFrame([[0.123456, 0.234567, 0.567567],
+                        [12.32112, 123123.2, 321321.2]],
+                       index=['A', 'B'], columns=['X', 'Y', 'Z'])
+
+        with ensure_clean() as filename:
+
+            df.to_csv(filename, compression="gzip")
+
+            # test the round trip - to_csv -> read_csv
+            rs = read_csv(filename, compression="gzip", index_col=0)
+            assert_frame_equal(df, rs)
+
+            # explicitly make sure file is gziped
+            import gzip
+            f = gzip.open(filename, 'rb')
+            text = f.read().decode('utf8')
+            f.close()
+            for col in df.columns:
+                self.assertIn(col, text)
+
+    def test_to_csv_compression_bz2(self):
+        ## GH7615
+        ## use the compression kw in to_csv
+        df = DataFrame([[0.123456, 0.234567, 0.567567],
+                        [12.32112, 123123.2, 321321.2]],
+                       index=['A', 'B'], columns=['X', 'Y', 'Z'])
+
+        with ensure_clean() as filename:
+
+            df.to_csv(filename, compression="bz2")
+
+            # test the round trip - to_csv -> read_csv
+            rs = read_csv(filename, compression="bz2", index_col=0)
+            assert_frame_equal(df, rs)
+
+            # explicitly make sure file is bz2ed
+            import bz2
+            f = bz2.BZ2File(filename, 'rb')
+            text = f.read().decode('utf8')
+            f.close()
+            for col in df.columns:
+                self.assertIn(col, text)
+
+    def test_to_csv_compression_value_error(self):
+        ## GH7615
+        ## use the compression kw in to_csv
+        df = DataFrame([[0.123456, 0.234567, 0.567567],
+                        [12.32112, 123123.2, 321321.2]],
+                       index=['A', 'B'], columns=['X', 'Y', 'Z'])
+
+        with ensure_clean() as filename:
+            # zip compression is not supported and should raise ValueError
+            self.assertRaises(ValueError, df.to_csv, filename, compression="zip")
+
     def test_info(self):
         io = StringIO()
         self.frame.info(buf=io)