Repository: pandas-dev/pandas
Pull request: EHN: to_csv compression accepts file-like object
- [x] closes #21227 
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Handle an unsupported case when a file-like object instead of path passed into to_csv with compression. According to documentation, compression keyword requires it to be a filename.

At the moment, when a handle is passed, it appears to be uncompressed.

Tentative enhancement.

Problem statement:
df.to_csv ignores compression when provided with a file handle
#### Code at issue

```python
with open(outputfile, "w") as file_handle:
    df2.to_csv(file_handle, compression='gzip')
```
#### Problem description
The written file is not gzip compressed.

#### Expected Output
The output should be a gzip compressed csv file. Similar to what is obtained when using:
```python
df2.to_csv('/path/to/file.csv.gz',compression='gzip')
```

Hints from issue discussion:
Maybe related to #21144
```
>>> import os
>>> import pandas as pd
>>> from pandas import *
>>>
>>> pd.__version__
'0.20.3'
>>>
>>> df = DataFrame(100 * [[123, 234, 435]])
>>>
>>> with open('test_compressed', 'w') as fh:
...     df.to_csv(fh, compression='gzip')
...
>>> fh_size = os.path.getsize('test_compressed')
>>> df.to_csv('test_compressed', compression='gzip')
>>> f_size = os.path.getsize('test_compressed')
>>>
>>> os.remove('test_compressed')
>>> assert fh_size == f_size
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError
```

looks like an existing behaviour dating back to version 0.20 or earlier.

Actually, the documentation of ``compression=`` says: 
> compression : string, optional
> 
> A string representing the compression to use in the output file. Allowed values are ‘gzip’, ‘bz2’, ‘zip’, ‘xz’. **This input is only used when the first argument is a filename.**

so it is not supported but may be a new use case perhaps?

Commit message: update docs

Patch to review:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index 22677b19192e1..0899e9cd87aba 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -1689,8 +1689,7 @@ def to_csv(self, path_or_buf=None, sep=",", na_rep='', float_format=None,
             defaults to 'ascii' on Python 2 and 'utf-8' on Python 3.
         compression : string, optional
             A string representing the compression to use in the output file.
-            Allowed values are 'gzip', 'bz2', 'zip', 'xz'. This input is only
-            used when the first argument is a filename.
+            Allowed values are 'gzip', 'bz2', 'zip', 'xz'.
         line_terminator : string, default ``'\n'``
             The newline character or character sequence to use in the output
             file
diff --git a/pandas/core/series.py b/pandas/core/series.py
index c9329e8b9e572..f25f73513df30 100644
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -3761,8 +3761,7 @@ def to_csv(self, path=None, index=True, sep=",", na_rep='',
             non-ascii, for python versions prior to 3
         compression : string, optional
             A string representing the compression to use in the output file.
-            Allowed values are 'gzip', 'bz2', 'zip', 'xz'. This input is only
-            used when the first argument is a filename.
+            Allowed values are 'gzip', 'bz2', 'zip', 'xz'.
         date_format: string, default None
             Format string for datetime objects.
         decimal: string, default '.'
diff --git a/pandas/io/formats/csvs.py b/pandas/io/formats/csvs.py
index 29b8d29af0808..0be2a180fbfa2 100644
--- a/pandas/io/formats/csvs.py
+++ b/pandas/io/formats/csvs.py
@@ -154,9 +154,9 @@ def save(self):
             # GH 17778 handles compression for byte strings.
             if not close and self.compression:
                 f.close()
-                with open(self.path_or_buf, 'r') as f:
+                with open(f.name, 'r') as f:
                     data = f.read()
-                f, handles = _get_handle(self.path_or_buf, self.mode,
+                f, handles = _get_handle(f.name, self.mode,
                                          encoding=encoding,
                                          compression=self.compression)
                 f.write(data)
diff --git a/pandas/tests/test_common.py b/pandas/tests/test_common.py
index bb7ee1b911fee..1470781d26556 100644
--- a/pandas/tests/test_common.py
+++ b/pandas/tests/test_common.py
@@ -241,3 +241,24 @@ def test_compression_size(obj, method, compression):
         getattr(obj, method)(filename, compression=None)
         uncompressed = os.path.getsize(filename)
         assert uncompressed > compressed
+
+
+@pytest.mark.parametrize('obj', [
+    DataFrame(100 * [[0.123456, 0.234567, 0.567567],
+                     [12.32112, 123123.2, 321321.2]],
+              columns=['X', 'Y', 'Z']),
+    Series(100 * [0.123456, 0.234567, 0.567567], name='X')])
+@pytest.mark.parametrize('method', ['to_csv'])
+def test_compression_size_fh(obj, method, compression):
+    if not compression:
+        pytest.skip("only test compression case.")
+
+    with tm.ensure_clean() as filename:
+        with open(filename, 'w') as fh:
+            getattr(obj, method)(fh, compression=compression)
+        compressed = os.path.getsize(filename)
+    with tm.ensure_clean() as filename:
+        with open(filename, 'w') as fh:
+            getattr(obj, method)(fh, compression=None)
+        uncompressed = os.path.getsize(filename)
+        assert uncompressed > compressed


Code review comments:
diff_hunk: @@ -241,3 +241,24 @@ def test_compression_size(obj, method, compression):
         getattr(obj, method)(filename, compression=None)
         uncompressed = os.path.getsize(filename)
         assert uncompressed > compressed
+
+
+@pytest.mark.parametrize('obj', [
+    DataFrame(100 * [[0.123456, 0.234567, 0.567567],
+                     [12.32112, 123123.2, 321321.2]],
+              columns=['X', 'Y', 'Z']),
+    Series(100 * [0.123456, 0.234567, 0.567567], name='X')])
+@pytest.mark.parametrize('method', ['to_csv'])
+def test_compression_size_fh(obj, method, compression):
path: pandas/tests/test_common.py
text: @user1:
can you make a new fixture alongside the existing to just have the compression options (exclude None), e.g. maybe compression_only (and can change the above test to use as well)

@author:
yeah sure.