Repository: pandas-dev/pandas
Pull request: BUG: to_csv line endings with compression
- [x] closes #25311
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Problem statement:
DataFrame to_csv line_terminator inconsistency when using compression
#### Code Sample, a copy-pastable example if possible

```python
df.to_csv('uncompressed.csv')
df.to_csv('compressed-wrong-line-terminator.csv.gz')
df.to_csv('compressed-good-line-terminator.csv.gz', line_terminator='\n')
```
#### Problem description

Current line_terminator defaults when using compression and when not using compression are different (Windows OS, pandas 0.24.1).

When uncompressing the gzip file created using the default line_terminator, we can clearly see that the files are different (compressed-wrong-line-terminator.csv vs uncompressed.csv); only when using the explicit line_termintor='\n' the uncompressed file is identical to the not compressed file (compressed-good-line-terminator.csv.gz vs. uncompressed.csv).

It is emphasized that if we use the explicit line_terminator='\n' for non-compressed files, the output file is different than the ones created without explicit assignment of the line_terminator - forcing the user the need to explicitly specify the line_terminator only for compressed files.

This behavior is problematic, especially using the latest pandas version, where compression is inferred from the file extension, and one would expect that also the line_separator will undergo the same inference.

#### Expected Output
As stated above, it is expected that the command in python line 2 (after uncompressing it) will produce the same file as produced by the command in python line 1.
However, we see that only the command in python line 3 (after uncompressing it) produces the same file as produced by the command in python line 1.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.1
pytest: None
pip: 19.0.1
setuptools: 40.4.3
Cython: None
numpy: 1.15.2
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.0
openpyxl: 2.5.12
xlrd: 1.2.0
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

</details>

Hints from issue discussion:
Hmm OK. Can you provide code to roundtrip back from the compressed file just so nothing is ambiguous here?
I've the same problem.
Code snippet:
```py
import pandas as pd
import numpy as np
d = pd.DataFrame(np.random.randint(1,10,size=(10,10)), columns=list('qwertyuiop'))
d.to_csv('foo.csv.gz', index=False)
```

The saved files has two line terminators in each line
```csv
q,w,e,r,t,y,u,i,o,p

6,7,9,9,1,7,2,6,9,8

9,9,1,8,2,7,2,5,9,9

4,8,3,8,1,3,9,3,4,1

4,3,8,4,6,6,9,5,2,6

4,2,7,3,3,4,4,7,5,3

8,2,5,8,8,6,9,5,6,3

3,1,8,6,9,7,9,6,8,3

1,6,7,8,6,7,5,3,5,3

8,4,9,4,8,3,5,5,6,2

6,7,8,6,6,2,8,2,3,4

```
The saved file in the example:
[foo.csv.gz](https://github.com/pandas-dev/pandas/files/2889368/foo.csv.gz)

Saving without compression does works as expected.

Setting the line_terminator resolves it

It seems to be limited to Windows, I've tried on Linux and it does not have the same problem, using the same pandas version


<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.8.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.1
pytest: 4.2.0
pip: 19.0.1
setuptools: 40.7.3
Cython: 0.29.4
numpy: 1.15.4
scipy: 1.2.0
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: 1.8.2
patsy: None
dateutil: 2.7.5
pytz: 2018.9
blosc: None
bottleneck: None
tables: 3.4.4
numexpr: 2.6.9
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: 0.9999999
sqlalchemy: 1.2.16
pymysql: None
psycopg2: 2.7.4 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
</details>

This looks like a duplicate of https://github.com/pandas-dev/pandas/issues/25048. LMK if not.
After carefully reading the details of **#25048**, it seems that sed task refers to a scenario where a **file handler** is passed to pandas.to_csv().
However, **in my case**, the call to pandas.to_csv() is with a filename (and **not a file handler**), and behave different when giving a filename that ends with .gz (inferring a request for a compressed file).
I believe we are talking about **two different problems** here (unless proven that they originate from the same *bug* and that fixing one fixes the other too).
I believe that we create a file handler internally when we detect a
compression scheme, and so go down the same path as the other issue. Is
that right?

On Wed, Mar 6, 2019 at 9:52 AM jointfull <notifications@github.com> wrote:

> After carefully reading the details of *#25048
> <https://github.com/pandas-dev/pandas/issues/25048>*, it seems that sed
> task refers to a scenario where a *file handler* is passed to
> pandas.to_csv().
> However, *in my case*, the call to pandas.to_csv() is with a filename
> (and *not a file handler*), and behave different when giving a filename
> that ends with .gz (inferring a request for a compressed file).
> I believe we are talking about *two different problems* here (unless
> proven that they originate from the same *bug* and that fixing one fixes
> the other too).
>
> —
> You are receiving this because you modified the open/close state.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/25311#issuecomment-470160705>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIqumnI1b7cUyMC7kgthzGp60Lylrks5vT-RagaJpZM4a6gZw>
> .
>

Though, perhaps we can work around for this specific case, by supplying the line_terminator for the user if necessary? cc @gfyoung @chris-b1.
I am pretty certain that these two issues are tied to the same underlying issue, so let's see what comes of #25048 and then return to this one if need be.

Commit message: fixup

Patch to review:
diff --git a/doc/source/user_guide/io.rst b/doc/source/user_guide/io.rst
index b23a0f10e9e2b..1b5d96fa9c146 100644
--- a/doc/source/user_guide/io.rst
+++ b/doc/source/user_guide/io.rst
@@ -1689,7 +1689,7 @@ The ``Series`` and ``DataFrame`` objects have an instance method ``to_csv`` whic
 allows storing the contents of the object as a comma-separated-values file. The
 function takes a number of arguments. Only the first is required.
 
-* ``path_or_buf``: A string path to the file to write or a StringIO
+* ``path_or_buf``: A string path to the file to write or a file object.  If a file object it must be opened with `newline=''`
 * ``sep`` : Field delimiter for the output file (default ",")
 * ``na_rep``: A string representation of a missing value (default '')
 * ``float_format``: Format string for floating point numbers
@@ -1702,7 +1702,7 @@ function takes a number of arguments. Only the first is required.
 * ``mode`` : Python write mode, default 'w'
 * ``encoding``: a string representing the encoding to use if the contents are
   non-ASCII, for Python versions prior to 3
-* ``line_terminator``: Character sequence denoting line end (default '\\n')
+* ``line_terminator``: Character sequence denoting line end (default `os.linesep`)
 * ``quoting``: Set quoting rules as in csv module (default csv.QUOTE_MINIMAL). Note that if you have set a `float_format` then floats are converted to strings and csv.QUOTE_NONNUMERIC will treat them as non-numeric
 * ``quotechar``: Character used to quote fields (default '"')
 * ``doublequote``: Control quoting of ``quotechar`` in fields (default True)
diff --git a/doc/source/whatsnew/v0.24.2.rst b/doc/source/whatsnew/v0.24.2.rst
index 2c6d1e01ed89b..0f603515c61cc 100644
--- a/doc/source/whatsnew/v0.24.2.rst
+++ b/doc/source/whatsnew/v0.24.2.rst
@@ -32,6 +32,7 @@ Fixed Regressions
 - Fixed regression in creating a period-dtype array from a read-only NumPy array of period objects. (:issue:`25403`)
 - Fixed regression in :class:`Categorical`, where constructing it from a categorical ``Series`` and an explicit ``categories=`` that differed from that in the ``Series`` created an invalid object which could trigger segfaults. (:issue:`25318`)
 - Fixed pip installing from source into an environment without NumPy (:issue:`25193`)
+- Fixed regression in :meth:`DataFrame.to_csv` writing duplicate line endings with gzip compress (:issue:`25311`)
 
 .. _whatsnew_0242.enhancements:
 
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index 0b81576404e2f..a37b745d3082a 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -2920,7 +2920,8 @@ def to_csv(self, path_or_buf=None, sep=",", na_rep='', float_format=None,
         ----------
         path_or_buf : str or file handle, default None
             File path or object, if None is provided the result is returned as
-            a string.
+            a string.  If a file object is passed it should be opened with
+            `newline=''`, disabling universal newlines.
 
             .. versionchanged:: 0.24.0
 
diff --git a/pandas/io/common.py b/pandas/io/common.py
index ad054d77b3bc8..c1cacf39c5b08 100644
--- a/pandas/io/common.py
+++ b/pandas/io/common.py
@@ -434,7 +434,7 @@ def _get_handle(path_or_buf, mode, encoding=None, compression=None,
     if (compat.PY3 and is_text and
             (compression or isinstance(f, need_text_wrapping))):
         from io import TextIOWrapper
-        f = TextIOWrapper(f, encoding=encoding)
+        f = TextIOWrapper(f, encoding=encoding, newline='')
         handles.append(f)
 
     if memory_map and hasattr(f, 'fileno'):
diff --git a/pandas/tests/frame/test_to_csv.py b/pandas/tests/frame/test_to_csv.py
index 54a8712a9c645..a5765d82f276a 100644
--- a/pandas/tests/frame/test_to_csv.py
+++ b/pandas/tests/frame/test_to_csv.py
@@ -4,6 +4,7 @@
 
 import csv
 import os
+import gzip
 
 import numpy as np
 import pytest
@@ -1221,3 +1222,14 @@ def test_multi_index_header(self):
                          '1,5,6,7,8']
         expected = tm.convert_rows_list_to_csv_str(expected_rows)
         assert result == expected
+
+    def test_gz_lineend(self):
+        # GH 25311
+        df = pd.DataFrame({'a': [1, 2]})
+        expected_rows = ['a', '1', '2']
+        expected = tm.convert_rows_list_to_csv_str(expected_rows)
+        with ensure_clean('__test_gz_lineend.csv.gz') as path:
+            df.to_csv(path, index=False)
+            result = gzip.open(path, mode='rt', newline='').read()
+
+        assert result == expected


Code review comments:
diff_hunk: @@ -1221,3 +1222,14 @@ def test_multi_index_header(self):
                          '1,5,6,7,8']
         expected = tm.convert_rows_list_to_csv_str(expected_rows)
         assert result == expected
+
+    def test_gz_lineend(self):
+        # GH 25311
+        df = pd.DataFrame({'a': [1, 2]})
+        expected_rows = ['a', '1', '2']
+        expected = tm.convert_rows_list_to_csv_str(expected_rows)
+        with ensure_clean('__test_gz_lineend.csv.gz') as path:
+            df.to_csv(path, index=False)
+            result = gzip.open(path, mode='rt', newline='').read()
path: pandas/tests/frame/test_to_csv.py
text: @user1:
this needs a with to close it

@user1:
but better to use pandas.util.testing.decompress_file anyhow