Repository: pandas-dev/pandas
Pull request: BUG: encoding error in to_csv compression
- [x] closes #21241 
closes #21118
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Fix a problem where encoding wasn't handled properly in to_csv compression in Python 3. It was caused by dumping uncompressed csv on disk and reading it back to memory without passing specified encoding. Then platform tried to decode using default locale encoding which may or may not succeed. 

This PR added tests for non-ascii data with csv compression. Also, by using string buffer, this PR removed repeated disk roundtrip and redundant encoding/decoding which caused UnicodeDecodeError. There is performance improvement compared to 0.22 and 0.23. It also supports file-like object as input path_or_buf. 

Before this PR:
```
>>> df = DataFrame(100 * [[123, "abc", u"样本1", u"样本2"]], columns=['A', 'B', 'C', 'D'])
>>>
>>> def test_to_csv(df):
...     df.to_csv(
...         path_or_buf='test',
...         encoding='utf8',
...         compression='zip',
...         quoting=1,
...         sep='\t',
...         index=False)
...
>>> timeit(lambda: test_to_csv(df), number=5000)
11.856349980007508
```
After this PR:

```
>>> df = DataFrame(100 * [[123, "abc", u"样本1", u"样本2"]], columns=['A', 'B', 'C', 'D'])
>>>
>>> def test_to_csv(df):
...     df.to_csv(
...         path_or_buf='test',
...         encoding='utf8',
...         compression='zip',
...         quoting=1,
...         sep='\t',
...         index=False)
...
>>> timeit(lambda: test_to_csv(df), number=5000)
5.459916951993364
```

Problem statement:
to_csv failing with encoding='utf-16'
#### Code Sample:

```python
df.to_csv('test.gz', sep='~',  header=False, index=False,compression='gzip',line_terminator='\r\n',encoding='utf-16', na_rep='')

```
/opt/anaconda/lib/python3.6/encodings/ascii.py in decode(self, input, final)
     24 class IncrementalDecoder(codecs.IncrementalDecoder):
     25     def decode(self, input, final=False):
---> 26         return codecs.ascii_decode(input, self.errors)[0]
     27 
     28 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)


#### Problem description

In first place, big thank you for supporting pandas, my life is easier and fun with pandas in the toolkit.
In previous version 0.22 we were able to do _to_csv_ with encoding='utf-16' to handle Japanese, Chinese among other content properly. Need the utf-16 encoding for next steps like upload data to MSSQL server in bulk mode.

I would like to know if I can use a workaround to continue have the support of uft-16.

Any other suggestions are welcome.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.114-42-default
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: POSIX
LOCALE: None.None

pandas: 0.23.0
pytest: 3.5.1
pip: 10.0.1
setuptools: 39.1.0
Cython: 0.28.2
numpy: 1.14.2
scipy: 1.1.0
pyarrow: 0.9.0
xarray: None
IPython: 6.4.0
sphinx: 1.7.4
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.3
numexpr: 2.6.5
feather: None
matplotlib: 2.2.2
openpyxl: 2.5.3
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.4
lxml: 4.2.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>


Issue with compression in to_csv method
#### Problem description

Hi there,

after upgrading to the lastest version of pandas I have an issue with the code, that worked fine on the previous version (0.22.0):


```python
            df.to_csv(
                path_or_buf=csv_path,
                encoding='utf8',
                compression='gz',
                quoting=1,
                sep='\t',
                index=False)

```

With pandas 0.23.0 I get:

> Traceback (most recent call last):
>   File "C:\_script.py", line 74, in <module>
>     index=False)
>   File "C:\Anaconda3\lib\site-packages\pandas\core\frame.py", line 1745, in to_csv
>     formatter.save()
>   File "C:\Anaconda3\lib\site-packages\pandas\io\formats\csvs.py", line 158, in save
>     data = f.read()
>   File "C:\Anaconda3\lib\encodings\cp1252.py", line 23, in decode
>     return codecs.charmap_decode(input,self.errors,decoding_table)[0]
> UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 298: character maps to <undefined>

If I comment **compression='gz'** the code works fine.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Windows
OS-release: 2012ServerR2
machine: AMD64
processor: Intel64 Family 6 Model 63 Stepping 2, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.23.0
pytest: 3.5.1
pip: 10.0.1
setuptools: 39.1.0
Cython: 0.28.2
numpy: 1.14.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: 1.7.4
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.3
numexpr: 2.6.5
feather: None
matplotlib: 2.2.2
openpyxl: 2.5.3
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.4
lxml: 4.2.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: None
psycopg2: 2.7.4 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
Can you please post a reproducible example? Tried locally with the below on master and it worked fine:

```python
buf = io.BytesIO(b'\xff\x34')
df = pd.read_csv(buf, encoding='utf16')

outbuf = buf.StringIO()
df.to_csv(outbuf, encoding='utf-16')
```
Notice my code is failing not due to data content.
Here an example using sklearn datasets

```python
from sklearn import datasets
iris = datasets.load_iris()

data1 = pd.DataFrame(data= np.c_[iris['data'], iris['target']],
                     columns= iris['feature_names'] + ['target'])

data1.to_csv('test.gz'
         , sep='~'
         ,  header=False, index=False
         ,compression='gzip'
         ,line_terminator='\r\n'
         ,encoding='utf-16'
         , na_rep=''
        )
```
If I use compression parameter I got the _UnicodeDecodeError_; but without the parameter, runs properly.
you would have to show a reproducible example
My apologies if I'm missing something, but shouldn't the compression value for GNU zip be, 'gzip', rather than 'gz'?  [https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.DataFrame.to_csv.html](url)
Thank you for the responses.

Well, I can't reproduce the issue with pandas 0.23.0 on my home machine, but it appears on two of my windows servers at work. I will try to make a clean install of anaconda on one of them and see what happens.

yeah, @CamWhitelaw, you're absolutely right. I've actually tried to change my code before pasted it here,  in original version of my code it uses gzip.
Please let us know if re-install anaconda solved the issue. 
did some digging, it might be different default encoding which `open()` falls to when not specified. then on windows it tried to decode with CP1252 when your file is UTF-8 encoded.

Commit message: default bytes strings unless requires unicode string

Patch to review:
diff --git a/doc/source/whatsnew/v0.23.1.txt b/doc/source/whatsnew/v0.23.1.txt
index f2bc81eea186b..223073271033a 100644
--- a/doc/source/whatsnew/v0.23.1.txt
+++ b/doc/source/whatsnew/v0.23.1.txt
@@ -92,6 +92,7 @@ I/O
 
 - Bug in IO methods specifying ``compression='zip'`` which produced uncompressed zip archives (:issue:`17778`, :issue:`21144`)
 - Bug in :meth:`DataFrame.to_stata` which prevented exporting DataFrames to buffers and most file-like objects (:issue:`21041`)
+- Bug in :meth:`DataFrame.to_csv` using compression causes encoding error (:issue:`21241`)
 -
 
 Plotting
diff --git a/pandas/io/formats/csvs.py b/pandas/io/formats/csvs.py
index 0be2a180fbfa2..ffed1c0c30477 100644
--- a/pandas/io/formats/csvs.py
+++ b/pandas/io/formats/csvs.py
@@ -151,11 +151,14 @@ def save(self):
             self._save()
 
         finally:
-            # GH 17778 handles compression for byte strings.
+            # GH 17778 handles zip compression for byte strings separately to
+            # support Python 2, also allow compression file handle
             if not close and self.compression:
                 f.close()
-                with open(f.name, 'r') as f:
+                with open(f.name, 'rb') as f:
                     data = f.read()
+                if not compat.PY2:
+                    data = data.decode(encoding)
                 f, handles = _get_handle(f.name, self.mode,
                                          encoding=encoding,
                                          compression=self.compression)
diff --git a/pandas/tests/frame/test_to_csv.py b/pandas/tests/frame/test_to_csv.py
index e4829ebf48561..44fa5468f03b2 100644
--- a/pandas/tests/frame/test_to_csv.py
+++ b/pandas/tests/frame/test_to_csv.py
@@ -919,29 +919,37 @@ def test_to_csv_path_is_none(self):
         recons = pd.read_csv(StringIO(csv_str), index_col=0)
         assert_frame_equal(self.frame, recons)
 
-    def test_to_csv_compression(self, compression):
-
-        df = DataFrame([[0.123456, 0.234567, 0.567567],
-                        [12.32112, 123123.2, 321321.2]],
-                       index=['A', 'B'], columns=['X', 'Y', 'Z'])
+    @pytest.mark.parametrize('frame, encoding', [
+        (DataFrame([[0.123456, 0.234567, 0.567567],
+                    [12.32112, 123123.2, 321321.2]],
+                   index=['A', 'B'], columns=['X', 'Y', 'Z']), None),
+        (DataFrame([['abc', 'def', 'ghi']], columns=['X', 'Y', 'Z']), 'ascii'),
+        (DataFrame(5 * [[123, u"你好", u"世界"]],
+                   columns=['X', 'Y', 'Z']), 'gb2312'),
+        (DataFrame(5 * [[123, u"Γειά σου", u"Κόσμε"]],
+                   columns=['X', 'Y', 'Z']), 'cp737')
+    ])
+    def test_to_csv_compression(self, frame, encoding, compression):
 
         with ensure_clean() as filename:
 
-            df.to_csv(filename, compression=compression)
+            frame.to_csv(filename, compression=compression, encoding=encoding)
 
             # test the round trip - to_csv -> read_csv
             rs = read_csv(filename, compression=compression,
-                          index_col=0)
-            assert_frame_equal(df, rs)
+                          index_col=0, encoding=encoding)
+            assert_frame_equal(frame, rs)
 
             # explicitly make sure file is compressed
             with tm.decompress_file(filename, compression) as fh:
-                text = fh.read().decode('utf8')
-                for col in df.columns:
+                text = fh.read().decode(encoding or 'utf8')
+                for col in frame.columns:
                     assert col in text
 
             with tm.decompress_file(filename, compression) as fh:
-                assert_frame_equal(df, read_csv(fh, index_col=0))
+                assert_frame_equal(frame, read_csv(fh,
+                                                   index_col=0,
+                                                   encoding=encoding))
 
     def test_to_csv_date_format(self):
         with ensure_clean('__tmp_to_csv_date_format__') as path:
diff --git a/pandas/tests/series/test_io.py b/pandas/tests/series/test_io.py
index e369dfda6deac..7ecf26786742a 100644
--- a/pandas/tests/series/test_io.py
+++ b/pandas/tests/series/test_io.py
@@ -137,29 +137,35 @@ def test_to_csv_path_is_none(self):
         csv_str = s.to_csv(path=None)
         assert isinstance(csv_str, str)
 
-    def test_to_csv_compression(self, compression):
-
-        s = Series([0.123456, 0.234567, 0.567567], index=['A', 'B', 'C'],
-                   name='X')
+    @pytest.mark.parametrize('s, encoding', [
+        (Series([0.123456, 0.234567, 0.567567], index=['A', 'B', 'C'],
+                name='X'), None),
+        (Series(['abc', 'def', 'ghi'], name='X'), 'ascii'),
+        (Series(["123", u"你好", u"世界"], name=u"中文"), 'gb2312'),
+        (Series(["123", u"Γειά σου", u"Κόσμε"], name=u"Ελληνικά"), 'cp737')
+    ])
+    def test_to_csv_compression(self, s, encoding, compression):
 
         with ensure_clean() as filename:
 
-            s.to_csv(filename, compression=compression, header=True)
+            s.to_csv(filename, compression=compression, encoding=encoding,
+                     header=True)
 
             # test the round trip - to_csv -> read_csv
             rs = pd.read_csv(filename, compression=compression,
-                             index_col=0, squeeze=True)
+                             encoding=encoding, index_col=0, squeeze=True)
             assert_series_equal(s, rs)
 
             # explicitly ensure file was compressed
             with tm.decompress_file(filename, compression) as fh:
-                text = fh.read().decode('utf8')
+                text = fh.read().decode(encoding or 'utf8')
                 assert s.name in text
 
             with tm.decompress_file(filename, compression) as fh:
                 assert_series_equal(s, pd.read_csv(fh,
                                                    index_col=0,
-                                                   squeeze=True))
+                                                   squeeze=True,
+                                                   encoding=encoding))
 
 
 class TestSeriesIO(TestData):


Code review comments:
diff_hunk: @@ -919,29 +919,37 @@ def test_to_csv_path_is_none(self):
         recons = pd.read_csv(StringIO(csv_str), index_col=0)
         assert_frame_equal(self.frame, recons)
 
-    def test_to_csv_compression(self, compression):
-
-        df = DataFrame([[0.123456, 0.234567, 0.567567],
-                        [12.32112, 123123.2, 321321.2]],
-                       index=['A', 'B'], columns=['X', 'Y', 'Z'])
+    @pytest.mark.parametrize('frame, encoding', [
path: pandas/tests/frame/test_to_csv.py
text: @user1:
Use `df` instead of `frame` and remove the whitespace between the parameter names here

@author:
sure.

diff_hunk: @@ -919,29 +919,37 @@ def test_to_csv_path_is_none(self):
         recons = pd.read_csv(StringIO(csv_str), index_col=0)
         assert_frame_equal(self.frame, recons)
 
-    def test_to_csv_compression(self, compression):
-
-        df = DataFrame([[0.123456, 0.234567, 0.567567],
-                        [12.32112, 123123.2, 321321.2]],
-                       index=['A', 'B'], columns=['X', 'Y', 'Z'])
+    @pytest.mark.parametrize('frame, encoding', [
+        (DataFrame([[0.123456, 0.234567, 0.567567],
+                    [12.32112, 123123.2, 321321.2]],
+                   index=['A', 'B'], columns=['X', 'Y', 'Z']), None),
+        (DataFrame([['abc', 'def', 'ghi']], columns=['X', 'Y', 'Z']), 'ascii'),
+        (DataFrame(5 * [[123, u"你好", u"世界"]],
+                   columns=['X', 'Y', 'Z']), 'gb2312'),
+        (DataFrame(5 * [[123, u"Γειά σου", u"Κόσμε"]],
+                   columns=['X', 'Y', 'Z']), 'cp737')
+    ])
+    def test_to_csv_compression(self, frame, encoding, compression):
 
         with ensure_clean() as filename:
 
-            df.to_csv(filename, compression=compression)
+            frame.to_csv(filename, compression=compression, encoding=encoding)
 
             # test the round trip - to_csv -> read_csv
             rs = read_csv(filename, compression=compression,
path: pandas/tests/frame/test_to_csv.py
text: @user1:
Can you change `rs` to `result`?

@author:
done.