Repository: pandas-dev/pandas

Issue:
Bug: On Python 3 to_csv() encoding defaults to ascii if the dataframe contains special characters.
#### Code Sample, a copy-pastable example if possible

```
import pandas as pd

L1 = ["AAAAA","BBBBB","TTTTT","77777"]
df1 = pd.DataFrame({"L1":L1})
df1.to_csv("test1.csv")

L2 = ["AAAAA","ÄÄÄÄÄ","ßßßßß","聞聞聞聞聞"]
df2 = pd.DataFrame({"L2":L2})
df2.to_csv("test2.csv")

df2.to_csv("test3.csv",encoding='utf8')
```
#### Problem description
The [to-csv doc](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html) says about `encoding`:
>A string representing the encoding to use in the output file, defaults to ‘ascii’ on Python 2 and ‘utf-8’ on Python 3.

Therefore being on Python 3 I expect `test1.csv` and `test2.csv` to be `utf8`.

However while `test1.csv` is encoded in `utf8`, `test2.csv` is encoded in `ascii`, if I want the correct encoding I have to explicitely add the encoding to produce the correct result as `test3.csv`.

Correspondingly doing
```
pd.read_csv("test2.csv")
```
leads to

`UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 0: invalid continuation byte`

#### EDIT: Additional info from first comment:

The output of the `to_csv()` calls looks correct:

```
df1.to_csv()
Out[9]: ',L1\n0,AAAAA\n1,BBBBB\n2,TTTTT\n3,77777\n'

df2.to_csv()
Out[10]: ',L2\n0,AAAAA\n1,ÄÄÄÄÄ\n2,ßßßßß\n3,聞聞聞聞聞\n'
```

Regarding the `read_csv()` part it's like this:

I can read `test1.csv` and `test3.csv` fine, regardless of specifying `encoding='utf8'` or not.

Likewise I can not read `test2.csv` at all, regardless of specifying `encoding='utf8'` or not. The error message is returned in both cases.

The problem is only solved by explicitely specifying `encoding='utf8'` in `to_csv()`.

#### EDIT 2:

I can only read `test2.csv` when I explicitely state `encoding='ansi'`, so `read_csv()` definitely expects `utf-8`.

#### Output of ``pd.show_versions()``

<details>
python: 3.6.2.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64

pandas: 0.20.3
</details>



Discussion:
@Khris777 : Thanks for the report!  I can address the `read_csv` part in that we don't use a default encoding for that part (per the docs), so you would be expected to explicitly specify `UTF8`.  As for `to_csv`, that does seem a little strange...

For reference, could you show us the output of your `to_csv` calls?  You can do that by just typing `df.to_csv()` without a filepath, and it will print out the CSV file to your console / terminal.
The output of the `to_csv()` calls looks correct:

```
df1.to_csv()
Out[9]: ',L1\n0,AAAAA\n1,BBBBB\n2,TTTTT\n3,77777\n'

df2.to_csv()
Out[10]: ',L2\n0,AAAAA\n1,ÄÄÄÄÄ\n2,ßßßßß\n3,聞聞聞聞聞\n'
```

Regarding the `read_csv()` part it's like this:

I can read `test1.csv` and `test3.csv` fine, regardless of specifying `encoding='utf8'` or not.

Likewise I can not read `test2.csv` at all, regardless of specifying `encoding='utf8'` or not. The error message is returned in both cases.

The problem is only solved by explicitely specifying `encoding='utf8'` in `to_csv()`.
@Khris777 : Interesting!  That's definitely a little unusual.  Do me a favor, could you copy + paste that output into your original issue description?  That will making reviewing this a little easier.

BTW, you are more than welcome to investigate what's going on here and submit a PR to fix this admittedly strange behavior.
@gfyoung : Can you reproduce the problem on your side?
@Khris777 : Yes, I can.
@gfyoung When `encoding=None`, write a file by `csv` library.
https://github.com/pandas-dev/pandas/blob/master/pandas/io/formats/format.py#L1630

The `csv` library seems to write a file by system default encoding, at least on Windows.
So I cannot reproduce the example on Mac which default language is Japanese, but can do on Windows 10 Japanese edition.
Also I found that we can read the file by the system default encoding.

### Input
```python
import pandas as pd

L2 = ["AAAAA","ÄÄÄÄÄ","ßßßßß","聞聞聞聞聞"]
df2 = pd.DataFrame({"L2":L2})
df2.to_csv("test2.csv")

# Japanese Windows default encoding is cp932
print(pd.read_csv("test2.csv", encoding='cp932'))
```

### Output
```python
   Unnamed: 0     L2
0           0  AAAAA
1           1  ?????
2           2  ?????
3           3  聞聞聞聞聞
```



@Licht-T : That's true, so are you suggesting then that workaround is specifying the correct encoding since `UTF8` is evidently insufficient?

Resolving patch:
diff --git a/pandas/io/formats/format.py b/pandas/io/formats/format.py
--- a/pandas/io/formats/format.py
+++ b/pandas/io/formats/format.py
@@ -1573,12 +1573,20 @@ def __init__(self, obj, path_or_buf=None, sep=",", na_rep='',
 
     def save(self):
         # create the writer & save
+        if self.encoding is None:
+            if compat.PY2:
+                encoding = 'ascii'
+            else:
+                encoding = 'utf-8'
+        else:
+            encoding = self.encoding
+
         if hasattr(self.path_or_buf, 'write'):
             f = self.path_or_buf
             close = False
         else:
             f, handles = _get_handle(self.path_or_buf, self.mode,
-                                     encoding=self.encoding,
+                                     encoding=encoding,
                                      compression=self.compression)
             close = True
 
@@ -1588,11 +1596,11 @@ def save(self):
                                  doublequote=self.doublequote,
                                  escapechar=self.escapechar,
                                  quotechar=self.quotechar)
-            if self.encoding is not None:
-                writer_kwargs['encoding'] = self.encoding
-                self.writer = UnicodeWriter(f, **writer_kwargs)
-            else:
+            if encoding == 'ascii':
                 self.writer = csv.writer(f, **writer_kwargs)
+            else:
+                writer_kwargs['encoding'] = encoding
+                self.writer = UnicodeWriter(f, **writer_kwargs)
 
             self._save()
 


Test patch:
diff --git a/pandas/tests/io/formats/test_to_csv.py b/pandas/tests/io/formats/test_to_csv.py
--- a/pandas/tests/io/formats/test_to_csv.py
+++ b/pandas/tests/io/formats/test_to_csv.py
@@ -1,3 +1,5 @@
+# -*- coding: utf-8 -*-
+
 from pandas import DataFrame
 import numpy as np
 import pandas as pd
@@ -6,6 +8,21 @@
 
 class TestToCSV(object):
 
+    def test_to_csv_defualt_encoding(self):
+        # GH17097
+        df = DataFrame({'col': [u"AAAAA", u"ÄÄÄÄÄ", u"ßßßßß", u"聞聞聞聞聞"]})
+
+        with tm.ensure_clean('test.csv') as path:
+            # the default to_csv encoding in Python 2 is ascii, and that in
+            # Python 3 is uft-8.
+            if pd.compat.PY2:
+                # the encoding argument parameter should be utf-8
+                with tm.assert_raises_regex(UnicodeEncodeError, 'ascii'):
+                    df.to_csv(path)
+            else:
+                df.to_csv(path)
+                tm.assert_frame_equal(pd.read_csv(path, index_col=0), df)
+
     def test_to_csv_quotechar(self):
         df = DataFrame({'col': [1, 2]})
         expected = """\