Fix #12529 / Improve to_clipboard for objects containing unicode
State: closed

Superceded by #14599

---

- [x] closes #12529
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

I added the whatsnew entry to 0.18.0.


Labels:
name: Unicode
description: Unicode strings
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
move to 0.18.1

use the option display.encoding for the actual encoding 

pls add a test

> use the option display.encoding for the actual encoding 

`pandas.options.display.encoding` is `utf-8` on my German Win10 or Win7. However, the encoding I have to use to get correct behaviour is "cp1250" (which is what is returned by `locale.getdefaultlocale()[1]`). 

I'll look into adding a test tomorrow.

hmm ok

look in the code and see how display.encoding is determined as well

it may be that that is wrong (iow not taking into account the locale) or this is just different 

Before writing a new test I ran the tests in "test_clipboard.py" on the unmodified 0.18.0rc1 installation. Surprisingly they fail:

```
py-2|C:\dev\Python27_64\Lib\site-packages\pandas\io\tests>nosetests test_clipboard.py
.EE.
======================================================================
ERROR: test_round_trip_frame (pandas.io.tests.test_clipboard.TestClipboard)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 81, in test_round_trip_frame
    self.check_round_trip_frame(dt)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 69, in check_round_trip_frame
    tm.assert_frame_equal(data, result, check_dtype=False)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 1115, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 999, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas\src\testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:3809)
  File "pandas\src\testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2685)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 866, in raise_assert_detail
    [right]: {3}""".format(obj, message, left, right)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 9: ordinal not in range(128)

======================================================================
ERROR: test_round_trip_frame_sep (pandas.io.tests.test_clipboard.TestClipboard)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 73, in test_round_trip_frame_sep
    self.check_round_trip_frame(dt, sep=',')
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 69, in check_round_trip_frame
    tm.assert_frame_equal(data, result, check_dtype=False)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 1115, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 999, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas\src\testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:3809)
  File "pandas\src\testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2685)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 866, in raise_assert_detail
    [right]: {3}""".format(obj, message, left, right)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 9: ordinal not in range(128)

----------------------------------------------------------------------
Ran 4 tests in 0.328s

FAILED (errors=2)
```

After fixing pandas\util\testing.py, I still get a failure:

```
py-2|C:\dev\Python27_64\Lib\site-packages\pandas\io\tests>nosetests test_clipboard.py
.FF.
======================================================================
FAIL: test_round_trip_frame (pandas.io.tests.test_clipboard.TestClipboard)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 81, in test_round_trip_frame
    self.check_round_trip_frame(dt)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 69, in check_round_trip_frame
    tm.assert_frame_equal(data, result, check_dtype=False)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 1115, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 999, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas\src\testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:3809)
  File "pandas\src\testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2685)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 867, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.iloc[:, 1] are different

DataFrame.iloc[:, 1] values are different (50.0 %)
[left]:  [en, espa\xf1ol]
[right]: [en, espa\xc3\xb1ol]

======================================================================
FAIL: test_round_trip_frame_sep (pandas.io.tests.test_clipboard.TestClipboard)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 73, in test_round_trip_frame_sep
    self.check_round_trip_frame(dt, sep=',')
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 69, in check_round_trip_frame
    tm.assert_frame_equal(data, result, check_dtype=False)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 1115, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 999, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas\src\testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:3809)
  File "pandas\src\testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2685)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 867, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.iloc[:, 1] are different

DataFrame.iloc[:, 1] values are different (50.0 %)
[left]:  [en, espa\xf1ol]
[right]: [en, espa\xc3\xb1ol]

----------------------------------------------------------------------
Ran 4 tests in 0.391s

FAILED (failures=2)

```

yeah these currently fail on windows py2 (and I have a single/different one failing on macosx), but nothing fails on linux (which is where Travis runs). They all work on py3.

I think we have an issue about that. But if not, can you open one. Something is still not being encoded correctly. Of course if you can figure out would be great!

can you rebase / update

@dalito Are you still working on it?

If I'm not mistaken the errors you're getting are caused by `pandas.util.clipboard` (pyperclip), which expects unicode input.

This solution: https://github.com/pijucha/pandas/commit/e53dcb099b02172afa3ee5e0a4ff80e69e1100de works for me in python2 (both windows and linux). You can test it and include in your PR if it also works for you.

can you rebase / update?

Closing this PR as this will be handled in #14599. @dalito Thanks for your work on this, and you are certainly welcome to test the other PR.