BUG in clipboard (linux, python2) with unicode and separator (GH13747)
State: closed

 - [x] closes #13747, closes #14362, closes #12807, closes #12529
 - [x] tests added / passed pandas/io/tests/test_clipboard.py for OSX
 - [x] passes ``git diff upstream/master | flake8 --diff``


Labels:
name: Bug
name: IO Data
description: IO issues that don't fit into a more specific label
name: Unicode
description: Unicode strings

Comments:
I believe this fix should also close #12529 (at least it worked for me a few months ago on linux and windows). 

Good practice is to [add some tests](http://pandas-docs.github.io/pandas-docs-travis/contributing.html#test-driven-development-code-writing) for a fix, for example as in `pandas/io/tests/test_clipboard.py`.

xref #12580

@pijucha , i thought we already have one unicode character test in test_clipboard.py (GH 13747) that used to fail earliers (in OSX) and it passes with this change, do i need to add more?

Ah, you're right. I forgot.

I played with some non-latin characters and also °C for the other issue - just to be sure. The existing test contains only `'en español'` and I wondered whether it could still pass with locale set to iso8859-1 or the like. I don't remember now, maybe it's not an issue.

@pijucha and none of the existing tests failed post this change so i think we can prolly merge it

I ran this test case in local post changes and it succeeds 

df = pd.DataFrame({'a':['µasd','Ωœ∑´'], 'b':['øπ∆˚¬','œ∑´®']})
df.to_clipboard(excel=None, sep =sep)
result = read_clipboard(sep = sep, index_col = 0)
tm.assert_frame_equal(df, result, check_dtype=False)

wouldn't hurt to add that as a test

should i add this as a individual test in pandas/io/tests/test_clipboard.py itself?

yep

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14599?src=pr) is 85.20% (diff: 30.09%)

> Merging [#14599](https://codecov.io/gh/pandas-dev/pandas/pull/14599?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **0.08%**

``` diff
@@             master     #14599   diff @@
==========================================
  Files           140        143     +3   
  Lines         50706      50773    +67   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43248      43262    +14   
- Misses         7458       7511    +53   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [45543ec...2aafb66](https://codecov.io/gh/pandas-dev/pandas/compare/45543ec22bb01d5c46eff9384491d218cffddd64...2aafb66cbf50f39e593a5e4b63667b62a47928bb?src=pr)

@jreback  i have added the test case in the file

@pijucha I have refactored the test case to be in sync with the rest of the file

@jreback @pijucha  i can see that travis ci check has errored but i cant see any error message related to the changes that i have made. The logs just say that no output was received in last 10 minutes. Could it be an issue with the travis at the time?

that might just be a stalled build it happens

in any event i would ideally like to have a test that actually fails on linux (and os x) before this change (so that once this fix is applied the test passes)

iow this is not currently tested on linux (though it fails for me locally as well)

I think we just disabled the test in the linked issue - can u re enable 

@jreback if i remove the "@disabled" tag from the class (pandas/io/tests/test_clipboard.py) , will that be sufficient?

yes - ideally see if that fails w/o your change then passes with / that was out there a long time ago (so the test might not work well anyhow)

@jreback , when i run all tests in my local machine, using the script, test.sh (at root directory), it does run these clipboard tests (that is how i came across this issue in the first place). So now i wonder if the disabled tag works at all, as the test.sh was able to call test_clipboard.py even with the disabled tag in place and now that i have removed the disabled tag, it would definitely execute the clipboard tests      

what i mean is travis doesn't run the disabled tests by default
that's what i want to see is s test that fails there (create a new branch and just remove the disabled ) thrn push up to test

FYI I think they will fail on travis. In [this job](https://travis-ci.org/pandas-dev/pandas/jobs/174096670#L1108) which is under my pytest branch, I had messed up the `disabled`, so the clipboard tests got run and they failed.

@aileronajay ok looks like if u can get these to pass on your other PR would be great
closing this as was a test

@jreback i have created another PR (https://github.com/pandas-dev/pandas/pull/14619), that tests if the clipboard tests currently fail, (for that PR the ci fails). Can you please re open this one, in this current PR, the tests will ideally pass as this PR has the unicode change

sorry thought was closing the other one

@jreback so the change did not fix the failiing tests in clipboard (that have been disabled). I mean it works for me locally in osx but not in travis

can u compare your env and see what's different vs travis
ideally set up as close an env as possible and try to repro 

@jreback i am able to reproduce this issue on a fedora vm. Also this issue manifests when the library used by clipboard is pyqt4 (this call can also be seen in the failing trace in travis)

Traceback (most recent call last):
  File "/home/travis/build/pandas-dev/pandas/pandas/io/tests/test_clipboard.py", line 86, in test_round_trip_frame
    self.check_round_trip_frame(dt)
  File "/home/travis/build/pandas-dev/pandas/pandas/io/tests/test_clipboard.py", line 71, in check_round_trip_frame
    result = read_clipboard()
  File "/home/travis/build/pandas-dev/pandas/pandas/io/clipboard.py", line 23, in read_clipboard
    text = clipboard_get()
  File "/home/travis/build/pandas-dev/pandas/pandas/util/clipboard.py", line 141, in _pasteQt
    return str(cb.text())
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb5' in position 7: ordinal not in range(128)

. If clipboard uses xclip or xsel we dont get this error. I think it might have to do with how pandas.util.clipboard or pyperclip handles encoding for the pasteQt and copyQt calls. No encoding is specified and i believe it tries to use ascii by default and fails

def _pasteQt():
    return str(cb.text())

def _copyQt(text):
    cb.setText(text) 

PS: I was able to execute all clipboard tests successfully in the fedora vm using xsel

@jreback do we change the pyperclip, python.util.clipboard file to fix this issue?

kicked off https://github.com/pandas-dev/pandas/pull/14640 with the return type changed to unicode in pandas.util.clipboard. The clipboard tests are passing now, some other un related tests are failing though

Old pyperclip had some bug related with Qt AFAIR - it may be this. How about replacing `pandas/util/clipboard.py` (which is pyperclip v1.5.15) with the latest pyperclip v1.5.27?

@pijucha or we could take the relevant change from 1.5.27. In 1.5.27 _pasteQt method has been changed to return the type based on the python version, the method returns unicode in case of python2 and str otherwise (i have made a similar change in the PR https://github.com/pandas-dev/pandas/pull/14640 that i created for testing purposes) .

easiest is to upgrade pyperclip I think (also this may fix some other open issues -  have a look)

just replace the file with the new one