BUG: to_clipboard text truncated for Python 3 on Windows for UTF-16 text
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

For windows users where Python is compiled with UCS-4 (Python 3 primarily), tables copied to clipboard are missing data from the end when there are any unicode characters in the dataframe that have a 4-byte representation in UTF-16 (i.e. in the U+010000 to U+10FFFF range).  The bug can be reproduced here:

```python
import pandas
obj=pandas.DataFrame([u'\U0001f44d\U0001f44d',
              u'12345'])
obj.to_clipboard()
```
where the clipboard text results in 
```
	0
0	👍👍
1	1234
```
One character is chopped from the end of the clipboard string for each 4-byte unicode character copied.

or more to the point:
```python
pandas.io.clipboard.clipboard_set(u'\U0001f44d 12345')
```
produces
```
👍 1234
```

The cause of this issue is that ```len(u'\U0001f44d')==1``` when python is in UCS-4, and Pandas allocates 2 bytes per python character in the clipboard buffer but the character consumes 4 bytes, displacing another character at the end of the string to be copied.  In UCS-2 (most Python 2 builds), ```len(u'\U0001f44d')==2``` and so 4 bytes are allocated and consumed by the character.

My proposed change (affecting only windows clipboard operations) first converts the text to UTF-16 little endian because that is the format used by windows, then measures the length of the resulting byte string, rather than using Python's ```len(text) * 2``` to measure how many bytes should be allocated to the clipboard buffer.

I've tested this change in python 3.6 and 2.7 on windows 7 x64.  I don't expect this causing other issues with other versions of windows but I would appreciate if anyone on older versions of windows would double check.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Windows
description: Windows OS

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=h1) Report
> Merging [#25040](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b9f2e2bd526ad69fe3d737d395742193e29fd0e5?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `25%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25040/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25040      +/-   ##
==========================================
+ Coverage   92.38%   92.38%   +<.01%     
==========================================
  Files         166      166              
  Lines       52401    52402       +1     
==========================================
+ Hits        48409    48412       +3     
+ Misses       3992     3990       -2
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.8% <25%> (-0.01%)` | :arrow_down: |
| #single | `42.88% <25%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/clipboard/windows.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC93aW5kb3dzLnB5) | `11.49% <25%> (-0.14%)` | :arrow_down: |
| [pandas/core/internals/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbmNhdC5weQ==) | `96.45% <0%> (ø)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `88.13% <0%> (+0.09%)` | :arrow_up: |
| [pandas/io/clipboard/clipboards.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9jbGlwYm9hcmRzLnB5) | `30.58% <0%> (+2.35%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=footer). Last update [b9f2e2b...2a39a50](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=h1) Report
> Merging [#25040](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b9f2e2bd526ad69fe3d737d395742193e29fd0e5?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25040/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25040      +/-   ##
==========================================
- Coverage   92.38%   92.36%   -0.02%     
==========================================
  Files         166      166              
  Lines       52401    52408       +7     
==========================================
- Hits        48409    48408       -1     
- Misses       3992     4000       +8
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.78% <0%> (-0.02%)` | :arrow_down: |
| #single | `42.87% <0%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/clipboard/windows.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC93aW5kb3dzLnB5) | `11.11% <0%> (-0.52%)` | :arrow_down: |
| [pandas/io/formats/console.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvY29uc29sZS5weQ==) | `66.66% <0%> (-5.8%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.82% <0%> (-0.11%)` | :arrow_down: |
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `94.17% <0%> (-0.09%)` | :arrow_down: |
| [pandas/core/tools/numeric.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvbnVtZXJpYy5weQ==) | `100% <0%> (ø)` | :arrow_up: |
| [pandas/core/internals/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbmNhdC5weQ==) | `96.45% <0%> (ø)` | :arrow_up: |
| [pandas/compat/pickle\_compat.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9waWNrbGVfY29tcGF0LnB5) | `75.6% <0%> (ø)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.63% <0%> (ø)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `88.13% <0%> (+0.09%)` | :arrow_up: |
| [pandas/io/clipboard/clipboards.py](https://codecov.io/gh/pandas-dev/pandas/pull/25040/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9jbGlwYm9hcmRzLnB5) | `30.58% <0%> (+2.35%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=footer). Last update [b9f2e2b...0c1b914](https://codecov.io/gh/pandas-dev/pandas/pull/25040?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@gfyoung how does it look now?
@david-liu-brattle-1 : Looks better!  Can you also add a `whatsnew` entry to `0.25.0` ?
yes would prefer to simply update the vendored copy
@jreback 

All set.  Also added whatsnew entry to 25.0
@david-liu-brattle-1 can you run all of the clipboard tests, I am not convinced we actually really run these on CI.

``pytest -m clipboard pandas``
@jreback 
Tests run fine on my end

```
============================= test session starts =============================
platform win32 -- Python 3.6.2, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
hypothesis profile 'ci' -> deadline=500, timeout=unlimited, suppress_health_check=[HealthCheck.too_slow], database=DirectoryBasedExampleDatabase('C:\\Users\\david liu\\Documents\\Code Projects2\\Python\\pandas\\pandas\\.hypothesis\\examples')
rootdir: C:\Users\david liu\Documents\Code Projects2\Python\pandas\pandas, inifile: setup.cfg
plugins: hypothesis-4.4.3
collected 47117 items / 46892 deselected / 6 skipped

pandas\tests\io\test_clipboard.py ...................................... [ 16%]
........................................................................ [ 48%]
........................................................................ [ 80%]
...........................................                              [100%]

========== 225 passed, 6 skipped, 46892 deselected in 32.43 seconds ===========
```

```
============================= test session starts =============================
platform win32 -- Python 2.7.11, pytest-3.6.1, py-1.5.3, pluggy-0.6.0
hypothesis profile 'ci' -> deadline=500, timeout=unlimited, suppress_health_check=[HealthCheck.too_slow], database=DirectoryBasedExampleDatabase('C:\\Users\\david liu\\Documents\\Code Projects2\\Python\\pandas\\pandas\\.hypothesis\\examples')
rootdir: C:\Users\david liu\Documents\Code Projects2\Python\pandas\pandas, inifile: setup.cfg
plugins: hypothesis-4.4.3
collected 47343 items / 47118 deselected / 6 skipped

pandas\tests\io\test_clipboard.py ...................................... [ 16%]
........................................................................ [ 48%]
........................................................................ [ 80%]
...........................................                              [100%]

========== 225 passed, 6 skipped, 47118 deselected in 87.43 seconds ===========

```
thanks @david-liu-brattle-1