BUG: encode filepaths on windows in 3.6
State: closed

Not for Merge, just a small test for #15086 with AppVeyor.

Labels:
name: Bug
name: Windows
description: Windows OS

Comments:
so looks your test failed (good!)

so if you can update the test to make more generic like I showed. then you can do the fix (which will be to encode the filename on windows : https://docs.python.org/3/whatsnew/3.6.html#pep-529-change-windows-filesystem-encoding-to-utf-8

so I would probably do this *only* on windows and only on >= 3.6

```
from pandas.compat import PY3, is_platform_windows

if PY3 and is_platform_windows():
    fn = fn.encode(sys.getfilesystemencoding())
```
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15092?src=pr&el=h1) Report
> Merging [#15092](https://codecov.io/gh/pandas-dev/pandas/pull/15092?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c26e5bbed826aaf9a643c604f7f754f29af4bd64?src=pr&el=desc) will **increase** coverage by `-1.58%`.


```diff
@@            Coverage Diff             @@
##           master   #15092      +/-   ##
==========================================
- Coverage   86.33%   84.75%   -1.58%     
==========================================
  Files         139      145       +6     
  Lines       51149    51279     +130     
==========================================
- Hits        44157    43464     -693     
- Misses       6992     7815     +823
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15092?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/compare/c26e5bbed826aaf9a643c604f7f754f29af4bd64...04d3d2d1fc52be97fb721cf83da2203ad834bb94?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `67.65% <80%> (+0.26%)` | :white_check_mark: |
| [pandas/core/groupby.py](https://codecov.io/gh/pandas-dev/pandas/compare/c26e5bbed826aaf9a643c604f7f754f29af4bd64...04d3d2d1fc52be97fb721cf83da2203ad834bb94?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS5weQ==) | `77.24% <ø> (-17.91%)` | :x: |
| [pandas/tools/plotting.py](https://codecov.io/gh/pandas-dev/pandas/compare/c26e5bbed826aaf9a643c604f7f754f29af4bd64...04d3d2d1fc52be97fb721cf83da2203ad834bb94?src=pr&el=tree#diff-cGFuZGFzL3Rvb2xzL3Bsb3R0aW5nLnB5) | `68.48% <ø> (-3.31%)` | :x: |
| [pandas/types/cast.py](https://codecov.io/gh/pandas-dev/pandas/compare/c26e5bbed826aaf9a643c604f7f754f29af4bd64...04d3d2d1fc52be97fb721cf83da2203ad834bb94?src=pr&el=tree#diff-cGFuZGFzL3R5cGVzL2Nhc3QucHk=) | `84.18% <ø> (-1.24%)` | :x: |
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/compare/c26e5bbed826aaf9a643c604f7f754f29af4bd64...04d3d2d1fc52be97fb721cf83da2203ad834bb94?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `94.05% <ø> (-1.11%)` | :x: |
| [pandas/types/dtypes.py](https://codecov.io/gh/pandas-dev/pandas/compare/c26e5bbed826aaf9a643c604f7f754f29af4bd64...04d3d2d1fc52be97fb721cf83da2203ad834bb94?src=pr&el=tree#diff-cGFuZGFzL3R5cGVzL2R0eXBlcy5weQ==) | `94.3% <ø> (-1.04%)` | :x: |
| [pandas/tools/hashing.py](https://codecov.io/gh/pandas-dev/pandas/compare/c26e5bbed826aaf9a643c604f7f754f29af4bd64...04d3d2d1fc52be97fb721cf83da2203ad834bb94?src=pr&el=tree#diff-cGFuZGFzL3Rvb2xzL2hhc2hpbmcucHk=) | `98.14% <ø> (-0.88%)` | :x: |
| [pandas/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/compare/c26e5bbed826aaf9a643c604f7f754f29af4bd64...04d3d2d1fc52be97fb721cf83da2203ad834bb94?src=pr&el=tree#diff-cGFuZGFzL2luZGV4ZXMvbXVsdGkucHk=) | `95.81% <ø> (-0.77%)` | :x: |
| [pandas/compat/numpy/function.py](https://codecov.io/gh/pandas-dev/pandas/compare/c26e5bbed826aaf9a643c604f7f754f29af4bd64...04d3d2d1fc52be97fb721cf83da2203ad834bb94?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9udW1weS9mdW5jdGlvbi5weQ==) | `93.5% <ø> (-0.77%)` | :x: |
| [pandas/compat/__init__.py](https://codecov.io/gh/pandas-dev/pandas/compare/c26e5bbed826aaf9a643c604f7f754f29af4bd64...04d3d2d1fc52be97fb721cf83da2203ad834bb94?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `61.71% <ø> (-0.46%)` | :x: |
| ... and [34 more](https://codecov.io/gh/pandas-dev/pandas/pull/15092/changes?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15092?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15092?src=pr&el=footer). Last update [c26e5bb...dc5877b](https://codecov.io/gh/pandas-dev/pandas/compare/c26e5bbed826aaf9a643c604f7f754f29af4bd64...dc5877b156d767502548f4e528e04d0372629b2e?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
With Python 3.6 `sys.getfilesystemencoding()` return `utf-8`.

Calling `sys._enablelegacywindowsfsencoding()` before call Pandas fix this bug. This function set files system encoding to `mbcs` (Like on Python < 3.6).
this looks reasonable so far.

can you add some tests for excel, hdf5 similar to the above; I think you will need to modify
about here:
https://github.com/pandas-dev/pandas/blob/master/pandas/io/common.py#L228

to call (and encode) the filename, IOW, I think *after* the ``_expand_user`` you can have a line which checks if its a string, if so, then get the encoding and encode it before returning it.

alternatively might need to do this in ``_get_handle`` instead, right before actually opening the file.

The problem is not present for all I/O functions from pandas. Actually, I see it only with read_csv and read_table, but not for read_excel or read_fwf by example. This need to be tested before modifying anything else.
@JGoutin agreed. If you want to add tests for non-parser things would be great. Its possible this *only* affects the c-parser as that is the only thing that actually encodes the filenames (others use ``open(...)`` directly) and might not be affected by this change.

can you rebase and push again to force the ci.
can you add a similar test for

``read_excel``, ``read_hdf``. I think they will work, but just checking.
pls add a whatsnew note
In which file is your whatsnew ? 
https://github.com/pandas-dev/pandas/blob/master/doc/source/whatsnew/v0.20.0.txt
can you rebase and push again. I want to see all green on the CI.
All work was lost when trying to rebasing... I never did this before and read a bad tutorial on this.  :unamused:

Sorry, I don't have the time to re-doing this.
@JGoutin do you have your existing branch? (you can recover doing ``reflog``). If you push up what you had (or anything even close), I will fix it.
or even copy-past your changes here is ok too.
http://stackoverflow.com/questions/3973994/how-can-i-recover-from-an-erronous-git-push-f-origin-master
All work seem to be lost. I tried reflog and somes other methods but the repo seem to be reseted to its early post-fork state.