COMPAT: Properly encode filenames in read_csv
State: closed

Python 3.6+ changes the default encoding to `utf8` ([PEP 529](https://www.python.org/dev/peps/pep-0529/)), which conflicts with the encoding of Windows (`mbcs`).  This conflicts rears its head when using only the C engine of `read_csv` because of the low level functionality that we implement ourselves.

This fix checks if we're using Python 3.6+ and on Windows, after which we force the encoding to `mbcs`

Closes #15086.

Labels:
name: Bug
name: Unicode
description: Unicode strings
name: IO CSV
description: read_csv, to_csv
name: Windows
description: Windows OS
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
lgtm. ping on green.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=h1) Report
> Merging [#24758](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/f160c7d7c63c11f8807ddd1e653a95821e82269d?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24758/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #24758   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files         166      166           
  Lines       52363    52363           
=======================================
  Hits        48376    48376           
  Misses       3987     3987
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.81% <ø> (ø)` | :arrow_up: |
| #single | `42.91% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=footer). Last update [f160c7d...cfad6ec](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=6891

@jreback : The lint failure is unrelated to my changes.  However, I'll rebase and push again to be safe.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=h1) Report
> Merging [#24758](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/453fa85a8b88ca22c7b878a3fcf97e068f11b6c4?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24758/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #24758   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files         166      166           
  Lines       52363    52363           
=======================================
  Hits        48376    48376           
  Misses       3987     3987
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.81% <ø> (ø)` | :arrow_up: |
| #single | `42.91% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=footer). Last update [453fa85...ab663b4](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=h1) Report
> Merging [#24758](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/453fa85a8b88ca22c7b878a3fcf97e068f11b6c4?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24758/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #24758   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files         166      166           
  Lines       52363    52363           
=======================================
  Hits        48376    48376           
  Misses       3987     3987
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.81% <ø> (ø)` | :arrow_up: |
| #single | `42.91% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=footer). Last update [453fa85...ab663b4](https://codecov.io/gh/pandas-dev/pandas/pull/24758?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback : Rebased, and all is green.  PTAL.
thanks @gfyoung 
Thank @gfyoung. Do you know if the CI would have failed on that path without your fix? IIUC, the failure was somewhat system-dependent, though I don't recall the exact details.
> Do you know if the CI would have failed on that path without your fix? IIUC, the failure was somewhat system-dependent, though I don't recall the exact details.

@TomAugspurger : Yes, it would have.  It's a Windows Python 3.6+ issue.