BUG: Gracefully handle all utf-8 characters in json urls 
State: closed

- [x] closes #17918  
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Url is passed through `quote` before doing the request.  The `safe` parameter characters are the reserved character set defined in [RFC 2396](https://www.ietf.org/rfc/rfc2396.txt)(See section 2.2). 

Labels:
name: Bug
name: IO Data
description: IO issues that don't fit into a more specific label
name: Unicode
description: Unicode strings

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=h1) Report
> Merging [#17933](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/77b4bb30ed5afc95f2c264c6b25e96ea8a9a5aee?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `50%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17933/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17933      +/-   ##
==========================================
- Coverage   91.23%   91.22%   -0.02%     
==========================================
  Files         163      163              
  Lines       50113    50114       +1     
==========================================
- Hits        45723    45714       -9     
- Misses       4390     4400      +10
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.03% <50%> (-0.01%)` | :arrow_down: |
| #single | `40.31% <50%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `69.19% <50%> (-0.3%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.75% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=footer). Last update [77b4bb3...ec9cc9a](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=h1) Report
> Merging [#17933](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/77b4bb30ed5afc95f2c264c6b25e96ea8a9a5aee?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `33.33%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17933/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17933      +/-   ##
==========================================
- Coverage   91.23%   91.22%   -0.02%     
==========================================
  Files         163      163              
  Lines       50113    50114       +1     
==========================================
- Hits        45723    45714       -9     
- Misses       4390     4400      +10
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.03% <33.33%> (-0.01%)` | :arrow_down: |
| #single | `40.31% <33.33%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `69.19% <33.33%> (-0.3%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.75% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=footer). Last update [77b4bb3...ec9cc9a](https://codecov.io/gh/pandas-dev/pandas/pull/17933?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback Ok so the more I dig into this issue, the less confident I am that this is a worthwhile change.  The issue was raised due to an error raised when a url passed to `read_json` contained whitespace.  We could encode the url but that might be a bad idea since there is no way to know when a reserved character is used for its reserved purpose and we run the risk of doubly encoding a url.  For example, if we are passed
```
http://example.com/to/json?key=some?value
```
we run the risk of not encoding the `?` in the query part of the url.

In addition, [`quote` is intended for quoting the path
    section of a URL](https://github.com/python/cpython/blob/master/Lib/urllib/parse.py#L762-L766), further reinforcing the feeling that trying to encode the entire url, including the scheme and query, is something that we shouldn't be doing.

Perhaps a better approach would be to add a note to the docs stating that properly encoding the url is the responsibility of the user.  I would be happy to open a PR if you think that would be a good idea.
actually that sounds like a good idea to add to docs and the doc-strings of all the read_* functions that take urls (most of them)
Please remove me from the list

On Sat, Oct 21, 2017 at 9:07 AM Grant Cooksey <notifications@github.com>
wrote:

>
>    - closes #xxxx
>    - tests added / passed
>    - passes git diff upstream/master -u -- "*.py" | flake8 --diff
>    - whatsnew entry
>
> Url is passed through quote before doing the request. The safe parameter
> characters are the reserved character set defined in RFC 2396
> <https://www.ietf.org/rfc/rfc2396.txt>(See section 2.2).
> ------------------------------
> You can view, comment on, or merge this pull request online at:
>
>   https://github.com/pandas-dev/pandas/pull/17933
> Commit Summary
>
>    - BUG: Gracefully handle all utf-8 characters in urls GH17918
>
> File Changes
>
>    - *M* doc/source/whatsnew/v0.21.0.txt
>    <https://github.com/pandas-dev/pandas/pull/17933/files#diff-0> (1)
>    - *M* pandas/io/common.py
>    <https://github.com/pandas-dev/pandas/pull/17933/files#diff-1> (3)
>    - *M* pandas/tests/io/json/test_pandas.py
>    <https://github.com/pandas-dev/pandas/pull/17933/files#diff-2> (10)
>
> Patch Links:
>
>    - https://github.com/pandas-dev/pandas/pull/17933.patch
>    - https://github.com/pandas-dev/pandas/pull/17933.diff
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/17933>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAJfa3JzCLqUaiLmYH-0Y6N7Ql2b7r_Xks5suZgqgaJpZM4QBdko>
> .
>
-- 
Greg Gurevich
greggurevich@gmail.com
cell: 1-917-504-7105

@jreback Ok I will work on that and close this PR.  Thanks!