wrap urlopen with requests
State: closed

- [X] closes #16716
- [ ] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Alternative to https://github.com/pandas-dev/pandas/pull/17087

I'll write the whatsnew entry, tests, and the session option for `html`, `json`, and `excel` if this path is OK with the devs.

Ping @skynss who is the original author of #17087 and @gfyoung who reviewed it.

Note that the main difference in this approach is that I made the option slightly simpler by allowing only the `session` instead of `http_params`.

You can see an example of this in action in:

http://nbviewer.jupyter.org/urls/gist.githubusercontent.com/ocefpaf/d6e9ab2c3569ff8fa181fc7885b6524d/raw/5d868fd4cbe2b81f84ccab7760b80251ef6e4651/pandas_test.ipynb

Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21504?src=pr&el=h1) Report
> Merging [#21504](https://codecov.io/gh/pandas-dev/pandas/pull/21504?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/e5c90e5e355077c4ee5245a4a72833f4e8af72b9?src=pr&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `26.47%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21504/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/21504?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #21504      +/-   ##
==========================================
- Coverage   92.29%   92.01%   -0.29%     
==========================================
  Files         161      169       +8     
  Lines       51497    50797     -700     
==========================================
- Hits        47530    46740     -790     
- Misses       3967     4057      +90
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.42% <26.47%> (-0.27%)` | :arrow_down: |
| #single | `42.21% <14.7%> (-0.22%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21504?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/21504/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `92.47% <ø> (-0.63%)` | :arrow_down: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/21504/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.48% <100%> (+0.18%)` | :arrow_up: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/21504/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `66.78% <4.16%> (-6.08%)` | :arrow_down: |
| [pandas/io/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/21504/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2h0bWwucHk=) | `89.48% <77.77%> (-1.72%)` | :arrow_down: |
| [pandas/io/parquet.py](https://codecov.io/gh/pandas-dev/pandas/pull/21504/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnF1ZXQucHk=) | `71.79% <0%> (-12.83%)` | :arrow_down: |
| [pandas/io/formats/console.py](https://codecov.io/gh/pandas-dev/pandas/pull/21504/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvY29uc29sZS5weQ==) | `65.15% <0%> (-10.61%)` | :arrow_down: |
| [pandas/core/arrays/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/21504/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2Jhc2UucHk=) | `88% <0%> (-9.36%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/21504/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `88.81% <0%> (-8.87%)` | :arrow_down: |
| [pandas/errors/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/21504/diff?src=pr&el=tree#diff-cGFuZGFzL2Vycm9ycy9fX2luaXRfXy5weQ==) | `92.3% <0%> (-7.7%)` | :arrow_down: |
| [pandas/core/dtypes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/21504/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Jhc2UucHk=) | `92.68% <0%> (-7.32%)` | :arrow_down: |
| ... and [138 more](https://codecov.io/gh/pandas-dev/pandas/pull/21504/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21504?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21504?src=pr&el=footer). Last update [e5c90e5...b46dd04](https://codecov.io/gh/pandas-dev/pandas/pull/21504?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Tested this on a username/password protected CSV endpoint and it's working!
```python
import requests
session = requests.sessions.Session()
session.auth = ('Nope', 'NoWay!')
url = 'https://cgoms.coas.oregonstate.edu/erddap/tabledap/CE01ISSM-BUOY-001-GPS.csv'

import pandas as pd
df = pd.read_csv(url, session=session)
```
Hello @ocefpaf! Thanks for updating the PR.











Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on August 25, 2018 at 15:30 Hours UTC
@ocefpaf can you address latest comments and merge in changes from master?
> @ocefpaf can you address latest comments and merge in changes from master?

I'm in transit now but I'll have some time next week to check this. However, if someone wants to take it over please do.

I do recall asking about a documenting strategy to avoid the repetition and the comments above. It would seems that to address some of the comments I would touch current docs, and I'm not really comfortable with that. I'll re-visit this soon...
Can you merge master?
> Can you merge master?

Done! But I'm not sure of I got it right, git rebases are not my forte :grimacing: 

BTW I'm happy to iterate on the docs questions in https://github.com/pandas-dev/pandas/pull/21504/files#r196079159 but I'm quite confused where I should start (it is my first time contributing here).
You are better off doing the following on your branch locally:

```bash
git fetch upstream
git merge upstream/master
```

That retains the history of comments on GitHub so makes it easier to track.

Looks like you have some test failures with the latest push - if you could take a look would be great
> You are better off doing the following on your branch locally:
> 
> ```shell
> git fetch upstream
> git merge upstream/master
> ```

That is what I did :-/

> Looks like you have some test failures with the latest push - if you could take a look would be great

I'll take a look ASAP.
> That is what I did :-/

Hmm OK. Saw the force push here so that's why I thought you had done something else. Shouldn't need the `-f` flag when you do it this way
@ocefpaf CI is red, can you fix any problem, and push again so there are no tests failures.
> @ocefpaf CI is red, can you fix any problem, and push again so there are no tests failures.

I believe that when I first submitted this PR the [`with-statement` wrapper](https://github.com/pandas-dev/pandas/pull/21504/files#r195869676) was not used or untested, that is why it passed. I tried to fix that but my Python knowledge and time is limited to do it before the end of the year. If anyone has an idea please let me know. (IMHO I would remove the wrapper b/c it will be removed anyway when pandas drop py27.)
Not sure what the problem is. I'll close this for now, as we can't merge with the conflicts and the CI in red, but if you find a solution for what you said, feel free to reopen.
> Not sure what the problem is. I'll close this for now, as we can't merge with the conflicts and the CI in red, but if you find a solution for what you said, feel free to reopen.

Would you be interested in a Python 3 only version of this PR? That is working and I guess that all the Python 2.7 code will be removed soon anyway.
@ocefpaf we drop Py2 support for 0.25 so sure I think Python 3 only is fine for that milestone onwards