Use `python-requests` library, if installed, to support broad http(s) scenarios including Basic Authentication
State: closed

Use `python-requests` ([Python HTTP Requests for Humans](https://github.com/requests/requests)) , if installed, to enable additional scenarios as Basic Authentication, permit self-signed SSL certs, and other advanced scenarios.

A new optional parameter  `url_params` is added into `read_csv`, `read_html`, `read_excel` and `read_json` to allow passing in either a dict containing authentication, etc or even a requests session. It also allows passing in basic auth in url: http://<usr>:<pwd>@<fqdn>:<port>/  and fixes #Issue #16910 Also, better handling of specified `:<port>` in url.

If `python-requests` library is not installed, existing behavior prior to this check-in should continue as-is, which directly calls `urllib\urllib3`

 - [x] closes #16910, #16716
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master -u -- "*.py" | flake8 --diff``
 - [x] whatsnew entry


Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv

Comments:
The `whatsnew` is outdated and wrong. I will push it out later today.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=h1) Report
> Merging [#17087](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/f9a552dc12262d1d208f9cdc2c5ffd1731f9c361?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `70.58%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17087/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17087      +/-   ##
==========================================
- Coverage   91.02%   90.99%   -0.04%     
==========================================
  Files         161      161              
  Lines       49396    49434      +38     
==========================================
+ Hits        44965    44984      +19     
- Misses       4431     4450      +19
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.77% <70.58%> (-0.02%)` | :arrow_down: |
| #single | `40.27% <27.45%> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.44% <100%> (ø)` | :arrow_up: |
| [pandas/io/excel.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsLnB5) | `80.34% <57.14%> (-0.21%)` | :arrow_down: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `68.51% <69.44%> (+0.72%)` | :arrow_up: |
| [pandas/io/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=tree#diff-cGFuZGFzL2lvL2h0bWwucHk=) | `85.17% <85.71%> (+0.31%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.66% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=footer). Last update [f9a552d...0b1a10f](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=h1) Report
> Merging [#17087](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c3c04e266cbc5e176cc6ef4dc385cdd88fda0669?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `65.3%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17087/graphs/tree.svg?token=eZ4WkYLtcO&width=650&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17087      +/-   ##
==========================================
- Coverage   91.58%   91.54%   -0.04%     
==========================================
  Files         153      153              
  Lines       51250    51288      +38     
==========================================
+ Hits        46935    46952      +17     
- Misses       4315     4336      +21
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.41% <65.3%> (-0.02%)` | :arrow_down: |
| #single | `40.66% <24.48%> (-0.13%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `91.75% <ø> (ø)` | :arrow_up: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.55% <100%> (ø)` | :arrow_up: |
| [pandas/io/excel.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsLnB5) | `89.85% <40%> (-0.28%)` | :arrow_down: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `69.25% <63.88%> (-0.24%)` | :arrow_down: |
| [pandas/io/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2h0bWwucHk=) | `86.29% <85.71%> (+0.31%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17087/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.81% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=footer). Last update [c3c04e2...2e34854](https://codecov.io/gh/pandas-dev/pandas/pull/17087?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

so let's make this really simple. If auth (and I would use ``auth=`` instead of ``url_params`` I don't find that intuitive. Use request, if ``auth`` is passed and not ``None`` and ``requests`` is not installed raise. Otherwise use existing code path. Let's do this first, then you can add fancier things.
@jreback 
- please see comment [here](https://github.com/pandas-dev/pandas/pull/16910#issuecomment-317070708) on why I proposed we use 'url_params' instead of 'auth'
- please see comment [here](https://github.com/pandas-dev/pandas/pull/16910#issuecomment-317093902) on why @gfyoung agreed with me to use 'url_params' instead of 'auth'

Only after I got that feedback from @gfyoung  did I implement `url_params`. Now if we are going back to suggesting `auth` only, unfortunately, I fail to understand why that is better, I have the exact same questions as I have raised previously. 

While I would like to help the project, I lack bandwidth on my end that this async conversation requires. And perhaps it might be best that I just hand what is here to you fine folks to take this forward. I have provided you multiple working solutions which you can choose to pick and modify (or just discard them):
1. `auth` and `verify_ssl` using only urllib\urllib3 + passing live test cases (previous fork)
2. `auth` and `verify_ssl` using using requests when installed + passing live test cases (previous fork)
3. `url_params` using requests, if installed, while keep using urllib in existing scenarios if `url_params` is not provided + passing live test cases (this fork)
In my benevolent and sincere view, the current check-in in this pull request is a really good/working/passing solution that I personally do not see any issues with - and as a consumer, it would work for me.

If after reviewing all comments in previous pull request, and this pull request, if there are any minor modifications I can make that seem reasonable, I would be happy to do. Please let me know.. else, my apologies that I am unable to contribute to the level of depth that you folks need at the current time.. and request you folks to do what is best. 
@skynss 

> so let's make this really simple. If auth (and I would use auth= instead of url_params I don't find that intuitive. Use request, if auth is passed and not None and requests is not installed raise. Otherwise use existing code path. Let's do this first, then you can add fancier things.

Here is my previous comment. I am not sure why you are going back to the previous conversation.I think my comment is quite clear.

 I suggested you do a more limited PR that *only* implements using requests if ``auth`` (instead of the name ``url_parms``) is passed. That's it. That gets us a lot of the way w/o any complication about falling back or anything else.
@jreback 
Can you clarify what would the api call look in your implementation where I need to perform basic auth and need to be able to bypass ssl verification?
```python
url = 'https://aaa.com/t.csv'
df = pd.read_csv(url, url_params = { 'auth':('uname', 'pwd'), 'verify': False}) # Option 1: current check-in
df = pd.read_csv(url, auth=('uname', 'pwd'), verify_ssl= False) # Option 2: previous fork/pull request
df = pd.read_csv(url, auth=('uname', 'pwd')) # Option 3: New suggestion that dont even handle scenario allowing self signed certs
``` 

option 3

> df = pd.read_csv(url, auth=('uname', 'pwd')) # Option 3: New suggestion that dont even handle scenario allowing self signed certs

I just don't want more keywords in all of the ``read_*`` calls. (and I don't particulaly like``url_params`` as a kw). ``auth`` is fine. ``verify_ssl`` if you really really need it.

> I just don't want more keywords in all of the read_* calls. (and I don't particulaly likeurl_params as a kw). auth is fine. verify_ssl if you really really need it

Almost no-one does basic auth over http (without ssl). Getting officially verified ssl certs is difficult especially in private environments and testing scenarios. So in those scenarios, the compromise is to use self-signed ssl certs or expired certs or certs whose domains dont match in those cases, and it becomes essential to bypass ssl cert verification to be able to test in that scenario. That is why requests library has a parameter `verify` (with a warning logged every time such request is called). 

None of us want to add additional params to `read_csv` so the suggestion that use `url_params` for anything `url` related.. be it auth or ssl verification (today) or potential future user asks of changing headers, or timeouts, or proxies, or any other stuff that is http(s) related. That way `read_csv` params never expand beyond `url_params` for http(s) related params.  If a new issue is filed in pandas to allow override of `http timeout`, or allow `proxy server` would we add new parameter to read_csv( timeout=5000, http_proxy=['my.proxy.com'])? Or does it make sense to just add them to 'url_params' along with auth, ssl, etc (or pass in a requests.Session())? 

I often read data that requires http timeout to be non-default. I would like to use the callback functionality that requests library provides to see at times "_what the heck is going on in the http layer and capture it_". If I wont be able to set this within pandas, then for me, its just easy to leave anything http (auth/ssl/timeout/callback/proxy etc) related out of pandas as a standard practice, first use requests separately and then use StringIO with read_csv. The currently checked code with `url_params` would allow me to "opt-in" to handle every scenario mentioned above that requests supports, while leaving current http flow unaffected.

If there is no plan for supporting above mentioned http scenarios, why even take a requests soft dependency? There is already a checked in solution with basic auth, that was already rejected (in previous fork) that worked with urllib/urllib3.
@skynss I DO like the idea of only adding 1 new parameter (to basically all read_* routines). I object to it being called ``url_parms``. maybe ``auth_params`` is better or something similar. As I said a PR that:

- adds 1 new parameter (with a dict of a well-defined structure of keys)
- adds requests to handle this paramaater (and raises if its not None and requests is not installed)
- leaves all existing functionaility alone

would be fine. let's do step-by-step and simpler is better.
> I DO like the idea of only adding 1 new parameter (to basically all read_* routines). I object to it being called url_parms. maybe auth_params is better or something similar

@jreback This PR already meets all your new requirements (except for renaming of `url_params` - perhaps they can be renamed to `http_params` because only http(s) is handled or `req_params` because we pass thru this to requests).  We shouldn't call them `auth_params` because they do and will contain non-auth related parameters such as `verify`. SSL certificate verification is unrelated to authentication.

The dict is already well defined (and very short): It can contain `auth` or `verify` and these parameters are passthru to similarly named parameters in `requests`

There is one exception in this PR that doesnt meet your requirement: `url_params` can be one of three:
* `None` -- per your requirement
* `dict` containing `auth` or `verify`  -- per your requirement. If anything else is passed, its just ignored but used as signal to force use of `requests` library.. which is actually desirable as it enables uses of `requests` in existing http scenarios.
* a `requests.Session()` object. If it is passed passed, we basically use this session instead of creating our own new `requests.Session()` object. I implemented it this way because this does not interfere with other requirements, but allows a knowledgeable user to really configure requests session object  without requiring pandas feature requests: `Custom Auth`, `proxies`, `timeout`, `callback urls`, etc. 

So, after this PR merge, if I want to implement `timeout` in my `read_csv`, even though it is not a feature implemented by pandas, I can still do the following:
```python
from requests import Session
s = Session()
s.timeout = (1.0, 3.5)
df = pd.read_csv( url, url_params=s) # great... now my read_csv will timeout as specified.
```
So far, I cannot see of any disadvantage of allowing passing of `requests.Session()`. I see the advantage of no longer having to play catch-up with changes from requests library, and moving the focus of pandas team from `io` related issues to `core pandas` related issues - which everyone really values pandas for.
great if u can change to http_params i'll take a look tom
> great if u can change to http_params i'll take a look tom

@jreback checked in.
this is not being tested on travis; rather all of the new code is skipped. you need to add requests to the pandas/ci/requirements-* files. But only add to 1 for 3.6 and 1 for 2.7 (of the main jobs).
@skynss if you want to rebase / update can take a look.
@jreback m trying to make time but just not getting enough time to the final fit and finish. I had thought i had checked in the correct solution with passing tests and rebased. What timeline is it for next milestone? If someone else wants to pick up you are welcome.. it might be few weeks for me. It would also help if i know what the outstanding things remain. There are lots of comments.
@skynss 0.21.0 is soon, we just released the rc for it. I'd like this to be merged in master for a while, so not a problem to do for the next milestone. This is a nice PR, but still need to test in the broader community after merge.
pls rebase / update when you can
@skynss , we would love to see this capability in pandas.   Are you still working on this?
@rsignell-usgs I have been too occupied with my work.. but will try to see if I can. I am not a routine contributor, so the logistics of submitting is whats timeconsuming, not the actual fix.
@jreback @gfyoung which branch should I rebase to? master or 0.21.x? Is it better to just close this pull request and create new one based on new branch so we don't have to filter thru all the prior comments?
@skynss : Keep the PR as is and rebase onto `master`.  We definitely want to merge this!
@jreback @gfyoung 
Can you take a look at following code of simple test case - it works in python2 but fails in python3

```python
import pandas.util.testing as tm

from urllib3.exceptions import InsecureRequestWarning
import requests

def test_this():
    # succeeds in PY2, fails in PY3
    with tm.assert_produces_warning(InsecureRequestWarning):
        r = requests.get('https://cnn.com/', verify=False)

```
This is the only issue remaining.
Also please review the code. Next steps?
needs a rebase
Set skip human verification set and lounch in my com
closing as stale. would like to have this, if one of the core wanted to pick this up, or if @skynss can rebase. pls ping to reopen.
@skynss are you still interested in this? If not, do you mind if I try to re-submitted this PR?
@ocefpaf : It's been inactive for some time now.  Go for it!
thanks @gfyoung 
@ocefpaf  please go ahead. I am just not getting any time with my work. Thx.
The last of mine commit worked on python2. But python3 failed for no apparent reason.. see my comment above on December 12. 2017. I think python3 failure had nothing to do with this pull, but with some other pandas issue at that time.