BUG, ENH: Read Data From Password-Protected URL's and allow self signed SSL certs
State: closed

 - [x] closes #16716
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master --name-only -- '*.py' | flake8 --diff``
 - [x] whatsnew entry

Labels:
name: Bug
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
@skynss : In addition to failing your own test, you seemed to have broken several others because the file paths on some of the tests didn't work anymore.  Could you run tests locally on your machine to confirm and investigate this?
@skynss : whoa, what happened to `test_common.py` ?  Why did you delete everything there?
@gfyoung Sorry I had made a silly mistake. I ran the tests that I changed and they seem to work. I cannot run all tests because of my current environment (ssh does not allow for Displayable tests to run)
The issue was I had wrongfully overwritten `python/tests/test_common.py` with `python/tests/io/test_common.py` originally and submitted it - which obviously caused tests to fail. I think I have corrected the mistake.  I am a bit new at this process.
@skynss : No worries!  No one's going to get mad if you make mistakes like this, especially since you are very new to the process :smile: 

I would focus right now on trying to address all of our comments as well as you can and ensuring that tests do not fail.  Make as many commits as you need to get that done.
@gfyoung  I think I have addressed all issues (except the whatsnew documentation issue which I am waiting for final recommendation from you folks) 
@skynss : Nice!  Well done.  Now that things are a little greener on the CI side, why don't you now tackle all of the style errors pointed out on Travis (starting right <a href="https://travis-ci.org/pandas-dev/pandas/jobs/253451474#L1777">here</a>).  This is good progress from where it was when you initially submitted it for review.
@gfyoung I looked at logs for "Travis CI build failed" It seems to have 8 failures, and to me it appeared like they did not appear to be related to my check-ins. Now, it is possible I accidentally did something else wrong (not that I am aware of) . Almost all of them have nothing to do with /pandas/io - there are couple of test_excel but these seem to be issues with using other libraries unrelated to the changes I made.  If possible, please take a look. Besides this issue, do I need to do anything else?
@skynss : Ah, I think I see your confusion.  "XFAIL" means that the tests are expected to fail and will therefore not break the build if they don't pass.  Don't worry about those!

You have no blocking test failures on that build.  You have *style errors*, meaning that your code violated style errors that we expect to be followed before merging any code.  Starting at line 1778, they should be listed there for you.  You can check them yourself by running `flake8` on any of the listed files in the build:

~~~
flake8 <filepath>
~~~
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=h1) Report
> Merging [#16910](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4ca9fcd73225af9cb1dc2acc99bb494dc5f8926a?src=pr&el=desc) will **decrease** coverage by `0.05%`.
> The diff coverage is `46%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16910/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16910      +/-   ##
==========================================
- Coverage   90.99%   90.93%   -0.06%     
==========================================
  Files         161      161              
  Lines       49303    49344      +41     
==========================================
+ Hits        44863    44871       +8     
- Misses       4440     4473      +33
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.7% <46%> (-0.04%)` | :arrow_down: |
| #single | `40.17% <18%> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.44% <100%> (ø)` | :arrow_up: |
| [pandas/io/excel.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsLnB5) | `80.32% <25%> (-0.24%)` | :arrow_down: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `64.07% <38.88%> (-3.73%)` | :arrow_down: |
| [pandas/io/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL2h0bWwucHk=) | `84.68% <75%> (-0.18%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.71% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=footer). Last update [4ca9fcd...437e0a2](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=h1) Report
> Merging [#16910](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4ca9fcd73225af9cb1dc2acc99bb494dc5f8926a?src=pr&el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `46%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16910/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16910      +/-   ##
==========================================
- Coverage   90.99%   90.93%   -0.07%     
==========================================
  Files         161      161              
  Lines       49303    49344      +41     
==========================================
+ Hits        44863    44870       +7     
- Misses       4440     4474      +34
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.7% <46%> (-0.05%)` | :arrow_down: |
| #single | `40.17% <18%> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.44% <100%> (ø)` | :arrow_up: |
| [pandas/io/excel.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsLnB5) | `80.32% <25%> (-0.24%)` | :arrow_down: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `64.07% <38.88%> (-3.73%)` | :arrow_down: |
| [pandas/io/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL2h0bWwucHk=) | `84.68% <75%> (-0.18%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.71% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.23% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=footer). Last update [4ca9fcd...f520f7b](https://codecov.io/gh/pandas-dev/pandas/pull/16910?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

dask added quite a bit of machinery to tests things like this, so might be worth a look. https://github.com/dask/distributed/blob/ea2405e541551b694b0fa095ef1da81d1cfb383e/distributed/tests/test_security.py
@skynss : Your last commit says you added a test case, but I don't see it?
Hello @skynss! Thanks for updating the PR.













 - In the file [`pandas/tests/io/test_http_auth.py`](https://github.com/pandas-dev/pandas/blob/f520f7b9050186fe84301508daec27e310010049/pandas/tests/io/test_http_auth.py), following are the PEP8 issues :

> [Line 7:1](https://github.com/pandas-dev/pandas/blob/f520f7b9050186fe84301508daec27e310010049/pandas/tests/io/test_http_auth.py#L7): [E722](https://duckduckgo.com/?q=pep8%20E722) do not use bare except'



##### Comment last updated on July 20, 2017 at 03:45 Hours UTC
@gfyoung  @jreback  Sorry.. was booked past couple of days. I just did the following:
1. Checked in the test case. `pandas/tests/io/test_http_auth.py` I didnt add it to `test_common.py` until I got opinion from you folks. I will elaborate the testability a bit more later. 
2. I decided to add a `warning` when disabling SSL cert strict checking.. just as `requests` rightfully does. I need to figure out how to handle the warning in pytest as expected. 
3. I fixed `whatsnew`. If you want me to elaborate, can you tell me which section of `whatsnew` the elaboration should go under?
4. I am aware latest test broke pep8 as I edited the string a bit and forgot to run the pep8 check. Ill fix it. 
5. I did strict checking of `uname` and `pwd` should either both be present or neither should be. 

Is there anything else that is needed?
> Checked in the test case. pandas/tests/io/test_http_auth.py I didnt add it to test_common.py until I got opinion from you folks. I will elaborate the testability a bit more later.

Okay, so I think you need to break these tests up because this will not be easy to read:

1) I'm not even sure if we should be hitting those endpoints during our tests (@jreback ?).

2) Divide up the tests between those that use SSL verification and those that don't.  Also divide based on what type of authentication is used.

3) It would be preferable that we also check the contents of the returned object.  That means also breaking up the tests by which function is called (i.e. `pd.read_csv`, `pd.read_json`, etc.)

Start with 1 and 2 first.  We can address 3 afterwards.
**Testing**:
I added `pandas/tests/io/test_http_auth.py` - let me know if this should be merged into `pandas/tests/io/test_common.py` or keep it in a separate file.
I can break it into separate tests... but I am not sure what is wrong with keeping it as it is now (except maybe adding better assert label)? It follows the pytest `parameterize` as you suggested and generates 12 separate tests. 

A bit more about the test: It utilizes a free hosting service. So it should continue to work (for the foreseeable future). It works well except this http server doesnt like port numbers being specified (see bug #[17019](https://github.com/pandas-dev/pandas/issues/17019) )

I am proposing adding live cases, because it is easier to test valid SSL certs.. and I saw some tests already use external urls `pandas/tests/io/test_html.py`
@skynss if we used ``requests`` as an optional dep to handle url like things would pandas code be much simpler? (e.g. we could still fall back to the current path now, maybe eventually removing that code and fully rely on requests for url things though).
> if we used requests as an optional dep to handle url like things would `pandas` code be much simpler?

@jreback : Yes, absolutely.  We wouldn't need to setup any authentication, as `requests` would handle that for us.  You can look at the code examples from the issue to see the difference.
i would make this an optional dep then 
> i would make this an optional dep then

Makes sense.  Probably should first clean-up the PR as is before adding this in.
@jreback  @gfyoung  wrt `requests` as optional dependency, what do we expect as end user experience?

Here are few options to and their advantages disadvantages: 
**Option 1**: Add [`requests.Session()`](http://docs.python-requests.org/en/master/user/advanced/) as a optional param to `read_csv( url, req_session=my_session)` or `read_csv( url, use_requests_lib=True)`- in which case, the user is knowingly opting to bypass urllib from pandas and use requests instead. If `read_csv( url, username='darth', password='mtfbwy', verify_ssl=False, req_session=my_session)`, only if my_session is valid or `use_requests_lib=True` is passed, is requests library used, else urllib is used even if `requests` package is already installed. 
Advantage: user has greater control. Passing requests session might enable power user functionality such as accessing AWS API gateway with token, and other scenarios. 
Disadvantage: Existing code stays.. Add new code to optionally handle requests

**Option 2**: Auto-detect and use `requests` package if installed, and use `urllib` if not installed. So end user would possibly call `read_csv( url, username='darth', password='mtfbwy', verify_ssl=False)` and has no control over whether `urllib` is used or `requests` is used. 
Advantage: `read_csv` api is cleaner as it doesnt have additional parameter of `use_requests_lib`. 
Disadvantage: User may notice inconsistent behavior in different environments without warning - and might be very hard to detect the cause is the presence or absence of `requests` package. Example: Imagine AWS Lambda does not have `requests` package installed in their python runtime.. and I upload a Lambda python function with locally packaged pandas v.0.20 in single zip. AWS decides that `requests` is common ask and includes it in their runtime. The behavior of my lambda function might change without me having any clue of why.

**Option 3**: Forget adding `username`, `password`, `verify_ssl` to `read_csv` and just add `req_session` parameter to `read_csv` instead. Expect end user to know how to build a requests session with username/pwd/verify set and pass that object to `read_csv`. 
Advantage: Cleaner and powerful `read_csv`. 
Disadvantage: Dev now has to know about auth/ssl using requests. Possibly, if end user is going to do this, it might just be easier to just utilize `requests` + `StringIO` and bypass the http functionality in `read_csv`

**Option 4**: Stop using urllib and switch with strong dependency to `requests`. 
Advantages: Simpler code 
Disadvantages: Potential regressions. Bulkier packages going up AWS Lambda. `read_csv` will still need to be able to accept a `req_session` object because there is no way to add all potential scenarios as parameters to `read_csv`

**Option 5**: Forget requests, allow user to just pass in urllib.request.Request object and possibly other headers etc directly to `read_csv`. 
Advantage: No regressions. No external dependency. Great for power user eg: change useragent string, etc. Use for other scenarios such as AWS API gateway authentication. Easy to code it in existing codebase. 
Disadvantage: For simple use cases, a novice user has to learn urllib.

I am not happy with **Option 2** as it will produce inconsistent behavior eg Issue #17019 and has greater chance of causing regression.
While **Option 3** is better than released version, I might as well consider getting data separately and passing in StringIO object. 

I dont recommend **Option4** or **Option 5** due to complexity esp in Py2/Py3 scenario.

I think **Option 1** is best but it be checked in 2 phases:
Phase 1: without requests i.e this issue.
Phase 2: allow  requests session as parameter. Have you folks observed lot of issues that `requests` might resolve? If yes, Phase 2 should be sooner rather than later. I think making the request call is probably easy to implement, but handling and testing various scenarios of response (mimetypes, gzip, multipart, redirections, chunked, various verbs, encoding) will probably need time and expertise. I'm bit too tight on bandwidth to dig deeper into this.
@skynss : The signature for `read_csv` is bloated enough as it is.  Username and password should be the only way to authenticate and is indeed the easiest.  Thus, I would be against option 1.

Auto-detecting would be the best way to incorporate `requests` behavior.  I agree that inconsistencies might be a concern, but that's bound to happen no matter how much you try to standardize.  That's why we're making it an optional dependency.  We would also need to document that potential issue so that users are aware of this.

I agree with you on option 3: username and password are the simplest hands down.
I agree with you on option 4: we are not relying on `requests` beyond being an optional dependency.

Option 5 is our backup.  We don't need to add `requests` support at the moment.  You could even do that as a follow-up.
the way to do this is:

- if no auth is provided use current behavior
- if auth is provide, import requests, if its not available, raise a nice error message that auth requires requests
- use request to service the request.

the idea is that we can push as much code as possible to use requests, rather that re-invent the wheel
@jreback Why push only auth scenarios to requests if installed? Instead, if requests is installed, use it by default for all http(s) cases. If not installed, use existing codepath - as-is?

Also, instead of passing in `username`, `pwd`, `verify_ssl`, it is probably easier to instead just pass in optional only `requests.session` object which accepts not only auth but also enables a ton of other scenarios such as  adding `proxies`, `client certs`, `useragent` changes, and `custom auth` scenarios such as 'Calling [AWS api-gateway with cognito](https://stackoverflow.com/questions/37336286/how-do-i-call-an-api-gateway-with-cognito-credentials-in-python) (just googled a sample scenario)'

Also at a quick glance, verb other than `GET` such as `POST`, etc might be [automatically supported using prepared requests](http://docs.python-requests.org/en/master/user/advanced/#prepared-requests)

Right now, I don't see how pandas returns status code, and response headers. By utilizing requests `event hooks` functionality built into requests, this can be handled.

```python
import pandas as pd
from requests import Session

# req_session is optional and replaces username, password, 
# by default, no requests.session needed. Backwards compatible api.
df = pd.read_csv('https://uname:pwd@aa.com/bb.csv') # will work automatically because 
# requests will handles it if installed, else handle with existing codebase

# custom auth
s = Session()
s.auth = MyAuthProvider('secret-key') # custom auth provider supported by requests
df = pd.read_csv(url, req_session=s)

# optional advanced scenarios
s = Session()
s.auth = ('darth', 'l0rd')  # if user wants to perform basic auth Skip if url itself contains username and pwd
s.timeout = (3.05, 27)                           # if user wants to modify timeout
s.verify = False                                      # if user wants to disable ssl cert verification
s.headers.update( {'User-Agent': 'Custom user agent'} )  # extensible to set any custom header needed
s.proxies = { 'http': 'http://a.com:100'}  # if user has proxies 
s.cert = '/path/client.cert'                     # if custom cert is needed
df = pd.read_csv( 'https://aa.com/bbb.csv', req_session=s)

# support verbs other than 'GET' such as 'POST'
r = Request('POST', 'http://joker:pwd@nlp_service.api/email_sentiment_extract?out=json')
prepped = req.prepare()
prepped.body = 'from: aaa@aa.bb\nto: cc@dd.ee\nsubject:Complaint letter\n\nbody: I am feeling :(' # multiple lines
df = pd.read_json( prepped) # minor update pandas code to detect type(Request) and submit it using requests session in lieu of URL.
"""
[{
  'from': 'aaa@aa.bb',
  'to': 'cc@dd.ee',
  'email_type': 'complaint',
  'sentiment': 'unhappy',
}]
"""

# Event hooks callback (eg log http status codes)
def print_http_status(r, *args, **kwargs):
    print(r.status_code)
    print(r.headers['Content-Length'])
s = Session()
s.hooks = dict(response=print_http_status)
df = pd.read_csv( 'https://aa.com/bbb.csv', req_session=s)

```
It does mean that some of the work I did becomes unnecessary, but if you are ok to use requests, I think the most extensible and simplest scenario is to allow passing in optional request.session parameter.

For the above scenarios, following changes would need to be made to code checked into `master`:
1. `read_*` accept new optional parameter `req_session`, and detect if 'url' is passed or 'prepared' is.
2. if requests installed, use url/prepared/req_session as specified. Call session.send()
3. if requests not installed, don't change current behavior of `read_*`
4. Testing: Don't have to test stuff that requests already tests.

I am far from being an expert at `requests` library and encourage others to review and correct me.

EDIT: I think I might already have this mostly working (except for read_html)
@jreback  @gfyoung : Take a look at a new branch [`use-requests`](https://github.com/skynss/pandas/tree/use-requests) . Its a completely different codebase which uses requests

- If installed uses `python-requests` by default.. else use existing codebase of `urlopen`
- only one new optional param of `req_session` added in lieu of username/pwd, etc
- fully working
- all pytest unit tests passed in `pandas/tests/io` - only thing remaining is to figure out how to add test case to emulate `python-requests` is not installed?
- whatsnew updated
- codebase rebased with pandas-dev/pandas master as of few mins ago. 

It is probably ready to be merged in. please review. let me know if you want me to start a different pull request. 
> the idea is that we can push as much code as possible to use requests, rather that re-invent the wheel

@jreback : I wouldn't say we're re-inventing the wheel.  It's rather than `requests` came up with a nice way to provide an even higher-level interface to Python `urllib` libraries.  That being said, I'm perfectly fine with using `requests` so long as we're okay with making it an optional dependency that is *required* when authentication with data IO is needed.
> Why push only auth scenarios to requests if installed? Instead, if requests is installed, use it by default for all http(s) cases. If not installed, use existing codepath - as-is?

@skynss You mentioned before that you were concerned about inconsistencies between `requests` and `urllib`.  If we were to use `requests` (at least for now) *solely* for authentication, we would not have to worry about this issue.  Not to mention, it will make things easier to test for the time being.
> Also, instead of passing in username, pwd, verify_ssl, it is probably easier to instead just pass in optional only requests.session object which accepts not only auth but also enables a ton of other scenarios

@skynss : That's a bit of reversal since your comment earlier <a href="https://github.com/pandas-dev/pandas/pull/16910#issuecomment-316528721">here</a>.  That being said, I'm still wary about allowing a user to pass this in.

Since it seems we are moving to incorporate `requests` into `pandas`, IMO we should keep the interface relatively agnostic to the user i.e., instead of accepting a session object, we accept an `auth` parameter, which is a dictionary mapping to different types of authentication that we could accept such as username / password with verify_ssl or a session object.

This interface allows us to have the flexibility of implementing whatever handling we would like in terms of authentication in the future WITHOUT impacting how the user has to interact with it.  Internally, we would check if certain fields existed and utilize them for authentication (e.g. `if 'session' in auth`, then use the session to authenticate).
so the current behavior must work w/o requests installed at all.

I would push to have as minimal code added as possible, IOW use requests for the auth and just use the current code for existing things; that said if you want to make a mock requests internally then this might be simpler.
@gfyoung 

> IMO we should keep the interface relatively agnostic to the user i.e., instead of accepting a session object, we accept an auth parameter, which is a dictionary mapping to different types of authentication that we could accept such as username / password with verify_ssl or a session object.

Can you clarify?
```python
# for basic auth
df = pd.read_csv(url, auth={ 'uname':'aa', 'pwd':'bb'})

# to bypass ssl cert verification (which is not authentication) it would be confusing to pass it into auth param
df = pd.read_csv(url, auth={ 'verify_ssl':False})
```
Perhaps what you meant (and what I think is good idea) was you prefer end user to not make any imports of `requests` but allow to pass in dict or requests.Session such as following:
```python
# simple user never has to import requests
up = {  'auth' : ('user','pwd'), 'verify_ssl' : False } 
df = pd.read_csv( url, url_params=up)

# or power user can choose to use request.Session directly
import requests
up = requests.Session()
up.auth = ('uname','pwd')
up.header.modify( {'User-Agent' : 'My custom user agent'})
df = pd.read_csv( url, url_params=up)

# internally we check 
if type(url_params) is requests.Session:
    sess = url_params
elif type(url_params) is dict:
    sess = requests.Session()
    # then we add in auth/verify_ssl in here
```

@jreback 

> so the current behavior must work w/o requests installed at all

Agreed. Both [basic-auth-https-self-signed](https://github.com/skynss/pandas/tree/basic-auth-https-self-signed)  as well as [use-requests](https://github.com/skynss/pandas/tree/use-requests) already satisfy them.

> use requests for the auth and just use the current code for existing things

If we just want to use requests for auth (and I assume you meant ssl cert verification bypass too) then why are we even having a dependency on requests? First option of `basic-auth-https-self-signed` already does that, and its tested. 

[use-requests](https://github.com/skynss/pandas/tree/use-requests) has the least amount of code change ( compared to [basic-auth-https-self-signed](https://github.com/skynss/pandas/tree/basic-auth-https-self-signed) ). It uses requests for all scenarios, only if it is installed. This functionality is passing all `pytest` . Also, if requests is not installed, existing functionality continues to work as is. 

I got engaged as I opened an issue for basic auth + self signed certs.
Tomorrow someone might open an issue that pandas doesnt support today eg: proxies. Yet another issue might be setting custom headers, etc. 
Here is a bug in released version of pandas. This does not involve https, or auth of any kind. This is supposed to work
```python
df = pd.read_csv('http://handsome-equator.000webhostapp.com/no_auth/aaa.csv') # works fine
df = pd.read_csv('http://handsome-equator.000webhostapp.com:80/no_auth/aaa.csv') # fails with 404 if port number is added. it shouldnt. It doesnt in browser, or requests
```
It fails because `urllib` behavior. It works in [use-requests](https://github.com/skynss/pandas/tree/use-requests) if requests is installed and it is used even for existing scenarios when installed.

So, what is the direction for http(s) in pandas? Natively use `urllib` or move to `requests`? If the goal is to try to use requests the most we can, then what is the test acceptance criteria for trying to use `requests` most when available? [use-requests](https://github.com/skynss/pandas/tree/use-requests)  passes all tests. Besides api design, what else remains? 

Wish there was a bit more synchronous way to communicate, to hammer out the issues. faster