Requests v0.14.2 - requests.session() fails on post() with KeyError exception

Hello All,

I am a _big_ fan of Requests, and have just upgraded from v0.14.0 on Python 3.2.3 to v0.14.2 on Python 3.3.0. I think that the upgrade to Requests broke my code when using a `requests.session()` object. I use Fiddler for some transport debugging, which is why I'm using the loopback address on port 8888.

_Please note_ that everything works as expected if I use a `requests.post()` object instead of using a `requests.session()` object for the request (obviously bypassing the proxy).

My code looks as follows:

``` python
s = requests.session(proxies={'http': '127.0.0.1:8888'})
r = s.post(url, data=soap_req, headers=headers)
```

The call to `s.post()` results in a `KeyError: None` exception in `connection_from_host()` in `class PoolManager` in poolmanager.py. 

The offending line is: 

``` python
    pool_cls = pool_classes_by_scheme[scheme]
```

Debugging shows `pool_key = (None, '127.0.0.1', 8888)`

What puzzles me is that scheme shows up as _None_ (as shown in the 1st element in the tuple above). Debugging indeed verifies scheme is NoneType, which makes no sense, since it has a default value of 'http' when it enters the method.

I would appreciate anyone who can shed light and/or provide a fix for the above. I'm at the limit of my debugging skills. Please advise if you need more info from my side for further troubleshooting.

Thanks in advance,

Jean-Marc


I took a quick look at this before I left for work this morning and couldn't reproduce it. I spent literally 5 minutes on it though, so I suspect it's real. =)

It shouldn't happen though, Requests creates sessions under the hood when you use the verb methods. If you're familiar with how to use `PDB`, it'd be interesting to see what's going on around [line 555 of models.py](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L555). If you don't get around to it today I'll take a look when I get back from work, but that won't be until like 6pm GMT.


Lukasa,

Appreciate the feedback. 

I did some debugging around line 555 (I'm using Wing, so I used breakpoints to examine state etc).

`_p` is of type `ParseResult` and looks normal (ie. it has a correctly populated scheme, netloc and path).

`no_proxy` points to a `filter` object with a bunch of built-in methods (nothing obvious to inspect).

`proxy` contains `'127.0.0.1:8888'`

The code then steps into the line:

`if proxy and not any(map(_p.hostname.endswith, no_proxy)):`

and calls `get_connection_for_url()`

`if self.config.get('keep_alive'):` is `True` so it then calls

`conn = self._poolmanager.connection_from_url(url)`

which calls `connection_from_url()`

u = `Url(scheme=None, auth=None, host='127.0.0.1', port=8888, path=None, query=None, fragment=None)`

it then calls `self.connection_from_host()` with the following values

```
u.host = '127.0.0.1'
u.port = 8888
u.scheme = None
```

finally, the code enters `connection_from_host()` and since `u.scheme` is not undefined, it is not assigned the default value of `'http'`, which causes `pool_cls = pool_classes_by_scheme[scheme]` to fail a couple of lines further on.

Hopefully the above is enough detail for you to troubleshoot?

Appreciate the help!

JM


Lukasa,

All this points to the way in which `parse_url()` handles a missing scheme. I noted that if the `url` passed in doesn't contain a scheme, then this is set to `None`, which blows up a little later.

I went back and modded my code from

`s = requests.session(proxies={'http': '127.0.0.1:8888'})`

to

`s = requests.session(proxies={'http': 'http://127.0.0.1:8888'})`

and now all works as expected.

Based on the code in the Proxies section at http://docs.python-requests.org/en/latest/user/advanced/

I haven't done anything wrong, so can I assume that this is a bug?

Please let me know.

Thanks,

JM


Yes, this is definitely a bug. =) Good spot, and excellent detail in your report, thanks so much!

I think Requests' documentation is documenting the behaviour we want. We can get this by adding a function call in line 558 (or some inline code) that puts the scheme into the proxy URL if it's missing. Thus, to carry on the example from [here](http://docs.python-requests.org/en/latest/user/advanced/#proxies), we'd change `"10.10.1.10:3128"` to `"http://10.10.1.10:3128"` and `"10.10.1.10:1080"` into `"https://10.10.1.10:1080"`.

The only worry here is that people might have code that passes their HTTPS traffic through an HTTP proxy, which this would break (unless the expressly indicate the scheme in their proxy URL). @kennethreitz, is this a problem?


You're welcome. It's the least I can do to make some small contribution to such a great library!

JM


@jeanmarcrousseau is this still an issue with v1.0.3?


I'm not sure - Lukasa was going to address, so rather check with him...

JM

On Tue, Dec 18, 2012 at 9:47 PM, Ian Cordasco notifications@github.comwrote:

> @jeanmarcrousseau https://github.com/jeanmarcrousseau is this still an
> issue with v1.0.3?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/997#issuecomment-11515467.


As of v1.0.4, we don't prepend the scheme to a proxy URL that doesn't have one, and we probably should.


I'm pretty sure this is resolved as of sometime around v1.1, so I'm closing this. Thanks for the report! =)


My pleasure. :-)

JM

On Sat, Jan 19, 2013 at 11:38 AM, Cory Benfield notifications@github.comwrote:

> I'm pretty sure this is resolved as of sometime around v1.1, so I'm
> closing this. Thanks for the report! =)
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/997#issuecomment-12457069.