gevent adapter

Hello.

As a matter of feature request, it would be useful to have a gevent-compatible adapter for requests which would not require any dangerous monkey-patching.

By now, I've tried combining `requests` with `geventhttpclient`; that works, but it is a large hack if it is done in a separate module ( https://gist.github.com/anonymous/699cf4beba490024e52c
 ) (and I don't want to break external compatbility with requests by trying to use completely different pool and response classes).


I presume that you've seen [grequests](https://github.com/kennethreitz/grequests) and that it's not suitable for your needs?


@Lukasa as you might see in its source code, it applies gevent.monkeys.patch_all, which is known to break various things and is a generally dangerous thing to do.  (worse, it does that on import-)


What specifically does it break for you @HoverHell?


@sigmavirus24 I think that's mostly out of the issue. Regardless, it threw an “this call would block forever” exception (on a code which otherwise works normally) which is enough to be suspicious; additionally (but not critically), it is known to have some problems with celery, raven, possibly uwsgi, and various other things. Not quite something I would want to deal with in an actively used django application.


@HoverHell grequests isn't exactly an alternative implementation; it's a little more like an example script or basic wrapper with some convenience methods (like mapping).

Some slightly less ham-handed monkeypatching might accomplish what you want, perhaps by doing `httplib.socket = gevent.socket; urllib3.socket = gevent.socket; requests.socket = gevent.socket`. Otherwise, there is no real way of doing it "the right way" due to requests' dependence on urllib3, and urllib3's dependence on httplib from the standard lib. Your geventhttpclient adapter would likely be the overall safest bet in that regard (and would probably give you some speed increase as well, due to geventhttpclient being written specifically for performance with gevent), but of course it's going to bring its own problems as you say.

For what it's worth, though, I've been using `gevent.monkey.patch_all()` in many of my applications for a long time with utterly no issues. I can see how it might cause issues with other big network-based third party libraries, but I think it's worth trying the monkey patching in new code before looking at alternatives. I can confirm that global monkey patching seems to work just fine with requests at least, without any need for grequests.

> it threw an “this call would block forever” exception (on a code which otherwise works normally) which is enough to be suspicious

It is rare to see such an exception when using gevent as intended, I believe.


> It is rare to see such an extension when using gevent as intended, I believe.

I agree. I don't think I've ever seen that problem before when using gevent correctly.


@Anorov 

> Your geventhttpclient adapter would likely be the overall safest bet in that regard
> Certainly; my question is: can it be done in a less hack-ish / more future-proof way, perhaps by integrating it more with the requests library e.g. making few minor changes in the methods arhitecture of the requests connection / connectionpool modules?

@sigmavirus24 

> I agree. I don't think I've ever seen that problem before when using gevent correctly.

I wasn't even using gevent at that point – I simply installed grequests, and another imported library (which wasn't being used either) imported grequests; in all the other regards it is simply a django application.

(though, thinking of it, I could guess that it is the late importing that made it problematic; still, I expect other problems regardless)


@HoverHell I'm starting to question whether this issue is even suited for this project. Your trouble seems to arise from that project, not requests, and you've been dismissive of efforts to try to help you in using it. I'm also surprised you're intent on using gevent when you're so convinced that it's going to cause you so many problems.


@sigmavirus24 I'm intent on using it (or an alternative, really) without _potentially_ problematic monkeypatching (sometimes even small chances of breaking something are too much); that's why I need more support from the requests.


@HoverHell you haven't told us how we can support you further though? There aren't changes that need to be made to requests to support gevent either by monkey patching everything or only a small number of things. We've tried to help but with every suggestion we're told it is insufficient.


@HoverHell I believe my suggestion of monkeypatching the `socket` module in `requests` and all of its dependencies are the quickest "less-hacky" ways of accomplishing this.

gevent itself was kind of designed as a hack: a way to add green threads into Python without a new interpreter and/or new standard lib. The gevent designer(s) tried to make the monkeypatched modules as backwards compatible as possible, but obviously monkeypatching will never be perfect.

Your absolute safest bet, if you wanted to make a fully compatible clone that defers to geventhttpclient and does no actual monkeypatching, is probably to make an alternative urllib3 implementation that uses geventhttpclient's swap-in httplib as described in [its README](https://github.com/gwik/geventhttpclient/blob/master/README.mdown):

```
from geventhttpclient.httplib import HTTPConnection
```

urllib3 doesn't really support adapters, so you'd have to make a new `geurllib3` module I suppose.

You'd basically just change a few of the imports in `connection.py` [here](https://github.com/shazow/urllib3/blob/98c6fbfc27d7d51327ad85a85a80dd4fe096cc79/urllib3/connection.py#L14-L30).

Then you should probably be able to subclass `HTTPAdapter` and use the new `geurllib3` module's classes and functions instead.

This would have to rely on geventhttpclient's version of httplib being 100% compatible with the standard lib's. I'd recommend running through the full urllib3 and requests test suite after making these changes. If you discover a bug, you can open an issue on the geventhttpclient repo. 

Either way, I agree this is out of scope as a requests issue. It's just a personal decision you'll have to make.


@sigmavirus24, @Anorov  as you can see by the link in the first message, I've already made a no-monkeypatching adapter through subclassing. However, it is variably hack-ish and incomplete and unreliable.

My question, thus, is: what changes can be made to requests that would make it better? The very least would be making `pool_classes_by_scheme` an attribute of the PoolManager; and, likely, adding more attributes like ConnectionCls.

Basically, my problem is that the whole default _HTTPAdapter_ of the requests is nearly a monolithic untweakable thing if you don't count in monkeypatching.


> The very least would be making pool_classes_by_scheme an attribute of the PoolManager; and, likely, adding more attributes like ConnectionCls.

Those are all changes to `urllib3`, not `requests`. 

> Basically, my problem is that the whole default HTTPAdapter of the requests is nearly a monolithic untweakable thing

It's actually quite tweakable and there are many examples of it being tweakable. It might be easier for you to explain yourself if you **start** a PR with some of the changes you think are necessary, but I have to admit that they'll have to be obviously beneficial changes to the overall project, especially since there's already a way to use requests with gevent.


Most of the changes will indeed need to be made in `urllib3`. I could see some of those being accepted; you'd need to open a PR with `urllib3` though.


Closing due to inactivity