HTTP keep alive support

I know keep alive is currently beyond the scope of urllib2, but I think it is a worthwhile addition to requests. I use requests mainly to consume HTTP based APIs created by our own or by 3rd party developers. Generally I need to do tasks like searching for some data, which is returned as a json document, afterwards I download 200-2000 images referenced by the json document.

I maintain a lib which is based on urllib2/requests, which is nice to read because of requests. But the production version is based on httplib2 which provides keep alive support. The code is not as nice as with requests, but a typical job takes 30-200% less time, because the http connection is reused.

urlgrabber (http://urlgrabber.baseurl.org/) provided such a HTTPHandler class. (See also http://stackoverflow.com/questions/1037406/python-urllib2-with-keep-alive) May be something like that can be integrated into requests? 

I will take a look at the url grabber code myself. May be I can provided a useable patch.


Sorry for the delay in my response. 

I would LOVE to have keep-alive support in Requests. My main concern is keeping the API for it as clean as possible.

I'm thinking it could be an optional flag to requests.session(), maybe. 


This sounds like a good idea. 


Hi Kenneth,

Based on urlgrabber keepalive module, I have created a branch for adding support for this, using requests sessions.
https://github.com/maraujop/requests/tree/feature/keepalive

It took me the whole afternoon to realize old Twitter REST API doesn't have support for Keep-alive connections which made me feel mental retarted BTW. Anyway, I pushed my branch just in case you think this is useful. HTTPS support is commented out, because it depends on another urlgrabber module called `sslfactory.py`. If you like it, it would be as easy as uncommenting the code and adding the module.

Cheers,
Miguel


:sparkles: BEAUTIFUL! :sparkles: 

Thanks! :cake:

I'll have to rip out the module you added and reimpliment myself because it's GPL. It looks really strait forward, so shouldn't be a problem. 


Also, I'm trying to keep all code original, because this effects the Debian packaging rules and potential standard library inclusion. The vendorized version of Poster that's included is considered a fork, which is the only reason I've been able to get away with it so far. 

Anyway, I can't wait to play with this! 


Thanks,

I'm not an expert on open source licenses, could you please briefly explain me what is the problem with GPL? 

Anyway I was going to say, the best thing would be rewriting the module, the code is quite messy, methods don't have docstrings (something I particularly hate) and I honestly think it could be clearer and cleaner.


https://github.com/kennethreitz/requests/blob/develop/docs/user/intro.rst read "Licence“ part


The biggest reason is that I couldn't distribute Requests without making it _all_ GPL. If anyone used Requests in a project, that project would be bound to the GPL. It would have to excluded from most commercial projects, as they can't be offered as open source. 

In short, it would make Requests a liability instead of an asset, to many people. 

Requests has already gone through the hurtles of making it into the Fedora and Debian package repositories. I'm speculating, but changing the license would likely lead to a re-evaluation. Adding a new packaged dependency would affect that as well.

A personal goal of mine is to have Requests be added to the Python Standard Library. This simply couldn't happen if it was GPL, unless I (as the copyright holder) could change the license. If it was because of a GPL'd internal dependency, I could not.

Anyway, there's a good brief overview :)


Thanks Simonas for the link, I think I missed it the first time I was reading the docs.

Thanks Kenneth for the explanation, I wasn't aware of these details and I will have them into account for the future.


Requests sessions are going to have keep-alive / connection pooling soon!


Cool :)


Done.