Too many open files

I am building a basic load generator and started running into file descriptor limits, I havent seen any documentation pertaining to how to release resources, so either I am doing it wrong and the docs need updated, or requests is leaking file descriptors somewhere (without support for keepalive I am slightly confused at why any files would be left open at all)


Where you using `requests.async`?


nope, all requests were reasonably plain requests.get / requests.post, I am still seeing a few in there

```
$ lsof | grep localhost | wc -l
110
```

all but 4/5 of them are of the format

```
Python    82117 daleharvey  123u    IPv4 0xffffff800da304e0       0t0      TCP localhost:61488->localhost:http (CLOSE_WAIT)
```


I'm a bit baffled by this, to be honest. 


Hah ill take another shot at reproducing it reliably, if I cant ill close


I've seen this happening to me, but only when I'm using the async module w/ 200+ simultaneous connections.


Hi,
I got exactly the same problem using requests and monkey patching with gevent : some connections staying in CLOSE_WAIT .
Maybe a problem with gevent so .


It may be problem of ulimit -n. Try with a higher value.


"Too many open files" is the result of the bug caused by sockets staying in CLOSE_WAIT .
So ulimit won't fix just make a workaround .


@tamiel how do we fix this?


I will do more tests asap and try to fix .


I've looked into it, and seems to be a problem with all libraries using httplib.HTTPSConnection. 


Posted an example here :

https://gist.github.com/1512329


I just encountered a very similar error using an async pool with only HTTP connections - I'm still investigating but passing a pool size to async.map makes the error reproduce quickly.


Any fixes to this? This makes Requests unusable with gevent..


It's all about the `CLOSE_WAIT`s. Just have to close them. I'm not sure why they're still open though.


Is it a urllib3 issue? Having to close these by ourselves isnt a great idea i feel.


It's more of a general issue. We can keep the conversation here. 


Ok just to give you a perspective, we are trying to move from httplib2 to requests, and we dont see this issue with httplib2. So its not a general issue for sure.


By general i mean that it's a very serious issue that effects everyone involved.


so how do we solve this? we really want to use requests + slumber moving forward


I'd love to know the answer to that.


The leak appears to be due to the internal redirect handling, which causes new requests to be generated before the pending responses have been consumed. In testing acdha@730c0e2e2bef77968a86962f9d5f2bebba4d19ec has an under-satisfying but effective fix simply by forcing each response to be consumed before continuing.

This required changes in two places which makes me want to refactor the interface slightly but I'm out of time to continue currently.


#399 has a fix which works well in my async load generator (https://github.com/acdha/webtoolbox/blob/master/bin/http_bench.py) with thousands of requests and a low fd ulimit


I have run into the same issue when using async -- kludged a workaround by chunking requests and deleting responses / calling gc.collect


I believe I was running into this today connecting to a licensed server that only allows 5 connections.  

Using async I could only GET 4 things before it paused for 60 seconds.

Using the normal GET with consumption I could fetch about 150 things serially in under 40 seconds.

Haven't made my kludge yet since I saw this issue.  


Just got this error while using ipython and got this message. This is just making each request one at a time, but I think I got something similar when using async. 

```
ERROR: Internal Python error in the inspect module.
Below is the traceback from this internal error.
Traceback (most recent call last):
    File "/Library/Python/2.7/site-packages/IPython/core/ultratb.py", line 756, in structured_traceback
    File "/Library/Python/2.7/site-packages/IPython/core/ultratb.py", line 242, in _fixed_getinnerframes
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1035, in getinnerframes
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 995, in getframeinfo
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 456, in getsourcefile
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 485, in getmodule
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 469, in getabsfile
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 347, in abspath
OSError: [Errno 24] Too many open files

Unfortunately, your original traceback can not be constructed.
```

Oddly, I think when using just the normal Python interpreter I get a "Max Retries Error" but I think that is another issue with me doing requests on all the same domain, but not sure.


I ran into this on the first project I had where allow_redirects was True; it appears to be caused by the redirection chain leaking response objects which aren't released even with prefetch=True. This fixed it in my initial testing:

```
        [i.raw.release_conn() for i in resp.history]
        resp.raw.release_conn()
```


Hmmm..


@acdha setting:

```
requests.defaults.defaults['allow_redirects'] = False
```

before I make any requests still results in the same error, but I think this isn't an option for my implementation as all the requests I'm making will require a redirect =/ 


@dalanmiller How are you processing your responses? I was previously using `async.map` with a response hook and it _appears_ to be more stable using a simple loop over `async.imap`:

``` python
for resp in requests.async.imap(reqs, size=8):
    try:
        print resp.status_code, resp.url
    finally:
        [i.raw.release_conn() for i in resp.history]
        resp.raw.release_conn()
```