requests.packages.urllib3.connection.HTTPConnection object at 0x00007effa13b1ec0>: Failed to establish a new connection: [Errno -11] System error

While using grequests I get the following error after running a program which repeatedly checks a batch of urls to determine if the sites are up. 

For simplicity say I do this : 

I have 50 urls.  
I created an httpmonitor module that is handed the urls and then the module is sent to a background/thread-with-Queue.  
The httpmonitor checks all the urls using standard grequests structure such as this  : 
max_workers = 10  now  but I have tried from 50 down to 3 

r = (grequests.head(url, callback=timeme, timeout=3, allow_redirects=False, stream = False) for url in urls) 
results = grequests.map(r, size=max_workers, exception_handler=handle_errors) 

I deal with the results, update a database and then the thread finishes and independently  a new one is added to the Queue (for this example, with the same 50 urls)  -- and this goes on and on.. 

Note.  I explicitly close connections after each url is processed with either a Response.close() or a AsyncRequest.session.close() so I believe all connections are closed after each block of urls is processed.  

Then I set it running and it runs until approximately 1500 requests are processed ( 30 blocks of 50 urls ) and then for a period of time all of the grequests fail with the error above Errno -11, System Error ( which I believe means OUT OF RESOURCES) ..  Sometimes it will return to normal operation for a period, most of the time . it fails on all further attempts to access a connection. 

1) Is there anymore information on this error that someone might share ?  
2) Is there a way to ensure all resources are released outside of grequests?  
3) Is there something else I have might have missed?  

Note: I was able to increase the number times the blocks of urls could be checked by explicitly closing  connections -- gaining an addition 400+ url checks -- but then the same error returned.  It would seem some resource is not closing and accumulating over the batch runs.. 

Here is the exact error (url does not matter)  

HTTPConnectionPool(host='www.yahoo.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x00007effa13b1ec0>: Failed to establish a new connection: [Errno -11] System error',))" 

Any help or ideas would be appreciated, Thank you 




It seems likely that this is an FD limit. If you raise the ulimit of the process does it delay the error?

Thank you for the response .. I have not changed the ulimit -- currently at 1024 -- Is there a standard procedure for how to do this and have it remain across reboots.. sorry I have not ventured down that rabbit hole yet .. I will try it though .. However if it is a out-of-resources issue, won't increasing the ulimit simply delay the response ( as you suggested) -- what might I do to prevent opening and locking up too many File Descriptors ? 

No need to retain it across reboots, I'm simply interested in ruling that cause in or out. So it's enough to have it last for the process only.

Yes .. I upped the ulimit to 64000 soft and hard and it has been running for a long while now.. into the 4000s instead of 1500.  

Ok, is it possible to get a small reproducer for this bug?

please explain small reproducer?  I have built a rather large project based on this general idea, but I should be able to provide you with a snapshot version .. meaning it does not require the 5 modules interacting which exist now to update the urls for testing and manage the background ... I'll think about it -- The database becomes available when the program uses up the available FDs -- so I am not sure how important keeping the DB in the example might be .. 

A small reproducer would be the minimal amount of code from your project required to reproduce the same behaviour, removing all extraneous logic. You'd try to slowly delete code until the problem goes away. 

Sounds good .. I am working a minimal version of just the portion that
includes grequests and the repetition of checking sites .. Thank you for
helping with this issue .. the hours spent have been very anxious :)
​

On Wed, Apr 19, 2017 at 5:31 PM, Cory Benfield <notifications@github.com>
wrote:

> A small reproducer would be the minimal amount of code from your project
> required to reproduce the same behaviour, removing all extraneous logic.
> You'd try to slowly delete code until the problem goes away.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/kennethreitz/requests/issues/3975#issuecomment-295452369>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AE4HdnHD5onYU54X-mZyfHh0T7kU9vFAks5rxn1OgaJpZM4NB3Ql>
> .
>


Cory I am completing a simple version of the software, which I believe will demonstrate the issue i have -- but while i was writing this, I have been monitoring the current running version and I see open FDs building up in real time  -- the same blocks of urls I check remain open with http (CLOSE_WAIT) descriptions.  My setup is on a Ubuntu 16.x version of linux using pypy and the latest versions of all software. When I run  lsof -a -p PID_NUMBER I am seeing hundreds of (1800) versions of the same open connections waiting to be closed -- I assume.  I could send you a print out privately if you would like to see the information .. 

lines of data looking like 47664->www.yahoo.com:http (CLOSE_WAIT)

In the time it took to write this I counted the FD for the this PID .. 

I actually have 1 gig of memory .. and only one processor .. 

(nexusrt) root@lemp-512mb-nyc3-01:/nexusrt/www/nexusrt.com# lsof -a -p 6453 |wc -l
1894
(nexusrt) root@lemp-512mb-nyc3-01:/nexusrt/www/nexusrt.com# lsof -a -p 6453 |wc -l
2227




So how many different hosts are you hitting? Connection pooling should be preventing the CLOSE_WAIT getting out of hand.

I am only hitting 50 at the moment  - but expect that to go up (as the limits of the resources of the box permit and then adding other machines - I would expect that 300 would a typical number.  However, I wanted to make this scaleable.  Now there are 3000 open connections with CLOSE_WAIT.  

So what code do you have dealing with sessions? Do you consistently use just one? Do you use any?

The portions of the code that deal with sessions are only contained within grequests.  I do explicitly close() the session (if there is an error, I am using an AsyncRequest.session.close() ) and Response.close() for returning response objects from the map function.  I do not use requests or request.Session() explicitly at all in this application.  My code looks basically like this -- 

r = (grequests.head(url, callback=timeme, timeout=3, allow_redirects=False, stream = False) for url in urls)
results = grequests.map(r, size=max_workers, exception_handler=handle_errors)

i have a simple generator of the issue.  It takes 50 urls, and repeatedly does a grequest.head() request (every 15 seconds)  .  The number of open CLOSE_WAIT stabilizes at around 380 and rises to 481 and back to 309 and this continues.  The difference in this app is that I am not using a background process to start the same type of code.  I use a background thread which manages a Queue -- batches of jobs just like the one running standing alone are queued and started when the one before it ends -- Not sure how that can make a difference..  But the Close_Waits clear faster in this test case.  I dropped the sleep time to 5 seconds and it still stabilizes, albeit with a higher number of Close_Waits on avg.  

interestingly -- I run it on my mac and it works with no residual open connections .. 

@steelliberty Is your Mac running PyPy or CPython?

The reason I ask is that the likely problem here is that you're letting the Session leak out of scope and therefore are causing all the CPython connections to get closed by the refcounting GC, while PyPy's non-refcounting GC hasn't yet encountered enough memory pressure to evict those sockets from memory (and so close them).

You can resolve this by changing your code to this:

```python
with requests.Session() as s:
    r = (grequests.head(url, callback=timeme, timeout=3, allow_redirects=False, stream = False, session=s) for url in urls)
    results = grequests.map(r, size=max_workers, exception_handler=handle_errors)
```

This will force all the requests to use the same `Session` object, which will ensure that you don't leak the connections. Additionally, when you exit the `with` block, all the `Session` connections will be destroyed eagerly, rather than relying on the GC to clean them up.

This sounds like what is happening is  exactly what I think can solve the problem. I failed to mention to you I act on the sessions right away, and after the code below if there are status_codes which warrant retesting, I basically hand those urls off to a similar block of code.  So I am not sure how they are staying open. Thank you .. I will quickly try this. 

r = (grequests.head(url, callback=timeme, timeout=5, allow_redirects=False, stream = False) for url in urls)
            results = grequests.map(r, size=max_workers, exception_handler=handle_errors)

            if results:
                for x in list(results):  #list of request objects for each url
                    if x == None:  #if none then url error_handled (where I close socket during processing)
                        continue
                    try:
                        url = x.url.rstrip('/') 
                        if x.history:
                            print "HISTORY ",url, x.history
                        sc = x.status_code  #results status_code
                        x.close()  #explicitly closing the socket here 
                    ....
                    further processing. 

                    


@steelliberty `Response.close` does *not* close the underlying socket, it just stops making it available to the response and returns it to the connection pool. You have to use `Session` objects to get the control over lifetime you want.

I see .. makes sense .. in the error_handler I have access at AsyncRequest.session.close() . .. 

Let me test this and I'll let you know.  Thank you. 


I *strongly* recommend using the `with` statement and `Session` instead.

seems more efficient then creating separate sessions -- and clearly easy to eliminate. 

That did the trick ! .. Thank you Cory.  You saved me many more hours of searching for a solution. 


Happy to help! I hope you find that your script now behaves much better.

It is running perfectly now .. other tiny issues vanished with that fix :)


On Fri, Apr 21, 2017 at 10:56 AM, Cory Benfield <notifications@github.com>
wrote:

> Closed #3975 <https://github.com/kennethreitz/requests/issues/3975>.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/kennethreitz/requests/issues/3975#event-1052580401>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AE4Hdv-LcMEy-Ys_6NO9QAUTQkoNBi9sks5ryMOhgaJpZM4NB3Ql>
> .
>