Repository: benoitc/gunicorn

Issue:
eventlet worker occasionally stops in the middle of handling requests when max_requests is reached
I'm not sure if this is a bug in our application, or if it's a bug with the eventlet worker in gunicorn. To give some background, in trying to reproduce a different issue in production (worker timeouts), I was able to reproduce an issue where I occasionally get 502s from nginx (in front of gunicorn) using a concurrency of 7 clients against 1 eventlet worker (usually I can get it to happen within 700 requests w/ an approximate concurrency of 7).  

These 502s don't show up in gunicorn's access log and I was able to narrow it down to max_requests being exceeded, thereby setting self.alive = False, even though there are still requests being processed by the worker. Then, in the run loop of the eventlet worker, there is a "while self.alive" block that checks itself every second before deciding to kill all greenlets. I didn't see any sort of checks as to whether there are any requests currently being handled before deciding to kill greenlets and the worker to stop itself. 

To test this theory, I just kept a set of uniquely generated request ids and added a request id every time the worker was handling a new request, and removed the request_id when it was returning a response. I then changed the "while self.alive" block to "while self.alive or self.request_ids" instead, and that overcame the 502 issue I was seeing.

Is there something my application is doing that could cause an issue with the eventlet gunicorn worker to kill greenlets that are currently serving a request w/o any warning or logs?

It seems that "while self.alive" is not sufficient to determine this.

Btw, I'm using gunicorn 0.17.2 and eventlet 0.12.1.

Thanks so much for any insight into what may be going on here. Let me know if there's any other details I can provide.



Discussion:
By design (please @davisp confirm) The max_requests is here to prevent any memory leak and such. So it's expected that the worker is killed at this point to allow new connections since the number of workers isn't increased automatically.

Imo that part can probably be improved introducing some rules to extend the number of wokers.

Thanks for the reply. I'm just concerned that the worker can be killed mid request and nginx returns an error for that request, while gunicorn does not log it (since the request never finished) and we had no clue (until digging deeper and custom logging) why the request was not completed. 

Yes, I'd agree that this seems like a misfeature. Just because a worker has reached its max_requests doesn't mean that in-flight requests should be killed. Once max_requests is hit the worker needs to be allowed to finish any outstanding requests before being killed.


Resolving patch:
diff --git a/gunicorn/workers/geventlet.py b/gunicorn/workers/geventlet.py
--- a/gunicorn/workers/geventlet.py
+++ b/gunicorn/workers/geventlet.py
@@ -40,6 +40,9 @@ def handle(self, listener, client, addr):
 
         super(EventletWorker, self).handle(listener, client, addr)
 
+        if not self.alive:
+            raise eventlet.StopServe()
+
     def run(self):
         acceptors = []
         for sock in self.sockets:
@@ -60,5 +63,10 @@ def run(self):
             eventlet.sleep(1.0)
 
         self.notify()
-        with eventlet.Timeout(self.cfg.graceful_timeout, False):
-            [eventlet.kill(a, eventlet.StopServe) for a in acceptors]
+        try:
+            with eventlet.Timeout(self.cfg.graceful_timeout) as t:
+                [a.wait() for a in acceptors]
+        except eventlet.Timeout as te:
+            if te != t:
+                raise
+            [a.kill() for a in acceptors]


Test patch: