Repository: benoitc/gunicorn

Issue:
Threaded workers accept connections even when no ready thread exist
When using threads (with sync worker class), worker accepts connection even when all threads is busy. In our environment,  sometimes (because a deadlock in cairocffi library) all threads of a worker are locked, but worker continues accepting connections until limit of open-files is reached.

I looked at the code in `gthread.py` and it regards the `worker_connections` setting (despite what the documentation says, that it only affects Eventlet and Gevent workers).

I think in threaded workers, `worker_connections` always should be set to number of threads.



Discussion:
what do you mean by "When using threads (with sync worker class), worker accepts connection even when all threads is busy. " ? With the `--thread` option?

yes, (actually `--threads` option)

thanks for your answer. will work on it during the week. Expect a fix for the coming release this we.

Hrm, I am not sure how to reproduce that issue. When I look at the code of the threaded workers it appears, that the worker should crash if the number of active connections is greater than the number of accepted worker connections:

https://github.com/benoitc/gunicorn/blob/master/gunicorn/workers/gthread.py#L174-L178

When it happen and no futures have returned in the timeout the connections breaks. Are you sure the connections are not handled by another worker?

Steps to regenerate
- create `foo.py`. It's a helloworld app which waits 100s if url is '?wait'

``` python
import time
import os

def application(env, start):
   t = time.time()
   print('---- start in ', os.getpid())
   if env['QUERY_STRING'] == 'wait':
       while time.time() - t < 100: time.sleep(1)
   print('--- end in', os.getpid())
   start('200 OK', []);
   return [b'Hello World']
```
- Run gunicorn with:
  `gunicorn --error-logfile - --workers 2 --threads 2 hello`. The server has 2 workers, each with 2 threads.
- Open a browser and make exactly 3 requests to `http://127.0.0.1:8000/?wait'`. Now log looks like:

```
[2014-10-20 10:26:49 +0330] [5419] [INFO] Starting gunicorn 19.1.1
[2014-10-20 10:26:49 +0330] [5419] [INFO] Listening at: http://127.0.0.1:8000 (5419)
[2014-10-20 10:26:49 +0330] [5419] [INFO] Using worker: threads
[2014-10-20 10:26:49 +0330] [5422] [INFO] Booting worker with pid: 5422
[2014-10-20 10:26:49 +0330] [5423] [INFO] Booting worker with pid: 5423
---- start in  5423
---- start in  5423
---- start in  5422
```

3 of 4 threads are now busy, and one is idle.
- Now make several requests to `http://127.0.0.1:8000/` (without cache, use Control+F5). Some of requests will succeed, but some of them won't generate any output (and stuck in "Waiting for 127.0.0.1" status) , even always there is a 1 idle thread.

We're having the same problem, settings workers to 2+ threads causes exact same issue as explained above.

on which OS? kernel?

Description:    Ubuntu 12.04.4 LTS
Linux code 3.8.0-34-generic 49~precise1-Ubuntu SMP Wed Nov 13 18:05:00 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

I've run the example provided above and I think I  understood the problem.

So the issue is that the threaded worker is not designed to accept long connections right now. It is mostly working like the sync worker. Using 2 threads / workers mean that only 4 simultaneous connections can be handled.  Trying to wait like this will rapidly lock exhaust the number of threads. However since the accepting loop is async and handled in its own loop, you may have more connections accepted that will all wait for a free worker,

I need to think more about that design. I am thinking to remove any async accepting loop and handle the pool synchronously instead.

:+1: better to handle the pool synchronously. That is better for upstream load balancers because they will see a slow connect and might fail over.

Could it be that this error is also present even if you have 1 threaded worker ? Seems that we have a odd situation that even after commenting threads=2 in our config we're able to reproduce state when a deadlocked worker accepts connections.

EDIT:

The interesting thing is that the deadlock is during the requests initialization phase.

user - > nginx -> gunicorn(webapp) -> rpcserver the deadlock happens on connecting to rpcserver, and when that happens  we have gunicorn accept connections and wait...

Listening socket is always open, and connecting clients will be queued in TCP backlog (even gunicorn does not accept a new connection). That's not related to this issue.

But there is another issue: After sending stop signal, while it waits for current requests to complete, gunicorn does not close listen socket.

@tahajahangir this actually by desig. But why would you want it? Though this is not related to this ticket, can you open a new ticket about it?

I have raised this issue before and tried to solve it. I forget why I
failed. I could dig up the comments maybe. But we should stop listening as
soon as we start shutting down.

:+1: opening a new issue for it.
On Oct 25, 2014 12:53 AM, "Benoit Chesneau" notifications@github.com
wrote:

> @tahajahangir https://github.com/tahajahangir this actually by desig.
> But why would you want it? Though this is not related to this ticket, can
> you open a new ticket about it?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/benoitc/gunicorn/issues/908#issuecomment-60475056.

@tilgovi If I recall correctly, the reason we are not doing it is for the case where we d a USR2 and stop the old master. In that case we don't want to close socket so the master can still listen on it. I am actually wondering if the system doesn't take care of that for us.... (also if it's portable)

That was indeed the reason. I'll follow up on the other issue.
On Oct 25, 2014 3:21 AM, "Benoit Chesneau" notifications@github.com wrote:

> @tilgovi https://github.com/tilgovi If I recall correctly, the reason
> we are not doing it is for the case where we d a USR2 and stop the old
> master. In that case we don't want to close socket so the master can still
> listen on it. I am actually wondering if the system doesn't take care of
> that for us.... (also if it's portable)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/benoitc/gunicorn/issues/908#issuecomment-60478205.

When the max_requests option is in use, and there are pending requests queued up for the worker, but the worker is shutting down, the queued up requests are closed off forcely 5xx error.

i am slightly revisiting the gthreads worker. The new workflow is quite more simple:
1. listeners are put in the poller. On read we try to accept on them. 
2. When a connection is accepted it is put in the execution queue
3. When a request is done and the socket can be kept alived, we put it in the poller, on read event we will try to handle the new request. If it is not put out of the poller before the keepalive timeout the socket will be closed.
4. if all threads are busy we are waiting until one request complet. If it doesn't complete before the timeout we kill the worker.

I will have a patch available tomorrow.


Resolving patch:
diff --git a/gunicorn/workers/gthread.py b/gunicorn/workers/gthread.py
--- a/gunicorn/workers/gthread.py
+++ b/gunicorn/workers/gthread.py
@@ -18,6 +18,7 @@
 import socket
 import ssl
 import sys
+from threading import RLock
 import time
 
 from .. import http
@@ -65,13 +66,14 @@ def init(self):
 
             # initialize the parser
             self.parser = http.RequestParser(self.cfg, self.sock)
-            return True
-        return False
 
     def set_timeout(self):
         # set the timeout
         self.timeout = time.time() + self.cfg.keepalive
 
+    def close(self):
+        util.close(self.sock)
+
     def __lt__(self, other):
         return self.timeout < other.timeout
 
@@ -83,68 +85,94 @@ class ThreadWorker(base.Worker):
     def __init__(self, *args, **kwargs):
         super(ThreadWorker, self).__init__(*args, **kwargs)
         self.worker_connections = self.cfg.worker_connections
+        self.max_keepalived = self.cfg.worker_connections - self.cfg.threads
 
         # initialise the pool
         self.tpool = None
         self.poller = None
+        self._lock = None
         self.futures = deque()
         self._keep = deque()
 
+    def init_process(self):
+        self.tpool = futures.ThreadPoolExecutor(max_workers=self.cfg.threads)
+        self.poller = selectors.DefaultSelector()
+        self._lock = RLock()
+        super(ThreadWorker, self).init_process()
+
     def _wrap_future(self, fs, conn):
         fs.conn = conn
         self.futures.append(fs)
         fs.add_done_callback(self.finish_request)
 
-    def init_process(self):
-        self.tpool = futures.ThreadPoolExecutor(max_workers=self.cfg.threads)
-        self.poller = selectors.DefaultSelector()
-        super(ThreadWorker, self).init_process()
+    def enqueue_req(self, conn):
+        conn.init()
+        # submit the connection to a worker
+        fs = self.tpool.submit(self.handle, conn)
+        self._wrap_future(fs, conn)
 
     def accept(self, listener):
-        if not self.alive:
-            return
-
         try:
             client, addr = listener.accept()
+            # initialize the connection object
             conn = TConn(self.cfg, listener, client, addr)
-
-            # wait for the read event to handle the connection
-            self.poller.register(client, selectors.EVENT_READ,
-                    partial(self.handle_client, conn))
-
+            self.nr += 1
+            # enqueue the job
+            self.enqueue_req(conn)
         except socket.error as e:
             if e.args[0] not in (errno.EAGAIN,
                     errno.ECONNABORTED, errno.EWOULDBLOCK):
                 raise
 
-    def handle_client(self, conn, client):
-        # unregister the client from the poller
-        self.poller.unregister(client)
+    def reuse_connection(self, conn, client):
+        with self._lock:
+            # unregister the client from the poller
+            self.poller.unregister(client)
+            # remove the connection from keepalive
+            try:
+                self._keep.remove(conn)
+            except ValueError:
+                # race condition
+                return
 
         # submit the connection to a worker
-        fs = self.tpool.submit(self.handle, conn)
-        self._wrap_future(fs, conn)
+        self.enqueue_req(conn)
 
     def murder_keepalived(self):
         now = time.time()
         while True:
-            try:
-                # remove the connection from the queue
-                conn = self._keep.popleft()
-            except IndexError:
-                break
+            with self._lock:
+                try:
+                    # remove the connection from the queue
+                    conn = self._keep.popleft()
+                except IndexError:
+                    break
 
             delta = conn.timeout - now
             if delta > 0:
                 # add the connection back to the queue
-                self._keep.appendleft(conn)
+                with self._lock:
+                    self._keep.appendleft(conn)
                 break
             else:
+                self.nr -= 1
                 # remove the socket from the poller
-                self.poller.unregister(conn.sock)
+                with self._lock:
+                    try:
+                        self.poller.unregister(conn.sock)
+                    except socket.error as e:
+                        if e.args[0] != errno.EBADF:
+                            raise
 
                 # close the socket
-                util.close(conn.sock)
+                conn.close()
+
+    def is_parent_alive(self):
+        # If our parent changed then we shut down.
+        if self.ppid != os.getppid():
+            self.log.info("Parent changed, shutting down: %s", self)
+            return False
+        return True
 
     def run(self):
         # init listeners, add them to the event loop
@@ -155,55 +183,46 @@ def run(self):
         timeout = self.cfg.timeout or 0.5
 
         while self.alive:
-            # If our parent changed then we shut down.
-            if self.ppid != os.getppid():
-                self.log.info("Parent changed, shutting down: %s", self)
-                return
-
             # notify the arbiter we are alive
             self.notify()
 
-            events = self.poller.select(0.2)
-            for key, mask in events:
-                callback = key.data
-                callback(key.fileobj)
+            # can we accept more connections?
+            if self.nr < self.worker_connections:
+                # wait for an event
+                events = self.poller.select(0.02)
+                for key, mask in events:
+                    callback = key.data
+                    callback(key.fileobj)
+
+            if not self.is_parent_alive():
+                break
 
             # hanle keepalive timeouts
             self.murder_keepalived()
 
-            # if we more connections than the max number of connections
-            # accepted on a worker, wait until some complete or exit.
-            if len(self.futures) >= self.worker_connections:
-                res = futures.wait(self.futures, timeout=timeout)
-                if not res:
-                    self.alive = False
-                    self.log.info("max requests achieved")
-                    break
-
-        # shutdown the pool
-        self.poller.close()
-        self.tpool.shutdown(False)
+            # if the number of connections is < to the max we can handle at
+            # the same time there is no need to wait for one
+            if len(self.futures) < self.cfg.threads:
+                continue
 
-        # wait for the workers
-        futures.wait(self.futures, timeout=self.cfg.graceful_timeout)
+            result = futures.wait(self.futures, timeout=timeout,
+                    return_when=futures.FIRST_COMPLETED)
 
-        # if we have still fures running, try to close them
-        while True:
-            try:
-                fs = self.futures.popleft()
-            except IndexError:
+            if not result.done:
                 break
 
-            sock = fs.conn.sock
+            else:
+                [self.futures.remove(f) for f in result.done]
 
-            # the future is not running, cancel it
-            if not fs.done() and not fs.running():
-                fs.cancel()
+        self.tpool.shutdown(False)
+        self.poller.close()
 
-            # make sure we close the sockets after the graceful timeout
-            util.close(sock)
 
     def finish_request(self, fs):
+        if fs.cancelled():
+            fs.conn.close()
+            return
+
         try:
             (keepalive, conn) = fs.result()
             # if the connection should be kept alived add it
@@ -214,32 +233,22 @@ def finish_request(self, fs):
 
                 # register the connection
                 conn.set_timeout()
-                self._keep.append(conn)
+                with self._lock:
+                    self._keep.append(conn)
 
-                # add the socket to the event loop
-                self.poller.register(conn.sock, selectors.EVENT_READ,
-                        partial(self.handle_client, conn))
+                    # add the socket to the event loop
+                    self.poller.register(conn.sock, selectors.EVENT_READ,
+                            partial(self.reuse_connection, conn))
             else:
-                util.close(conn.sock)
+                self.nr -= 1
+                conn.close()
         except:
             # an exception happened, make sure to close the
             # socket.
-            util.close(fs.conn.sock)
-        finally:
-            # remove the future from our list
-            try:
-                self.futures.remove(fs)
-            except ValueError:
-                pass
+            self.nr -= 1
+            fs.conn.close()
 
     def handle(self, conn):
-        if not conn.init():
-            # connection kept alive
-            try:
-                self._keep.remove(conn)
-            except ValueError:
-                pass
-
         keepalive = False
         req = None
         try:
@@ -287,8 +296,6 @@ def handle_request(self, req, conn):
                     conn.listener.getsockname(), self.cfg)
             environ["wsgi.multithread"] = True
 
-            self.nr += 1
-
             if self.alive and self.nr >= self.max_requests:
                 self.log.info("Autorestarting worker after current request.")
                 resp.force_close()
@@ -296,6 +303,8 @@ def handle_request(self, req, conn):
 
             if not self.cfg.keepalive:
                 resp.force_close()
+            elif len(self._keep) >= self.max_keepalived:
+                resp.force_close()
 
             respiter = self.wsgi(environ, resp.start_response)
             try:


Test patch: