Repository: benoitc/gunicorn

Issue:
unix socket goes missing when max_requests is used
nginx: 1.6.2
gunicorn: 19.1.1
eventlet workers
ubuntu 12.04

When I define a max_requests config option, after that limit has been reached and the worker has been restarted, the unix socket I specify goes missing.  I have to completely restart the master process for the socket to reappear again. 

A stacktrace also appears in the log with the following: 

```
2015-01-14_22:37:12.28211 [2015-01-14 14:37:12 +0000] [20408] [INFO] Autorestarting worker after current request.
2015-01-14_22:37:12.28921 Traceback (most recent call last):
2015-01-14_22:37:12.28924   File "/home/appuser/python_envs/application/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 455, in fire_timers
2015-01-14_22:37:12.28969     timer()
2015-01-14_22:37:12.28971   File "/home/appuser/python_envs/application/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
2015-01-14_22:37:12.28987     cb(*args, **kw)
2015-01-14_22:37:12.28989   File "/home/appuser/python_envs/application/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main
2015-01-14_22:37:12.29013     result = function(*args, **kwargs)
2015-01-14_22:37:12.29015   File "/home/appuser/python_envs/application/local/lib/python2.7/site-packages/gunicorn/workers/geventlet.py", line 64, in handle
2015-01-14_22:37:12.29031     raise eventlet.StopServe()
2015-01-14_22:37:12.29037 StopServe
```



Discussion:
That traceback is expected, but should be fixed already for the next release.
If the socket is disappearing, though, that is a serious problem.

Serious problem indeed.  I haven't had much time to trace it, but using a tcp bind survives a worker reset.   I can also manually SIGTERM workers and the socket remains.  Only the workers that restart themselves with max_requests appear to rm the socket.

do you reproduce the same without eventlet?

bump.

I can reproduce this with eventlet but not with the sync worker.

This can be solved easily like so:

``` patch
diff --git a/gunicorn/workers/geventlet.py b/gunicorn/workers/geventlet.py
index 48b63fb..061d4a8 100644
--- a/gunicorn/workers/geventlet.py
+++ b/gunicorn/workers/geventlet.py
@@ -108,7 +108,7 @@ class EventletWorker(AsyncWorker):
     def run(self):
         acceptors = []
         for sock in self.sockets:
-            gsock = GreenSocket(sock)
+            gsock = GreenSocket(sock.dup())
             gsock.setblocking(1)
             hfun = partial(self.handle, gsock)
             acceptor = eventlet.spawn(_eventlet_serve, gsock, hfun,
```

The issue is that eventlet is removing the socket when it closes it.

Rather, eventlet's `GreenSocket` has a `__del__` handler that calls close.

Duplicating the socket seems to prevent the close from cleaning it up.

Found the same issue when using eventlet, tried patch provided by @tilgovi and it looks good.

I got distracted looking into the behavior across the different workers and trying to figure out what should happen when gunicorn exits, but maybe that can be left to another issue. This brings us closer to parity. There's still the issue that the socket isn't cleaned up after gunicorn exits completely, at least on some workers (including eventlet with this patch).

@benoitc how should I proceed?


Resolving patch:
diff --git a/gunicorn/sock.py b/gunicorn/sock.py
--- a/gunicorn/sock.py
+++ b/gunicorn/sock.py
@@ -57,7 +57,6 @@ def close(self):
             self.sock.close()
         except socket.error as e:
             self.log.info("Error while closing socket %s", str(e))
-        time.sleep(0.3)
         del self.sock
 
 
@@ -104,6 +103,7 @@ def __init__(self, addr, conf, log, fd=None):
                     os.remove(addr)
                 else:
                     raise ValueError("%r is not a socket" % addr)
+        self.parent = os.getpid()
         super(UnixSocket, self).__init__(addr, conf, log, fd=fd)
 
     def __str__(self):
@@ -118,7 +118,8 @@ def bind(self, sock):
 
     def close(self):
         super(UnixSocket, self).close()
-        os.unlink(self.cfg_addr)
+        if self.parent == os.getpid():
+            os.unlink(self.cfg_addr)
 
 
 def _sock_type(addr):
diff --git a/gunicorn/workers/geventlet.py b/gunicorn/workers/geventlet.py
--- a/gunicorn/workers/geventlet.py
+++ b/gunicorn/workers/geventlet.py
@@ -25,7 +25,6 @@
 from gunicorn.http.wsgi import sendfile as o_sendfile
 from gunicorn.workers.async import AsyncWorker
 
-
 def _eventlet_sendfile(fdout, fdin, offset, nbytes):
     while True:
         try:
@@ -87,11 +86,11 @@ def patch_sendfile():
 class EventletWorker(AsyncWorker):
 
     def patch(self):
+        hubs.use_hub()
         eventlet.monkey_patch(os=False)
         patch_sendfile()
 
     def init_process(self):
-        hubs.use_hub()
         self.patch()
         super(EventletWorker, self).init_process()
 
@@ -119,7 +118,11 @@ def run(self):
 
         while self.alive:
             self.notify()
-            eventlet.sleep(1.0)
+            try:
+                eventlet.sleep(1.0)
+            except AssertionError:
+                self.alive = False
+                break
 
         self.notify()
         try:


Test patch: