Repository: benoitc/gunicorn

Issue:
'I/O operation on closed file' on SIGHUP
```
2014-08-25 13:51:42 [16744] [INFO] Handling signal: hup
2014-08-25 13:51:42 [16744] [INFO] Hang up: Master
2014-08-25 13:51:42 [7683] [INFO] Booting worker with pid: 7683
2014-08-25 13:51:42 [7684] [INFO] Booting worker with pid: 7684
2014-08-25 13:51:42 [7685] [INFO] Booting worker with pid: 7685
2014-08-25 13:51:42 [7686] [INFO] Booting worker with pid: 7686
2014-08-25 13:51:42 [7687] [INFO] Booting worker with pid: 7687
2014-08-25 13:51:42 [7688] [INFO] Booting worker with pid: 7688
2014-08-25 13:51:42 [7689] [INFO] Booting worker with pid: 7689
2014-08-25 13:51:42 [38739] [INFO] Worker exiting (pid: 38739)
2014-08-25 13:51:42 [38743] [INFO] Worker exiting (pid: 38743)
2014-08-25 13:51:42 [38745] [INFO] Worker exiting (pid: 38745)
2014-08-25 13:51:42 [38744] [INFO] Worker exiting (pid: 38744)
2014-08-25 13:51:42 [38741] [INFO] Worker exiting (pid: 38741)
2014-08-25 13:51:42 [38742] [INFO] Worker exiting (pid: 38742)
2014-08-25 13:51:42 [38740] [INFO] Worker exiting (pid: 38740)
2014-08-25 13:51:42 [38746] [INFO] Worker exiting (pid: 38746)
2014-08-25 13:51:42 [7690] [INFO] Booting worker with pid: 7690
2014-08-25 13:51:43 [16744] [INFO] Unhandled exception in main loop:
Traceback (most recent call last):
  File "/home/httpapi/.virtualenvs/hizonenum/lib/python2.7/site-packages/gunicorn/arbiter.py", line 174, in run
    self.sleep()
  File "/home/httpapi/.virtualenvs/hizonenum/lib/python2.7/site-packages/gunicorn/arbiter.py", line 308, in sleep
    oldest = min(w.tmp.last_update() for w in worker_values)
  File "/home/httpapi/.virtualenvs/hizonenum/lib/python2.7/site-packages/gunicorn/arbiter.py", line 308, in <genexpr>
    oldest = min(w.tmp.last_update() for w in worker_values)
  File "/home/httpapi/.virtualenvs/hizonenum/lib/python2.7/site-packages/gunicorn/workers/workertmp.py", line 50, in last_update
    return os.fstat(self._tmp.fileno()).st_ctime
ValueError: I/O operation on closed file

2014-08-25 13:51:43 [7689] [INFO] Worker exiting (pid: 7689)
2014-08-25 13:51:43 [7683] [INFO] Worker exiting (pid: 7683)
2014-08-25 13:51:43 [7687] [INFO] Worker exiting (pid: 7687)
2014-08-25 13:51:43 [7690] [INFO] Worker exiting (pid: 7690)
2014-08-25 13:51:43 [7684] [INFO] Worker exiting (pid: 7684)
2014-08-25 13:51:43 [7688] [INFO] Worker exiting (pid: 7688)
2014-08-25 13:51:43 [7686] [INFO] Worker exiting (pid: 7686)
2014-08-25 13:51:43 [7685] [INFO] Worker exiting (pid: 7685)
```



Discussion:
Interestingly, I had the same issue, but at a random point in time:

```
[2014-08-30 03:13:25 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:15513)
[2014-08-30 03:13:26 +0000] [18391] [INFO] Booting worker with pid: 18391
[2014-08-30 03:13:27 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:16607)
[2014-08-30 03:13:28 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:17159)
[2014-08-30 03:13:28 +0000] [18428] [INFO] Booting worker with pid: 18428
[2014-08-30 03:13:28 +0000] [18429] [INFO] Booting worker with pid: 18429
[2014-08-30 03:13:30 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:16850)
[2014-08-30 03:13:31 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:16998)
[2014-08-30 03:13:31 +0000] [18538] [INFO] Booting worker with pid: 18538
[2014-08-30 03:13:31 +0000] [18539] [INFO] Booting worker with pid: 18539
[2014-08-30 03:13:34 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:17032)
[2014-08-30 03:13:35 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:17125)
[2014-08-30 03:13:35 +0000] [18609] [INFO] Booting worker with pid: 18609
[2014-08-30 03:13:35 +0000] [18610] [INFO] Booting worker with pid: 18610
[2014-08-30 03:13:36 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:16812)
[2014-08-30 03:13:37 +0000] [16812] [INFO] Worker exiting (pid: 16812)
Exception _mysql_exceptions.ProgrammingError: (2014, "Commands out of sync; you can't run this command now") in <bound method Cursor.__del__ of <MySQLdb.cursors.Cursor object at 0x38ae210>> ignored
[2014-08-30 03:13:37 +0000] [18682] [INFO] Booting worker with pid: 18682
127.0.0.1 - - [30/Aug/2014:03:13:37 +0200] "POST [redacted] HTTP/1.0" 200 - "[redacted]" "Mozilla/5.0 (Linux; Android 4.3; SAMSUNG-SGH-I747 Build/JSS15J) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.141 Mobile Safari/537.36"
127.0.0.1 - - [30/Aug/2014:03:13:38 +0200] "GET [redacted] HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0"
[2014-08-30 03:15:00 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:18539)
[2014-08-30 03:15:01 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:18391)
[2014-08-30 03:15:01 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:18538)
[2014-08-30 03:15:01 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:18609)
[2014-08-30 03:15:01 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:18610)
[2014-08-30 03:15:01 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:18428)
[2014-08-30 03:15:01 +0000] [9531] [CRITICAL] WORKER TIMEOUT (pid:18429)
[2014-08-30 03:15:01 +0000] [18919] [INFO] Booting worker with pid: 18919
[2014-08-30 03:15:01 +0000] [9531] [INFO] Unhandled exception in main loop:
Traceback (most recent call last):
  File "/app/shared/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 174, in run
    self.sleep()
  File "/app/shared/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 308, in sleep
    oldest = min(w.tmp.last_update() for w in worker_values)
  File "/app/shared/venv/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 308, in <genexpr>
    oldest = min(w.tmp.last_update() for w in worker_values)
  File "/app/shared/venv/local/lib/python2.7/site-packages/gunicorn/workers/workertmp.py", line 50, in last_update
    return os.fstat(self._tmp.fileno()).st_ctime
ValueError: I/O operation on closed file

[2014-08-30 03:15:01 +0000] [18919] [INFO] Worker exiting (pid: 18919)
[2014-08-30 03:15:01 +0000] [18682] [INFO] Worker exiting (pid: 18682)
```

gunicorn==19.1.1


Resolving patch:
diff --git a/gunicorn/arbiter.py b/gunicorn/arbiter.py
--- a/gunicorn/arbiter.py
+++ b/gunicorn/arbiter.py
@@ -305,7 +305,16 @@ def sleep(self):
         """
         if self.WORKERS:
             worker_values = list(self.WORKERS.values())
-            oldest = min(w.tmp.last_update() for w in worker_values)
+
+            oldest = time.time()
+            for w in worker_values:
+                try:
+                    last_update = w.tmp.last_update()
+                    if last_update < oldest:
+                        oldest = last_update
+                except ValueError:
+                    pass
+
             timeout = self.timeout - (time.time() - oldest)
             # The timeout can be reached, so don't wait for a negative value
             timeout = max(timeout, 1.0)


Test patch: