Repository: benoitc/gunicorn

Issue:
About gunicorn [CRITICAL] Worker Timeout
Recently, I found that my worker appears timeout at about 9 o 'clock every day.

Finally, one day I found that when the system time was modified (more than the current time), the worker unexpectedly appeared timeout! At this time, I can confirm that the daily worker timeout is caused by time synchronization.

I checked the relevant source code and found that the system time was used to determine whether the worker was timeout.

https://github.com/benoitc/gunicorn/blob/7af6f651c04511e631cbb888f527897d4273703c/gunicorn/arbiter.py#L497
About gunicorn [CRITICAL] Worker Timeout
Recently, I found that my worker appears timeout at about 9 o 'clock every day.

Finally, one day I found that when the system time was modified (more than the current time), the worker unexpectedly appeared timeout! At this time, I can confirm that the daily worker timeout is caused by time synchronization.

I checked the relevant source code and found that the system time was used to determine whether the worker was timeout.

https://github.com/benoitc/gunicorn/blob/7af6f651c04511e631cbb888f527897d4273703c/gunicorn/arbiter.py#L497


Discussion:
This probably cannot be fixed right now. The workers `chmod` a temporary file to mark liveness so it's not possible to use a monotonic time there.

The best workaround is probably to use a time daemon that handles smoothing the system clock changes.
hrm wouldn’t fic the file myime to a monotonic timestamp work?

On Wed 23 Jan 2019 at 07:15, Randall Leeds <notifications@github.com> wrote:

> This probably cannot be fixed right now. The workers chmod a temporary
> file to mark liveness so it's not possible to use a monotonic time there.
>
> The best workaround is probably to use a time daemon that handles
> smoothing the system clock changes.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/benoitc/gunicorn/issues/1965#issuecomment-456683618>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAA4osEE-2PpUbFKeGGD5mZpVW_2Wu6qks5vF_37gaJpZM4aODxy>
> .
>
-- 
Sent from my Mobile

Maybe it would work. I am reading now that `os.utime` supports a file descriptor argument for Python 3.3+, so we could use it on the worker temp files.

https://docs.python.org/3/library/os.html#os.utime

We could set the time using `time.monotonic` and I think it might work!
@skytoup do you want to try to make a patch?
This probably cannot be fixed right now. The workers `chmod` a temporary file to mark liveness so it's not possible to use a monotonic time there.

The best workaround is probably to use a time daemon that handles smoothing the system clock changes.
hrm wouldn’t fic the file myime to a monotonic timestamp work?

On Wed 23 Jan 2019 at 07:15, Randall Leeds <notifications@github.com> wrote:

> This probably cannot be fixed right now. The workers chmod a temporary
> file to mark liveness so it's not possible to use a monotonic time there.
>
> The best workaround is probably to use a time daemon that handles
> smoothing the system clock changes.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/benoitc/gunicorn/issues/1965#issuecomment-456683618>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAA4osEE-2PpUbFKeGGD5mZpVW_2Wu6qks5vF_37gaJpZM4aODxy>
> .
>
-- 
Sent from my Mobile

Maybe it would work. I am reading now that `os.utime` supports a file descriptor argument for Python 3.3+, so we could use it on the worker temp files.

https://docs.python.org/3/library/os.html#os.utime

We could set the time using `time.monotonic` and I think it might work!
@skytoup do you want to try to make a patch?

Resolving patch:
diff --git a/gunicorn/arbiter.py b/gunicorn/arbiter.py
--- a/gunicorn/arbiter.py
+++ b/gunicorn/arbiter.py
@@ -495,7 +495,7 @@ def murder_workers(self):
         workers = list(self.WORKERS.items())
         for (pid, worker) in workers:
             try:
-                if time.time() - worker.tmp.last_update() <= self.timeout:
+                if time.monotonic() - worker.tmp.last_update() <= self.timeout:
                     continue
             except (OSError, ValueError):
                 continue
diff --git a/gunicorn/workers/workertmp.py b/gunicorn/workers/workertmp.py
--- a/gunicorn/workers/workertmp.py
+++ b/gunicorn/workers/workertmp.py
@@ -4,6 +4,7 @@
 # See the NOTICE for more information.
 
 import os
+import time
 import platform
 import tempfile
 
@@ -40,10 +41,11 @@ def __init__(self, cfg):
             raise
 
     def notify(self):
-        os.utime(self._tmp.fileno())
+        new_time = time.monotonic()
+        os.utime(self._tmp.fileno(), (new_time, new_time))
 
     def last_update(self):
-        return os.fstat(self._tmp.fileno()).st_ctime
+        return os.fstat(self._tmp.fileno()).st_mtime
 
     def fileno(self):
         return self._tmp.fileno()


Test patch: