Repository: celery/celery

Issue:
Restarting celery issues and better supervisord config file
I use supervisord config file based on the example in celery repository and have some issues when celeryd restart is occured: sometimes task processing silently stops after restarting celery without any error messages in logs. Processes remains visible in process list.

Finally I figured out that sometimes when processes are restarted celery spawns additional process that is not managed by supervisord and this leads to these bugs. So I started to watch `ps` output carefully after each restart and kill extra processes manually via kill <id>. After killing these processes tasks begin to be executed properly. This is sort of hack that solve a problem for a week or so.

And today I think the real reason is found. The default supervisord value for 'stopwaitsecs' option is 10s. This means that after 10s celery process will be killed with KILL signal instead of TERM. It seems that celery don't like being killed and try to spawn additional process in that case.

So I think it'll be good to add something like 'stopwaitsecs=600' to all supervisord example config files (from faq: "You should never stop celeryd with the KILL signal (-9), unless you’ve tried TERM a few times and waited a few minutes to let it get a chance to shut down.") and investigate celeryd behaviour on KILL signal: it is mentioned in docs that tasks will be lost (and it is tolerably in many cases) but the issue with spawned process is a bit weird.



Discussion:
Processes spawned when receiving the `KILL` signal is weird indeed. I don't see that behavior when used outside of `supervisord`, so maybe this is something caused by it?

If you install the `setproctitle` module, celery should report the kind of process in `ps` listings, could you do that to investigate what kind of process is created?

(`easy_install setproctitle`)

Setting the timeout to `600` is probably good. Is there any setting for infinity (maybe with a warning if it takes too long)? When `celeryd` is killed via `TERM` (which is the preferred shutdown signal) it stops receiving messages and waits for the currently executing tasks to finish. And I guess for most applications, termination mid-execution is not acceptable.

As for process spawning: setproctitle and watching for process ids was helpful. It is not process spawning. Worker processes remain alive when parent process is killed. 
This is a simulation of supervisord restart with manual killing and zero timeout:

```
 4976 ?        Ss     0:00 /usr/bin/python /usr/bin/supervisord --pidfile /var/run/supervisord.pid
 5422 ?        S      0:01  \_ [celerybeat] --schedule=/var/lib/celery/celerybeat-schedule-nadovmeste --loglevel=INFO                                                             
 6101 ?        Sl     0:00  \_ [celeryd.MainProcess] Running... (--loglevel=INFO)                                                           
 6108 ?        S      0:00      \_ [celeryd.PoolWorker-1]                                                                                       
 nadovmeste:~# kill 6101 & kill -9 6101 &
```

ps -afx:

```
 4976 ?        Ss     0:00 /usr/bin/python /usr/bin/supervisord --pidfile /var/run/supervisord.pid
 5422 ?        S      0:01  \_ [celerybeat] --schedule=/var/lib/celery/celerybeat-schedule-nadovmeste --loglevel=INFO                                                             
 6867 ?        Sl     0:00  \_ [celeryd.MainProcess] Running... (--loglevel=INFO)                                                           
 6875 ?        S      0:00      \_ [celeryd.PoolWorker-1]                                                                                       
 6108 ?        S      0:00 [celeryd.PoolWorker-1]       
```

I was able reproduce this only with such artifical race between `kill` and `kill -9`. Sometimes worker gets killed properly. The issue seems to be supervisord-specific because when I start celeryd from console I have no luck reproducing it.

I was able to reproduce this with console-started scripts after several attempts:

```
/home/nadovmeste/envs/nadovmeste/bin/python /home/nadovmeste/src/nadovmeste/manage.py celeryd -B --loglevel=INFO&
```

and then in another terminal session:

```
nadovmeste:~# ps -afx

 6450 ?        Ss     0:00  \_ sshd: root@pts/2 
 6452 pts/2    Ss+    0:00      \_ -bash
 9343 pts/2    Sl     0:00          \_ [celeryd.MainProcess] Running... (-B --loglevel=INFO)                                                           
 9350 pts/2    S      0:00              \_ [celeryd.PoolWorker-2]                                                                                          
 9355 pts/2    S      0:00              \_ [celerybeat]     

nadovmeste:~# kill 9343 & kill -9 9343

nadovmeste:~# ps -afx

 4526 ?        Ss     0:00  \_ sshd: root@pts/1 
 4529 pts/1    Ss     0:00  |   \_ -bash
 9366 pts/1    R+     0:00  |       \_ ps -afx
 6450 ?        Ss     0:00  \_ sshd: root@pts/2 
 6452 pts/2    Ss+    0:00      \_ -bash    
 ...
 9350 pts/2    S      0:00 [celeryd.PoolWorker-2]                                                                                          
 9355 pts/2    S      0:00 [celerybeat]
```

Haven't found any special option for infinite timeout with warning in supervisord docs. Probably very large number will suffice if it is what we want.

Maybe it's something related to celerybeat because the I was able to reproduce the issue for console-started celeryd only after using the `-B` option.

If i am testing some celery tasks locally and i use the -B option sometimes the process is not killed when i used ctrl-c.

I can't reproduce this locally. Btw, are you running the master branch? I just fixed a bug that could hang shutdown. If you could test with this it would be nice.

Yes, I'm running the latest master branch. I saw your bug-fixing commit and hoped that it will help but it seems that it doesn't help in my case: the latest celery seems to behave the same. But it is possible that the initial problem is solved - I check this only with an immediate kill. Can't wrap my hand around it now :) The ctrl-c issue is not reproducible with my setup.

So the bug report, simplified: http://gist.github.com/401028 . The results are always the same (not sometimes). I have some periodic tasks and some non-periodic. Tasks are simple and don't take much time to finish. Is it a bug that children processes stay alive after killing the main process? If so and you can't reproduce it then I'll try to provide the minimal project.

The celerybeat killing behaviour is interesting: when I kill hanging(?) celerybeat process the hanging(?) worker process also shutdowns.

@kmike I still can't reproduce with the commands above. Maybe because I'm on OS X, or maybe you're running Python 2.5? (I'm running 2.6.1)

Could run it with `--loglevel=DEBUG?` It could provide some info about where it stops.

The celerybeat process is started by the main process, so I'm assuming the main process is waiting
for the celerybeat to exit before it kills the remaining pool processes.

I thought that the main process was killed: it is not visible in process list. Don't have much experience with process management though.

My setup was Debian Lenny + python 2.5. 

I'll try to run celeryd with --loglevel=DEBUG and to reproduce it on my macbook. 

hmm, you're right of course. It's almost like the beat process takes ownership of the pool processes.

I just tried to reproduce on Debian Lenny with python 2.5, and it works just right there.
Tried killing with both TERM and INT.

Ask, thank you for help.

I think that initial problem was solved with increased supervisord timeout and your bug-fixing commit. The simulation was incorrect because I use `kill -9` commands and they send KILL signal instead of TERM. With TERM signal processes are getting killed properly.

Supervisord use TERM signal so all should be fine.

But the thing that scares me a bit is that the initial bug wasn't investigated. I'll try to reproduce it and let you know.

Ah! I'm so sorry. I didn't read the issue carefully enough. Yes! That's exactly what happens when you kill it with SIGKILL. The 9 signal can't be catched, so there's nothing we can do about this AFAIK.

In case you are still having issues terminating your Celery workers, you might try setting `stopasgroup=true` before increasing your `stopwaitsecs`.


Resolving patch:
diff --git a/celery/concurrency/prefork.py b/celery/concurrency/prefork.py
--- a/celery/concurrency/prefork.py
+++ b/celery/concurrency/prefork.py
@@ -41,6 +41,8 @@ def process_initializer(app, hostname):
     Initialize the child pool process to ensure the correct
     app instance is used and things like logging works.
     """
+    # Each running worker gets SIGKILL by OS when main process exits.
+    platforms.set_pdeathsig('SIGKILL')
     _set_task_join_will_block(True)
     platforms.signals.reset(*WORKER_SIGRESET)
     platforms.signals.ignore(*WORKER_SIGIGNORE)
diff --git a/celery/platforms.py b/celery/platforms.py
--- a/celery/platforms.py
+++ b/celery/platforms.py
@@ -17,6 +17,7 @@
 from contextlib import contextmanager
 
 from billiard.compat import close_open_fds, get_fdmax
+from billiard.util import set_pdeathsig as _set_pdeathsig
 # fileno used to be in this module
 from kombu.utils.compat import maybe_fileno
 from kombu.utils.encoding import safe_str
@@ -708,6 +709,16 @@ def strargv(argv):
     return ''
 
 
+def set_pdeathsig(name):
+    """Sends signal ``name`` to process when parent process terminates."""
+    if signals.supported('SIGKILL'):
+        try:
+            _set_pdeathsig(signals.signum('SIGKILL'))
+        except OSError:
+            # We ignore when OS does not support set_pdeathsig
+            pass
+
+
 def set_process_title(progname, info=None):
     """Set the :command:`ps` name for the currently running process.
 


Test patch:
diff --git a/t/unit/concurrency/test_prefork.py b/t/unit/concurrency/test_prefork.py
--- a/t/unit/concurrency/test_prefork.py
+++ b/t/unit/concurrency/test_prefork.py
@@ -53,11 +53,18 @@ def get(self):
         return self.value
 
 
+@patch('celery.platforms.set_mp_process_title')
 class test_process_initializer:
 
+    @staticmethod
+    def Loader(*args, **kwargs):
+        loader = Mock(*args, **kwargs)
+        loader.conf = {}
+        loader.override_backends = {}
+        return loader
+
     @patch('celery.platforms.signals')
-    @patch('celery.platforms.set_mp_process_title')
-    def test_process_initializer(self, set_mp_process_title, _signals):
+    def test_process_initializer(self, _signals, set_mp_process_title):
         with mock.restore_logging():
             from celery import signals
             from celery._state import _tls
@@ -67,13 +74,7 @@ def test_process_initializer(self, set_mp_process_title, _signals):
             on_worker_process_init = Mock()
             signals.worker_process_init.connect(on_worker_process_init)
 
-            def Loader(*args, **kwargs):
-                loader = Mock(*args, **kwargs)
-                loader.conf = {}
-                loader.override_backends = {}
-                return loader
-
-            with self.Celery(loader=Loader) as app:
+            with self.Celery(loader=self.Loader) as app:
                 app.conf = AttributeDict(DEFAULTS)
                 process_initializer(app, 'awesome.worker.com')
                 _signals.ignore.assert_any_call(*WORKER_SIGIGNORE)
@@ -100,6 +101,19 @@ def Loader(*args, **kwargs):
                 finally:
                     os.environ.pop('CELERY_LOG_FILE', None)
 
+    @patch('celery.platforms.set_pdeathsig')
+    def test_pdeath_sig(self, _set_pdeathsig, set_mp_process_title):
+        with mock.restore_logging():
+            from celery import signals
+            on_worker_process_init = Mock()
+            signals.worker_process_init.connect(on_worker_process_init)
+            from celery.concurrency.prefork import process_initializer
+
+            with self.Celery(loader=self.Loader) as app:
+                app.conf = AttributeDict(DEFAULTS)
+                process_initializer(app, 'awesome.worker.com')
+            _set_pdeathsig.assert_called_once_with('SIGKILL')
+
 
 class test_process_destructor:
 
diff --git a/t/unit/utils/test_platforms.py b/t/unit/utils/test_platforms.py
--- a/t/unit/utils/test_platforms.py
+++ b/t/unit/utils/test_platforms.py
@@ -18,7 +18,7 @@
                               close_open_fds, create_pidlock, detached,
                               fd_by_path, get_fdmax, ignore_errno, initgroups,
                               isatty, maybe_drop_privileges, parse_gid,
-                              parse_uid, set_mp_process_title,
+                              parse_uid, set_mp_process_title, set_pdeathsig,
                               set_process_title, setgid, setgroups, setuid,
                               signals)
 from celery.utils.text import WhateverIO
@@ -170,6 +170,18 @@ def test_setitem_raises(self, set):
         signals['INT'] = lambda *a: a
 
 
+class test_set_pdeathsig:
+
+    def test_call(self):
+        set_pdeathsig('SIGKILL')
+
+    @t.skip.if_win32
+    def test_call_with_correct_parameter(self):
+        with patch('celery.platforms._set_pdeathsig') as _set_pdeathsig:
+            set_pdeathsig('SIGKILL')
+            _set_pdeathsig.assert_called_once_with(signal.SIGKILL)
+
+
 @t.skip.if_win32
 class test_get_fdmax: