Repository: celery/celery

Issue:
SIGTERM does not do a warm shutdown.
According to the documentation here: http://celery.readthedocs.org/en/latest/userguide/workers.html#process-signals

I should be able to send a SIGTERM to my running worker process and have it finish the task it is currently working on before it shuts down.  However when I call sigterm the process exits immediately with the following traceback.

```
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/billiard/pool.py", line 1171, in mark_as_worker_lost
    human_status(exitcode)),
WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
```

I'm using Celery 3.1.18 with django 1.4, is there something special that has to be done to integrate the signal handlers with django?

Also posted here on the mailing list: https://groups.google.com/forum/#!topic/celery-users/t8g5KvIvQZ8



Discussion:
Closing this, as we don't have the resources to complete this task.

May be fixed in master, let's see if comes back after 4.0 release.

@ask any suggestions on where to start with trying to fix this?  I'm happy to try to dig into it if you can provide me a starting point.

I'm not sure, are you running on Heroku perhaps?  TERM does not propagate to child processes, so I'm unsure why this would happen.

We are running in ec2 and in vagrant, the signal is sent from supervisord to the celery process.

Are you using stopasgroup or killasgroup in your supervisor config?  If so, you may not want to do that.

We are doing that.  What's the reason why that would be causing issues?

Supervisor will send the SIGTERM down to all the forks (with stopasgroup) which I doubt would handle the shutdown that way.  I have tried in the past (unsuccessfully) to catch the SIGTERM for gracefully exiting long operations, however I haven't had time to dig into why I had trouble doing so. 

If the work itself only gets the SIGTERM, it will wait for the forks to complete their tasks before exiting I think. 

If you want the worker to drop currently executing tasks you should send SIGQUIT to parent, not send TERM to child processes.

@ask Thanks :-)  I'll play with that to see the behavior of it.  

As for this issue, I think it is solved really. Having supervisor set to stopasgroup will cause the described problem and @feanil says they are doing that. Best, I think, would be just to set to stopasgroup back to false and be done as the SIGTERMed worker will get will wait on all tasks to complete (what he was looking for). 

Thanks! Then I guess we can close again until further information :)

I'm looking into this problem with @feanil, and I am mystified about the right way to get the behavior we want.
What we want: for "supervisor restart" to restart celery workers, but let them finish their current task.
We found that the default supervisor signal (TERM) will restart the workers, but will cause them to abort their current task.
Changing the signal to INT makes the worker restart if it is idle, but if busy, will finish its task and then restart.
Does this match your understanding of how celery workers behave?

What we are trying to debug is a situation where workers (even idle workers) get stuck in a supervisor STOPPING state.  We don't know why it is.  Switching back to TERM for the stop signal seems to fix the problem, but aborts in-progress tasks.  Is there a way to get what we need? 

I guess I would need to know more about your configuration first, but sending TERM to the master/parent process initiates a warm shutdown sequence.  It closes the connection to the broker, waits for all currently executing tasks to finish and then terminates.

Currently executing tasks, means any task in the 'active' state,  tasks merely reserved by the worker
is left unacknowledged so that the broker resends them as soon as the worker closes the broker connection.

I am definitely seeing celery tasks aborting when supervisor sends TERM.  I don't know how to figure out why that is different than your description.

@nedbat When you have stopasgroup enabled, this will send a TERM signal to the worker plus all worker forks causing the forks to abort. You want to turn off stopasgroup (and update supervisor) and the TERM signal will only get sent to the worker. 

Apologies if I am not understanding the issue properly. I use supervisor with celery very often, though. If you want to post a gist to your supervisor config I'll be happy to look at it.

The TERM signal being sent to the process _group_ would certainly explain what you're seeing!

@ask @mverrilli thanks for walking me through it.  Removing stopasgroup, and using the default signal of TERM seems good. :)

I am finding celery multi is exiting immediately on SIGTERM rather than waiting for tasks to finish, is that case applicable to this issue? Also I'm sending SIGTERM to all processes as reported by `ps aux`, should we perhaps only send to the root process or something?

@carn1x you should only be sending the SIGTERM signal to the parent process, not the child threads.

Something like this would probably work for you -- it's been working for me.

```
# find parent celery process id                                                                                                                                                                             
CELERY_PID=$(ps aux --sort=start_time | grep 'celery worker' | grep 'bin/celery' | head -1 | awk '{print $2}')

# warm shutdown builder via SIGTERM signal to parent celery process                                                                                                                                         
kill -TERM $CELERY_PID
```

I use this in a bash script before doing some cleanup maint. I actually wait to confirm the thread stopped before continuing:

```
while kill -0 $CELERY_PID 2> /dev/null; do
  #make sure you're not waiting forever
done
```

If you read the comments above by @ask, he clarified that the process group shouldn't get killed, just the parent. I can confirm the code above works great for me on production systems. I hope this helps.

BTW, that above assumes you only have one worker running -- with a set of threads. If you have multiple workers, you'll have to adjust that to parse out the correct parent process ID. The main point is that you need to kill the parent (the first celery worker thread).

thanks @rocksfrow, working great :)

@ask current example [supervisord/celeryd.conf](https://github.com/celery/celery/blob/master/extra/supervisord/celeryd.conf) uses `killasgroup=true`. Maybe the default should be false?
Expected behavior of acks_late and a SIGTERM

Going off ask's reply above..

So let's assume you have a worker working on a 10 second task with acks_late=True. Halfway through, the parent process gets a SIGTERM.

Based on what Ask said, and what I have experienced.. what happens is

1) The Parent immediately cuts the connection to the broker. This triggers the late_ack to trip, and another task to get assigned the task that is in progress.

2) The parent waits for the child to finish executing his current task, completing the task once

3) Another worker is re-given this task (due to 1 and late_ack), and this task is duplicated a second time.


Does this sound correct?  Is there any clean way to both use late_ack AND do a graceful SIGTERM for a shutdown?
Great comment @brianwawok, warm shutdown with acks_late looks broken. The task being immediately reassigned does not meet any definition of warm shutdown. Although this should not cause correctness issues because acks_late tasks are supposed to be idempotent.
I have small workaround to fix it now
```python
import signal

from celery import Celery, Task

class MyBaseTask(Task):
`logger = get_task_logger('my_logger)`
    def __call__(self, *args, **kwargs):
        signal.signal(signal.SIGTERM,
                      lambda signum, frame: self.logger.info('SIGTERM received, wait till the task finished'))
        return super().__call__(*args, **kwargs)

app = Celery('my_app')
app.Task = MyBaseTask
````
For me it worked

could you please add a pr with your some proposed fix? @namevic 

Resolving patch:
diff --git a/celery/app/task.py b/celery/app/task.py
--- a/celery/app/task.py
+++ b/celery/app/task.py
@@ -2,6 +2,7 @@
 """Task implementation: request context and the task base class."""
 from __future__ import absolute_import, unicode_literals
 
+import signal
 import sys
 
 from billiard.einfo import ExceptionInfo
@@ -20,6 +21,7 @@
 from celery.utils import abstract
 from celery.utils.functional import mattrgetter, maybe_list
 from celery.utils.imports import instantiate
+from celery.utils.log import get_logger
 from celery.utils.nodenames import gethostname
 from celery.utils.serialization import raise_with_context
 
@@ -388,6 +390,10 @@ def add_around(cls, attr, around):
         setattr(cls, attr, meth)
 
     def __call__(self, *args, **kwargs):
+        logger = get_logger(__name__)
+        handle_sigterm = lambda signum, frame: \
+            logger.info('SIGTERM received, waiting till the task finished')
+        signal.signal(signal.SIGTERM, handle_sigterm)
         _task_stack.push(self)
         self.push_request(args=args, kwargs=kwargs)
         try:


Test patch: