Repository: celery/celery

Issue:
Retry tasks which failed because of SIGKILL
Hello,

We are currently using celery with Broker=RabbitMQ & Backend=Redis.  Our setup runs on Kubernetes with auto-scaling enabled.  (Our number of pods scale up / down based on resource utilization).  We use the Forked Process mode to run our celery worker.

When the size of our cluster increases, everything works great.  However, when the pods turn down, the tasks that were running on the pods get marked as FAILED.  The sequence of actions is as follows:
1. Resource consumption decreases
2. Kubernetes realizes that pods that were scaled up now need to be turned down
3. Kubernetes sends `SIGKILL` signal to the pods that should turn down
4. Celery intercepts the signals and turns down all the Forked Processes
5. The tasks that were running on the processes return their execution back to the Main Process
6. The main process marks all the running tasks as FAILED

When the main process is the one that marks the running operations as FAILED, the `retry` code is never executed.

We do want to recover these tasks as they are part of a long running process.  Celery should provide a way to recover these tasks.


Discussion:
The thing here is about acknowledgement.
By default Celery will acknowledge a task right before a worker executes it. If the worker then fails unexpectedly the task is already marked as acknowledged and no retry policy will apply. This might seem a bit counterintuitive but it is the default because of the alternative.
The alternative is to set `acks_late=True` which will make Celery acknowledge a task *after* it's been run. That means if a broker fail unexpectedly while running a task then that task will not be marked as acknowledged and the retry policy will apply.

Now, the issue with using `acks_late=True` is that you have to make sure your task takes this into consideration and either is able to pickup from where it left or whatever it does is idempotent. Meaning, running the same code over and over again will always yield the same result.
Because of this caveat Celery chooses to acknowledge a task right after a worker executes it in case the task has some undesired result if executed multiple times.
Thanks a lot for the reply.  I should have clarified: we are setting `acks_late=True` for the tasks.  These are the parameter list for our task signature:
```
@cel.task(bind=True, acks_late=True, ignore_result=True, autoretry_for=(Exception,))
```

We have made it such that the task execution is idempotent, so even if there is a retry on the task, we should be OK.
OK, so I looked at the documentation again, and looks like I missed the config setting: `task_reject_on_worker_lost` which is exactly the [feature](http://docs.celeryproject.org/en/latest/userguide/configuration.html#task-reject-on-worker-lost) I was looking for. 

While this solves part-1 of the problem, it still does't solve the second part.  If this particular task happens to be part of a `group` in a `chord`, the error is propagated to a chord-error and the chord completely fails.  Not sure if there is a way to fix this correctly. 
Ah, yes. Forgot you also need `task_reject_on_worker_lost`.
Can you show the stacktrace when the group example happens?
Here is a small program to reproduce the error:

-- server.py
```
import celery
import time

cel = celery.Celery(
	'experiments',
	backend='redis://localhost:6379',
	broker='amqp://localhost:5672'
)

cel.conf.update(
  CELERYD_PREFETCH_MULTIPLIER = 1,
  CELERY_REJECT_ON_WORKER_LOST=True,
  CELERY_TASK_REJECT_ON_WORKER_LOST=True,
)

@cel.task(bind=True, acks_late=True, name='tasks.long_running_task', queue='worker.experiment')
def long_running_task(self, index):
	if index >= 0 and index <= 3:
		print("Sleeping for 30 secs")
		time.sleep(30.0)


@cel.task(bind=True, acks_late=True, name='tasks.callback_task', queue='worker.experiment')
def callback_task(self):
	print("Completed the job successfully.")
```
Run the server using the following command:
```
celery -A worker_failure worker -l info --concurrency=4 -Q worker.experiment
```

-- client.py
```
import celery
import worker_failure

if __name__ == '__main__':
	res = celery.chord(
		worker_failure.long_running_task.s(i) for i in range(4)
	)(worker_failure.callback_task.s())
	res.get()
```

Run the client using the following command: `python main.py`

Once the tasks are received, kill one of the processes by using the following command:
```
ps -awwwx | grep celery | tail -n 1 | awk '{print $1}' | xargs kill -9
```

This should kill the last process (ideally :) )

On the server side, you'll see the following trace:
```
[2018-10-14 10:01:55,639: ERROR/ForkPoolWorker-1] Chord callback for '2af041bb-809d-4fbe-b6df-2afb457f693b' raised: ChordError(u"Dependency d1cb5de9-7a6b-4442-a52c-5cad85c04c10 raised WorkerLostError(u'Worker exited prematurely: signal 9 (SIGKILL).',)",)
Traceback (most recent call last):
  File "/Users/shaunak/venv/lib/python2.7/site-packages/celery/backends/redis.py", line 289, in on_chord_part_return
    callback.delay([unpack(tup, decode) for tup in resl])
  File "/Users/shaunak/venv/lib/python2.7/site-packages/celery/backends/redis.py", line 242, in _unpack_chord_result
    raise ChordError('Dependency {0} raised {1!r}'.format(tid, retval))
ChordError: Dependency d1cb5de9-7a6b-4442-a52c-5cad85c04c10 raised WorkerLostError(u'Worker exited prematurely: signal 9 (SIGKILL).',)
```

This is not correct, since I do see the task being successful:
```
[2018-10-14 10:01:24,008: INFO/ForkPoolWorker-5] Task tasks.long_running_task[d1cb5de9-7a6b-4442-a52c-5cad85c04c10] succeeded in 30.007873791s: None
```

I think that the order of handling of WorkerLostError in [request.py](https://github.com/celery/celery/blob/master/celery/worker/request.py#L368-L385) might be incorrect.

The current code looks as such:
```
        elif isinstance(exc, WorkerLostError) or not return_ok:
            self.task.backend.mark_as_failure(
                self.id, exc, request=self._context,
                store_result=self.store_errors,
            )
        # (acks_late) acknowledge after result stored.
        if self.task.acks_late:
            reject = (
                self.task.reject_on_worker_lost and
                isinstance(exc, WorkerLostError)
            )
            ack = self.task.acks_on_failure_or_timeout
            if reject:
                requeue = not self.delivery_info.get('redelivered')
                self.reject(requeue=requeue)
                send_failed_event = False
            elif ack:
                self.acknowledge()
```

Note that `task` is marked as failed as soon as we are able to determine that the `exc` is a `WorkerLostError`.   This internally causes the [`on_chord_return`](https://github.com/celery/celery/blob/master/celery/backends/base.py#L150-L162) function to mark the chord as failed. 

There are multiple ways in which we can try to fix the problem:
1. The determination of whether we should mark the task as failed should be done after we determine whether the task is marked as retryable.  If it is, the status of the task should be marked with `mark_as_retry`
2. If one still wants to mark the task as failed, adding an additional parameter as to whether `on_chord_part_return` could help.  

Note that the `on_chord_return` will also [run the `errbacks`](https://github.com/celery/celery/blob/master/celery/backends/base.py#L161-L162) regardless of whether the task is marked as retryable or not.  This also needs to be amended in case the task is marked as retryable in case of worker lost.
@xirdneh any update on when we can expect a fix, and whether it will be backported to previous releases?

I can also submit a patch as a fix, based on my limited understanding of the code. 
Very interesting case.
I believe you have a good idea on how to fix this issue. Could you submit that PR you have in mind as well as a test for this?
Let me know if you have any questions submitting the PR or writing the test :)
Thanks for the help 👍 
Thanks for the help.  I'll get the PR up in a few days.  I plan to take the approach of marking the task as `FAILED` to keep the state consistent, but pass a `None` `request` to [`mark_as_failure`](https://github.com/celery/celery/blob/master/celery/backends/base.py#L150)

This will prevent both `on_chord_part_return`, and `on_err_callback` to be invoked.
@shaunakgodbole Make sure to mention/relate this issue with the PR, please.
did you get the time for the PR?
Unfortunately did not get time to fix this.  I'll take care of it next week.  Sorry about the delay.
@shaunakgodbole Any progress on this?

Also, you might be able to mitigate this if you know how long your tasks are running by setting a `terminationGracePeriodSeconds` in your Kubernetes Deployment manifest.

Resolving patch:
diff --git a/celery/worker/request.py b/celery/worker/request.py
--- a/celery/worker/request.py
+++ b/celery/worker/request.py
@@ -496,18 +496,8 @@ def on_failure(self, exc_info, send_failed_event=True, return_ok=False):
         if isinstance(exc, Retry):
             return self.on_retry(exc_info)
 
-        # These are special cases where the process wouldn't've had
-        # time to write the result.
-        if isinstance(exc, Terminated):
-            self._announce_revoked(
-                'terminated', True, string(exc), False)
-            send_failed_event = False  # already sent revoked event
-        elif isinstance(exc, WorkerLostError) or not return_ok:
-            self.task.backend.mark_as_failure(
-                self.id, exc, request=self._context,
-                store_result=self.store_errors,
-            )
         # (acks_late) acknowledge after result stored.
+        requeue = False
         if self.task.acks_late:
             reject = (
                 self.task.reject_on_worker_lost and
@@ -521,6 +511,19 @@ def on_failure(self, exc_info, send_failed_event=True, return_ok=False):
             elif ack:
                 self.acknowledge()
 
+        # These are special cases where the process would not have had time
+        # to write the result.
+        if isinstance(exc, Terminated):
+            self._announce_revoked(
+                'terminated', True, string(exc), False)
+            send_failed_event = False  # already sent revoked event
+        elif not requeue and (isinstance(exc, WorkerLostError) or not return_ok):
+            # only mark as failure if task has not been requeued
+            self.task.backend.mark_as_failure(
+                self.id, exc, request=self._context,
+                store_result=self.store_errors,
+            )
+
         if send_failed_event:
             self.send_event(
                 'task-failed',


Test patch:
diff --git a/t/unit/tasks/test_result.py b/t/unit/tasks/test_result.py
--- a/t/unit/tasks/test_result.py
+++ b/t/unit/tasks/test_result.py
@@ -7,7 +7,7 @@
 
 import pytest
 
-from case import Mock, call, patch, skip, MagicMock
+from case import Mock, call, patch, skip
 from celery import states, uuid
 from celery.app.task import Context
 from celery.backends.base import SyncBackendMixin
diff --git a/t/unit/worker/test_request.py b/t/unit/worker/test_request.py
--- a/t/unit/worker/test_request.py
+++ b/t/unit/worker/test_request.py
@@ -635,6 +635,26 @@ def get_ei():
         job.on_failure(exc_info)
         assert self.mytask.backend.get_status(job.id) == states.PENDING
 
+    def test_on_failure_acks_late_reject_on_worker_lost_enabled(self):
+        try:
+            raise WorkerLostError()
+        except WorkerLostError:
+            exc_info = ExceptionInfo()
+        self.mytask.acks_late = True
+        self.mytask.reject_on_worker_lost = True
+
+        job = self.xRequest()
+        job.delivery_info['redelivered'] = False
+        job.on_failure(exc_info)
+
+        assert self.mytask.backend.get_status(job.id) == states.PENDING
+
+        job = self.xRequest()
+        job.delivery_info['redelivered'] = True
+        job.on_failure(exc_info)
+
+        assert self.mytask.backend.get_status(job.id) == states.FAILURE
+
     def test_on_failure_acks_late(self):
         job = self.xRequest()
         job.time_start = 1