Repository: celery/celery

Issue:
Message being acknowledged on WorkerLostError when CELERY_ACKS_LATE=True
When using celery v3.0.24, with `CELERY_ACKS_LATE = True` , if the OOM killer kills the celery worker, then the worker acknowledges the message.
As per [this](https://github.com/celery/celery/commit/e810420c) commit. The `exc_info.internal` comes in as `false`, which means it is not a internal error, due to which the message is acknowledged.
The desirable behaviour, in such a case would be to not acknowledge the message (and be able to know, whether its a OOM error), so that some other worker can pick it up. 
As a workaround, I've commented out the [code](https://github.com/siddharth96/celery/commit/427695d1b23034dadda85fd7a48f7367831be4fa), where celery acknowledges the message, because in such a case, message will be lost.



Discussion:
This is deliberate as if a task is killed it may mean that the next invocation will also cause the same to happen.  If the task is redelivered it may cause a loop where the same conditions occur again and again.  Also, sadly you cannot distinguish processes killed by OOM from processes killed by other means, and if an administrator kills -9 a task going amok, you usually don't want that task to be called again.

There could be a configuration option for not acking terminated tasks, but I'm not sure how useful that would be.
A better solution could be to use `basic_reject(requeue=False)` instead of `basic_ack`, that way you can configure
a dead letter queue so that the killed tasks will be sent to a queue for manual inspection.

I must say, regardless of the status of this feature request, the documentation is misleading. Specifically, [this FAQ makes it seem that process failures would NOT acknowledge messages](http://celery.readthedocs.org/en/latest/faq.html#faq-acks-late-vs-retry). And [this FAQ boldface states](http://celery.readthedocs.org/en/latest/faq.html#id54) that in the event of a kill signal (9), that acks_late will allow the task to re-run (which again, is patently wrong based on this poorly documented behavior). Nowhere in the docs have I found that if the process _dies_, the message will be acknowledged, regardless of acks_late or not. (for instance, I have a set of 10k+ tasks, and some 1% of tasks wind up acknowledged but incomplete when a WorkerLostError is thrown in connection with the worker, although there are no other errors of any kind in any of my logs related to that task).

TL;DR at the least, appropriately document the current state when describing the functionality and limitations of acks_late. A work-around would be helpful -- I'm not sure I understand the solution of using `basic_reject`, although I'll keep looking into it.

The docs are referring to killing the worker process with KILL, not the child processes.  The term worker will always refer to the worker instance, not the pool processes.  The section within about acks_late is probably not very helpful and should be removed


Resolving patch:
diff --git a/celery/app/defaults.py b/celery/app/defaults.py
--- a/celery/app/defaults.py
+++ b/celery/app/defaults.py
@@ -132,6 +132,7 @@ def __repr__(self):
         'REDIS_DB': Option(type='int', **_REDIS_OLD),
         'REDIS_PASSWORD': Option(type='string', **_REDIS_OLD),
         'REDIS_MAX_CONNECTIONS': Option(type='int'),
+        'REJECT_ON_WORKER_LOST': Option(type='bool'),
         'RESULT_BACKEND': Option(type='string'),
         'RESULT_DB_SHORT_LIVED_SESSIONS': Option(False, type='bool'),
         'RESULT_DB_TABLENAMES': Option(type='dict'),
diff --git a/celery/app/task.py b/celery/app/task.py
--- a/celery/app/task.py
+++ b/celery/app/task.py
@@ -220,6 +220,12 @@ class Task(object):
     #: :setting:`CELERY_ACKS_LATE` setting.
     acks_late = None
 
+    #: When CELERY_ACKS_LATE is set to True, the default behavior to
+    #: handle worker crash is to acknowledge the message. Setting
+    #: this to true allows the message to be rejected and requeued so
+    #: it will be executed again by another worker.
+    reject_on_worker_lost = None
+
     #: Tuple of expected exceptions.
     #:
     #: These are errors that are expected in normal operation
@@ -248,6 +254,7 @@ class Task(object):
         ('rate_limit', 'CELERY_DEFAULT_RATE_LIMIT'),
         ('track_started', 'CELERY_TRACK_STARTED'),
         ('acks_late', 'CELERY_ACKS_LATE'),
+        ('reject_on_worker_lost', 'CELERY_REJECT_ON_WORKER_LOST'),
         ('ignore_result', 'CELERY_IGNORE_RESULT'),
         ('store_errors_even_if_ignored',
             'CELERY_STORE_ERRORS_EVEN_IF_IGNORED'),
diff --git a/celery/worker/request.py b/celery/worker/request.py
--- a/celery/worker/request.py
+++ b/celery/worker/request.py
@@ -326,7 +326,6 @@ def on_retry(self, exc_info):
     def on_failure(self, exc_info, send_failed_event=True, return_ok=False):
         """Handler called if the task raised an exception."""
         task_ready(self)
-
         if isinstance(exc_info.exception, MemoryError):
             raise MemoryError('Process got: %s' % (exc_info.exception,))
         elif isinstance(exc_info.exception, Reject):
@@ -352,7 +351,13 @@ def on_failure(self, exc_info, send_failed_event=True, return_ok=False):
                 )
         # (acks_late) acknowledge after result stored.
         if self.task.acks_late:
-            self.acknowledge()
+            reject_and_requeue = (self.task.reject_on_worker_lost and
+                isinstance(exc, WorkerLostError) and
+                self.delivery_info.get('redelivered', False) is False)
+            if reject_and_requeue:
+                self.reject(requeue=True)
+            else:
+                self.acknowledge()
 
         if send_failed_event:
             self.send_event(


Test patch:
diff --git a/celery/tests/worker/test_request.py b/celery/tests/worker/test_request.py
--- a/celery/tests/worker/test_request.py
+++ b/celery/tests/worker/test_request.py
@@ -325,6 +325,20 @@ def test_on_failure_Reject_rejects_with_requeue(self):
             req_logger, req.connection_errors, True,
         )
 
+    def test_on_failure_WrokerLostError_rejects_with_requeue(self):
+        einfo = None
+        try:
+            raise WorkerLostError()
+        except:
+            einfo = ExceptionInfo(internal=True)
+        req = self.get_request(self.add.s(2, 2))
+        req.task.acks_late = True
+        req.task.reject_on_worker_lost = True
+        req.delivery_info['redelivered'] = False
+        req.on_failure(einfo)
+        req.on_reject.assert_called_with(req_logger,
+            req.connection_errors, True)
+
     def test_tzlocal_is_cached(self):
         req = self.get_request(self.add.s(2, 2))
         req._tzlocal = 'foo'