Repository: celery/celery

Issue:
Request on_timeout should ignore soft time limit exception
When Request.on_timeout receive a soft timeout from billiard, it does the same as if it was receiving a hard time limit exception. This is ran by the controller.

But the task may catch this exception and eg. return (this is what soft timeout are for).

This cause:
1. the result to be saved once as an exception by the controller (on_timeout) and another time with the result returned by the task
2. the task status to be passed to failure and to success on the same manner
3. if the task is participating to a chord, the chord result counter (at least with redis) is incremented twice (instead of once), making the chord to return prematurely and eventually loose tasks…

1, 2 and 3 can leads of course to strange race conditions…

## Steps to reproduce (Illustration)

with the program in test_timeout.py:

```python
import time
import celery


app = celery.Celery('test_timeout')
app.conf.update(
    result_backend="redis://localhost/0",
    broker_url="amqp://celery:celery@localhost:5672/host",
)

@app.task(soft_time_limit=1)
def test():
    try:
        time.sleep(2)
    except Exception:
        return 1

@app.task()
def add(args):
    print("### adding", args)
    return sum(args)

@app.task()
def on_error(context, exception, traceback, **kwargs):
    print("### on_error: ", exception)

if __name__ == "__main__":
    result = celery.chord([test.s().set(link_error=on_error.s()), test.s().set(link_error=on_error.s())])(add.s())
    result.get()
```

start a worker and the program:

```
$ celery -A test_timeout worker -l WARNING
$ python3 test_timeout.py
```

## Expected behavior

add method is called with `[1, 1]` as argument and test_timeout.py return normally

## Actual behavior

The test_timeout.py fails, with
```
celery.backends.base.ChordError: Callback error: ChordError("Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)",
```
On the worker side, the **on_error is called but the add method as well !**

```
[2017-11-29 23:07:25,538: WARNING/MainProcess] Soft time limit (1s) exceeded for test_timeout.test[15109e05-da43-449f-9081-85d839ac0ef2]
[2017-11-29 23:07:25,546: WARNING/MainProcess] ### on_error:
[2017-11-29 23:07:25,546: WARNING/MainProcess] SoftTimeLimitExceeded(True,)
[2017-11-29 23:07:25,547: WARNING/MainProcess] Soft time limit (1s) exceeded for test_timeout.test[38f3f7f2-4a89-4318-8ee9-36a987f73757]
[2017-11-29 23:07:25,553: ERROR/MainProcess] Chord callback for 'ef6d7a38-d1b4-40ad-b937-ffa84e40bb23' raised: ChordError("Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)",)
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 290, in on_chord_part_return
    callback.delay([unpack(tup, decode) for tup in resl])
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 290, in <listcomp>
    callback.delay([unpack(tup, decode) for tup in resl])
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 243, in _unpack_chord_result
    raise ChordError('Dependency {0} raised {1!r}'.format(tid, retval))
celery.exceptions.ChordError: Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)
[2017-11-29 23:07:25,565: WARNING/MainProcess] ### on_error:
[2017-11-29 23:07:25,565: WARNING/MainProcess] SoftTimeLimitExceeded(True,)
[2017-11-29 23:07:27,262: WARNING/PoolWorker-2] ### adding
[2017-11-29 23:07:27,264: WARNING/PoolWorker-2] [1, 1]
```

Of course, on purpose did I choose to call the test.s() twice, to show that the count in the chord continues. In fact:
- the chord result is incremented twice by the error of soft time limit
- the chord result is again incremented twice by the correct returning of `test` task

## Conclusion

Request.on_timeout  should not process soft time limit exception. 

here is a quick monkey patch (correction of celery is trivial)

```python
def patch_celery_request_on_timeout():
    from celery.worker import request
    orig = request.Request.on_timeout
    def patched_on_timeout(self, soft, timeout):
        if not soft:
            orig(self, soft, timeout)
    request.Request.on_timeout = patched_on_timeout
patch_celery_request_on_timeout()
```



## version info

software -> celery:4.1.0 (latentcall) kombu:4.0.2 py:3.4.3
            billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:redis://10.0.3.253/0

Request on_timeout should ignore soft time limit exception
When Request.on_timeout receive a soft timeout from billiard, it does the same as if it was receiving a hard time limit exception. This is ran by the controller.

But the task may catch this exception and eg. return (this is what soft timeout are for).

This cause:
1. the result to be saved once as an exception by the controller (on_timeout) and another time with the result returned by the task
2. the task status to be passed to failure and to success on the same manner
3. if the task is participating to a chord, the chord result counter (at least with redis) is incremented twice (instead of once), making the chord to return prematurely and eventually loose tasks…

1, 2 and 3 can leads of course to strange race conditions…

## Steps to reproduce (Illustration)

with the program in test_timeout.py:

```python
import time
import celery


app = celery.Celery('test_timeout')
app.conf.update(
    result_backend="redis://localhost/0",
    broker_url="amqp://celery:celery@localhost:5672/host",
)

@app.task(soft_time_limit=1)
def test():
    try:
        time.sleep(2)
    except Exception:
        return 1

@app.task()
def add(args):
    print("### adding", args)
    return sum(args)

@app.task()
def on_error(context, exception, traceback, **kwargs):
    print("### on_error: ", exception)

if __name__ == "__main__":
    result = celery.chord([test.s().set(link_error=on_error.s()), test.s().set(link_error=on_error.s())])(add.s())
    result.get()
```

start a worker and the program:

```
$ celery -A test_timeout worker -l WARNING
$ python3 test_timeout.py
```

## Expected behavior

add method is called with `[1, 1]` as argument and test_timeout.py return normally

## Actual behavior

The test_timeout.py fails, with
```
celery.backends.base.ChordError: Callback error: ChordError("Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)",
```
On the worker side, the **on_error is called but the add method as well !**

```
[2017-11-29 23:07:25,538: WARNING/MainProcess] Soft time limit (1s) exceeded for test_timeout.test[15109e05-da43-449f-9081-85d839ac0ef2]
[2017-11-29 23:07:25,546: WARNING/MainProcess] ### on_error:
[2017-11-29 23:07:25,546: WARNING/MainProcess] SoftTimeLimitExceeded(True,)
[2017-11-29 23:07:25,547: WARNING/MainProcess] Soft time limit (1s) exceeded for test_timeout.test[38f3f7f2-4a89-4318-8ee9-36a987f73757]
[2017-11-29 23:07:25,553: ERROR/MainProcess] Chord callback for 'ef6d7a38-d1b4-40ad-b937-ffa84e40bb23' raised: ChordError("Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)",)
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 290, in on_chord_part_return
    callback.delay([unpack(tup, decode) for tup in resl])
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 290, in <listcomp>
    callback.delay([unpack(tup, decode) for tup in resl])
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 243, in _unpack_chord_result
    raise ChordError('Dependency {0} raised {1!r}'.format(tid, retval))
celery.exceptions.ChordError: Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)
[2017-11-29 23:07:25,565: WARNING/MainProcess] ### on_error:
[2017-11-29 23:07:25,565: WARNING/MainProcess] SoftTimeLimitExceeded(True,)
[2017-11-29 23:07:27,262: WARNING/PoolWorker-2] ### adding
[2017-11-29 23:07:27,264: WARNING/PoolWorker-2] [1, 1]
```

Of course, on purpose did I choose to call the test.s() twice, to show that the count in the chord continues. In fact:
- the chord result is incremented twice by the error of soft time limit
- the chord result is again incremented twice by the correct returning of `test` task

## Conclusion

Request.on_timeout  should not process soft time limit exception. 

here is a quick monkey patch (correction of celery is trivial)

```python
def patch_celery_request_on_timeout():
    from celery.worker import request
    orig = request.Request.on_timeout
    def patched_on_timeout(self, soft, timeout):
        if not soft:
            orig(self, soft, timeout)
    request.Request.on_timeout = patched_on_timeout
patch_celery_request_on_timeout()
```



## version info

software -> celery:4.1.0 (latentcall) kombu:4.0.2 py:3.4.3
            billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:redis://10.0.3.253/0



Discussion:
@ask I think this is quite a big problem (with a trivial fix).

It requires attention though as it is bringging a new behaviour (but previous behaviour is not well documented, and, in my opinion, the new behaviour was the one expected)
This change in behaviour is what kept my team from upgrading to celery 4. Indeed, the chord callback was often not called at all.
I don't know if it is related, but I modified your code sample and it resulted in some `Exception raised outside body` errors and multiple other errors if you try running `python test_timeout.py` multiple times.

Here is my script:

```python
import time
import celery


app = celery.Celery(
    'test_timeout',
    broker='amqp://localhost',
    backend='redis://localhost')


@app.task(soft_time_limit=1)
def test(nb_seconds):
    try:
        time.sleep(nb_seconds)
        return nb_seconds
    except:
        print("### error handled")
        return nb_seconds


@app.task()
def add(args):
    print("### adding", args)
    return sum(args)


@app.task()
def on_error(context, exception, traceback, **kwargs):
    print("### on_error: ", exception)


if __name__ == "__main__":
    result = celery.chord([
        test.s(i).set(link_error=on_error.s()) for i in range(0, 2)
    ])(add.s())
    result.get()
```

NB: if you update the range for range(2, 4) for instance, the `Exception raised outside body` does not seem to happen. It seems this particular issue happens when the `SoftTimeLimitExceeded` is raised exactly during the `return`.
could you please send a PR with your proposed fix/workaround on master branch?
hi @auvipy, I won't have the time until jannuary. I'll need help on how to write the tests also (that's the reason why I didn't propose a PR).
OK. dont get afraid of sending logical changes just for the reason you don't know how to write test. we will certainly try to help you 
@ask I think this is quite a big problem (with a trivial fix).

It requires attention though as it is bringging a new behaviour (but previous behaviour is not well documented, and, in my opinion, the new behaviour was the one expected)
This change in behaviour is what kept my team from upgrading to celery 4. Indeed, the chord callback was often not called at all.
I don't know if it is related, but I modified your code sample and it resulted in some `Exception raised outside body` errors and multiple other errors if you try running `python test_timeout.py` multiple times.

Here is my script:

```python
import time
import celery


app = celery.Celery(
    'test_timeout',
    broker='amqp://localhost',
    backend='redis://localhost')


@app.task(soft_time_limit=1)
def test(nb_seconds):
    try:
        time.sleep(nb_seconds)
        return nb_seconds
    except:
        print("### error handled")
        return nb_seconds


@app.task()
def add(args):
    print("### adding", args)
    return sum(args)


@app.task()
def on_error(context, exception, traceback, **kwargs):
    print("### on_error: ", exception)


if __name__ == "__main__":
    result = celery.chord([
        test.s(i).set(link_error=on_error.s()) for i in range(0, 2)
    ])(add.s())
    result.get()
```

NB: if you update the range for range(2, 4) for instance, the `Exception raised outside body` does not seem to happen. It seems this particular issue happens when the `SoftTimeLimitExceeded` is raised exactly during the `return`.
could you please send a PR with your proposed fix/workaround on master branch?
hi @auvipy, I won't have the time until jannuary. I'll need help on how to write the tests also (that's the reason why I didn't propose a PR).
OK. dont get afraid of sending logical changes just for the reason you don't know how to write test. we will certainly try to help you 

Resolving patch:
diff --git a/celery/worker/request.py b/celery/worker/request.py
--- a/celery/worker/request.py
+++ b/celery/worker/request.py
@@ -18,8 +18,8 @@
 from celery import signals
 from celery.app.trace import trace_task, trace_task_ret
 from celery.exceptions import (Ignore, InvalidTaskError, Reject, Retry,
-                               SoftTimeLimitExceeded, TaskRevokedError,
-                               Terminated, TimeLimitExceeded, WorkerLostError)
+                               TaskRevokedError, Terminated,
+                               TimeLimitExceeded, WorkerLostError)
 from celery.five import python_2_unicode_compatible, string
 from celery.platforms import signals as _signals
 from celery.utils.functional import maybe, noop
@@ -299,22 +299,21 @@ def on_accepted(self, pid, time_accepted):
 
     def on_timeout(self, soft, timeout):
         """Handler called if the task times out."""
-        task_ready(self)
         if soft:
             warn('Soft time limit (%ss) exceeded for %s[%s]',
                  timeout, self.name, self.id)
-            exc = SoftTimeLimitExceeded(soft)
         else:
+            task_ready(self)
             error('Hard time limit (%ss) exceeded for %s[%s]',
                   timeout, self.name, self.id)
             exc = TimeLimitExceeded(timeout)
 
-        self.task.backend.mark_as_failure(
-            self.id, exc, request=self, store_result=self.store_errors,
-        )
+            self.task.backend.mark_as_failure(
+                self.id, exc, request=self, store_result=self.store_errors,
+            )
 
-        if self.task.acks_late:
-            self.acknowledge()
+            if self.task.acks_late:
+                self.acknowledge()
 
     def on_success(self, failed__retval__runtime, **kwargs):
         """Handler called if the task was successfully processed."""
diff --git a/t/integration/tasks.py b/t/integration/tasks.py
--- a/t/integration/tasks.py
+++ b/t/integration/tasks.py
@@ -4,6 +4,7 @@
 from time import sleep
 
 from celery import chain, group, shared_task
+from celery.exceptions import SoftTimeLimitExceeded
 from celery.utils.log import get_task_logger
 
 logger = get_task_logger(__name__)
@@ -24,6 +25,16 @@ def delayed_sum(numbers, pause_time=1):
     return sum(numbers)
 
 
+@shared_task
+def delayed_sum_with_soft_guard(numbers, pause_time=1):
+    """Sum the iterable of numbers."""
+    try:
+        sleep(pause_time)
+        return sum(numbers)
+    except SoftTimeLimitExceeded:
+        return 0
+
+
 @shared_task(bind=True)
 def add_replaced(self, x, y):
     """Add two numbers (via the add task)."""


Test patch:
diff --git a/t/integration/test_canvas.py b/t/integration/test_canvas.py
--- a/t/integration/test_canvas.py
+++ b/t/integration/test_canvas.py
@@ -11,7 +11,8 @@
 
 from .conftest import flaky
 from .tasks import (add, add_replaced, add_to_all, collect_ids, delayed_sum,
-                    ids, redis_echo, second_order_replace1)
+                    delayed_sum_with_soft_guard, ids, redis_echo,
+                    second_order_replace1)
 
 TIMEOUT = 120
 
@@ -110,6 +111,26 @@ def assert_ids(self, res, size):
             node = node.parent
             i -= 1
 
+    def test_chord_soft_timeout_recuperation(self, manager):
+        """Test that if soft timeout happens in task but is managed by task,
+        chord still get results normally
+        """
+        if not manager.app.conf.result_backend.startswith('redis'):
+            raise pytest.skip('Requires redis result backend.')
+
+        c = chord([
+            # return 3
+            add.s(1, 2),
+            # return 0 after managing soft timeout
+            delayed_sum_with_soft_guard.s(
+                [100], pause_time=2
+            ).set(
+                soft_time_limit=1
+            ),
+        ])
+        result = c(delayed_sum.s(pause_time=0)).get()
+        assert result == 3
+
 
 class test_group:
 
diff --git a/t/unit/worker/test_request.py b/t/unit/worker/test_request.py
--- a/t/unit/worker/test_request.py
+++ b/t/unit/worker/test_request.py
@@ -599,31 +599,39 @@ def test_from_message_invalid_kwargs(self):
         with pytest.raises(InvalidTaskError):
             raise req.execute().exception
 
-    def test_on_timeout(self, patching):
-        warn = patching('celery.worker.request.warn')
+    def test_on_hard_timeout(self, patching):
         error = patching('celery.worker.request.error')
 
         job = self.xRequest()
         job.acknowledge = Mock(name='ack')
         job.task.acks_late = True
-        job.on_timeout(soft=True, timeout=1337)
-        assert 'Soft time limit' in warn.call_args[0][0]
         job.on_timeout(soft=False, timeout=1337)
         assert 'Hard time limit' in error.call_args[0][0]
         assert self.mytask.backend.get_status(job.id) == states.FAILURE
         job.acknowledge.assert_called_with()
 
-        self.mytask.ignore_result = True
         job = self.xRequest()
-        job.on_timeout(soft=True, timeout=1336)
-        assert self.mytask.backend.get_status(job.id) == states.PENDING
+        job.acknowledge = Mock(name='ack')
+        job.task.acks_late = False
+        job.on_timeout(soft=False, timeout=1335)
+        job.acknowledge.assert_not_called()
+
+    def test_on_soft_timeout(self, patching):
+        warn = patching('celery.worker.request.warn')
 
         job = self.xRequest()
         job.acknowledge = Mock(name='ack')
-        job.task.acks_late = False
-        job.on_timeout(soft=True, timeout=1335)
+        job.task.acks_late = True
+        job.on_timeout(soft=True, timeout=1337)
+        assert 'Soft time limit' in warn.call_args[0][0]
+        assert self.mytask.backend.get_status(job.id) == states.PENDING
         job.acknowledge.assert_not_called()
 
+        self.mytask.ignore_result = True
+        job = self.xRequest()
+        job.on_timeout(soft=True, timeout=1336)
+        assert self.mytask.backend.get_status(job.id) == states.PENDING
+
     def test_fast_trace_task(self):
         from celery.app import trace
         setup_worker_optimizations(self.app)