Repository: celery/celery

Issue:
Request on_timeout should ignore soft time limit exception
When Request.on_timeout receive a soft timeout from billiard, it does the same as if it was receiving a hard time limit exception. This is ran by the controller.

But the task may catch this exception and eg. return (this is what soft timeout are for).

This cause:
1. the result to be saved once as an exception by the controller (on_timeout) and another time with the result returned by the task
2. the task status to be passed to failure and to success on the same manner
3. if the task is participating to a chord, the chord result counter (at least with redis) is incremented twice (instead of once), making the chord to return prematurely and eventually loose tasks…

1, 2 and 3 can leads of course to strange race conditions…

## Steps to reproduce (Illustration)

with the program in test_timeout.py:

```python
import time
import celery


app = celery.Celery('test_timeout')
app.conf.update(
    result_backend="redis://localhost/0",
    broker_url="amqp://celery:celery@localhost:5672/host",
)

@app.task(soft_time_limit=1)
def test():
    try:
        time.sleep(2)
    except Exception:
        return 1

@app.task()
def add(args):
    print("### adding", args)
    return sum(args)

@app.task()
def on_error(context, exception, traceback, **kwargs):
    print("### on_error: ", exception)

if __name__ == "__main__":
    result = celery.chord([test.s().set(link_error=on_error.s()), test.s().set(link_error=on_error.s())])(add.s())
    result.get()
```

start a worker and the program:

```
$ celery -A test_timeout worker -l WARNING
$ python3 test_timeout.py
```

## Expected behavior

add method is called with `[1, 1]` as argument and test_timeout.py return normally

## Actual behavior

The test_timeout.py fails, with
```
celery.backends.base.ChordError: Callback error: ChordError("Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)",
```
On the worker side, the **on_error is called but the add method as well !**

```
[2017-11-29 23:07:25,538: WARNING/MainProcess] Soft time limit (1s) exceeded for test_timeout.test[15109e05-da43-449f-9081-85d839ac0ef2]
[2017-11-29 23:07:25,546: WARNING/MainProcess] ### on_error:
[2017-11-29 23:07:25,546: WARNING/MainProcess] SoftTimeLimitExceeded(True,)
[2017-11-29 23:07:25,547: WARNING/MainProcess] Soft time limit (1s) exceeded for test_timeout.test[38f3f7f2-4a89-4318-8ee9-36a987f73757]
[2017-11-29 23:07:25,553: ERROR/MainProcess] Chord callback for 'ef6d7a38-d1b4-40ad-b937-ffa84e40bb23' raised: ChordError("Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)",)
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 290, in on_chord_part_return
    callback.delay([unpack(tup, decode) for tup in resl])
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 290, in <listcomp>
    callback.delay([unpack(tup, decode) for tup in resl])
  File "/usr/local/lib/python3.4/dist-packages/celery/backends/redis.py", line 243, in _unpack_chord_result
    raise ChordError('Dependency {0} raised {1!r}'.format(tid, retval))
celery.exceptions.ChordError: Dependency 15109e05-da43-449f-9081-85d839ac0ef2 raised SoftTimeLimitExceeded('SoftTimeLimitExceeded(True,)',)
[2017-11-29 23:07:25,565: WARNING/MainProcess] ### on_error:
[2017-11-29 23:07:25,565: WARNING/MainProcess] SoftTimeLimitExceeded(True,)
[2017-11-29 23:07:27,262: WARNING/PoolWorker-2] ### adding
[2017-11-29 23:07:27,264: WARNING/PoolWorker-2] [1, 1]
```

Of course, on purpose did I choose to call the test.s() twice, to show that the count in the chord continues. In fact:
- the chord result is incremented twice by the error of soft time limit
- the chord result is again incremented twice by the correct returning of `test` task

## Conclusion

Request.on_timeout  should not process soft time limit exception. 

here is a quick monkey patch (correction of celery is trivial)

```python
def patch_celery_request_on_timeout():
    from celery.worker import request
    orig = request.Request.on_timeout
    def patched_on_timeout(self, soft, timeout):
        if not soft:
            orig(self, soft, timeout)
    request.Request.on_timeout = patched_on_timeout
patch_celery_request_on_timeout()
```



## version info

software -> celery:4.1.0 (latentcall) kombu:4.0.2 py:3.4.3
            billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:redis://10.0.3.253/0



Discussion:

Resolving patch:
diff --git a/celery/beat.py b/celery/beat.py
--- a/celery/beat.py
+++ b/celery/beat.py
@@ -232,10 +232,21 @@ def adjust(self, n, drift=-0.010):
     def is_due(self, entry):
         return entry.is_due()
 
+    def _when(self, entry, next_time_to_run, mktime=time.mktime):
+        adjust = self.adjust
+
+        return (mktime(entry.schedule.now().timetuple()) +
+                (adjust(next_time_to_run) or 0))
+
+    def populate_heap(self, event_t=event_t, heapify=heapq.heapify):
+        """Populate the heap with the data contained in the schedule."""
+        self._heap = [event_t(self._when(e, e.is_due()[1]) or 0, 5, e)
+                      for e in values(self.schedule)]
+        heapify(self._heap)
+
     # pylint disable=redefined-outer-name
-    def tick(self, event_t=event_t, min=min,
-             heappop=heapq.heappop, heappush=heapq.heappush,
-             heapify=heapq.heapify, mktime=time.mktime):
+    def tick(self, event_t=event_t, min=min, heappop=heapq.heappop,
+             heappush=heapq.heappush):
         """Run a tick - one iteration of the scheduler.
 
         Executes one due task per call.
@@ -243,17 +254,14 @@ def tick(self, event_t=event_t, min=min,
         Returns:
             float: preferred delay in seconds for next call.
         """
-        def _when(entry, next_time_to_run):
-            return (mktime(entry.schedule.now().timetuple()) +
-                    (adjust(next_time_to_run) or 0))
-
         adjust = self.adjust
         max_interval = self.max_interval
+
+        if self._heap is None:
+            self.populate_heap()
+
         H = self._heap
-        if H is None:
-            H = self._heap = [event_t(_when(e, e.is_due()[1]) or 0, 5, e)
-                              for e in values(self.schedule)]
-            heapify(H)
+
         if not H:
             return max_interval
 
@@ -265,7 +273,7 @@ def _when(entry, next_time_to_run):
             if verify is event:
                 next_entry = self.reserve(entry)
                 self.apply_entry(entry, producer=self.producer)
-                heappush(H, event_t(_when(next_entry, next_time_to_run),
+                heappush(H, event_t(self._when(next_entry, next_time_to_run),
                                     event[1], next_entry))
                 return 0
             else:


Test patch:
diff --git a/t/unit/app/test_beat.py b/t/unit/app/test_beat.py
--- a/t/unit/app/test_beat.py
+++ b/t/unit/app/test_beat.py
@@ -6,6 +6,7 @@
 from case import Mock, call, patch, skip
 from celery import beat
 from celery import uuid
+from celery.beat import event_t
 from celery.five import keys, string_t
 from celery.schedules import schedule
 from celery.utils.objects import Bunch
@@ -339,6 +340,15 @@ def test_merge_inplace(self):
         assert 'baz' in a.schedule
         assert a.schedule['bar'].schedule._next_run_at == 40
 
+    @patch('celery.beat.Scheduler._when', return_value=1)
+    def test_populate_heap(self, _when):
+        scheduler = mScheduler(app=self.app)
+        scheduler.update_from_dict(
+            {'foo': {'schedule': mocked_schedule(True, 10)}}
+        )
+        scheduler.populate_heap()
+        assert scheduler._heap == [event_t(1, 5, scheduler.schedule['foo'])]
+
 
 def create_persistent_scheduler(shelv=None):
     if shelv is None: