Repository: celery/celery

Issue:
Synchronous subtask guard blocks chord() when CELERY_TASK_ALWAYS_EAGER=True
## Checklist

- [x] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.

## Steps to reproduce

```
$ celery -A minimal_eager_chord report
software -> celery:4.2.0 (windowlicker) kombu:4.2.1 py:3.5.3
            billiard:3.5.0.3 py-amqp:2.2.2
platform -> system:Darwin arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled
```

Script to reproduce: 
https://gist.github.com/jcushman/ea8adcb37dffdf5c3fdf756d02366ab2

## Expected behavior

No error occurs

## Actual behavior

```
In [1]: test.delay().get()

...

~/minimal_eager_chord.py in test()
     27 def test():
     28     chord(
---> 29         a.s(), b.s()
     30     ).apply_async()
     31

...

~/.pyenv/versions/3.5.3/envs/capstone/src/celery/celery/result.py in assert_will_not_block()
     39 def assert_will_not_block():
     40     if task_join_will_block():
---> 41         raise RuntimeError(E_WOULDBLOCK)
```

## Comments

This is a similar issue to #4576: `assert_will_not_block()` is triggered when using canvas functions with `CELERY_TASK_ALWAYS_EAGER=True`, even when they are used with apply_async(). 

The patch for that issue fixed `chain()` but not `chord()`, and the problem may exist for other canvas functions as well. I'm not sure whether it makes sense to add more calls to `with allow_join_result():` in `celery/canvas.py` (as in the patch for #4576), or if `assert_will_not_block()` could be modified to avoid false positives from `CELERY_TASK_ALWAYS_EAGER=True` in general.

The tricky bit of a general solution would be that the guard should still detect actual unsafe subtasks even if `CELERY_TASK_ALWAYS_EAGER=True`, since that's a natural way to run tests.
Synchronous subtask guard blocks chord() when CELERY_TASK_ALWAYS_EAGER=True
## Checklist

- [x] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.

## Steps to reproduce

```
$ celery -A minimal_eager_chord report
software -> celery:4.2.0 (windowlicker) kombu:4.2.1 py:3.5.3
            billiard:3.5.0.3 py-amqp:2.2.2
platform -> system:Darwin arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled
```

Script to reproduce: 
https://gist.github.com/jcushman/ea8adcb37dffdf5c3fdf756d02366ab2

## Expected behavior

No error occurs

## Actual behavior

```
In [1]: test.delay().get()

...

~/minimal_eager_chord.py in test()
     27 def test():
     28     chord(
---> 29         a.s(), b.s()
     30     ).apply_async()
     31

...

~/.pyenv/versions/3.5.3/envs/capstone/src/celery/celery/result.py in assert_will_not_block()
     39 def assert_will_not_block():
     40     if task_join_will_block():
---> 41         raise RuntimeError(E_WOULDBLOCK)
```

## Comments

This is a similar issue to #4576: `assert_will_not_block()` is triggered when using canvas functions with `CELERY_TASK_ALWAYS_EAGER=True`, even when they are used with apply_async(). 

The patch for that issue fixed `chain()` but not `chord()`, and the problem may exist for other canvas functions as well. I'm not sure whether it makes sense to add more calls to `with allow_join_result():` in `celery/canvas.py` (as in the patch for #4576), or if `assert_will_not_block()` could be modified to avoid false positives from `CELERY_TASK_ALWAYS_EAGER=True` in general.

The tricky bit of a general solution would be that the guard should still detect actual unsafe subtasks even if `CELERY_TASK_ALWAYS_EAGER=True`, since that's a natural way to run tests.


Discussion:
Same issue here: our tests are using `CELERY_TASK_ALWAYS_EAGER=True` and tests using `chords()` fail with the same stacktrace
fixes are welcome
Same issue here: our tests are using `CELERY_TASK_ALWAYS_EAGER=True` and tests using `chords()` fail with the same stacktrace
fixes are welcome

Resolving patch:
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -1226,8 +1226,9 @@ def apply_async(self, args=(), kwargs={}, task_id=None,
         tasks = (self.tasks.clone() if isinstance(self.tasks, group)
                  else group(self.tasks, app=app))
         if app.conf.task_always_eager:
-            return self.apply(args, kwargs,
-                              body=body, task_id=task_id, **options)
+            with allow_join_result():
+                return self.apply(args, kwargs,
+                                  body=body, task_id=task_id, **options)
         # chord([A, B, ...], C)
         return self.run(tasks, body, args, task_id=task_id, **options)
 
diff --git a/t/integration/tasks.py b/t/integration/tasks.py
--- a/t/integration/tasks.py
+++ b/t/integration/tasks.py
@@ -3,7 +3,7 @@
 
 from time import sleep
 
-from celery import chain, group, shared_task
+from celery import chain, chord, group, shared_task
 from celery.exceptions import SoftTimeLimitExceeded
 from celery.utils.log import get_task_logger
 
@@ -42,6 +42,11 @@ def chain_add(x, y):
     ).apply_async()
 
 
+@shared_task
+def chord_add(x, y):
+    chord(add.s(x, x), add.s(y)).apply_async()
+
+
 @shared_task
 def delayed_sum(numbers, pause_time=1):
     """Sum the iterable of numbers."""


Test patch:
diff --git a/t/integration/test_canvas.py b/t/integration/test_canvas.py
--- a/t/integration/test_canvas.py
+++ b/t/integration/test_canvas.py
@@ -366,6 +366,17 @@ def test_add_chord_to_chord(self, manager):
         res = c()
         assert res.get() == [0, 5 + 6 + 7]
 
+    @flaky
+    def test_eager_chord_inside_task(self, manager):
+        from .tasks import chord_add
+
+        prev = chord_add.app.conf.task_always_eager
+        chord_add.app.conf.task_always_eager = True
+
+        chord_add.apply_async(args=(4, 8), throw=True).get()
+
+        chord_add.app.conf.task_always_eager = prev
+
     @flaky
     def test_group_chain(self, manager):
         if not manager.app.conf.result_backend.startswith('redis'):
diff --git a/t/unit/tasks/test_canvas.py b/t/unit/tasks/test_canvas.py
--- a/t/unit/tasks/test_canvas.py
+++ b/t/unit/tasks/test_canvas.py
@@ -747,6 +747,29 @@ def test_freeze_tasks_is_not_group(self):
         x.tasks = [self.add.s(2, 2)]
         x.freeze()
 
+    def test_chain_always_eager(self):
+        self.app.conf.task_always_eager = True
+        from celery import _state
+        from celery import result
+
+        fixture_task_join_will_block = _state.task_join_will_block
+        try:
+            _state.task_join_will_block = _state.orig_task_join_will_block
+            result.task_join_will_block = _state.orig_task_join_will_block
+
+            @self.app.task(shared=False)
+            def finalize(*args):
+                pass
+
+            @self.app.task(shared=False)
+            def chord_add():
+                return chord([self.add.s(4, 4)], finalize.s()).apply_async()
+
+            chord_add.apply_async(throw=True).get()
+        finally:
+            _state.task_join_will_block = fixture_task_join_will_block
+            result.task_join_will_block = fixture_task_join_will_block
+
 
 class test_maybe_signature(CanvasCase):