Repository: celery/celery

Issue:
Chain with groups at start and end and more than 2 tasks in between never complete
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->

- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [ ] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [ ] I have verified that the issue exists against the `master` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [x] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

-  Subsequent groups within a chain fail #3585 
- Nested group(chain(group)) fails #5496
- group([chain(group(task(), task()), group(...)), chain(group(...), group(...))]) construct never finishes #2354
- Canvas with group-task-task-group does not work #5354


#### Possible Duplicates
- Nested group-chain-group structure never succeeds #2573

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
software -> celery:4.3.0 (rhubarb) kombu:4.5.0 py:3.6.5
            billiard:3.6.0.0 redis:3.2.1
platform -> system:Linux arch:64bit
            kernel version:4.9.0-9-amd64 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis:///

broker_url: 'redis://localhost:6379//'
result_backend: 'redis:///'
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: 3.6
* **Minimal Celery Version**: 4.3
* **Minimal Kombu Version**: 4.5.0
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: Redis
* **Minimal OS and/or Kernel Version**: Debian Stretch
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->

<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
amqp==2.4.2
billiard==3.6.0.0
celery==4.3.0
celery-eternal==0.1.1
celery-singleton==0.1.3
certifi==2018.4.16
kombu==4.5.0
pytz==2019.1
redis==3.2.1
vine==1.3.0
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
from time import sleep

from celery import Celery
from celery import group

app = Celery('tasks', broker = 'redis://')
app.conf['result_backend'] = app.conf.broker_url

@app.task
def prod(x, y):
    return x*y

@app.task
def subtract(args):
    return args[0]-args[1]

@app.task(shared=False)
def identity(args):
    """Identity task (returns its input)."""
    return args

x = (
    group( prod.s(1, 1), prod.s(2, 1) )
    |
    identity.s()
    |
    subtract.s()
    |
    group( prod.s(5), prod.s(6) )
)
r = x.delay()
sleep(10)
print(r.waiting())

# Another case
x = (
    group( prod.s(1, 1), prod.s(2, 1) )
    |
    subtract.si((3,4))
    |
    subtract.si((4,3))
    |
    subtract.si((6,5))
    |
    group( prod.si(5, 6), prod.si(6, 5) )
)
r = x.delay()
sleep(10)
print(r.waiting())

# However this works
# x = (
#    group( prod.s(1, 1), prod.s(2, 1) )
#    |
#    subtract.si((3,4))
#    |
#    group( prod.si(5, 6), prod.si(6, 5) )
# )
# r = x.delay()
# print([p.result for p in r.results])
# prints [30, 30]

# The adding more than one component in between 2 groups
# in a chain is causing the canvas to wait right before the final group
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->
Above code snippet should return a list of 2 values
([5, 6] in for first case, [30, 30] for second case)
# Actual Behavior
The code never never returns, keeps waiting
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->
Celery worker logs for the first case mentioned
```
[2019-05-10 21:12:42,361: INFO/MainProcess] Connected to redis://localhost:6379//
[2019-05-10 21:12:42,366: INFO/MainProcess] mingle: searching for neighbors
[2019-05-10 21:12:43,380: INFO/MainProcess] mingle: all alone
[2019-05-10 21:12:43,393: INFO/MainProcess] celery@debian ready.
[2019-05-10 21:13:43,557: INFO/MainProcess] Received task: celery_task.prod[e11748a0-dca2-41fb-b799-a2ae7879c9fd]  
[2019-05-10 21:13:43,559: INFO/MainProcess] Received task: celery_task.prod[6ec1c88f-cca7-417c-8ff1-eacd9bdef7a1]  
[2019-05-10 21:13:43,566: INFO/ForkPoolWorker-2] Task celery_task.prod[6ec1c88f-cca7-417c-8ff1-eacd9bdef7a1] succeeded in 0.0064055299990286585s: 2
[2019-05-10 21:13:43,585: INFO/ForkPoolWorker-1] Task celery_task.prod[e11748a0-dca2-41fb-b799-a2ae7879c9fd] succeeded in 0.02549504399939906s: 1
[2019-05-10 21:13:43,586: INFO/MainProcess] Received task: celery_task.identity[87148533-8da1-4fc6-bbfe-608470cb3c1a]  
[2019-05-10 21:13:43,588: INFO/ForkPoolWorker-1] Task celery_task.identity[87148533-8da1-4fc6-bbfe-608470cb3c1a] succeeded in 0.0011925399994652253s: [2, 1]
[2019-05-10 21:13:43,588: INFO/MainProcess] Received task: celery_task.subtract[e907af43-a886-45ea-bfe3-f709c8267e5d]  
[2019-05-10 21:13:43,589: INFO/ForkPoolWorker-1] Task celery_task.subtract[e907af43-a886-45ea-bfe3-f709c8267e5d] succeeded in 0.00028916799965372775s: 1
```
Considering the first case mentioned in the code snippet, you can see that value of the task before the second group, `subtract`, is returned but the last group (consisting 2 `prod`s) is never received, even after waiting many minutes. (I put the `sleep(10)` just as a proxy for some wait)
Nested group-chain-group structure never succeeds
The structure like 

```
workflow = chain(
    t.task1.s(),
    chord(
        [
            t.task2.s(),
            chain(
                t.task3.s(),
                chord(
                    [t.task4.s(), t.task5.s()],
                    t.task6.s()
                )
            )
        ],
        t.task7.s()
    )
)
```

never succeeds. Task `t.task7` is never called, while other tasks succeed just fine. 

Replacing chords with groups did nothing. When I remove the inner chord, workflow finishes without any problems. Workflow also succeeds if I remove an inner chain.

The exact code that can be used to reproduce this issue as well as log messages are in [this gist](https://gist.github.com/traut/fcf5d76e07dd3ed6e54d)



Discussion:
and this @tothegump 
Ok, let me have a look.
It appears that #4481 and #4690 possibly caused this regression.
@tothegump Any idea why?
I'm looking into this now.
I haven't figured out a way to fix it yet.
I can confirm that this works when run eagerly.

I tested both on the deprecated amqp backend and the redis backend.
The canvas never completes.

I still have no idea why.
@tothegump I need help here.
Any chance we can schedule a debugging session together?
> @tothegump I need help here.
> Any chance we can schedule a debugging session together?

Sure. Would you like to schedule a debugging session together by email?
Yes.
I just saw your email.
I'll respond.
I suspect that this issue is related to #2556 because this configuration works just fine:

```
workflow = chain(
    t.task1.s(),
    chord(
        [
            t.task2.s(),
            chain(
                t.task3.s(),
                chord(
                    [t.task4.s(), t.task5.s()],
                    t.task6.s()
                ),
                t.task3.s(),
                t.task4.s()
            )
        ],
        t.task7.s()
    )
)
```

Note 2 added tasks `t.task3` and `t.task4` calls after inner chord. The workflow will not work without them or with only one task there. This configuration still hangs: 

```
workflow = chain(
    t.task1.s(),
    chord(
        [
            t.task2.s(),
            chain(
                t.task3.s(),
                chord(
                    [t.task4.s(), t.task5.s()],
                    t.task6.s()
                ),
                t.task3.s()
            )
        ],
        t.task7.s()
    )
)
```

Just came across the same issue - Confirmed

Hello. Is there any progress on this? I ran into what might be a related issue today. I have something of the form:

```
group1 | group2 | task1 | task2
```

'task2' doesn't get executed even though everything else succeeds.

Interestingly, if  `group2` is composed of a single task, then the full pipeline succeeds. Does that give a clue into what is going on?

Example of failure:

```
from celery import Celery
from celery import chain
from celery import group

app = Celery('bug')
app.config_from_object('debug_celeryconfig')

@app.task
def t(v):
    print v
    return v


if __name__ == '__main__':

    # This works.
    group_1 = group([
        t.si('group_1_task_1'),
        t.si('group_1_task_2'),
    ])
    group_2 = group([
        t.si('group_2_task_1'),
    ])
    addtl_task_1 = t.si('addtl_task_1')
    addtl_task_2 = t.si('addtl_task_2')
    pipeline = group_1 | group_2 | addtl_task_1 | addtl_task_2
    assert pipeline.delay().get() == 'addtl_task_2'

    # This fails.
    group_1 = group([
        t.si('group_1_task_1'),
    ])
    group_2 = group([
        t.si('group_2_task_1'),
        t.si('group_2_task_2'),
    ])
    addtl_task_1 = t.si('addtl_task_1')
    addtl_task_2 = t.si('addtl_task_2')
    pipeline = group_1 | group_2 | addtl_task_1 | addtl_task_2
    assert pipeline.delay().get() == 'addtl_task_2'

    # This also fails.
    group_1 = group([
        t.si('group_1_task_1'),
        t.si('group_1_task_2')
    ])
    group_2 = group([
        t.si('group_2_task_1'),
        t.si('group_2_task_2') # NEW
    ])
    addtl_task_1 = t.si('addtl_task_1')
    addtl_task_2 = t.si('addtl_task_2')
    pipeline = group_1 | group_2 | addtl_task_1 | addtl_task_2
    assert pipeline.delay().get() == 'addtl_task_2'
```

@glebkuznetsov Can you please submit the failing test cases in a form of a PR?
That will certainly help resolve this issue?

@thedrow This issue is blocking for us. I can provide a PR for a failing test case and I can also work on a fix but I need a bit of guidance. Could you please tell me which kind of PR you expect for the failing test? Following code is a failing case with Redis as broker and backend, but it also fails with amqp as broker and backend. It passes when `CELERY_ALWAYS_EAGER = True`

```
from celery import Celery
from celery import group

app = Celery(
    'tasks',
    backend='redis://?new_join=1',
    broker='redis://'
)


@app.task
def t(v):
    print(v)
    return v


if __name__ == '__main__':

    # This doesn't work.
    group_1 = group([
        t.si('group_1_task_1'),
        t.si('group_1_task_2'),
    ])
    group_2 = group([
        t.si('group_2_task_1'),
    ])
    addtl_task_1 = t.si('addtl_task_1')
    pipeline = group_1 | group_2 | addtl_task_1
    result = pipeline.delay().get()
    assert result == 'addtl_task_1'
```

trying to chain groups will probably not work as you expect them to. See:

https://github.com/celery/celery/issues/1671#issuecomment-39901321

is anyone working on fixing it?

here is my example,
it's hanging when trying to do `result.get()` (ie `result.join_native()`) on waiting for results of inner group (`KeyValueStorage.get_many`) but works when doing  `result.join()`

```
job = group(
    group(
        add.s(16, 16),
        add.s(32, 32)
    ),
    (
        add.s(2, 2)
        |
        add.s(4)
    )
)
result = job.apply_async()
```

here is more complicated example, it hangs with either `join` or `native_join`, but on different routine (`BaseBackend.wait_for`)

```
job = group(
   (
        group(
            add.s(16, 16),
            add.s(32, 32)
        )
        |
        add.si(2, 4)
    ),
    (
        add.s(2, 2)
        |
        add.s(4)
    )
)
result = job.apply_async()
```

UPD:

this one (with chord instead of group and chain) works but only with `join` instead of `native_join`

```
job = group(
   (
        chord([
            add.s(16, 16),
            add.s(32, 32)
        ], add.si(2, 4))
    ),
    (
        add.s(2, 2)
        |
        add.s(4)
    )
)
result = job.apply_async()
```

UPD2: but some nested group workflows are still not working, but can be workarounded by turning them into a chord with a dummy task as a final note

@merwan how did you resolved your blocker? did you found some workaround? or some other similar library?

UPD: for our setup i applied this workaround: https://github.com/verifiedpixel/celery/commit/f42a70e04e46565443a10ff8f27e454707c2c977

I'd be happy to guide people regarding to how we're going to fix it.
@actionless Can you demonstrate how this fixes the issues?

@thedrow, to reproduce the problem and workaround you can use the last example i provided (group(chord, chain)) (with redis, obviously).

But it seems to be another unrelated to redis issue with "parsing" complicated task workflows (group-chain-group and for some cases like (group(group,group)) but they're harder to reproduce) -- it can be workarounded by using chord instead of group for nested taskset. Also sometimes during the debugging i had a feeling what `__init__` for `group` was called two times more than needed so the id of an executed task was not the same as for which parent task is waiting to be executed, i think i had that behavior with my second example.

These examples work for me in master.

Please open a new issue if you can reproduce a problem in that version!


Resolving patch:
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -599,7 +599,15 @@ def run(self, args=None, kwargs=None, group_id=None, chord=None,
             # chain option may already be set, resulting in
             # "multiple values for keyword argument 'chain'" error.
             # Issue #3379.
-            options['chain'] = tasks if not use_link else None
+            chain_ = tasks if not use_link else None
+            if 'chain' not in options:
+                options['chain'] = chain_
+            elif chain_ is not None:
+                # If a chain already exists, we need to extend it with the next
+                # tasks in the chain.
+                # Issue #5354.
+                options['chain'].extend(chain_)
+
             first_task.apply_async(**options)
             return results[0]
 
diff --git a/t/integration/tasks.py b/t/integration/tasks.py
--- a/t/integration/tasks.py
+++ b/t/integration/tasks.py
@@ -68,7 +68,7 @@ def delayed_sum_with_soft_guard(numbers, pause_time=1):
 
 @shared_task
 def tsum(nums):
-    """Sum an iterable of numbers"""
+    """Sum an iterable of numbers."""
     return sum(nums)
 
 


Test patch:
diff --git a/t/integration/test_canvas.py b/t/integration/test_canvas.py
--- a/t/integration/test_canvas.py
+++ b/t/integration/test_canvas.py
@@ -247,6 +247,103 @@ def test_chain_of_task_a_group_and_a_chord(self, manager):
         res = c()
         assert res.get(timeout=TIMEOUT) == 8
 
+    @flaky
+    def test_chain_of_chords_as_groups_chained_to_a_task_with_two_tasks(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        c = add.si(1, 0)
+        c = c | group(add.s(1), add.s(1))
+        c = c | tsum.s()
+        c = c | add.s(1)
+        c = c | group(add.s(1), add.s(1))
+        c = c | tsum.s()
+
+        res = c()
+        assert res.get(timeout=TIMEOUT) == 12
+
+    @flaky
+    def test_chain_of_chords_with_two_tasks(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        c = add.si(1, 0)
+        c = c | group(add.s(1), add.s(1))
+        c = c | tsum.s()
+        c = c | add.s(1)
+        c = c | chord(group(add.s(1), add.s(1)), tsum.s())
+
+        res = c()
+        assert res.get(timeout=TIMEOUT) == 12
+
+    @flaky
+    def test_chain_of_a_chord_and_a_group_with_two_tasks(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        c = add.si(1, 0)
+        c = c | group(add.s(1), add.s(1))
+        c = c | tsum.s()
+        c = c | add.s(1)
+        c = c | group(add.s(1), add.s(1))
+
+        res = c()
+        assert res.get(timeout=TIMEOUT) == [6, 6]
+
+    @flaky
+    def test_chain_of_a_chord_and_a_task_and_a_group(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        c = group(add.s(1, 1), add.s(1, 1))
+        c = c | tsum.s()
+        c = c | add.s(1)
+        c = c | group(add.s(1), add.s(1))
+
+        res = c()
+        assert res.get(timeout=TIMEOUT) == [6, 6]
+
+    @flaky
+    def test_chain_of_a_chord_and_two_tasks_and_a_group(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        c = group(add.s(1, 1), add.s(1, 1))
+        c = c | tsum.s()
+        c = c | add.s(1)
+        c = c | add.s(1)
+        c = c | group(add.s(1), add.s(1))
+
+        res = c()
+        assert res.get(timeout=TIMEOUT) == [7, 7]
+
+    @flaky
+    def test_chain_of_a_chord_and_three_tasks_and_a_group(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        c = group(add.s(1, 1), add.s(1, 1))
+        c = c | tsum.s()
+        c = c | add.s(1)
+        c = c | add.s(1)
+        c = c | add.s(1)
+        c = c | group(add.s(1), add.s(1))
+
+        res = c()
+        assert res.get(timeout=TIMEOUT) == [8, 8]
+
 
 class test_result_set:
 
@@ -338,7 +435,9 @@ def assert_ids(r, expected_value, expected_root_id, expected_parent_id):
 
 
 def assert_ping(manager):
-    ping_val = list(manager.inspect().ping().values())[0]
+    ping_result = manager.inspect().ping()
+    assert ping_result
+    ping_val = list(ping_result.values())[0]
     assert ping_val == {"ok": "pong"}