Repository: celery/celery

Issue:
task/group chains fails in some scenarios
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->

- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [x] I have tried reproducing the issue on more than one operating system.
- [x] I have tried reproducing the issue on more than one workers pool.
- [x] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [x] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.
## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Possible Duplicates

- [Subsequent groups within a chain fail #3585](https://github.com/celery/celery/issues/3585)
- [Consecutive groups in chain fails #4848 ](https://github.com/celery/celery/issues/3585)

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**: 4.2.2 (windowlicker)
**Celery report**:
software -> celery:4.2.2 (windowlicker)
                   kombu:4.3.0 py:2.7.5
                   billiard:3.5.0.5 redis:3.2.1
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/0

broker_url: redis://localhost:6379/0
result_backend: redis://localhost:6379/0

</p>
</details>

# Steps to Reproduce
## Minimally Reproducible Test Case

```
from celery import signature, group, chain
from tasks import task

task1 = task.si('task1')
task2 = task.si('task2')

group1 = group(task.si('group1 job1'), task.si('group1 job2'))
group2 = group(task.si('group2 job1'), task.si('group2 job2'))

# working sequences
#res = chain(group1, group2)
#res = chain(group1, task1, group2)
#res = chain(task1, group1, task2, group2)
#res = chain(group1, group2, task1)

# failing sequence
res = chain(task1, group1, group2)

res.delay()
```
# Expected Behavior
Tasks are passed to celery

# Actual Behavior
It looks like celery is making the wrong decision to convert this chain into a chord. There are some workarounds available, but no real fix is available yet.

```
Traceback (most recent call last):
  File "debug_run.py", line 19, in <module>
    res.delay()
  File "/home/ja04913/ocenter/ocenter-venv/lib/python2.7/site-packages/celery/canvas.py", line 179, in delay
    return self.apply_async(partial_args, partial_kwargs)
  File "/home/ja04913/ocenter/ocenter-venv/lib/python2.7/site-packages/celery/canvas.py", line 557, in apply_async
    dict(self.options, **options) if options else self.options))
  File "/home/ja04913/ocenter/ocenter-venv/lib/python2.7/site-packages/celery/canvas.py", line 573, in run
    task_id, group_id, chord,
  File "/home/ja04913/ocenter/ocenter-venv/lib/python2.7/site-packages/celery/canvas.py", line 655, in prepare_steps
    task_id=prev_res.task_id, root_id=root_id, app=app,
AttributeError: 'GroupResult' object has no attribute 'task_id'
```

Subsequent groups within a chain fail
```
software -> celery:4.0.0 (latentcall) kombu:4.0.0 py:2.7.10
            billiard:3.5.0.2 py-amqp:2.1.1
platform -> system:FreeBSD arch:64bit, ELF imp:PyPy
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:cache+memcached://10.1.1.2:11211/
```
```
amqp==2.1.1
celery==4.0.0
Django==1.10.3
librabbitmq==1.6.1
pylibmc==1.5.1
```

## Steps to reproduce

```python
# celery.py
from __future__ import absolute_import
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')

app = Celery('project')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(force=True)
```

```python
from celery import task, group, chain

@task
def add(x, y):
    return x + y

# chain( task, group(tasks) )
x = chain( add.si(1, 1), group([add.si(1, 1), add.si(1, 1)]) )
type(x) # celery.canvas._chain
x.apply_async() # works as expected

# chain( task, group(tasks), group(tasks) )
x = chain( add.si(1, 1), group([add.si(1, 1), add.si(1, 1)]), group([add.si(1, 1), add.si(1, 1)]) )
type(x) # celery.canvas._chain
x.apply_async() # fails, traceback below

# chain( task, group(tasks), task, group(tasks) )
x = chain( add.si(1, 1), group([add.si(1, 1), add.si(1, 1)]), add.si(1, 1), group([add.si(1, 1), add.si(1, 1)]) )
type(x) # celery.canvas._chain
x.apply_async() # works as expected
```

## Expected behavior

It seems like a chain containing subsequent groups fails. If there is a task in between, or just a single group, it works as expected. This is different behavior from 3.x and I can't seem to find any obvious differences in the docs. The only thing that looks relevant in the release notes is the fix about `group | group` being flattened into a single group (#2573).

## Actual behavior

```python
AttributeError                            Traceback (most recent call last)
<ipython-input-30-495e2ee230ea> in <module>()
----> 1 x.apply_async()

/project/site-packages/celery/canvas.pyc in delay(self, *partial_args, **partial_kwargs)
    180     def delay(self, *partial_args, **partial_kwargs):
    181         """Shortcut to :meth:`apply_async` using star arguments."""
--> 182         return self.apply_async(partial_args, partial_kwargs)
    183 
    184     def apply(self, args=(), kwargs={}, **options):

/project/site-packages/celery/canvas.pyc in apply_async(self, args, kwargs, **options)
    565             return self.apply(args, kwargs, **options)
    566         return self.run(args, kwargs, app=app, **(
--> 567             dict(self.options, **options) if options else self.options))
    568 
    569     def run(self, args=(), kwargs={}, group_id=None, chord=None,

/project/site-packages/celery/canvas.pyc in run(self, args, kwargs, group_id, chord, task_id, link, link_error, publisher, producer, root_id, parent_id, app, **options)
    584             tasks, results = self.prepare_steps(
    585                 args, self.tasks, root_id, parent_id, link_error, app,
--> 586                 task_id, group_id, chord,
    587             )
    588 

/project/site-packages/celery/canvas.pyc in prepare_steps(self, args, tasks, root_id, parent_id, link_error, app, last_task_id, group_id, chord_body, clone, from_dict)
    663                 task = chord(
    664                     task, body=prev_task,
--> 665                     task_id=prev_res.task_id, root_id=root_id, app=app,
    666                 )
    667 

AttributeError: 'GroupResult' object has no attribute 'task_id'
```


Discussion:
are you using celery 4.3?
I've tested it with 4.3 where it shows the same behavior.
I can confirm I also have this issue on 4.3. 
Hey there. Look's like i'm facing the same issue.
Any workaround ?

I'm working with :
```
amqp (2.1.4)
celery (4.0.2)
kombu (4.0.2)
```

Here is my stack trace:
```python
Apr  4 19:31:59: File ".python/3.5-dev/lib/python3.5/site-packages/celery/canvas.py", line 567, in apply_async
Apr  4 19:31:59: dict(self.options, **options) if options else self.options))
Apr  4 19:31:59: File ".python/3.5-dev/lib/python3.5/site-packages/celery/canvas.py", line 586, in run
Apr  4 19:31:59: task_id, group_id, chord,
Apr  4 19:31:59: File ".python/3.5-dev/lib/python3.5/site-packages/celery/canvas.py", line 665, in prepare_steps
Apr  4 19:31:59: task_id=prev_res.task_id, root_id=root_id, app=app,
Apr  4 19:31:59: AttributeError: 'GroupResult' object has no attribute 'task_id'
```


My workflow looks like:
(TaskA | Group B | Group C | Goup D | Task E) and all the tasks are immutable.
RabbitMQ is used as broker and Mysql as result backend.

Regards.
It's not that ideal, but you can throw an empty/dummy task in between.
@mheppner 
Yes i could do that but it is far from ideal...

I'll wait and hope for a bugfix.
I would be glad to do it but but i'm not familiar enough with Celery internals :/

> It's not that ideal, but you can throw an empty/dummy task in between.

It does avoid error, but it ends up with wrong exection.

It seems that the behavior with multiple groups in chain seems to be broken in Celery v4.
Following example illustrate it:

```
# myapp.py (run worker with: celery -A myapp worker)
from celery import Celery

app = Celery('tasks', backend='redis://localhost:6379', broker='redis://localhost:6379')

@app.task
def nop(*args):
    print(args)
```

```
# myapp_run.py
import celery
import myapp

celery.chain(myapp.nop.si("1"),
             celery.group(myapp.nop.si("2-a"), myapp.nop.si("2-b")),
             myapp.nop.si("3"),
             celery.group(myapp.nop.si("4-a"), myapp.nop.si("4-b")),
             myapp.nop.si("5")).delay().get()
```

worker output with Celery v3.1.25 (expected behavior)

```
[2017-05-01 10:44:46,159: WARNING/Worker-2] ('1',)
[2017-05-01 10:44:46,178: WARNING/Worker-1] ('2-a',)
[2017-05-01 10:44:46,188: WARNING/Worker-2] ('2-b',)
[2017-05-01 10:44:46,693: WARNING/Worker-1] ('3',)
[2017-05-01 10:44:46,708: WARNING/Worker-2] ('4-a',)
[2017-05-01 10:44:46,712: WARNING/Worker-1] ('4-b',)
[2017-05-01 10:44:47,217: WARNING/Worker-2] ('5',)
```

worker output with Celery v4.0.2 (unexpected behavior)

```
[2017-05-01 10:42:41,983: WARNING/PoolWorker-2] ('1',)
[2017-05-01 10:42:41,986: WARNING/PoolWorker-1] ('2-a',)
[2017-05-01 10:42:41,988: WARNING/PoolWorker-2] ('2-b',)
[2017-05-01 10:42:41,990: WARNING/PoolWorker-1] ('4-a',)
[2017-05-01 10:42:41,992: WARNING/PoolWorker-1] ('4-b',)
[2017-05-01 10:42:41,995: WARNING/PoolWorker-1] ('4-b',)
[2017-05-01 10:42:41,995: WARNING/PoolWorker-2] ('4-a',)
[2017-05-01 10:42:42,000: WARNING/PoolWorker-1] ('3',)
[2017-05-01 10:42:42,013: WARNING/PoolWorker-1] ('5',)
[2017-05-01 10:42:42,013: WARNING/PoolWorker-2] ('5',)
[2017-05-01 10:42:42,014: WARNING/PoolWorker-2] ('4-a',)
[2017-05-01 10:42:42,016: WARNING/PoolWorker-2] ('4-b',)
[2017-05-01 10:42:42,018: WARNING/PoolWorker-2] ('5',)
```

Tested with:

* OS: Ubuntu 16.04.1
* Python: 3.5.2
* Redis: 3.0.6


@yoichi I think the issue you are describing is fixed in `master`. @xavierp is it possible to give example tasks and a chain that reproduce the issue? Thanks.
@georgepsarakis Thanks for the information. I've confirmed the problem is not reproduced with Celery in master branch (e812c578).

chained groups are still broken; using a "placeholder task" bypasses error of:
AttributeError: 'GroupResult' object has no attribute 'task_id'
I can confirm that this group-following-group error still occurs on 4.1.0. I don't know when 4.1.0 diverged from master, but it was released it a couple months after @georgepsarakis and @yoichi tested master. Strangely, if one of the groups happens to have only one element, then the error does not occur, so there must actually be multiple signatures per group to reliably test for the bug.
+1 - Just encountered it as well.
could you try celery 4.2?
Updated to 4.2.1 and am still receive "AttributeError: 'GroupResult' object has no attribute 'task_id'" for two consecutive groups in a chain.
Doing this in the meantime:

@app.task(name='group_separator')
def group_separator() -> None:
    return True
Thanks @denniswalker for this workaround. Group separator worked for me as I commented here: https://github.com/celery/celery/issues/4848#issuecomment-447317343.

Resolving patch:
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -669,10 +669,20 @@ def prepare_steps(self, args, kwargs, tasks,
                 # signature instead of a group.
                 tasks.pop()
                 results.pop()
-                task = chord(
-                    task, body=prev_task,
-                    task_id=prev_res.task_id, root_id=root_id, app=app,
-                )
+                try:
+                    task = chord(
+                        task, body=prev_task,
+                        task_id=prev_res.task_id, root_id=root_id, app=app,
+                    )
+                except AttributeError:
+                    # A GroupResult does not have a task_id since it consists
+                    # of multiple tasks.
+                    # We therefore, have to construct the chord without it.
+                    # Issues #5467, #3585.
+                    task = chord(
+                        task, body=prev_task,
+                        root_id=root_id, app=app,
+                    )
 
             if is_last_task:
                 # chain(task_id=id) means task id is set for the last task


Test patch:
diff --git a/t/integration/test_canvas.py b/t/integration/test_canvas.py
--- a/t/integration/test_canvas.py
+++ b/t/integration/test_canvas.py
@@ -232,6 +232,21 @@ def test_groupresult_serialization(self, manager):
         assert len(result) == 2
         assert isinstance(result[0][1], list)
 
+    @flaky
+    def test_chain_of_task_a_group_and_a_chord(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        c = add.si(1, 0)
+        c = c | group(add.s(1), add.s(1))
+        c = c | group(tsum.s(), tsum.s())
+        c = c | tsum.s()
+
+        res = c()
+        assert res.get(timeout=TIMEOUT) == 8
+
 
 class test_result_set: