Repository: celery/celery

Issue:
5.1.0b1 chord of chains dont work
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
- [ ] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [ ] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [ ] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [ ] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [ ] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] I have included the contents of ``pip freeze`` in the issue.
- [ ] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [x] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- None

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:  ```5.1.0b1 (singularity)```
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
software -> celery:5.1.0b1 (singularity) kombu:5.1.0b1 py:3.8.5
            billiard:3.6.4.0 py-amqp:5.0.6
platform -> system:Linux arch:64bit, ELF
            kernel version:5.8.0-48-generic imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:redis://localhost/

broker_url: 'amqp://guest:********@localhost:5672//'
result_backend: 'redis://localhost/'
deprecated_settings: None
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: N/A or Unknown
* **Minimal Celery Version**: N/A or Unknown
* **Minimal Kombu Version**: N/A or Unknown
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: N/A or Unknown
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
import celery

app = celery.Celery(
    broker='amqp://localhost',
    backend='redis://localhost',
)


@app.task
def task_a(number):
    print(f'Task A: {number}')
    return number


@app.task
def task_b(number):
    print(f'Task B: {number}')
    return number


@app.task
def task_c(number):
    print(f'Task C: {number}')


if __name__ == '__main__':
    celery.chord(
      (celery.chain(task_a.s(i), task_b.s()))
      for i in range(3)
    )(task_c.s())  # yapf: disable

    app.start()
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->
Expected output:
```
[2021-04-11 16:44:51,364: WARNING/ForkPoolWorker-15] Task A: 0
[2021-04-11 16:44:51,364: WARNING/ForkPoolWorker-16] Task A: 1
[2021-04-11 16:44:51,364: WARNING/ForkPoolWorker-1] Task A: 2
[2021-04-11 16:44:51,387: WARNING/ForkPoolWorker-4] Task B: 2
[2021-04-11 16:44:51,387: WARNING/ForkPoolWorker-3] Task B: 0
[2021-04-11 16:44:51,387: WARNING/ForkPoolWorker-2] Task B: 1
[2021-04-11 16:44:51,413: WARNING/ForkPoolWorker-15] Task C: [0, 1, 2]
```

On `celery==5.0.5` it works fine

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->
```
[2021-04-11 16:33:24,583: WARNING/ForkPoolWorker-15] Task A: 0
[2021-04-11 16:33:24,583: WARNING/ForkPoolWorker-1] Task A: 2
[2021-04-11 16:33:24,583: WARNING/ForkPoolWorker-16] Task A: 1
[2021-04-11 16:33:24,608: WARNING/ForkPoolWorker-4] Task B: 2
[2021-04-11 16:33:24,608: WARNING/ForkPoolWorker-2] Task B: 0
[2021-04-11 16:33:24,608: WARNING/ForkPoolWorker-3] Task B: 1
```



Discussion:
Hey @Sobes76rus :wave:,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our [Open Collective](https://opencollective.com/celery) and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

@maybe-sybr If you can't get to it, I will.
I marked it as a blocker for 5.1.0 so we have to do it.
Looks like a bug in chord size counting again. For some reason we've counted 8 chord parts to return which is way off what it should be:
```
[2021-04-13 09:52:54,712: INFO/MainProcess] Connected to amqp://guest:**@127.13.37.1:5672//
[2021-04-13 09:52:54,717: INFO/MainProcess] mingle: searching for neighbors
[2021-04-13 09:52:55,737: INFO/MainProcess] mingle: all alone
[2021-04-13 09:52:55,752: INFO/MainProcess] celery@karma ready.
[2021-04-13 09:52:55,752: INFO/MainProcess] Received task: __main__.task_a[8c3bd499-cf53-4086-8c9d-73cc6bd981f4]
[2021-04-13 09:52:55,753: INFO/MainProcess] Received task: __main__.task_a[664d91de-4430-49ff-8965-476c2b4c80e2]
[2021-04-13 09:52:55,753: INFO/MainProcess] Received task: __main__.task_a[86ae66bb-4f43-46a8-8851-b62674cc3ad7]
[2021-04-13 09:52:55,855: WARNING/ForkPoolWorker-7] Task A: 0
[2021-04-13 09:52:55,855: WARNING/ForkPoolWorker-8] Task A: 1
[2021-04-13 09:52:55,856: WARNING/ForkPoolWorker-1] Task A: 2
[2021-04-13 09:52:55,868: INFO/MainProcess] Received task: __main__.task_b[d3161d3c-ab8f-4610-83ff-24fd3d12f538]
[2021-04-13 09:52:55,868: INFO/ForkPoolWorker-8] Task __main__.task_a[664d91de-4430-49ff-8965-476c2b4c80e2] succeeded in 0.013156214000446198s: 1
[2021-04-13 09:52:55,868: INFO/MainProcess] Received task: __main__.task_b[44dec7e8-152e-4e11-bd70-ce4e7cb8c84b]
[2021-04-13 09:52:55,868: INFO/ForkPoolWorker-7] Task __main__.task_a[8c3bd499-cf53-4086-8c9d-73cc6bd981f4] succeeded in 0.013253123000140477s: 0
[2021-04-13 09:52:55,868: INFO/ForkPoolWorker-1] Task __main__.task_a[86ae66bb-4f43-46a8-8851-b62674cc3ad7] succeeded in 0.01293888700001844s: 2
[2021-04-13 09:52:55,868: INFO/MainProcess] Received task: __main__.task_b[6da81e65-af87-4770-92fd-4ce3ce9e5937]
[2021-04-13 09:52:55,869: WARNING/ForkPoolWorker-2] Task B: 0
[2021-04-13 09:52:55,869: WARNING/ForkPoolWorker-3] Task B: 1
[2021-04-13 09:52:55,869: WARNING/ForkPoolWorker-4] Task B: 2
[2021-04-13 09:52:55,871: WARNING/ForkPoolWorker-2] OCPR: chord_size_bytes=b'8' ready=1
[2021-04-13 09:52:55,871: WARNING/ForkPoolWorker-4] OCPR: chord_size_bytes=b'8' ready=2
[2021-04-13 09:52:55,871: WARNING/ForkPoolWorker-3] OCPR: chord_size_bytes=b'8' ready=3
[2021-04-13 09:52:55,871: INFO/ForkPoolWorker-2] Task __main__.task_b[d3161d3c-ab8f-4610-83ff-24fd3d12f538] succeeded in 0.001984218999496079s: 0
[2021-04-13 09:52:55,871: INFO/ForkPoolWorker-4] Task __main__.task_b[6da81e65-af87-4770-92fd-4ce3ce9e5937] succeeded in 0.0020933550003974233s: 2
[2021-04-13 09:52:55,871: INFO/ForkPoolWorker-3] Task __main__.task_b[44dec7e8-152e-4e11-bd70-ce4e7cb8c84b] succeeded in 0.002132425999661791s: 1
```
Counting of chord size children of groups is busted.

https://github.com/celery/celery/blob/1901ea8594185c015d1518d89f3b90180275c0b9/celery/canvas.py#L1178-L1185

The else case here is counting the length of a simple signature which is the length of its top level attributes (args, kwargs, and friends). Running the MRTC with the following diff on celery master:

```diff
diff --git a/celery/backends/redis.py b/celery/backends/redis.py
index 74a2e18b5..c642fe88e 100644
--- a/celery/backends/redis.py
+++ b/celery/backends/redis.py
@@ -415,6 +415,7 @@ class RedisBackend(BaseKeyValueStoreBackend, AsyncBackendMixin):
         return retval
 
     def set_chord_size(self, group_id, chord_size):
+        raise Exception(chord_size)
         self.set(self.get_key_for_group(group_id, '.s'), chord_size)
 
     def apply_chord(self, header_result_args, body, **kwargs):
@@ -467,6 +468,7 @@ class RedisBackend(BaseKeyValueStoreBackend, AsyncBackendMixin):
             _, readycount, totaldiff, chord_size_bytes = pipeline.execute()[:4]
 
         totaldiff = int(totaldiff or 0)
+        print(f"OCPR: chord_size_bytes={chord_size_bytes} ready={readycount}")
 
         if chord_size_bytes:
             try:
diff --git a/celery/canvas.py b/celery/canvas.py
index 57b0aea06..8aa533f43 100644
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -1182,6 +1182,9 @@ class group(Signature):
                             chord_size = sig.tasks[-1].__length_hint__()
                         else:
                             chord_size = task_index + len(sig.tasks[-1])
+                            print(f"subtask is not chord -> {chord_size=}")
+                            print(f"{len(sig.tasks[-1])=}")
+                            print(f"{list(sig.tasks[-1])=}")
                     app.backend.set_chord_size(group_id, chord_size)
                 sig.apply_async(producer=producer, add_to_parent=False,
                                 chord=_chord, args=args, kwargs=kwargs,
```

-> 

```
subtask is not chord -> chord_size=8
len(sig.tasks[-1])=6
list(sig.tasks[-1])=['task', 'args', 'kwargs', 'options', 'subtask_type', 'immutable']
Traceback (most recent call last):
  File "/home/maybe/dev/foss/github/celery/scratch/6721/app.py", line 27, in <module>
    celery.chord(
  File "/home/maybe/dev/foss/github/celery/celery/canvas.py", line 1354, in __call__
    return self.apply_async((), {'body': body} if body else {}, **options)
  File "/home/maybe/dev/foss/github/celery/celery/canvas.py", line 1407, in apply_async
    return self.run(tasks, body, args, task_id=task_id, **merged_options)
  File "/home/maybe/dev/foss/github/celery/celery/canvas.py", line 1474, in run
    header_result = header(*partial_args, task_id=group_id, **options)
  File "/home/maybe/dev/foss/github/celery/celery/canvas.py", line 1064, in __call__
    return self.apply_async(partial_args, **options)
  File "/home/maybe/dev/foss/github/celery/celery/canvas.py", line 1089, in apply_async
    results = list(self._apply_tasks(tasks, producer, app, p,
  File "/home/maybe/dev/foss/github/celery/celery/canvas.py", line 1188, in _apply_tasks
    app.backend.set_chord_size(group_id, chord_size)
  File "/home/maybe/dev/foss/github/celery/celery/backends/redis.py", line 418, in set_chord_size
    raise Exception(chord_size)
Exception: 8
```

It looks like the else case on L1183 is assuming that the trailing task of the signature being checked (which is a chain since we passed the condition on L1180) is also a chain. In the MRTC the trailing task is a simple signature!

Couple of takeaways:
  * Using string equality on `subtask_type` is highly suspicious first of all. We should probably be doing an isinstance check since I think the objects we're touch at this stage are well formed `Signature` instances (otherwise `.subtask_type` would blow up).
  * But the actual fix for this is likely to be to modify that conditional to correctly handle a tail task which is a simple signature in the else case, and add a preceding elif to handle when the trailing task is a chain

I'll try to find some time to get a patch up this evening. We'll also want to add a couple of regression tests. This should affect all backends with chords since it's a counting error so unit tests and a chord supporting backend integration test would be nice.

*Edit:* FTR the broken lines were added in #6576 but IDK if it functioned as expected before then
```diff
diff --git a/celery/canvas.py b/celery/canvas.py
index 57b0aea06..e3e8e3d45 100644
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -1180,7 +1180,7 @@ class group(Signature):
                     if isinstance(sig, _chain):
                         if sig.tasks[-1].subtask_type == 'chord':
                             chord_size = sig.tasks[-1].__length_hint__()
-                        else:
+                        elif sig.tasks[-1].subtask_type == "group":
                             chord_size = task_index + len(sig.tasks[-1])
                     app.backend.set_chord_size(group_id, chord_size)
                 sig.apply_async(producer=producer, add_to_parent=False,
```

This diff does appear to run the MRTC as expected ->

```
[2021-04-13 16:11:47,828: INFO/MainProcess] celery@karma ready.
[2021-04-13 16:11:47,828: INFO/MainProcess] Received task: __main__.task_a[65331ad0-7ace-4939-b451-f3eda9ec157c]
[2021-04-13 16:11:47,829: INFO/MainProcess] Received task: __main__.task_a[c3524b98-ae11-441d-b68d-fa895253dea0]
[2021-04-13 16:11:47,829: INFO/MainProcess] Received task: __main__.task_a[5c6fe9ee-5a28-4571-8c09-57540503aeb0]
[2021-04-13 16:11:47,931: WARNING/ForkPoolWorker-8] Task A: 1
[2021-04-13 16:11:47,931: WARNING/ForkPoolWorker-7] Task A: 0
[2021-04-13 16:11:47,932: WARNING/ForkPoolWorker-1] Task A: 2
[2021-04-13 16:11:47,942: INFO/MainProcess] Received task: __main__.task_b[228c3d19-4a26-4e9f-9613-57c095cb0078]
[2021-04-13 16:11:47,943: INFO/ForkPoolWorker-7] Task __main__.task_a[65331ad0-7ace-4939-b451-f3eda9ec157c] succeeded in 0.011826527999801328s: 0
[2021-04-13 16:11:47,943: INFO/MainProcess] Received task: __main__.task_b[3fe44fe4-47a2-4b41-adac-cf0ea7e7cf44]
[2021-04-13 16:11:47,943: INFO/ForkPoolWorker-8] Task __main__.task_a[c3524b98-ae11-441d-b68d-fa895253dea0] succeeded in 0.011867351000546478s: 1
[2021-04-13 16:11:47,944: WARNING/ForkPoolWorker-2] Task B: 0
[2021-04-13 16:11:47,944: WARNING/ForkPoolWorker-3] Task B: 1
[2021-04-13 16:11:47,945: INFO/MainProcess] Received task: __main__.task_b[abe8305a-7c95-43f2-a519-074b0e36959b]
[2021-04-13 16:11:47,945: WARNING/ForkPoolWorker-3] OCPR: chord_size_bytes=b'3' ready=2
[2021-04-13 16:11:47,945: WARNING/ForkPoolWorker-2] OCPR: chord_size_bytes=b'3' ready=1
[2021-04-13 16:11:47,945: INFO/ForkPoolWorker-1] Task __main__.task_a[5c6fe9ee-5a28-4571-8c09-57540503aeb0] succeeded in 0.014302862000477035s: 2
[2021-04-13 16:11:47,946: INFO/ForkPoolWorker-3] Task __main__.task_b[3fe44fe4-47a2-4b41-adac-cf0ea7e7cf44] succeeded in 0.0021919120008533355s: 1
[2021-04-13 16:11:47,946: WARNING/ForkPoolWorker-7] Task B: 2
[2021-04-13 16:11:47,946: INFO/ForkPoolWorker-2] Task __main__.task_b[228c3d19-4a26-4e9f-9613-57c095cb0078] succeeded in 0.002304372999788029s: 0
[2021-04-13 16:11:47,946: WARNING/ForkPoolWorker-7] OCPR: chord_size_bytes=b'3' ready=3
[2021-04-13 16:11:47,948: INFO/ForkPoolWorker-7] Task __main__.task_b[abe8305a-7c95-43f2-a519-074b0e36959b] succeeded in 0.0020840869983658195s: 2
[2021-04-13 16:11:47,948: INFO/MainProcess] Received task: __main__.task_c[28abaab2-7c2a-48c3-a83a-d099d678cbbf]
[2021-04-13 16:11:47,948: WARNING/ForkPoolWorker-7] Task C: [0, 1, 2]
[2021-04-13 16:11:47,949: INFO/ForkPoolWorker-7] Task __main__.task_c[28abaab2-7c2a-48c3-a83a-d099d678cbbf] succeeded in 0.0005527139983314555s: None
^C
worker: Hitting Ctrl+C again will terminate all running tasks!
```

The logic is this block is actually a little more tortured than I thought. It's re-writing the chord size each time it loops but I think it needs to do this in an additive way which it doesn't appear to be doing, as well as possibly handling tail groups and chords but not chains. We'll have to think about this pretty carefully, like usual with these chord counting issues :/
Thank you for the analysis, @maybe-sybr. ❤️ 
Would you be able to come up with a fix?
> Thank you for the analysis, @maybe-sybr. heart
> Would you be able to come up with a fix?

I should have time to sit down and work through this is in the next day or two, yeah. I've actually found another edge case which might be related to this code and is affecting my own work so I'll be able to spend some time digging into it and hopefully get a PR up before this weekend.
This is on my list for today.

Resolving patch:
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -1170,21 +1170,25 @@ def _apply_tasks(self, tasks, producer=None, app=None, p=None,
             # we are able to tell when we are at the end by checking if
             # next_task is None.  This enables us to set the chord size
             # without burning through the entire generator.  See #3021.
+            chord_size = 0
             for task_index, (current_task, next_task) in enumerate(
                 lookahead(tasks)
             ):
+                # We expect that each task must be part of the same group which
+                # seems sensible enough. If that's somehow not the case we'll
+                # end up messing up chord counts and there are all sorts of
+                # awful race conditions to think about. We'll hope it's not!
                 sig, res, group_id = current_task
-                _chord = sig.options.get("chord") or chord
-                if _chord is not None and next_task is None:
-                    chord_size = task_index + 1
-                    if isinstance(sig, _chain):
-                        if sig.tasks[-1].subtask_type == 'chord':
-                            chord_size = sig.tasks[-1].__length_hint__()
-                        else:
-                            chord_size = task_index + len(sig.tasks[-1])
+                chord_obj = sig.options.get("chord") or chord
+                # We need to check the chord size of each contributing task so
+                # that when we get to the final one, we can correctly set the
+                # size in the backend and the chord can be sensible completed.
+                chord_size += _chord._descend(sig)
+                if chord_obj is not None and next_task is None:
+                    # Per above, sanity check that we only saw one group
                     app.backend.set_chord_size(group_id, chord_size)
                 sig.apply_async(producer=producer, add_to_parent=False,
-                                chord=_chord, args=args, kwargs=kwargs,
+                                chord=chord_obj, args=args, kwargs=kwargs,
                                 **options)
                 # adding callback to result, such that it will gradually
                 # fulfill the barrier.
@@ -1296,8 +1300,8 @@ def app(self):
         return app if app is not None else current_app
 
 
-@Signature.register_type()
-class chord(Signature):
+@Signature.register_type(name="chord")
+class _chord(Signature):
     r"""Barrier synchronization primitive.
 
     A chord consists of a header and a body.
@@ -1415,20 +1419,27 @@ def apply(self, args=None, kwargs=None,
         )
 
     @classmethod
-    def __descend(cls, sig_obj):
+    def _descend(cls, sig_obj):
         # Sometimes serialized signatures might make their way here
         if not isinstance(sig_obj, Signature) and isinstance(sig_obj, dict):
             sig_obj = Signature.from_dict(sig_obj)
         if isinstance(sig_obj, group):
             # Each task in a group counts toward this chord
             subtasks = getattr(sig_obj.tasks, "tasks", sig_obj.tasks)
-            return sum(cls.__descend(task) for task in subtasks)
+            return sum(cls._descend(task) for task in subtasks)
         elif isinstance(sig_obj, _chain):
-            # The last element in a chain counts toward this chord
-            return cls.__descend(sig_obj.tasks[-1])
+            # The last non-empty element in a chain counts toward this chord
+            for child_sig in sig_obj.tasks[-1::-1]:
+                child_size = cls._descend(child_sig)
+                if child_size > 0:
+                    return child_size
+            else:
+                # We have to just hope this chain is part of some encapsulating
+                # signature which is valid and can fire the chord body
+                return 0
         elif isinstance(sig_obj, chord):
             # The child chord's body counts toward this chord
-            return cls.__descend(sig_obj.body)
+            return cls._descend(sig_obj.body)
         elif isinstance(sig_obj, Signature):
             # Each simple signature counts as 1 completion for this chord
             return 1
@@ -1437,7 +1448,7 @@ def __descend(cls, sig_obj):
 
     def __length_hint__(self):
         tasks = getattr(self.tasks, "tasks", self.tasks)
-        return sum(self.__descend(task) for task in tasks)
+        return sum(self._descend(task) for task in tasks)
 
     def run(self, header, body, partial_args, app=None, interval=None,
             countdown=1, max_retries=None, eager=False,
@@ -1537,6 +1548,11 @@ def _get_app(self, body=None):
     body = getitem_property('kwargs.body', 'Body task of chord.')
 
 
+# Add a back-compat alias for the previous `chord` class name which conflicts
+# with keyword arguments elsewhere in this file
+chord = _chord
+
+
 def signature(varies, *args, **kwargs):
     """Create new signature.
 


Test patch:
diff --git a/t/integration/test_canvas.py b/t/integration/test_canvas.py
--- a/t/integration/test_canvas.py
+++ b/t/integration/test_canvas.py
@@ -704,6 +704,112 @@ def test_nested_group_group(self, manager):
         res = sig.delay()
         assert res.get(timeout=TIMEOUT) == [42, 42]
 
+    def test_nested_group_chord_counting_simple(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        gchild_sig = identity.si(42)
+        child_chord = chord((gchild_sig, ), identity.s())
+        group_sig = group((child_chord, ))
+        res = group_sig.delay()
+        # Wait for the result to land and confirm its value is as expected
+        assert res.get(timeout=TIMEOUT) == [[42]]
+
+    def test_nested_group_chord_counting_chain(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        gchild_count = 42
+        gchild_sig = chain((identity.si(1337), ) * gchild_count)
+        child_chord = chord((gchild_sig, ), identity.s())
+        group_sig = group((child_chord, ))
+        res = group_sig.delay()
+        # Wait for the result to land and confirm its value is as expected
+        assert res.get(timeout=TIMEOUT) == [[1337]]
+
+    def test_nested_group_chord_counting_group(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        gchild_count = 42
+        gchild_sig = group((identity.si(1337), ) * gchild_count)
+        child_chord = chord((gchild_sig, ), identity.s())
+        group_sig = group((child_chord, ))
+        res = group_sig.delay()
+        # Wait for the result to land and confirm its value is as expected
+        assert res.get(timeout=TIMEOUT) == [[1337] * gchild_count]
+
+    def test_nested_group_chord_counting_chord(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        gchild_count = 42
+        gchild_sig = chord(
+            (identity.si(1337), ) * gchild_count, identity.si(31337),
+        )
+        child_chord = chord((gchild_sig, ), identity.s())
+        group_sig = group((child_chord, ))
+        res = group_sig.delay()
+        # Wait for the result to land and confirm its value is as expected
+        assert res.get(timeout=TIMEOUT) == [[31337]]
+
+    def test_nested_group_chord_counting_mixed(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        gchild_count = 42
+        child_chord = chord(
+            (
+                identity.si(42),
+                chain((identity.si(42), ) * gchild_count),
+                group((identity.si(42), ) * gchild_count),
+                chord((identity.si(42), ) * gchild_count, identity.si(1337)),
+            ),
+            identity.s(),
+        )
+        group_sig = group((child_chord, ))
+        res = group_sig.delay()
+        # Wait for the result to land and confirm its value is as expected. The
+        # group result gets unrolled into the encapsulating chord, hence the
+        # weird unpacking below
+        assert res.get(timeout=TIMEOUT) == [
+            [42, 42, *((42, ) * gchild_count), 1337]
+        ]
+
+    @pytest.mark.xfail(raises=TimeoutError, reason="#6734")
+    def test_nested_group_chord_body_chain(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        child_chord = chord(identity.si(42), chain((identity.s(), )))
+        group_sig = group((child_chord, ))
+        res = group_sig.delay()
+        # The result can be expected to timeout since it seems like its
+        # underlying promise might not be getting fulfilled (ref #6734). Pick a
+        # short timeout since we don't want to block for ages and this is a
+        # fairly simple signature which should run pretty quickly.
+        expected_result = [[42]]
+        with pytest.raises(TimeoutError) as expected_excinfo:
+            res.get(timeout=TIMEOUT / 10)
+        # Get the child `AsyncResult` manually so that we don't have to wait
+        # again for the `GroupResult`
+        assert res.children[0].get(timeout=TIMEOUT) == expected_result[0]
+        assert res.get(timeout=TIMEOUT) == expected_result
+        # Re-raise the expected exception so this test will XFAIL
+        raise expected_excinfo.value
+
 
 def assert_ids(r, expected_value, expected_root_id, expected_parent_id):
     root_id, parent_id, value = r.get(timeout=TIMEOUT)
diff --git a/t/unit/tasks/test_canvas.py b/t/unit/tasks/test_canvas.py
--- a/t/unit/tasks/test_canvas.py
+++ b/t/unit/tasks/test_canvas.py
@@ -1,5 +1,5 @@
 import json
-from unittest.mock import MagicMock, Mock, call, patch, sentinel
+from unittest.mock import MagicMock, Mock, call, patch, sentinel, ANY
 
 import pytest
 import pytest_subtests  # noqa: F401
@@ -782,6 +782,194 @@ def test_kwargs_delay_partial(self):
         res = self.helper_test_get_delay(x.delay(y=1))
         assert res == [2, 2]
 
+    def test_apply_from_generator(self):
+        child_count = 42
+        child_sig = self.add.si(0, 0)
+        child_sigs_gen = (child_sig for _ in range(child_count))
+        group_sig = group(child_sigs_gen)
+        with patch("celery.canvas.Signature.apply_async") as mock_apply_async:
+            res_obj = group_sig.apply_async()
+        assert mock_apply_async.call_count == child_count
+        assert len(res_obj.children) == child_count
+
+    # This needs the current app for some reason not worth digging into
+    @pytest.mark.usefixtures('depends_on_current_app')
+    def test_apply_from_generator_empty(self):
+        empty_gen = (False for _ in range(0))
+        group_sig = group(empty_gen)
+        with patch("celery.canvas.Signature.apply_async") as mock_apply_async:
+            res_obj = group_sig.apply_async()
+        assert mock_apply_async.call_count == 0
+        assert len(res_obj.children) == 0
+
+    # In the following tests, getting the group ID is a pain so we just use
+    # `ANY` to wildcard it when we're checking on calls made to our mocks
+    def test_apply_contains_chord(self):
+        gchild_count = 42
+        gchild_sig = self.add.si(0, 0)
+        gchild_sigs = (gchild_sig, ) * gchild_count
+        child_chord = chord(gchild_sigs, gchild_sig)
+        group_sig = group((child_chord, ))
+        with patch.object(
+            self.app.backend, "set_chord_size",
+        ) as mock_set_chord_size, patch(
+            "celery.canvas.Signature.apply_async",
+        ) as mock_apply_async:
+            res_obj = group_sig.apply_async()
+        # We only see applies for the header grandchildren because the tasks
+        # are never actually run due to our mocking of `apply_async()`
+        assert mock_apply_async.call_count == gchild_count
+        assert len(res_obj.children) == len(group_sig.tasks)
+        # We must have set the chord size for the group of tasks which makes up
+        # the header of the `child_chord`, just before we apply the last task.
+        mock_set_chord_size.assert_called_once_with(ANY, gchild_count)
+
+    def test_apply_contains_chords_containing_chain(self):
+        ggchild_count = 42
+        ggchild_sig = self.add.si(0, 0)
+        gchild_sig = chain((ggchild_sig, ) * ggchild_count)
+        child_count = 24
+        child_chord = chord((gchild_sig, ), ggchild_sig)
+        group_sig = group((child_chord, ) * child_count)
+        with patch.object(
+            self.app.backend, "set_chord_size",
+        ) as mock_set_chord_size, patch(
+            "celery.canvas.Signature.apply_async",
+        ) as mock_apply_async:
+            res_obj = group_sig.apply_async()
+        # We only see applies for the header grandchildren because the tasks
+        # are never actually run due to our mocking of `apply_async()`
+        assert mock_apply_async.call_count == child_count
+        assert len(res_obj.children) == child_count
+        # We must have set the chord sizes based on the number of tail tasks of
+        # the encapsulated chains - in this case 1 for each child chord
+        mock_set_chord_size.assert_has_calls((call(ANY, 1), ) * child_count)
+
+    @pytest.mark.xfail(reason="Invalid canvas setup with bad exception")
+    def test_apply_contains_chords_containing_empty_chain(self):
+        gchild_sig = chain(tuple())
+        child_count = 24
+        child_chord = chord((gchild_sig, ), self.add.si(0, 0))
+        group_sig = group((child_chord, ) * child_count)
+        # This is an invalid setup because we can't complete a chord header if
+        # there are no actual tasks which will run in it. However, the current
+        # behaviour of an `IndexError` isn't particularly helpful to a user.
+        res_obj = group_sig.apply_async()
+
+    def test_apply_contains_chords_containing_chain_with_empty_tail(self):
+        ggchild_count = 42
+        ggchild_sig = self.add.si(0, 0)
+        tail_count = 24
+        gchild_sig = chain(
+            (ggchild_sig, ) * ggchild_count +
+            (group((ggchild_sig, ) * tail_count), group(tuple()), ),
+        )
+        child_chord = chord((gchild_sig, ), ggchild_sig)
+        group_sig = group((child_chord, ))
+        with patch.object(
+            self.app.backend, "set_chord_size",
+        ) as mock_set_chord_size, patch(
+            "celery.canvas.Signature.apply_async",
+        ) as mock_apply_async:
+            res_obj = group_sig.apply_async()
+        # We only see applies for the header grandchildren because the tasks
+        # are never actually run due to our mocking of `apply_async()`
+        assert mock_apply_async.call_count == 1
+        assert len(res_obj.children) == 1
+        # We must have set the chord sizes based on the size of the last
+        # non-empty task in the encapsulated chains - in this case `tail_count`
+        # for the group preceding the empty one in each grandchild chain
+        mock_set_chord_size.assert_called_once_with(ANY, tail_count)
+
+    def test_apply_contains_chords_containing_group(self):
+        ggchild_count = 42
+        ggchild_sig = self.add.si(0, 0)
+        gchild_sig = group((ggchild_sig, ) * ggchild_count)
+        child_count = 24
+        child_chord = chord((gchild_sig, ), ggchild_sig)
+        group_sig = group((child_chord, ) * child_count)
+        with patch.object(
+            self.app.backend, "set_chord_size",
+        ) as mock_set_chord_size, patch(
+            "celery.canvas.Signature.apply_async",
+        ) as mock_apply_async:
+            res_obj = group_sig.apply_async()
+        # We see applies for all of the header grandchildren because the tasks
+        # are never actually run due to our mocking of `apply_async()`
+        assert mock_apply_async.call_count == child_count * ggchild_count
+        assert len(res_obj.children) == child_count
+        # We must have set the chord sizes based on the number of tail tasks of
+        # the encapsulated groups - in this case `ggchild_count`
+        mock_set_chord_size.assert_has_calls(
+            (call(ANY, ggchild_count), ) * child_count,
+        )
+
+    @pytest.mark.xfail(reason="Invalid canvas setup but poor behaviour")
+    def test_apply_contains_chords_containing_empty_group(self):
+        gchild_sig = group(tuple())
+        child_count = 24
+        child_chord = chord((gchild_sig, ), self.add.si(0, 0))
+        group_sig = group((child_chord, ) * child_count)
+        with patch.object(
+            self.app.backend, "set_chord_size",
+        ) as mock_set_chord_size, patch(
+            "celery.canvas.Signature.apply_async",
+        ) as mock_apply_async:
+            res_obj = group_sig.apply_async()
+        # We only see applies for the header grandchildren because the tasks
+        # are never actually run due to our mocking of `apply_async()`
+        assert mock_apply_async.call_count == child_count
+        assert len(res_obj.children) == child_count
+        # This is actually kind of meaningless because, similar to the empty
+        # chain test, this is an invalid setup. However, we should probably
+        # expect that the chords are dealt with in some other way the probably
+        # being left incomplete forever...
+        mock_set_chord_size.assert_has_calls((call(ANY, 0), ) * child_count)
+
+    def test_apply_contains_chords_containing_chord(self):
+        ggchild_count = 42
+        ggchild_sig = self.add.si(0, 0)
+        gchild_sig = chord((ggchild_sig, ) * ggchild_count, ggchild_sig)
+        child_count = 24
+        child_chord = chord((gchild_sig, ), ggchild_sig)
+        group_sig = group((child_chord, ) * child_count)
+        with patch.object(
+            self.app.backend, "set_chord_size",
+        ) as mock_set_chord_size, patch(
+            "celery.canvas.Signature.apply_async",
+        ) as mock_apply_async:
+            res_obj = group_sig.apply_async()
+        # We see applies for all of the header great-grandchildren because the
+        # tasks are never actually run due to our mocking of `apply_async()`
+        assert mock_apply_async.call_count == child_count * ggchild_count
+        assert len(res_obj.children) == child_count
+        # We must have set the chord sizes based on the number of tail tasks of
+        # the deeply encapsulated chords' header tasks, as well as for each
+        # child chord. This means we have `child_count` interleaved calls to
+        # set chord sizes of 1 and `ggchild_count`.
+        mock_set_chord_size.assert_has_calls(
+            (call(ANY, 1), call(ANY, ggchild_count), ) * child_count,
+        )
+
+    def test_apply_contains_chords_containing_empty_chord(self):
+        gchild_sig = chord(tuple(), self.add.si(0, 0))
+        child_count = 24
+        child_chord = chord((gchild_sig, ), self.add.si(0, 0))
+        group_sig = group((child_chord, ) * child_count)
+        with patch.object(
+            self.app.backend, "set_chord_size",
+        ) as mock_set_chord_size, patch(
+            "celery.canvas.Signature.apply_async",
+        ) as mock_apply_async:
+            res_obj = group_sig.apply_async()
+        # We only see applies for the header grandchildren because the tasks
+        # are never actually run due to our mocking of `apply_async()`
+        assert mock_apply_async.call_count == child_count
+        assert len(res_obj.children) == child_count
+        # We must have set the chord sizes based on the number of tail tasks of
+        # the encapsulated chains - in this case 1 for each child chord
+        mock_set_chord_size.assert_has_calls((call(ANY, 1), ) * child_count)
+
 
 class test_chord(CanvasCase):