Repository: celery/celery

Issue:
Chords contained in a group raise `AttributeError`s during freezing if passed through a backend
When calling `self.replace()` with a signature reconstructed from a serialized dictionary (e.g. after having been passed through a backend), if that signature is (or contains) a group which itself contains a chord, celery explodes after attempting to treat deeply nested dictionaries as signature objects. See below for a minimal repro I got together today.

My gut feel is that instantiating a signature from a dictionary may not be recursing down through the structure far enough and it leaves some of the encapsulated tasks as dicts. I've also noticed that groups containing a group also break in the same way, but I think that's because there's an internal promotion to a chord happening somewhere.

# Checklist

- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [ ] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [ ] I have verified that the issue exists against the `master` branch of Celery.
- [ ] I have included the contents of ``pip freeze`` in the issue.
- [ ] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates

#### Related Issues
 - #4015

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: N/A or Unknown
* **Minimal Celery Version**: N/A or Unknown
* **Minimal Kombu Version**: N/A or Unknown
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: N/A or Unknown
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
amqp==5.0.1
billiard==3.6.3.0
# Editable install with no version control (celery==5.0.0rc3)
-e /home/maybe/tmp/capp/venv/lib/python3.8/site-packages
click==7.1.2
click-didyoumean==0.0.3
click-repl==0.1.6
future==0.18.2
kombu==5.0.2
prompt-toolkit==3.0.7
pytz==2020.1
redis==3.5.3
six==1.15.0
vine==5.0.0
wcwidth==0.2.5
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
import celery

app = celery.Celery("app", backend="redis://")

@app.task
def foo(*_):
    return 42

@app.task(bind=True)
def replace_with(self, sig):
    assert isinstance(sig, dict)
    sig = celery.Signature.from_dict(sig)
    raise self.replace(sig)

if __name__ == "__main__":
    sig = celery.group(
        celery.group(foo.s()),
    )
    res = sig.delay()
    print(res.get())

    sig.freeze()
    res = replace_with.delay(sig)
    print(res.get())
```

</p>
</details>

# Expected Behavior
It shouldn't explode. Presumably tasks within the group/chord should be signatures rather than dicts.

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->
Stack trace in the worker output:
```
[2020-09-08 12:44:05,453: DEBUG/MainProcess] Task accepted: app.replace_with[dcea02fd-23a3-404a-9fdd-b213eb51c0d1] pid:453431
[2020-09-08 12:44:05,457: ERROR/ForkPoolWorker-8] Task app.replace_with[dcea02fd-23a3-404a-9fdd-b213eb51c0d1] raised unexpected: AttributeError("'dict' object has no attribute '_app'")
Traceback (most recent call last):
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/kombu/utils/objects.py", line 41, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'app'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/app/trace.py", line 409, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/app/trace.py", line 701, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/maybe/tmp/capp/app.py", line 13, in replace_with
    raise self.replace(sig)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/app/task.py", line 894, in replace
    sig.freeze(self.request.id)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/canvas.py", line 1302, in freeze
    self.tasks = group(self.tasks, app=self.app)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/kombu/utils/objects.py", line 43, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/canvas.py", line 1456, in app
    return self._get_app(self.body)
  File "/home/maybe/tmp/capp/venv/lib64/python3.8/site-packages/celery/canvas.py", line 1466, in _get_app
    app = tasks[0]._app
AttributeError: 'dict' object has no attribute '_app'
```


Discussion:
This patch seems to fix the issue for the test case, and the idea might just need to be replicated to other signatures types that contain other signatures (e.g. chords, chains):
```patch
*** celery/canvas.py        2020-09-08 12:03:57.627712549 +1000
--- venv/lib/python3.8/site-packages/celery/canvas.py   2020-09-08 12:58:24.277149667 +1000
***************
*** 1022,1029 ****

      @classmethod
      def from_dict(cls, d, app=None):
          return _upgrade(
!             d, group(d['kwargs']['tasks'], app=app, **d['options']),
          )

      def __init__(self, *tasks, **options):
--- 1022,1033 ----

      @classmethod
      def from_dict(cls, d, app=None):
+         tasks = (
+             Signature.from_dict(dict(t))
+             for t in d['kwargs']['tasks']
+         )
          return _upgrade(
!             d, group(tasks, app=app, **d['options']),
          )

      def __init__(self, *tasks, **options):
***************
```

Edit: In fact, it looks like this issue may have already been fixed for `chain`s
https://github.com/celery/celery/blob/465d26766d6d959e9f871bf3663d3491e4b82883/celery/canvas.py#L583-L590

This is the commit which fixed the same issue for chains: 230b1ff7cb59b720464ffa9cc76fb19b4366d775 and #4015 extended the original fix to deserialize all tasks in a chain.

Resolving patch:
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -1047,8 +1047,16 @@ class group(Signature):
 
     @classmethod
     def from_dict(cls, d, app=None):
+        # We need to mutate the `kwargs` element in place to avoid confusing
+        # `freeze()` implementations which end up here and expect to be able to
+        # access elements from that dictionary later and refer to objects
+        # canonicalized here
+        orig_tasks = d["kwargs"]["tasks"]
+        d["kwargs"]["tasks"] = rebuilt_tasks = type(orig_tasks)((
+            maybe_signature(task, app=app) for task in orig_tasks
+        ))
         return _upgrade(
-            d, group(d['kwargs']['tasks'], app=app, **d['options']),
+            d, group(rebuilt_tasks, app=app, **d['options']),
         )
 
     def __init__(self, *tasks, **options):
diff --git a/t/integration/tasks.py b/t/integration/tasks.py
--- a/t/integration/tasks.py
+++ b/t/integration/tasks.py
@@ -1,6 +1,6 @@
 from time import sleep
 
-from celery import Task, chain, chord, group, shared_task
+from celery import Signature, Task, chain, chord, group, shared_task
 from celery.exceptions import SoftTimeLimitExceeded
 from celery.utils.log import get_task_logger
 
@@ -244,3 +244,67 @@ def run(self):
         if self.request.retries:
             return self.request.retries
         raise ValueError()
+
+
+# The signatures returned by these tasks wouldn't actually run because the
+# arguments wouldn't be fulfilled - we never actually delay them so it's fine
+@shared_task
+def return_nested_signature_chain_chain():
+    return chain(chain([add.s()]))
+
+
+@shared_task
+def return_nested_signature_chain_group():
+    return chain(group([add.s()]))
+
+
+@shared_task
+def return_nested_signature_chain_chord():
+    return chain(chord([add.s()], add.s()))
+
+
+@shared_task
+def return_nested_signature_group_chain():
+    return group(chain([add.s()]))
+
+
+@shared_task
+def return_nested_signature_group_group():
+    return group(group([add.s()]))
+
+
+@shared_task
+def return_nested_signature_group_chord():
+    return group(chord([add.s()], add.s()))
+
+
+@shared_task
+def return_nested_signature_chord_chain():
+    return chord(chain([add.s()]), add.s())
+
+
+@shared_task
+def return_nested_signature_chord_group():
+    return chord(group([add.s()]), add.s())
+
+
+@shared_task
+def return_nested_signature_chord_chord():
+    return chord(chord([add.s()], add.s()), add.s())
+
+
+@shared_task
+def rebuild_signature(sig_dict):
+    sig_obj = Signature.from_dict(sig_dict)
+
+    def _recurse(sig):
+        if not isinstance(sig, Signature):
+            raise TypeError("{!r} is not a signature object".format(sig))
+        # Most canvas types have a `tasks` attribute
+        if isinstance(sig, (chain, group, chord)):
+            for task in sig.tasks:
+                _recurse(task)
+        # `chord`s also have a `body` attribute
+        if isinstance(sig, chord):
+            _recurse(sig.body)
+    _recurse(sig_obj)


Test patch:
diff --git a/requirements/test.txt b/requirements/test.txt
--- a/requirements/test.txt
+++ b/requirements/test.txt
@@ -1,6 +1,7 @@
 case>=1.3.1
 pytest~=6.0
 pytest-celery
+pytest-subtests
 pytest-timeout~=1.4.2
 boto3>=1.9.178
 moto==1.3.7
diff --git a/t/integration/test_canvas.py b/t/integration/test_canvas.py
--- a/t/integration/test_canvas.py
+++ b/t/integration/test_canvas.py
@@ -9,6 +9,7 @@
 from celery.exceptions import TimeoutError
 from celery.result import AsyncResult, GroupResult, ResultSet
 
+from . import tasks
 from .conftest import get_active_redis_channels, get_redis_connection
 from .tasks import (ExpectedException, add, add_chord_to_chord, add_replaced,
                     add_to_all, add_to_all_to_chord, build_chain_inside_task,
@@ -1095,3 +1096,101 @@ def test_nested_chord_group_chain_group_tail(self, manager):
         )
         res = sig.delay()
         assert res.get(timeout=TIMEOUT) == [[42, 42]]
+
+
+class test_signature_serialization:
+    """
+    Confirm nested signatures can be rebuilt after passing through a backend.
+
+    These tests are expected to finish and return `None` or raise an exception
+    in the error case. The exception indicates that some element of a nested
+    signature object was not properly deserialized from its dictionary
+    representation, and would explode later on if it were used as a signature.
+    """
+    def test_rebuild_nested_chain_chain(self, manager):
+        sig = chain(
+            tasks.return_nested_signature_chain_chain.s(),
+            tasks.rebuild_signature.s()
+        )
+        sig.delay().get(timeout=TIMEOUT)
+
+    def test_rebuild_nested_chain_group(self, manager):
+        sig = chain(
+            tasks.return_nested_signature_chain_group.s(),
+            tasks.rebuild_signature.s()
+        )
+        sig.delay().get(timeout=TIMEOUT)
+
+    def test_rebuild_nested_chain_chord(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        sig = chain(
+            tasks.return_nested_signature_chain_chord.s(),
+            tasks.rebuild_signature.s()
+        )
+        sig.delay().get(timeout=TIMEOUT)
+
+    def test_rebuild_nested_group_chain(self, manager):
+        sig = chain(
+            tasks.return_nested_signature_group_chain.s(),
+            tasks.rebuild_signature.s()
+        )
+        sig.delay().get(timeout=TIMEOUT)
+
+    def test_rebuild_nested_group_group(self, manager):
+        sig = chain(
+            tasks.return_nested_signature_group_group.s(),
+            tasks.rebuild_signature.s()
+        )
+        sig.delay().get(timeout=TIMEOUT)
+
+    def test_rebuild_nested_group_chord(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        sig = chain(
+            tasks.return_nested_signature_group_chord.s(),
+            tasks.rebuild_signature.s()
+        )
+        sig.delay().get(timeout=TIMEOUT)
+
+    def test_rebuild_nested_chord_chain(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        sig = chain(
+            tasks.return_nested_signature_chord_chain.s(),
+            tasks.rebuild_signature.s()
+        )
+        sig.delay().get(timeout=TIMEOUT)
+
+    def test_rebuild_nested_chord_group(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        sig = chain(
+            tasks.return_nested_signature_chord_group.s(),
+            tasks.rebuild_signature.s()
+        )
+        sig.delay().get(timeout=TIMEOUT)
+
+    def test_rebuild_nested_chord_chord(self, manager):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        sig = chain(
+            tasks.return_nested_signature_chord_chord.s(),
+            tasks.rebuild_signature.s()
+        )
+        sig.delay().get(timeout=TIMEOUT)
diff --git a/t/unit/tasks/test_canvas.py b/t/unit/tasks/test_canvas.py
--- a/t/unit/tasks/test_canvas.py
+++ b/t/unit/tasks/test_canvas.py
@@ -2,6 +2,7 @@
 from unittest.mock import MagicMock, Mock, call, patch, sentinel
 
 import pytest
+import pytest_subtests  # noqa: F401
 
 from celery._state import _task_stack
 from celery.canvas import (Signature, _chain, _maybe_group, chain, chord,
@@ -326,13 +327,9 @@ def test_from_dict_no_tasks(self):
 
     def test_from_dict_full_subtasks(self):
         c = chain(self.add.si(1, 2), self.add.si(3, 4), self.add.si(5, 6))
-
         serialized = json.loads(json.dumps(c))
-
         deserialized = chain.from_dict(serialized)
-
-        for task in deserialized.tasks:
-            assert isinstance(task, Signature)
+        assert all(isinstance(task, Signature) for task in deserialized.tasks)
 
     @pytest.mark.usefixtures('depends_on_current_app')
     def test_app_falls_back_to_default(self):
@@ -345,9 +342,8 @@ def test_handles_dicts(self):
         )
         c.freeze()
         tasks, _ = c._frozen
-        for task in tasks:
-            assert isinstance(task, Signature)
-            assert task.app is self.app
+        assert all(isinstance(task, Signature) for task in tasks)
+        assert all(task.app is self.app for task in tasks)
 
     def test_groups_in_chain_to_chord(self):
         g1 = group([self.add.s(2, 2), self.add.s(4, 4)])
@@ -698,6 +694,30 @@ def test_from_dict(self):
         x['args'] = None
         assert group.from_dict(dict(x))
 
+    def test_from_dict_deep_deserialize(self):
+        original_group = group([self.add.s(1, 2)] * 42)
+        serialized_group = json.loads(json.dumps(original_group))
+        deserialized_group = group.from_dict(serialized_group)
+        assert all(
+            isinstance(child_task, Signature)
+            for child_task in deserialized_group.tasks
+        )
+
+    def test_from_dict_deeper_deserialize(self):
+        inner_group = group([self.add.s(1, 2)] * 42)
+        outer_group = group([inner_group] * 42)
+        serialized_group = json.loads(json.dumps(outer_group))
+        deserialized_group = group.from_dict(serialized_group)
+        assert all(
+            isinstance(child_task, Signature)
+            for child_task in deserialized_group.tasks
+        )
+        assert all(
+            isinstance(grandchild_task, Signature)
+            for child_task in deserialized_group.tasks
+            for grandchild_task in child_task.tasks
+        )
+
     def test_call_empty_group(self):
         x = group(app=self.app)
         assert not len(x())
@@ -1005,22 +1025,35 @@ def test_repr(self):
         x.kwargs['body'] = None
         assert 'without body' in repr(x)
 
-    def test_freeze_tasks_body_is_group(self):
-        # Confirm that `group index` is passed from a chord to elements of its
-        # body when the chord itself is encapsulated in a group
+    def test_freeze_tasks_body_is_group(self, subtests):
+        # Confirm that `group index` values counting up from 0 are set for
+        # elements of a chord's body when the chord is encapsulated in a group
         body_elem = self.add.s()
-        chord_body = group([body_elem])
+        chord_body = group([body_elem] * 42)
         chord_obj = chord(self.add.s(), body=chord_body)
         top_group = group([chord_obj])
         # We expect the body to be the signature we passed in before we freeze
-        (embedded_body_elem, ) = chord_obj.body.tasks
-        assert embedded_body_elem is body_elem
-        assert embedded_body_elem.options == dict()
-        # When we freeze the chord, its body will be clones and options set
+        with subtests.test(msg="Validate body tasks are retained"):
+            assert all(
+                embedded_body_elem is body_elem
+                for embedded_body_elem in chord_obj.body.tasks
+            )
+        # We also expect the body to have no initial options - since all of the
+        # embedded body elements are confirmed to be `body_elem` this is valid
+        assert body_elem.options == {}
+        # When we freeze the chord, its body will be cloned and options set
         top_group.freeze()
-        (embedded_body_elem, ) = chord_obj.body.tasks
-        assert embedded_body_elem is not body_elem
-        assert embedded_body_elem.options["group_index"] == 0   # 0th task
+        with subtests.test(
+            msg="Validate body group indicies count from 0 after freezing"
+        ):
+            assert all(
+                embedded_body_elem is not body_elem
+                for embedded_body_elem in chord_obj.body.tasks
+            )
+            assert all(
+                embedded_body_elem.options["group_index"] == i
+                for i, embedded_body_elem in enumerate(chord_obj.body.tasks)
+            )
 
     def test_freeze_tasks_is_not_group(self):
         x = chord([self.add.s(2, 2)], body=self.add.s(), app=self.app)
@@ -1050,6 +1083,117 @@ def chord_add():
             _state.task_join_will_block = fixture_task_join_will_block
             result.task_join_will_block = fixture_task_join_will_block
 
+    def test_from_dict(self):
+        header = self.add.s(1, 2)
+        original_chord = chord(header=header)
+        rebuilt_chord = chord.from_dict(dict(original_chord))
+        assert isinstance(rebuilt_chord, chord)
+
+    def test_from_dict_with_body(self):
+        header = body = self.add.s(1, 2)
+        original_chord = chord(header=header, body=body)
+        rebuilt_chord = chord.from_dict(dict(original_chord))
+        assert isinstance(rebuilt_chord, chord)
+
+    def test_from_dict_deep_deserialize(self, subtests):
+        header = body = self.add.s(1, 2)
+        original_chord = chord(header=header, body=body)
+        serialized_chord = json.loads(json.dumps(original_chord))
+        deserialized_chord = chord.from_dict(serialized_chord)
+        with subtests.test(msg="Verify chord is deserialized"):
+            assert isinstance(deserialized_chord, chord)
+        with subtests.test(msg="Validate chord header tasks is deserialized"):
+            assert all(
+                isinstance(child_task, Signature)
+                for child_task in deserialized_chord.tasks
+            )
+        with subtests.test(msg="Verify chord body is deserialized"):
+            assert isinstance(deserialized_chord.body, Signature)
+
+    def test_from_dict_deep_deserialize_group(self, subtests):
+        header = body = group([self.add.s(1, 2)] * 42)
+        original_chord = chord(header=header, body=body)
+        serialized_chord = json.loads(json.dumps(original_chord))
+        deserialized_chord = chord.from_dict(serialized_chord)
+        with subtests.test(msg="Verify chord is deserialized"):
+            assert isinstance(deserialized_chord, chord)
+        # A header which is a group gets unpacked into the chord's `tasks`
+        with subtests.test(
+            msg="Validate chord header tasks are deserialized and unpacked"
+        ):
+            assert all(
+                isinstance(child_task, Signature)
+                and not isinstance(child_task, group)
+                for child_task in deserialized_chord.tasks
+            )
+        # A body which is a group remains as it we passed in
+        with subtests.test(
+            msg="Validate chord body is deserialized and not unpacked"
+        ):
+            assert isinstance(deserialized_chord.body, group)
+            assert all(
+                isinstance(body_child_task, Signature)
+                for body_child_task in deserialized_chord.body.tasks
+            )
+
+    def test_from_dict_deeper_deserialize_group(self, subtests):
+        inner_group = group([self.add.s(1, 2)] * 42)
+        header = body = group([inner_group] * 42)
+        original_chord = chord(header=header, body=body)
+        serialized_chord = json.loads(json.dumps(original_chord))
+        deserialized_chord = chord.from_dict(serialized_chord)
+        with subtests.test(msg="Verify chord is deserialized"):
+            assert isinstance(deserialized_chord, chord)
+        # A header which is a group gets unpacked into the chord's `tasks`
+        with subtests.test(
+            msg="Validate chord header tasks are deserialized and unpacked"
+        ):
+            assert all(
+                isinstance(child_task, group)
+                for child_task in deserialized_chord.tasks
+            )
+            assert all(
+                isinstance(grandchild_task, Signature)
+                for child_task in deserialized_chord.tasks
+                for grandchild_task in child_task.tasks
+            )
+        # A body which is a group remains as it we passed in
+        with subtests.test(
+            msg="Validate chord body is deserialized and not unpacked"
+        ):
+            assert isinstance(deserialized_chord.body, group)
+            assert all(
+                isinstance(body_child_task, group)
+                for body_child_task in deserialized_chord.body.tasks
+            )
+            assert all(
+                isinstance(body_grandchild_task, Signature)
+                for body_child_task in deserialized_chord.body.tasks
+                for body_grandchild_task in body_child_task.tasks
+            )
+
+    def test_from_dict_deep_deserialize_chain(self, subtests):
+        header = body = chain([self.add.s(1, 2)] * 42)
+        original_chord = chord(header=header, body=body)
+        serialized_chord = json.loads(json.dumps(original_chord))
+        deserialized_chord = chord.from_dict(serialized_chord)
+        with subtests.test(msg="Verify chord is deserialized"):
+            assert isinstance(deserialized_chord, chord)
+        # A header which is a chain gets unpacked into the chord's `tasks`
+        with subtests.test(
+            msg="Validate chord header tasks are deserialized and unpacked"
+        ):
+            assert all(
+                isinstance(child_task, Signature)
+                and not isinstance(child_task, chain)
+                for child_task in deserialized_chord.tasks
+            )
+        # A body which is a chain gets mutatated into the hidden `_chain` class
+        with subtests.test(
+            msg="Validate chord body is deserialized and not unpacked"
+        ):
+            assert isinstance(deserialized_chord.body, _chain)
+
 
 class test_maybe_signature(CanvasCase):