Repository: celery/celery

Issue:
Chain-in-chord that fails before the last task in the chain doesn't trigger chord error handling
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
- [ ] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).


## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [x] I have tried reproducing the issue on more than one workers pool.
- [x] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [x] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.
Confirmed to exist on at least celery 4.3.0 as well.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- #3709  is related to error handling for chords, though this is more of an edge case in which the behavior is problematic, rather than just misalignment of docs and actual behavior.

#### Possible Duplicates

- There are several other issues that deal with chains-in-chords, e.g. #5229 seems similar, but nothing that addresses this exact problem. I don't think this duplicates them, but I haven't dug into the celery code enough to be sure that they aren't caused the the same underlying misbehavior in the redis backend.

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**: 4.4.6 (cliffs)
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
software -> celery:4.4.6 (cliffs) kombu:4.6.11 py:3.6.9
            billiard:3.6.3.0 redis:3.5.3
platform -> system:Linux arch:64bit, ELF
            kernel version:4.4.0-17763-Microsoft imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/

broker_url: 'redis://localhost:6379//'
result_backend: 'redis://localhost:6379/'
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
Not sure that any of these are minimal, but I'll include what I can test locally:
* **Minimal Python Version**: 3.6
* **Minimal Celery Version**: 4.3
* **Minimal Kombu Version**: N/A or Unknown
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: 6.0.4
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
amqp==2.6.0
billiard==3.6.3.0
celery==4.4.6
future==0.18.2
importlib-metadata==1.7.0
kombu==4.6.11
pkg-resources==0.0.0
pytz==2020.1
redis==3.5.3
vine==1.3.0
zipp==3.1.0
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>

Output of `redis-server --version`, in case it's useful:
```
Redis server v=6.0.4 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=d5d470262d47f1a0
```

</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>
Basic test case below. I'm still working on converting this to an integration test for your test suite.

```python
from celery import Celery

app = Celery('task_fail', backend='redis://localhost:6379', broker='redis://localhost:6379')

@app.task
def simple_pass():
    print("Passing")
    pass

@app.task
def simple_fail():
    print("Failing!")
    raise Exception('Fail!')

@app.task
def record_failure(*args, **kwargs):
    print(f'Fail with args={args}, kwargs={kwargs}')

if __name__ == "__main__":
    from celery import chord

    # Test 1: Passes - When the chain fails at the end, the failure logic on
    # the chord is executed.
    try:
        chain = simple_pass.si() | simple_fail.si()
        ~chord([simple_pass.si(), chain],
                simple_pass.si().on_error(record_failure.s(test="test_1")))
    except Exception:
        print("Test 1 worked")

    # Test 2: Fails - When the chain has an intermediate exception, the failure
    # logic on the chord is not executed & the client simply hangs.
    try:
        chain = simple_pass.si() | simple_fail.si() | simple_pass.si()
        ~chord([simple_pass.si(), chain],
                simple_pass.si().on_error(record_failure.s(test="test_2")))
    except Exception:
        print("Test 2 worked")
```
Note that the only difference between test 1 and test 2 is that test 2 add the `simple_pass.si()` call to the end of the chain. That seems to be enough to trigger this error.
</p>
</details>

# Expected Behavior
Test 2 should behave the same way that Test 1 does. Output of Test 1, with some simplification:
```
[2020-07-10 15:28:52,940: INFO/MainProcess] Received task: task_fail.simple_pass[65241973-d598-4bbe-a0c8-1f86694c9f33]
[2020-07-10 15:28:52,945: WARNING/ForkPoolWorker-8] Passing
[2020-07-10 15:28:52,954: INFO/ForkPoolWorker-8] Task task_fail.simple_pass[65241973-d598-4bbe-a0c8-1f86694c9f33] succeeded in 0.008173699956387281s: None
[2020-07-10 15:28:52,967: INFO/MainProcess] Received task: task_fail.simple_pass[fc15c163-0616-4b87-94c3-2c1b225f2e0e]
[2020-07-10 15:28:52,972: WARNING/ForkPoolWorker-8] Passing
[2020-07-10 15:28:52,985: INFO/ForkPoolWorker-8] Task task_fail.simple_pass[fc15c163-0616-4b87-94c3-2c1b225f2e0e] succeeded in 0.013192899990826845s: None
[2020-07-10 15:28:52,985: INFO/MainProcess] Received task: task_fail.simple_fail[0b0a3925-7ea8-4423-8e87-3a9f00d5c967]
[2020-07-10 15:28:52,992: WARNING/ForkPoolWorker-8] Failing!
[2020-07-10 15:28:53,010: ERROR/ForkPoolWorker-8] Chord callback for '76840678-80a3-4915-96b8-04c135ecb237' raised: ChordError("Dependency 0b0a3925-7ea8-4423-8e87-3a9f00d5c967 raised Exception('Fail!',)",)
Traceback (most recent call last):
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "~/celery/task_fail.py", line 13, in simple_fail
    raise Exception('Fail!')
Exception: Fail!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/redis.py", line 449, in on_chord_part_return
    callback.delay([unpack(tup, decode) for tup in resl])
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/redis.py", line 449, in <listcomp>
    callback.delay([unpack(tup, decode) for tup in resl])
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/redis.py", line 402, in _unpack_chord_result
    raise ChordError('Dependency {0} raised {1!r}'.format(tid, retval))
celery.exceptions.ChordError: Dependency 0b0a3925-7ea8-4423-8e87-3a9f00d5c967 raised Exception('Fail!',)
[2020-07-10 15:28:53,022: INFO/MainProcess] Received task: task_fail.record_failure[5f264f5d-a330-4f01-9274-662474d31a8f]
[2020-07-10 15:28:53,026: ERROR/ForkPoolWorker-8] Task task_fail.simple_fail[0b0a3925-7ea8-4423-8e87-3a9f00d5c967] raised unexpected: Exception('Fail!',)
Traceback (most recent call last):
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "~/celery/task_fail.py", line 13, in simple_fail
    raise Exception('Fail!')
Exception: Fail![2020-07-10 15:28:53,028: WARNING/ForkPoolWorker-1] Fail with args=('bc3aeb74-68c1-40e9-9c55-be9fb7860786',), kwargs={'test': 'test_1'}
[2020-07-10 15:28:53,036: INFO/ForkPoolWorker-1] Task task_fail.record_failure[5f264f5d-a330-4f01-9274-662474d31a8f] succeeded in 0.007674199994653463s: None
```
The most important thing to notice here is that the `record_failure` task is called & logs the failure, including the ID of the failing task. I expect this behavior to occur in both cases.

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->
The output of the worker on Test 2 is:
```
[2020-07-10 15:28:55,039: INFO/MainProcess] Received task: task_fail.simple_pass[5779ce6a-64af-40ea-9b26-8ff3bdad348d]
[2020-07-10 15:28:55,050: INFO/MainProcess] Received task: task_fail.simple_pass[39469dfa-401c-45f0-9314-5849f6e7ca5f]
[2020-07-10 15:28:55,051: WARNING/ForkPoolWorker-8] Passing
[2020-07-10 15:28:55,051: WARNING/ForkPoolWorker-1] Passing
[2020-07-10 15:28:55,058: INFO/ForkPoolWorker-8] Task task_fail.simple_pass[5779ce6a-64af-40ea-9b26-8ff3bdad348d] succeeded in 0.007474799989722669s: None
[2020-07-10 15:28:55,062: INFO/ForkPoolWorker-1] Task task_fail.simple_pass[39469dfa-401c-45f0-9314-5849f6e7ca5f] succeeded in 0.010612400015816092s: None
[2020-07-10 15:28:55,069: INFO/MainProcess] Received task: task_fail.simple_fail[2fead196-cfe2-4ce5-896f-303c2f9970e2]
[2020-07-10 15:28:55,073: WARNING/ForkPoolWorker-8] Failing!
[2020-07-10 15:28:55,081: ERROR/ForkPoolWorker-8] Task task_fail.simple_fail[2fead196-cfe2-4ce5-896f-303c2f9970e2] raised unexpected: Exception('Fail!',)
Traceback (most recent call last):
  File "/home/crawjor/test/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/crawjor/test/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/crawjor/test/celery/task_fail.py", line 13, in simple_fail
    raise Exception('Fail!')
Exception: Fail!
```
Then no more progress is made. Note that neither the error handler nor the callback is ever invoked, despite the chain failing (which should trigger failure handlers in the chord).

The client side also hangs waiting on redis:
```
File "task_fail.py", line 40, in <module>
    simple_pass.si().on_error(record_failure.s(test="test_2")))
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/canvas.py", line 475, in __invert__
    return self.apply_async().get()
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/result.py", line 237, in get
    on_message=on_message,
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 200, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 268, in _wait_for_pending
    on_interval=on_interval):
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 55, in drain_events_until
    yield self.wait_for(p, wait, timeout=interval)
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 64, in wait_for
    wait(timeout=timeout)
  File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/redis.py", line 161, in drain_events
    message = self._pubsub.get_message(timeout=timeout)
  File "~/celery/celery.venv/lib/python3.6/site-packages/redis/client.py", line 3617, in get_message
    response = self.parse_response(block=False, timeout=timeout)
  File "~/celery/celery.venv/lib/python3.6/site-packages/redis/client.py", line 3503, in parse_response
    if not block and not conn.can_read(timeout=timeout):
  File "~/celery/celery.venv/lib/python3.6/site-packages/redis/connection.py", line 734, in can_read
    return self._parser.can_read(timeout)
  File "~/celery/celery.venv/lib/python3.6/site-packages/redis/connection.py", line 321, in can_read
    return self._buffer and self._buffer.can_read(timeout)
  File "~/celery/celery.venv/lib/python3.6/site-packages/redis/connection.py", line 231, in can_read
    raise_on_timeout=False)
  File "~/celery/celery.venv/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket
    data = recv(self._sock, socket_read_size)
  File "~/celery/celery.venv/lib/python3.6/site-packages/redis/_compat.py", line 72, in recv
```

There are no active, revoked, scheduled, or reserved tasks, and the queue is empty:
```
~/celery$ celery -A task_fail inspect active
-> celery: OK
    - empty -
~/celery$ celery -A task_fail inspect scheduled
-> celery: OK
    - empty -
~/celery$ celery -A task_fail inspect revoked
-> celery: OK
    - empty -
~/celery$ celery -A task_fail inspect reserved
-> celery: OK
    - empty -
```



Discussion:
Test case added in this PR: https://github.com/celery/celery/pull/6226

Resolving patch:
diff --git a/celery/backends/base.py b/celery/backends/base.py
--- a/celery/backends/base.py
+++ b/celery/backends/base.py
@@ -22,6 +22,7 @@
 import celery.exceptions
 from celery import current_app, group, maybe_signature, states
 from celery._state import get_current_task
+from celery.app.task import Context
 from celery.exceptions import (BackendGetMetaError, BackendStoreError,
                                ChordError, ImproperlyConfigured,
                                NotRegistered, TaskRevokedError, TimeoutError)
@@ -170,8 +171,44 @@ def mark_as_failure(self, task_id, exc,
             self.store_result(task_id, exc, state,
                               traceback=traceback, request=request)
         if request:
+            # This task may be part of a chord
             if request.chord:
                 self.on_chord_part_return(request, state, exc)
+            # It might also have chained tasks which need to be propagated to,
+            # this is most likely to be exclusive with being a direct part of a
+            # chord but we'll handle both cases separately.
+            #
+            # The `chain_data` try block here is a bit tortured since we might
+            # have non-iterable objects here in tests and it's easier this way.
+            try:
+                chain_data = iter(request.chain)
+            except (AttributeError, TypeError):
+                chain_data = tuple()
+            for chain_elem in chain_data:
+                chain_elem_opts = chain_elem['options']
+                # If the state should be propagated, we'll do so for all
+                # elements of the chain. This is only truly important so
+                # that the last chain element which controls completion of
+                # the chain itself is marked as completed to avoid stalls.
+                if self.store_result and state in states.PROPAGATE_STATES:
+                    try:
+                        chained_task_id = chain_elem_opts['task_id']
+                    except KeyError:
+                        pass
+                    else:
+                        self.store_result(
+                            chained_task_id, exc, state,
+                            traceback=traceback, request=chain_elem
+                        )
+                # If the chain element is a member of a chord, we also need
+                # to call `on_chord_part_return()` as well to avoid stalls.
+                if 'chord' in chain_elem_opts:
+                    failed_ctx = Context(chain_elem)
+                    failed_ctx.update(failed_ctx.options)
+                    failed_ctx.id = failed_ctx.options['task_id']
+                    failed_ctx.group = failed_ctx.options['group_id']
+                    self.on_chord_part_return(failed_ctx, state, exc)
+            # And finally we'll fire any errbacks
             if call_errbacks and request.errbacks:
                 self._call_task_errbacks(request, exc, traceback)
 
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -1134,7 +1134,14 @@ def link_error(self, sig):
             # pass a Mock object as argument.
             sig['immutable'] = True
             sig = Signature.from_dict(sig)
-        return self.tasks[0].link_error(sig)
+        # Any child task might error so we need to ensure that they are all
+        # capable of calling the linked error signature. This opens the
+        # possibility that the task is called more than once but that's better
+        # than it not being called at all.
+        #
+        # We return a concretised tuple of the signatures actually applied to
+        # each child task signature, of which there might be none!
+        return tuple(child_task.link_error(sig) for child_task in self.tasks)
 
     def _prepared(self, tasks, partial_args, group_id, root_id, app,
                   CallableSignature=abstract.CallableSignature,
@@ -1179,7 +1186,7 @@ def _apply_tasks(self, tasks, producer=None, app=None, p=None,
                 # end up messing up chord counts and there are all sorts of
                 # awful race conditions to think about. We'll hope it's not!
                 sig, res, group_id = current_task
-                chord_obj = sig.options.get("chord") or chord
+                chord_obj = chord if chord is not None else sig.options.get("chord")
                 # We need to check the chord size of each contributing task so
                 # that when we get to the final one, we can correctly set the
                 # size in the backend and the chord can be sensible completed.
diff --git a/t/integration/tasks.py b/t/integration/tasks.py
--- a/t/integration/tasks.py
+++ b/t/integration/tasks.py
@@ -223,6 +223,13 @@ def redis_echo(message):
     redis_connection.rpush('redis-echo', message)
 
 
+@shared_task
+def redis_count():
+    """Task that increments a well-known redis key."""
+    redis_connection = get_redis_connection()
+    redis_connection.incr('redis-count')
+
+
 @shared_task(bind=True)
 def second_order_replace1(self, state=False):
     redis_connection = get_redis_connection()


Test patch:
diff --git a/t/integration/test_canvas.py b/t/integration/test_canvas.py
--- a/t/integration/test_canvas.py
+++ b/t/integration/test_canvas.py
@@ -1,9 +1,11 @@
 import re
 import tempfile
+import uuid
 from datetime import datetime, timedelta
 from time import sleep
 
 import pytest
+import pytest_subtests  # noqa: F401
 
 from celery import chain, chord, group, signature
 from celery.backends.base import BaseKeyValueStoreBackend
@@ -17,7 +19,7 @@
                     add_to_all, add_to_all_to_chord, build_chain_inside_task,
                     chord_error, collect_ids, delayed_sum,
                     delayed_sum_with_soft_guard, fail, identity, ids,
-                    print_unicode, raise_error, redis_echo,
+                    print_unicode, raise_error, redis_count, redis_echo,
                     replace_with_chain, replace_with_chain_which_raises,
                     replace_with_empty_chain, retry_once, return_exception,
                     return_priority, second_order_replace1, tsum,
@@ -810,6 +812,109 @@ def test_nested_group_chord_body_chain(self, manager):
         # Re-raise the expected exception so this test will XFAIL
         raise expected_excinfo.value
 
+    def test_callback_called_by_group(self, manager, subtests):
+        if not manager.app.conf.result_backend.startswith("redis"):
+            raise pytest.skip("Requires redis result backend.")
+        redis_connection = get_redis_connection()
+
+        callback_msg = str(uuid.uuid4()).encode()
+        callback = redis_echo.si(callback_msg)
+
+        group_sig = group(identity.si(42), identity.si(1337))
+        group_sig.link(callback)
+        redis_connection.delete("redis-echo")
+        with subtests.test(msg="Group result is returned"):
+            res = group_sig.delay()
+            assert res.get(timeout=TIMEOUT) == [42, 1337]
+        with subtests.test(msg="Callback is called after group is completed"):
+            maybe_key_msg = redis_connection.blpop("redis-echo", TIMEOUT)
+            if maybe_key_msg is None:
+                raise TimeoutError("Callback was not called in time")
+            _, msg = maybe_key_msg
+            assert msg == callback_msg
+
+    def test_errback_called_by_group_fail_first(self, manager, subtests):
+        if not manager.app.conf.result_backend.startswith("redis"):
+            raise pytest.skip("Requires redis result backend.")
+        redis_connection = get_redis_connection()
+
+        errback_msg = str(uuid.uuid4()).encode()
+        errback = redis_echo.si(errback_msg)
+
+        group_sig = group(fail.s(), identity.si(42))
+        group_sig.link_error(errback)
+        redis_connection.delete("redis-echo")
+        with subtests.test(msg="Error propagates from group"):
+            res = group_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(msg="Errback is called after group task fails"):
+            maybe_key_msg = redis_connection.blpop("redis-echo", TIMEOUT)
+            if maybe_key_msg is None:
+                raise TimeoutError("Errback was not called in time")
+            _, msg = maybe_key_msg
+            assert msg == errback_msg
+
+    def test_errback_called_by_group_fail_last(self, manager, subtests):
+        if not manager.app.conf.result_backend.startswith("redis"):
+            raise pytest.skip("Requires redis result backend.")
+        redis_connection = get_redis_connection()
+
+        errback_msg = str(uuid.uuid4()).encode()
+        errback = redis_echo.si(errback_msg)
+
+        group_sig = group(identity.si(42), fail.s())
+        group_sig.link_error(errback)
+        redis_connection.delete("redis-echo")
+        with subtests.test(msg="Error propagates from group"):
+            res = group_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(msg="Errback is called after group task fails"):
+            maybe_key_msg = redis_connection.blpop("redis-echo", TIMEOUT)
+            if maybe_key_msg is None:
+                raise TimeoutError("Errback was not called in time")
+            _, msg = maybe_key_msg
+            assert msg == errback_msg
+
+    def test_errback_called_by_group_fail_multiple(self, manager, subtests):
+        if not manager.app.conf.result_backend.startswith("redis"):
+            raise pytest.skip("Requires redis result backend.")
+        redis_connection = get_redis_connection()
+
+        expected_errback_count = 42
+        errback = redis_count.si()
+
+        # Include a mix of passing and failing tasks
+        group_sig = group(
+            *(identity.si(42) for _ in range(24)),  # arbitrary task count
+            *(fail.s() for _ in range(expected_errback_count)),
+        )
+        group_sig.link_error(errback)
+        redis_connection.delete("redis-count")
+        with subtests.test(msg="Error propagates from group"):
+            res = group_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(msg="Errback is called after group task fails"):
+            check_interval = 0.1
+            check_max = int(TIMEOUT * check_interval)
+            for i in range(check_max + 1):
+                maybe_count = redis_connection.get("redis-count")
+                # It's either `None` or a base-10 integer
+                count = int(maybe_count or b"0")
+                if count == expected_errback_count:
+                    # escape and pass
+                    break
+                elif i < check_max:
+                    # try again later
+                    sleep(check_interval)
+                else:
+                    # fail
+                    assert count == expected_errback_count
+            else:
+                raise TimeoutError("Errbacks were not called in time")
+
 
 def assert_ids(r, expected_value, expected_root_id, expected_parent_id):
     root_id, parent_id, value = r.get(timeout=TIMEOUT)
@@ -1406,6 +1511,335 @@ def test_error_propagates_from_chord2(self, manager):
         with pytest.raises(ExpectedException):
             res.get(timeout=TIMEOUT)
 
+    def test_error_propagates_to_chord_from_simple(self, manager, subtests):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        child_sig = fail.s()
+
+        chord_sig = chord((child_sig, ), identity.s())
+        with subtests.test(msg="Error propagates from simple header task"):
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+
+        chord_sig = chord((identity.si(42), ), child_sig)
+        with subtests.test(msg="Error propagates from simple body task"):
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+
+    def test_errback_called_by_chord_from_simple(self, manager, subtests):
+        if not manager.app.conf.result_backend.startswith("redis"):
+            raise pytest.skip("Requires redis result backend.")
+        redis_connection = get_redis_connection()
+
+        errback_msg = str(uuid.uuid4()).encode()
+        errback = redis_echo.si(errback_msg)
+        child_sig = fail.s()
+
+        chord_sig = chord((child_sig, ), identity.s())
+        chord_sig.link_error(errback)
+        with subtests.test(msg="Error propagates from simple header task"):
+            redis_connection.delete("redis-echo")
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(
+            msg="Errback is called after simple header task fails"
+        ):
+            maybe_key_msg = redis_connection.blpop("redis-echo", TIMEOUT)
+            if maybe_key_msg is None:
+                raise TimeoutError("Errback was not called in time")
+            _, msg = maybe_key_msg
+            assert msg == errback_msg
+
+        chord_sig = chord((identity.si(42), ), child_sig)
+        chord_sig.link_error(errback)
+        with subtests.test(msg="Error propagates from simple body task"):
+            redis_connection.delete("redis-echo")
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(
+            msg="Errback is called after simple body task fails"
+        ):
+            maybe_key_msg = redis_connection.blpop("redis-echo", TIMEOUT)
+            if maybe_key_msg is None:
+                raise TimeoutError("Errback was not called in time")
+            _, msg = maybe_key_msg
+            assert msg == errback_msg
+
+    def test_error_propagates_to_chord_from_chain(self, manager, subtests):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        child_sig = chain(identity.si(42), fail.s(), identity.si(42))
+
+        chord_sig = chord((child_sig, ), identity.s())
+        with subtests.test(
+            msg="Error propagates from header chain which fails before the end"
+        ):
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+
+        chord_sig = chord((identity.si(42), ), child_sig)
+        with subtests.test(
+            msg="Error propagates from body chain which fails before the end"
+        ):
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+
+    def test_errback_called_by_chord_from_chain(self, manager, subtests):
+        if not manager.app.conf.result_backend.startswith("redis"):
+            raise pytest.skip("Requires redis result backend.")
+        redis_connection = get_redis_connection()
+
+        errback_msg = str(uuid.uuid4()).encode()
+        errback = redis_echo.si(errback_msg)
+        child_sig = chain(identity.si(42), fail.s(), identity.si(42))
+
+        chord_sig = chord((child_sig, ), identity.s())
+        chord_sig.link_error(errback)
+        with subtests.test(
+            msg="Error propagates from header chain which fails before the end"
+        ):
+            redis_connection.delete("redis-echo")
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(
+            msg="Errback is called after header chain which fails before the end"
+        ):
+            maybe_key_msg = redis_connection.blpop("redis-echo", TIMEOUT)
+            if maybe_key_msg is None:
+                raise TimeoutError("Errback was not called in time")
+            _, msg = maybe_key_msg
+            assert msg == errback_msg
+
+        chord_sig = chord((identity.si(42), ), child_sig)
+        chord_sig.link_error(errback)
+        with subtests.test(
+            msg="Error propagates from body chain which fails before the end"
+        ):
+            redis_connection.delete("redis-echo")
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(
+            msg="Errback is called after body chain which fails before the end"
+        ):
+            maybe_key_msg = redis_connection.blpop("redis-echo", TIMEOUT)
+            if maybe_key_msg is None:
+                raise TimeoutError("Errback was not called in time")
+            _, msg = maybe_key_msg
+            assert msg == errback_msg
+
+    def test_error_propagates_to_chord_from_chain_tail(self, manager, subtests):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        child_sig = chain(identity.si(42), fail.s())
+
+        chord_sig = chord((child_sig, ), identity.s())
+        with subtests.test(
+            msg="Error propagates from header chain which fails at the end"
+        ):
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+
+        chord_sig = chord((identity.si(42), ), child_sig)
+        with subtests.test(
+            msg="Error propagates from body chain which fails at the end"
+        ):
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+
+    def test_errback_called_by_chord_from_chain_tail(self, manager, subtests):
+        if not manager.app.conf.result_backend.startswith("redis"):
+            raise pytest.skip("Requires redis result backend.")
+        redis_connection = get_redis_connection()
+
+        errback_msg = str(uuid.uuid4()).encode()
+        errback = redis_echo.si(errback_msg)
+        child_sig = chain(identity.si(42), fail.s())
+
+        chord_sig = chord((child_sig, ), identity.s())
+        chord_sig.link_error(errback)
+        with subtests.test(
+            msg="Error propagates from header chain which fails at the end"
+        ):
+            redis_connection.delete("redis-echo")
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(
+            msg="Errback is called after header chain which fails at the end"
+        ):
+            maybe_key_msg = redis_connection.blpop("redis-echo", TIMEOUT)
+            if maybe_key_msg is None:
+                raise TimeoutError("Errback was not called in time")
+            _, msg = maybe_key_msg
+            assert msg == errback_msg
+
+        chord_sig = chord((identity.si(42), ), child_sig)
+        chord_sig.link_error(errback)
+        with subtests.test(
+            msg="Error propagates from body chain which fails at the end"
+        ):
+            redis_connection.delete("redis-echo")
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(
+            msg="Errback is called after body chain which fails at the end"
+        ):
+            maybe_key_msg = redis_connection.blpop("redis-echo", TIMEOUT)
+            if maybe_key_msg is None:
+                raise TimeoutError("Errback was not called in time")
+            _, msg = maybe_key_msg
+            assert msg == errback_msg
+
+    def test_error_propagates_to_chord_from_group(self, manager, subtests):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        child_sig = group(identity.si(42), fail.s())
+
+        chord_sig = chord((child_sig, ), identity.s())
+        with subtests.test(msg="Error propagates from header group"):
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+
+        chord_sig = chord((identity.si(42), ), child_sig)
+        with subtests.test(msg="Error propagates from body group"):
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+
+    def test_errback_called_by_chord_from_group(self, manager, subtests):
+        if not manager.app.conf.result_backend.startswith("redis"):
+            raise pytest.skip("Requires redis result backend.")
+        redis_connection = get_redis_connection()
+
+        errback_msg = str(uuid.uuid4()).encode()
+        errback = redis_echo.si(errback_msg)
+        child_sig = group(identity.si(42), fail.s())
+
+        chord_sig = chord((child_sig, ), identity.s())
+        chord_sig.link_error(errback)
+        with subtests.test(msg="Error propagates from header group"):
+            redis_connection.delete("redis-echo")
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(msg="Errback is called after header group fails"):
+            maybe_key_msg = redis_connection.blpop("redis-echo", TIMEOUT)
+            if maybe_key_msg is None:
+                raise TimeoutError("Errback was not called in time")
+            _, msg = maybe_key_msg
+            assert msg == errback_msg
+
+        chord_sig = chord((identity.si(42), ), child_sig)
+        chord_sig.link_error(errback)
+        with subtests.test(msg="Error propagates from body group"):
+            redis_connection.delete("redis-echo")
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(msg="Errback is called after body group fails"):
+            maybe_key_msg = redis_connection.blpop("redis-echo", TIMEOUT)
+            if maybe_key_msg is None:
+                raise TimeoutError("Errback was not called in time")
+            _, msg = maybe_key_msg
+            assert msg == errback_msg
+
+    def test_errback_called_by_chord_from_group_fail_multiple(
+        self, manager, subtests
+    ):
+        if not manager.app.conf.result_backend.startswith("redis"):
+            raise pytest.skip("Requires redis result backend.")
+        redis_connection = get_redis_connection()
+
+        fail_task_count = 42
+        errback = redis_count.si()
+        # Include a mix of passing and failing tasks
+        child_sig = group(
+            *(identity.si(42) for _ in range(24)),  # arbitrary task count
+            *(fail.s() for _ in range(fail_task_count)),
+        )
+
+        chord_sig = chord((child_sig, ), identity.s())
+        chord_sig.link_error(errback)
+        with subtests.test(msg="Error propagates from header group"):
+            redis_connection.delete("redis-count")
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(msg="Errback is called after header group fails"):
+            # NOTE: Here we only expect the errback to be called once since it
+            # is attached to the chord body which is a single task!
+            expected_errback_count = 1
+            check_interval = 0.1
+            check_max = int(TIMEOUT * check_interval)
+            for i in range(check_max + 1):
+                maybe_count = redis_connection.get("redis-count")
+                # It's either `None` or a base-10 integer
+                count = int(maybe_count or b"0")
+                if count == expected_errback_count:
+                    # escape and pass
+                    break
+                elif i < check_max:
+                    # try again later
+                    sleep(check_interval)
+                else:
+                    # fail
+                    assert count == expected_errback_count
+            else:
+                raise TimeoutError("Errbacks were not called in time")
+
+        chord_sig = chord((identity.si(42), ), child_sig)
+        chord_sig.link_error(errback)
+        with subtests.test(msg="Error propagates from body group"):
+            redis_connection.delete("redis-count")
+            res = chord_sig.delay()
+            with pytest.raises(ExpectedException):
+                res.get(timeout=TIMEOUT)
+        with subtests.test(msg="Errback is called after body group fails"):
+            # NOTE: Here we expect the errback to be called once per failing
+            # task in the chord body since it is a group
+            expected_errback_count = fail_task_count
+            check_interval = 0.1
+            check_max = int(TIMEOUT * check_interval)
+            for i in range(check_max + 1):
+                maybe_count = redis_connection.get("redis-count")
+                # It's either `None` or a base-10 integer
+                count = int(maybe_count or b"0")
+                if count == expected_errback_count:
+                    # escape and pass
+                    break
+                elif i < check_max:
+                    # try again later
+                    sleep(check_interval)
+                else:
+                    # fail
+                    assert count == expected_errback_count
+            else:
+                raise TimeoutError("Errbacks were not called in time")
+
 
 class test_signature_serialization:
     """
diff --git a/t/unit/tasks/test_canvas.py b/t/unit/tasks/test_canvas.py
--- a/t/unit/tasks/test_canvas.py
+++ b/t/unit/tasks/test_canvas.py
@@ -1,5 +1,5 @@
 import json
-from unittest.mock import MagicMock, Mock, call, patch, sentinel, ANY
+from unittest.mock import ANY, MagicMock, Mock, call, patch, sentinel
 
 import pytest
 import pytest_subtests  # noqa: F401