Repository: celery/celery

Issue:
Error handler on a task chain not called (Celery 5.1.x with amqp broker and rpc backend)
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [ ] I have verified that the issue exists against the `master` branch of Celery.
- [ ] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [ ] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [ ] I have verified that the issue exists against the `master` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [ ] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [x] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- None

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
software -> celery:5.1.2 (sun-harmonics) kombu:5.1.0 py:3.8.7
            billiard:3.6.4.0 py-amqp:5.0.6
platform -> system:Darwin arch:64bit
            kernel version:19.6.0 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:rpc:///

deprecated_settings: None
broker_url: 'amqp://****:********@localhost:5672//'
result_backend: 'rpc:///'
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: N/A or Unknown
* **Minimal Celery Version**: 5.1.0
* **Minimal Kombu Version**: N/A or Unknown
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: N/A or Unknown
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
amqp==5.0.6
billiard==3.6.4.0
celery==5.1.2
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.2.0
kombu==5.1.0
prompt-toolkit==3.0.19
pytz==2021.1
six==1.16.0
vine==5.0.0
wcwidth==0.2.5
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

We are chaining tasks together and then adding an errback to the chain.
The expectation is that if a task in the chain fails, then the errback will be called,
and the rest of the chain skipped. Our broker is `amqp` (RabbitMQ), and we are using the `rpc` backend.

This feature worked for 4.4.x and works in 5.0.5. Starting in 5.1.0, the Celery
worker will throw an internal exception and the errback is not called. Below is a simple test case:
<details>
<p>

*tasks.py*

```python
from celery import Celery
from celery.utils.log import get_task_logger

app = Celery()
app.conf.broker_url = "amqp://****:****@localhost:5672/"
app.conf.result_backend = "rpc://"

logger = get_task_logger(__name__)


@app.task
def task1():
    logger.info("TASK1")
    raise ValueError('foo')


@app.task
def task2():
    logger.info("TASK2")
    

@app.task
def error_handler(request, exc, traceback):
    logger.error("ERROR HANDLER")
    logger.error('Task {0} raised exception: {1!r}'.format(
          request.id, exc))

```

Then run `task1` and `task2` in a chain using `error_handler` as the errback:

```python
from tasks import error_handler, task1, task2

chain = (task1.s() | task2.s())
x = chain.apply_async(link_error=error_handler.s())
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->

`task1` will fail and `error_handler` should be called:

```
[INFO/MainProcess] Task tasks.task1[35e2f20e-fc5b-4889-8ae0-9c1a593a15ec] received
[INFO/ForkPoolWorker-7] tasks.task1[35e2f20e-fc5b-4889-8ae0-9c1a593a15ec]: TASK1
[ERROR/ForkPoolWorker-7] tasks.error_handler[None]: ERROR HANDLER
[ERROR/ForkPoolWorker-7] tasks.error_handler[None]: Task 35e2f20e-fc5b-4889-8ae0-9c1a593a15ec raised exception: ValueError('foo')
[ERROR/ForkPoolWorker-7] Task tasks.task1[35e2f20e-fc5b-4889-8ae0-9c1a593a15ec] raised unexpected: ValueError('foo')
```

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->

The Celery worker logs the following stack trace and `error_handler` is never called:

```
[ERROR/MainProcess] Pool callback raised exception: AttributeError("'dict' object has no attribute 'reply_to'")
Traceback (most recent call last):
  File "/****/share/virtualenvs/celery5-reply-to-reL48Jin/lib/python3.8/site-packages/billiard/pool.py", line 1796, in safe_apply_callback
    fun(*args, **kwargs)
  File "/****/share/virtualenvs/celery5-reply-to-reL48Jin/lib/python3.8/site-packages/celery/worker/request.py", line 571, in on_failure
    self.task.backend.mark_as_failure(
  File "/****/share/virtualenvs/celery5-reply-to-reL48Jin/lib/python3.8/site-packages/celery/backends/base.py", line 199, in mark_as_failure
    self.store_result(
  File "/****/share/virtualenvs/celery5-reply-to-reL48Jin/lib/python3.8/site-packages/celery/backends/rpc.py", line 198, in store_result
    routing_key, correlation_id = self.destination_for(task_id, request)
  File "/****/share/virtualenvs/celery5-reply-to-reL48Jin/lib/python3.8/site-packages/celery/backends/rpc.py", line 179, in destination_for
    return request.reply_to, request.correlation_id or task_id
AttributeError: 'dict' object has no attribute 'reply_to'
```



Discussion:
Hey @ppiatko :wave:,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our [Open Collective](https://opencollective.com/celery) and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

can  you add the test script?
There's a small example to reproduce the error in the "Details" section of "Minimally Reproducible Test Case". Did you need something more? Thanks!
got it
I've got the same issue, after some investigation I realized that could be related to this piece of code in backends/base.py:173

```python
        if request:
            # This task may be part of a chord
            if request.chord:
                self.on_chord_part_return(request, state, exc)
            # It might also have chained tasks which need to be propagated to,
            # this is most likely to be exclusive with being a direct part of a
            # chord but we'll handle both cases separately.
            #
            # The `chain_data` try block here is a bit tortured since we might
            # have non-iterable objects here in tests and it's easier this way.
            try:
                chain_data = iter(request.chain)
            except (AttributeError, TypeError):
                chain_data = tuple()
            for chain_elem in chain_data:
                chain_elem_opts = chain_elem['options']
                # If the state should be propagated, we'll do so for all
                # elements of the chain. This is only truly important so
                # that the last chain element which controls completion of
                # the chain itself is marked as completed to avoid stalls.
                if self.store_result and state in states.PROPAGATE_STATES:
                    try:
                        chained_task_id = chain_elem_opts['task_id']
                    except KeyError:
                        pass
                    else:
                        self.store_result(
                            chained_task_id, exc, state,
                            traceback=traceback, request=chain_elem
                        )
```

Looks like after raising the Exception, the chain iters again and a dictionary is sent to the rpc backend instead of a request instance.
Ah, interestingly I reconstructed a `Context` for the `chain_elem` to be failed a few lines below that busted call  to `self.store_result()` when the same `chain_elem` is passed to `on_chord_part_return()`. I must have assumed that backends would be fine with a dictionary but did know that the redis OCPR implementation expects a `Context` object.

I think this should be a fairly straightforward fix.

Resolving patch:
diff --git a/celery/backends/base.py b/celery/backends/base.py
--- a/celery/backends/base.py
+++ b/celery/backends/base.py
@@ -185,29 +185,35 @@ def mark_as_failure(self, task_id, exc,
             except (AttributeError, TypeError):
                 chain_data = tuple()
             for chain_elem in chain_data:
-                chain_elem_opts = chain_elem['options']
+                # Reconstruct a `Context` object for the chained task which has
+                # enough information to for backends to work with
+                chain_elem_ctx = Context(chain_elem)
+                chain_elem_ctx.update(chain_elem_ctx.options)
+                chain_elem_ctx.id = chain_elem_ctx.options.get('task_id')
+                chain_elem_ctx.group = chain_elem_ctx.options.get('group_id')
                 # If the state should be propagated, we'll do so for all
                 # elements of the chain. This is only truly important so
                 # that the last chain element which controls completion of
                 # the chain itself is marked as completed to avoid stalls.
-                if store_result and state in states.PROPAGATE_STATES:
-                    try:
-                        chained_task_id = chain_elem_opts['task_id']
-                    except KeyError:
-                        pass
-                    else:
-                        self.store_result(
-                            chained_task_id, exc, state,
-                            traceback=traceback, request=chain_elem
-                        )
+                #
+                # Some chained elements may be complex signatures and have no
+                # task ID of their own, so we skip them hoping that not
+                # descending through them is OK. If the last chain element is
+                # complex, we assume it must have been uplifted to a chord by
+                # the canvas code and therefore the condition below will ensure
+                # that we mark something as being complete as avoid stalling.
+                if (
+                    store_result and state in states.PROPAGATE_STATES and
+                    chain_elem_ctx.task_id is not None
+                ):
+                    self.store_result(
+                        chain_elem_ctx.task_id, exc, state,
+                        traceback=traceback, request=chain_elem_ctx,
+                    )
                 # If the chain element is a member of a chord, we also need
                 # to call `on_chord_part_return()` as well to avoid stalls.
-                if 'chord' in chain_elem_opts:
-                    failed_ctx = Context(chain_elem)
-                    failed_ctx.update(failed_ctx.options)
-                    failed_ctx.id = failed_ctx.options['task_id']
-                    failed_ctx.group = failed_ctx.options['group_id']
-                    self.on_chord_part_return(failed_ctx, state, exc)
+                if 'chord' in chain_elem_ctx.options:
+                    self.on_chord_part_return(chain_elem_ctx, state, exc)
             # And finally we'll fire any errbacks
             if call_errbacks and request.errbacks:
                 self._call_task_errbacks(request, exc, traceback)


Test patch: