Repository: celery/celery

Issue:
`AttributeError` thrown when using `link_error`/`on_error` on a rewritten chain when `task_allow_error_cb_on_chord_header` is enabled
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `main` branch of Celery.
- [x] This has already been asked to the [discussions forum](https://github.com/celery/celery/discussions) first.
- [x] I have read the relevant section in the
  [contribution guide](https://docs.celeryq.dev/en/main/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/main)
  to find out if the bug was already fixed in the main branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `main` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [x] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [x] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- This appears to be a similar class of issue as #4848 and #5265, since it covers some of the same ground

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**: 5.3.1 (emerald-rush)
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
software -> celery:5.3.1 (emerald-rush) kombu:5.3.1 py:3.10.7
            billiard:4.1.0 redis:5.0.0
platform -> system:Darwin arch:64bit
            kernel version:22.6.0 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost/

broker_url: 'redis://localhost:6379//'
result_backend: 'redis://localhost/'
deprecated_settings: None
task_allow_error_cb_on_chord_header: True
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
- **Minimal Python Version**: 3.10.x
- **Minimal Celery Version**: 5.3.1
- **Minimal Kombu Version**: N/A
- **Minimal Broker Version**: N/A
- **Minimal Result Backend Version**: N/A
- **Minimal OS and/or Kernel Version**: N/A (we've reproduced with our own deployment on Linux, and locally on macOS)
- **Minimal Broker Client Version**: N/A
- **Minimal Result Backend Client Version**: N/A

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
amqp==5.1.1
async-timeout==4.0.3
billiard==4.1.0
celery @ git+https://github.com/celery/celery.git@8ae0b229596cc8aeea4fb71020d9358a59338e08
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
kombu==5.3.1
prompt-toolkit==3.0.39
python-dateutil==2.8.2
redis==5.0.0
six==1.16.0
tzdata==2023.3
vine==5.0.0
wcwidth==0.2.6
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
from celery import Celery, chain, group, chord

app = Celery('tasks', broker='redis://localhost/', backend='redis://localhost/')
app.conf.task_allow_error_cb_on_chord_header = True


@app.task
def fake_work_unit(x):
    raise Exception


@app.task
def nothing_work_unit():
    return "hi"


@app.task
def fake_error_handler(*args):
    print("oops")


def test():
    f = app.signature(
        "test_celery.nothing_work_unit",
        immutable=True,
    )

    f2 = app.signature(
        "test_celery.fake_error_handler",
    )

    # This formulation works correctly...
    # chain(
    #    f.on_error(f2),
    #    group(
    #        app.signature(
    #            "test_celery.fake_work_unit",
    #            args=(1,),
    #            immutable=True,
    #        ),
    #        app.signature(
    #            "test_celery.fake_work_unit",
    #            args=(2,),
    #            immutable=True,
    #        ),
    #    ).on_error(f2),
    #    f.on_error(f2),
    # ).apply_async()

    # ...but not this one.
    chain(
        f,
        group(
            app.signature(
                "test_celery.fake_work_unit",
                args=(1,),
                immutable=True,
            ),
            app.signature(
                "test_celery.fake_work_unit",
                args=(2,),
                immutable=True,
            ),
        ),
        f,
    ).on_error(f2).apply_async()
```

Run `celery -A test_celery worker` and then in a different shell:
```
from test_celery import test
test()
```

Then comment out the second formulation and uncomment the first. Notice that it works as intended.

This test case also works if you disable `task_allow_error_cb_on_chord_header`, or remove the `on_error`/`link_error` call, but either will result in the error handler being invoked.

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->

Either formulation described should work. The second formulation should not be throwing `AttributeError`.

The intended result is:
* `test_celery.nothing_work_unit` succeeds.
* One of the `test_celery.fake_work_unit` tasks runs and raises an exception, causing `fake_error_handler` to be called.

# Actual Behavior

When trying to run the second formulation, I get an `AttributeError` and no task is ever run:

```
>>> from test_celery import test
>>> test()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/asteinborn/Ramp/misc/celery-test/test_celery.py", line 68, in test
    ).on_error(f2).apply_async()
  File "/Users/asteinborn/Ramp/misc/celery-test/venv/lib/python3.10/site-packages/celery/canvas.py", line 1035, in apply_async
    return self.run(args, kwargs, app=app, **(
  File "/Users/asteinborn/Ramp/misc/celery-test/venv/lib/python3.10/site-packages/celery/canvas.py", line 1060, in run
    tasks, results_from_prepare = self.prepare_steps(
  File "/Users/asteinborn/Ramp/misc/celery-test/venv/lib/python3.10/site-packages/celery/canvas.py", line 1246, in prepare_steps
    task.link_error(errback)
  File "/Users/asteinborn/Ramp/misc/celery-test/venv/lib/python3.10/site-packages/celery/canvas.py", line 2276, in link_error
    task.link_error(errback.clone(immutable=True))
AttributeError: 'str' object has no attribute 'link_error'
```


Discussion:
Thank you for the informed report!
I'll check it out in the next few days.

Resolving patch:
diff --git a/celery/canvas.py b/celery/canvas.py
--- a/celery/canvas.py
+++ b/celery/canvas.py
@@ -653,7 +653,7 @@ def stamp_links(self, visitor, append_stamps=False, **headers):
 
         # Stamp all of the callbacks of this signature
         headers = deepcopy(non_visitor_headers)
-        for link in self.options.get('link', []) or []:
+        for link in maybe_list(self.options.get('link')) or []:
             link = maybe_signature(link, app=self.app)
             visitor_headers = None
             if visitor is not None:
@@ -668,7 +668,7 @@ def stamp_links(self, visitor, append_stamps=False, **headers):
 
         # Stamp all of the errbacks of this signature
         headers = deepcopy(non_visitor_headers)
-        for link in self.options.get('link_error', []) or []:
+        for link in maybe_list(self.options.get('link_error')) or []:
             link = maybe_signature(link, app=self.app)
             visitor_headers = None
             if visitor is not None:
@@ -1016,9 +1016,9 @@ def unchain_tasks(self):
         # Clone chain's tasks assigning signatures from link_error
         # to each task and adding the chain's links to the last task.
         tasks = [t.clone() for t in self.tasks]
-        for sig in self.options.get('link', []):
+        for sig in maybe_list(self.options.get('link')) or []:
             tasks[-1].link(sig)
-        for sig in self.options.get('link_error', []):
+        for sig in maybe_list(self.options.get('link_error')) or []:
             for task in tasks:
                 task.link_error(sig)
         return tasks
@@ -2272,7 +2272,7 @@ def link_error(self, errback):
             applied to the body.
         """
         if self.app.conf.task_allow_error_cb_on_chord_header:
-            for task in self.tasks:
+            for task in maybe_list(self.tasks) or []:
                 task.link_error(errback.clone(immutable=True))
         else:
             # Once this warning is removed, the whole method needs to be refactored to:


Test patch:
diff --git a/t/integration/test_canvas.py b/t/integration/test_canvas.py
--- a/t/integration/test_canvas.py
+++ b/t/integration/test_canvas.py
@@ -2962,6 +2962,43 @@ def test_flag_allow_error_cb_on_chord_header_on_upgraded_chord(self, manager, su
             # Cleanup
             redis_connection.delete(errback_key)
 
+    def test_upgraded_chord_link_error_with_header_errback_enabled(self, manager, subtests):
+        try:
+            manager.app.backend.ensure_chords_allowed()
+        except NotImplementedError as e:
+            raise pytest.skip(e.args[0])
+
+        if not manager.app.conf.result_backend.startswith('redis'):
+            raise pytest.skip('Requires redis result backend.')
+        redis_connection = get_redis_connection()
+
+        manager.app.conf.task_allow_error_cb_on_chord_header = True
+
+        body_msg = 'chord body called'
+        body_key = 'echo_body'
+        body_sig = redis_echo.si(body_msg, redis_key=body_key)
+
+        errback_msg = 'errback called'
+        errback_key = 'echo_errback'
+        errback_sig = redis_echo.si(errback_msg, redis_key=errback_key)
+
+        redis_connection.delete(errback_key, body_key)
+
+        sig = chain(
+            identity.si(42),
+            group(
+                fail.si(),
+                fail.si(),
+            ),
+            body_sig,
+        ).on_error(errback_sig)
+
+        with subtests.test(msg='Error propagates from failure in header'):
+            with pytest.raises(ExpectedException):
+                sig.apply_async().get(timeout=TIMEOUT)
+
+        redis_connection.delete(errback_key, body_key)
+
 
 class test_signature_serialization:
     """
@@ -3441,3 +3478,27 @@ def on_signature(self, sig, **headers) -> dict:
         res = stamped_task.delay()
         res.get(timeout=TIMEOUT)
         assert assertion_result
+
+    def test_stamp_canvas_with_dictionary_link(self, manager, subtests):
+        class CustomStampingVisitor(StampingVisitor):
+            def on_signature(self, sig, **headers) -> dict:
+                return {"on_signature": 42}
+
+        with subtests.test("Stamp canvas with dictionary link"):
+            canvas = identity.si(42)
+            canvas.options["link"] = dict(identity.si(42))
+            canvas.stamp(visitor=CustomStampingVisitor())
+
+    def test_stamp_canvas_with_dictionary_link_error(self, manager, subtests):
+        class CustomStampingVisitor(StampingVisitor):
+            def on_signature(self, sig, **headers) -> dict:
+                return {"on_signature": 42}
+
+        with subtests.test("Stamp canvas with dictionary link error"):
+            canvas = fail.si()
+            canvas.options["link_error"] = dict(fail.si())
+            canvas.stamp(visitor=CustomStampingVisitor())
+
+        with subtests.test(msg='Expect canvas to fail'):
+            with pytest.raises(ExpectedException):
+                canvas.apply_async().get(timeout=TIMEOUT)