Repository: celery/celery

Issue:
Using a class based task as errback results in AttributeError '_header_'
## Checklist

- [X] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
[celery-report.txt](https://github.com/celery/celery/files/1456808/celery-report.txt)
- [ ] I have verified that the issue exists against the `master` branch of Celery.

## Steps to reproduce
Create a tasks file:
```
from celery import Celery,Task
celery = Celery('tasks', broker='redis://localhost:6379/6',backend='redis://localhost:6379/6')

@celery.task
def ErrTask1(req):
    print('failure')

class CustomTask(Task):
    def run(self,var):
        print('running %s' % var)
        if var == "err":
            raise Exception("err")
CustomTask = celery.register_task(CustomTask())

class ErrTask2(Task):
        def run(self):
                    print('failure')
ErrTask2 = celery.register_task(ErrTask2())

```
start celery with it:
`celery worker -A tasks:celery -l info`

Start a python shell and start the CustomTask with an error callback:

```
from tasks import CustomTask,ErrTask1,ErrTask2
a = CustomTask.apply_async(['err'],link_error=ErrTask1.s())
a.result
b = CustomTask.apply_async(['err'],link_error=ErrTask2.s())
b.result
```
## Expected behavior
When running with ErrTask1 as error callback ( the one with the decorator) we get the expected result:
```
>>> a = CustomTask.apply_async(['err'],link_error=ErrTask1.s())
>>> a.result
Exception(u'err',)
```
## Actual behavior
When running with ErrTask2 as error callback ( the one that is defined classbased) we get the Attributeerror:
```
>>> b = CustomTask.apply_async(['err'],link_error=ErrTask2.s())
>>> b.result
AttributeError(u"'ErrTask2' object has no attribute '__header__'",)
```
It seems the changed behaviour on the error callback in celery 4 has something to do with it. The class based tasks miss some attributes which doesn't seem to be an issue for normal execution but only for the error callback. Probably there is a difference in initialization between `celery.register_task()` and the decorator.

Workaround is to define the error callbacks with the task decorator. 
Using a class based task as errback results in AttributeError '_header_'
## Checklist

- [X] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
[celery-report.txt](https://github.com/celery/celery/files/1456808/celery-report.txt)
- [ ] I have verified that the issue exists against the `master` branch of Celery.

## Steps to reproduce
Create a tasks file:
```
from celery import Celery,Task
celery = Celery('tasks', broker='redis://localhost:6379/6',backend='redis://localhost:6379/6')

@celery.task
def ErrTask1(req):
    print('failure')

class CustomTask(Task):
    def run(self,var):
        print('running %s' % var)
        if var == "err":
            raise Exception("err")
CustomTask = celery.register_task(CustomTask())

class ErrTask2(Task):
        def run(self):
                    print('failure')
ErrTask2 = celery.register_task(ErrTask2())

```
start celery with it:
`celery worker -A tasks:celery -l info`

Start a python shell and start the CustomTask with an error callback:

```
from tasks import CustomTask,ErrTask1,ErrTask2
a = CustomTask.apply_async(['err'],link_error=ErrTask1.s())
a.result
b = CustomTask.apply_async(['err'],link_error=ErrTask2.s())
b.result
```
## Expected behavior
When running with ErrTask1 as error callback ( the one with the decorator) we get the expected result:
```
>>> a = CustomTask.apply_async(['err'],link_error=ErrTask1.s())
>>> a.result
Exception(u'err',)
```
## Actual behavior
When running with ErrTask2 as error callback ( the one that is defined classbased) we get the Attributeerror:
```
>>> b = CustomTask.apply_async(['err'],link_error=ErrTask2.s())
>>> b.result
AttributeError(u"'ErrTask2' object has no attribute '__header__'",)
```
It seems the changed behaviour on the error callback in celery 4 has something to do with it. The class based tasks miss some attributes which doesn't seem to be an issue for normal execution but only for the error callback. Probably there is a difference in initialization between `celery.register_task()` and the decorator.

Workaround is to define the error callbacks with the task decorator. 


Discussion:
@AlexHill @georgepsarakis could you check
I see that this issue #3723 is related too.

Edit: Somehow related #4022 too.
Hi,

is there any resolution to this issue? We are facing it on 4.1.0 (and 4.1.1). I see that the task is closed but couldn't find any code related to it.
@livenson this should be on master 4.2rc versions as you can see the milestone 4.2 tag
https://github.com/celery/celery/pull/4545
@auvipy thanks for reply, but I still don't get it –– #4545 adds additional handling but still expects __header__ to be there, which is not happening for class-based Tasks, at least I could only find a single place where it is set in Celery - https://github.com/celery/celery/blob/2636251a1249634258c1910c6400bf2ccf28b8ed/celery/app/base.py#L445 . I.e. only in function decorator.
reopening, could you plz dig bit more and let us know? could you try master?
for a fix, you can use this solution https://github.com/opennode/waldur-core/blob/develop/waldur_core/core/tasks.py#L31
anyone up for a PR? with @maximprokopenko proposed solution?
PR could be made if you think it makes sense -- basically, our problem in Waldur was that we are  still using "legacy" base Task class with self-registration. So the question is if it's a correct workaround to apply -- or perhaps it should rather be fixed in self-registration of legacy class. @auvipy , any feelings about it?   
which version of celery are you using? is you code open sourced?
4.1.0 (tried with 4.1.1 -- but it's the same for that part of the code).

Yes. @maximprokopenko actually linked to our project.
@thedrow @georgepsarakis whats yours thoughts on this guys?
@ask Could you please validate proposed fix?
@auvipy @livenson @stevenwbe just tested the given example with 4.2.0rc3+ and it seems to work, as it has the expected behavior, both the error callback is executed on the worker and the result property contains `Exception('err')` . Can you please cross-check again?
The issue is back in 4.2.1.
@sbj-ss If you claim we have a regression, we'll need you to provide a test case in order for us to verify it.
@thedrow you can use the test case from the 1st post.
@thedrow @sbj-ss ,
It is indeed correct that it occurs again. Output of the above testcase:

(venv2)$ python
Python 2.7.5 (default, May 31 2018, 09:41:32)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from tasks import CustomTask,ErrTask1,ErrTask2
>>> a = CustomTask.apply_async(['err'],link_error=ErrTask1.s())
>>> a.result
Exception(u'err',)
>>> b = CustomTask.apply_async(['err'],link_error=ErrTask2.s())
>>> b.result
AttributeError(u"'ErrTask2' object has no attribute '__header__'",)
>>>
(venv2) $ pip freeze | grep celery
celery==4.2.1
celery-redis-sentinel==0.3.0

@AlexHill @georgepsarakis could you check
I see that this issue #3723 is related too.

Edit: Somehow related #4022 too.
Hi,

is there any resolution to this issue? We are facing it on 4.1.0 (and 4.1.1). I see that the task is closed but couldn't find any code related to it.
@livenson this should be on master 4.2rc versions as you can see the milestone 4.2 tag
https://github.com/celery/celery/pull/4545
@auvipy thanks for reply, but I still don't get it –– #4545 adds additional handling but still expects __header__ to be there, which is not happening for class-based Tasks, at least I could only find a single place where it is set in Celery - https://github.com/celery/celery/blob/2636251a1249634258c1910c6400bf2ccf28b8ed/celery/app/base.py#L445 . I.e. only in function decorator.
reopening, could you plz dig bit more and let us know? could you try master?
for a fix, you can use this solution https://github.com/opennode/waldur-core/blob/develop/waldur_core/core/tasks.py#L31
anyone up for a PR? with @maximprokopenko proposed solution?
PR could be made if you think it makes sense -- basically, our problem in Waldur was that we are  still using "legacy" base Task class with self-registration. So the question is if it's a correct workaround to apply -- or perhaps it should rather be fixed in self-registration of legacy class. @auvipy , any feelings about it?   
which version of celery are you using? is you code open sourced?
4.1.0 (tried with 4.1.1 -- but it's the same for that part of the code).

Yes. @maximprokopenko actually linked to our project.
@thedrow @georgepsarakis whats yours thoughts on this guys?
@ask Could you please validate proposed fix?
@auvipy @livenson @stevenwbe just tested the given example with 4.2.0rc3+ and it seems to work, as it has the expected behavior, both the error callback is executed on the worker and the result property contains `Exception('err')` . Can you please cross-check again?
The issue is back in 4.2.1.
@sbj-ss If you claim we have a regression, we'll need you to provide a test case in order for us to verify it.
@thedrow you can use the test case from the 1st post.
@thedrow @sbj-ss ,
It is indeed correct that it occurs again. Output of the above testcase:

(venv2)$ python
Python 2.7.5 (default, May 31 2018, 09:41:32)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from tasks import CustomTask,ErrTask1,ErrTask2
>>> a = CustomTask.apply_async(['err'],link_error=ErrTask1.s())
>>> a.result
Exception(u'err',)
>>> b = CustomTask.apply_async(['err'],link_error=ErrTask2.s())
>>> b.result
AttributeError(u"'ErrTask2' object has no attribute '__header__'",)
>>>
(venv2) $ pip freeze | grep celery
celery==4.2.1
celery-redis-sentinel==0.3.0


Resolving patch:
diff --git a/celery/backends/base.py b/celery/backends/base.py
--- a/celery/backends/base.py
+++ b/celery/backends/base.py
@@ -170,6 +170,13 @@ def _call_task_errbacks(self, request, exc, traceback):
         for errback in request.errbacks:
             errback = self.app.signature(errback)
             if (
+                # Celery tasks type created with the @task decorator have the
+                # __header__ property, but Celery task created from Task
+                # class do not have this property.
+                # That's why we have to check if this property exists before
+                # checking is it partial function.
+                hasattr(errback.type, '__header__') and
+
                 # workaround to support tasks with bind=True executed as
                 # link errors. Otherwise retries can't be used
                 not isinstance(errback.type.__header__, partial) and


Test patch:
diff --git a/t/unit/backends/test_base.py b/t/unit/backends/test_base.py
--- a/t/unit/backends/test_base.py
+++ b/t/unit/backends/test_base.py
@@ -8,7 +8,7 @@
 from case import ANY, Mock, call, patch, skip
 
 from celery import chord, group, states, uuid
-from celery.app.task import Context
+from celery.app.task import Context, Task
 from celery.backends.base import (BaseBackend, DisabledBackend,
                                   KeyValueStoreBackend, _nulldict)
 from celery.exceptions import ChordError, TimeoutError
@@ -383,6 +383,23 @@ def test_mark_as_failure__errback(self):
         b.mark_as_failure('id', exc, request=request)
         assert self.errback.last_result == 5
 
+    @patch('celery.backends.base.group')
+    def test_class_based_task_can_be_used_as_error_callback(self, mock_group):
+        b = BaseBackend(app=self.app)
+        b._store_result = Mock()
+
+        class TaskBasedClass(Task):
+            def run(self):
+                pass
+
+        TaskBasedClass = self.app.register_task(TaskBasedClass())
+
+        request = Mock(name='request')
+        request.errbacks = [TaskBasedClass.subtask(args=[], immutable=True)]
+        exc = KeyError()
+        b.mark_as_failure('id', exc, request=request)
+        mock_group.assert_called_once_with(request.errbacks, app=self.app)
+
     def test_mark_as_failure__chord(self):
         b = BaseBackend(app=self.app)
         b._store_result = Mock()