Repository: celery/celery

Issue:
Bound tasks as link error raises TypeError exception (celery 4.0.2, 4.2.0)
## Checklist

- [x] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.


```
software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:2.7.6
            billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:db+mysql://celery:**@127.0.0.1/celery

CELERY_ACKS_LATE: True
BROKER_URL: u'amqp://celery:********@127.0.0.1:****/celery'
CELERY_RESULT_ENGINE_OPTIONS: {
    'isolation_level': 'READ_COMMITTED'}
CELERY_RESULT_SERIALIZER: 'json'
CELERY_ACCEPT_CONTENT: ['json']
CELERYD_MAX_TASKS_PER_CHILD: 100
CELERYD_PREFETCH_MULTIPLIER: 1
CELERY_REDIRECT_STDOUTS_LEVEL: 'INFO'
CELERY_TRACK_STARTED: True
CELERYD_TASK_SOFT_TIME_LIMIT: 900
CELERYD_TASK_TIME_LIMIT: 910
CELERY_TASK_SERIALIZER: 'json'
CELERY_RESULT_BACKEND: u'db+mysql://celery:********@127.0.0.1/celery'
CELERY_SEND_TASK_ERROR_EMAILS: True
```

## Steps to reproduce
Try to apply_async (or async, it doesn't matter) any task with link_error set to bound task. Till now I was using celery 3.1.18 and bound tasks were working fine as link errors.

See example below:

```python
@app.task(name="raise_exception", bind=True)
def raise_exception(self):
    raise Exception("Bad things happened")


@app.task(name="handle_task_exception", bind=True)
def handle_task_exception(self):
    print("Exception detected")
```

```python
subtask = raise_exception.subtask()

subtask.apply_async(link_error=handle_task_exception.s())
```

## Expected behavior
Bound tasks can be used as link errors. 

## Actual behavior
Worker fails with following exception
```
[2016-12-28 07:36:12,692: INFO/MainProcess] Received task: raise_exception[b10b8451-3f4d-4cf0-b4b0-f964105cf849]   
[2016-12-28 07:36:12,847: INFO/PoolWorker-5] /usr/local/lib/python2.7/dist-packages/celery/app/trace.py:542: RuntimeWarning: Exception raised outside body: TypeError('<functools.partial object at 0x7f74848dffc8> is not a Python function',): 
Traceback (most recent call last): 
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 381, in trace_task 
    I, R, state, retval = on_error(task_request, exc, uuid) 
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 323, in on_error 
    task, request, eager=eager, call_errbacks=call_errbacks, 
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 157, in handle_error_state 
    call_errbacks=call_errbacks) 
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 202, in handle_failure 
    call_errbacks=call_errbacks, 
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 168, in mark_as_failure 
    self._call_task_errbacks(request, exc, traceback) 
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 174, in _call_task_errbacks 
    if arity_greater(errback.type.__header__, 1): 
  File "/usr/local/lib/python2.7/dist-packages/celery/utils/functional.py", line 292, in arity_greater 
    argspec = getfullargspec(fun) 
  File "/usr/local/lib/python2.7/dist-packages/vine/five.py", line 350, in getfullargspec 
    s = _getargspec(fun) 
  File "/usr/lib/python2.7/inspect.py", line 816, in getargspec 
    raise TypeError('{!r} is not a Python function'.format(func)) 
TypeError: <functools.partial object at 0x7f74848dffc8> is not a Python function 

  exc, exc_info.traceback))) 
[2016-12-28 07:36:12,922: ERROR/MainProcess] Pool callback raised exception: TypeError('<functools.partial object at 0x7f74848dffc8> is not a Python function',) 
Traceback (most recent call last): 
  File "/usr/local/lib/python2.7/dist-packages/billiard/pool.py", line 1748, in safe_apply_callback 
    fun(*args, **kwargs) 
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/request.py", line 366, in on_failure 
    self.id, exc, request=self, store_result=self.store_errors, 
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 168, in mark_as_failure 
    self._call_task_errbacks(request, exc, traceback) 
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 174, in _call_task_errbacks 
    if arity_greater(errback.type.__header__, 1): 
  File "/usr/local/lib/python2.7/dist-packages/celery/utils/functional.py", line 292, in arity_greater 
    argspec = getfullargspec(fun) 
  File "/usr/local/lib/python2.7/dist-packages/vine/five.py", line 350, in getfullargspec 
    s = _getargspec(fun) 
  File "/usr/lib/python2.7/inspect.py", line 816, in getargspec 
    raise TypeError('{!r} is not a Python function'.format(func)) 
TypeError: <functools.partial object at 0x7f74848dffc8> is not a Python function 
```


Discussion:
Also encountering this issue when using a celery task with `on_error` of a signature object, just adding `bind=True` causes the error to occur.
```
software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:2.7.13
            billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled
```
any workaround? got the same problem..
maybe different approach to fail a task instead of raising exception?
I think celery code should be patched.
As far as I remember there is an if statement before launching link error handle function.
The if checks whether handler is a function and it fails when function is "partial".

Handler should be launched if it is a function or partial.
I decided to omit the `bind=True` and ref my `base` task class via the task name.. 
Hi, folks! I've also encountered this issue on Mac OS X 10.11.6 (El Capitan).
I have bound tasks also, they are linked together via task signature special method -
```
fail_job_si = fail_job.si(*args)
run_job_si = run_job.si(*args)
run_job_si.link_error(fail_job_si)
```
Here is my configuration:
```
software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:2.7.13
            billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Darwin arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:rpc:///
```
So I did some digging lately... It turns out that when vine/five.py is patched to handle partials

```python
try:  # pragma: no cover
    from inspect import formatargspec, getfullargspec
except ImportError:  # Py2
    from collections import namedtuple
    from inspect import formatargspec, getargspec as _getargspec  # noqa

    FullArgSpec = namedtuple('FullArgSpec', (
        'args', 'varargs', 'varkw', 'defaults',
        'kwonlyargs', 'kwonlydefaults', 'annotations',
    ))

    def getfullargspec(fun, _fill=(None, ) * 3):  # noqa
        """For compatibility with Python 3."""
        # HERE GOES THE AWFUL IF
        if type(fun) is partial:
            s = _getargspec(fun.func)
        else:
            s = _getargspec(fun)
        return FullArgSpec(*s + _fill)                                  
```
Bound link errors are executed properly, although when I try to do self.retry inside link error I get

```
/env/local/lib/python2.7/site-packages/celery/app/trace.py:549: RuntimeWarning: Exception raised outside body: Retry(Retry(...), None, None): 
Traceback (most recent call last): 
  File "/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 388, in trace_task 
    I, R, state, retval = on_error(task_request, exc, uuid) 
  File "/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 330, in on_error  
    task, request, eager=eager, call_errbacks=call_errbacks, 
  File "/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 164, in handle_error_state 
    call_errbacks=call_errbacks) 
  File "/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 209, in handle_failure 
    call_errbacks=call_errbacks, 
  File "/env/local/lib/python2.7/site-packages/celery/backends/base.py", line 168, in mark_as_failure 
    self._call_task_errbacks(request, exc, traceback) 
  File "/env/local/lib/python2.7/site-packages/celery/backends/base.py", line 175, in _call_task_errbacks 
    errback(request, exc, traceback) 
  File "/env/local/lib/python2.7/site-packages/celery/canvas.py", line 178, in __call__  
    return self.type(*args, **kwargs) 
  File "/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 630, in __protected_call__ 
    return orig(self, *args, **kwargs) 
  File "/env/local/lib/python2.7/site-packages/celery/app/task.py", line 380, in __call__  
    return self.run(*args, **kwargs) 
  File "/lib/tasks.py", line 1225, in handle_task_exception 
    return self.retry(countdown=1, max_retries=10, throw=False) 
  File "/env/local/lib/python2.7/site-packages/celery/app/task.py", line 653, in retry 
    raise_with_context(exc or Retry('Task can be retried', None)) 
  File "/env/local/lib/python2.7/site-packages/celery/utils/serialization.py", line 276, in raise_with_context 
    reraise(type(exc), exc, exc_info[2] 
  File "/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 374, in trace_task 
    R = retval = fun(*args, **kwargs) 
  File "/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 629, in __protected_call__ 
    return self.run(*args, **kwargs) 
  File "/lib/tasks.py", line 1218, in raise_exception 
    raise Exception("My custom exception") 
Retry: Task can be retried
```

Perhaps @ask could give us some hint? :)
Getting exact same error using celery 4.10 using on_error to a bound task.

We were experiencing the same thing. Our workaround is here: https://github.com/WikiWatershed/model-my-watershed/pull/2287/commits/0871277f665b301ad42b5309e74fd70ddde70e69

Previously we would use the `self.app` reference to get the `AsyncResult` from the given `uuid` and get the exception and traceback through that. But now with Celery 4 it turns out we didn't need the error handler to be a bound task anymore, because it now gets `request`, `exc`, and `traceback` as arguments. See the documentation here: http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#canvas-refactor and https://github.com/celery/celery/pull/2538#issuecomment-227865509.

I think if you _still_ need access to `self.app` within the task, you could just import `app` from wherever it is defined in the file, and use that.
I've just tested against master, unfortunately the bug still exists.

```
software -> celery:4.2.0 (latentcall) kombu:4.1.0 py:2.7.6
            billiard:3.5.0.3 py-amqp:2.2.2
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:db+mysql://celery:**@127.0.0.1/celery

task_serializer: 'json'
broker_heartbeat: 360
result_serializer: 'json'
task_time_limit: 910
event_queue_ttl: 60
worker_max_tasks_per_child: 100
worker_prefetch_multiplier: 1
task_acks_late: True
task_soft_time_limit: 900
task_track_started: True
```

The result is

```
[2018-01-31 11:40:20,556: INFO/MainProcess] Received task: raise_exception[21dfd0c5-1aa1-4e18-9326-b7f8326c441b]  
[2018-01-31 11:40:20,719: INFO/ForkPoolWorker-1] /localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/app/trace.py:561: RuntimeWarning: Exception raised outside body: TypeError('<functools.partial object at 0x7ff86fe95c58> is not a Python function',):
Traceback (most recent call last):
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 396, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 338, in on_error
    task, request, eager=eager, call_errbacks=call_errbacks,
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 172, in handle_error_state
    call_errbacks=call_errbacks)
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 217, in handle_failure
    call_errbacks=call_errbacks,
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/backends/base.py", line 160, in mark_as_failure
    self._call_task_errbacks(request, exc, traceback)
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/backends/base.py", line 166, in _call_task_errbacks
    if arity_greater(errback.type.__header__, 1):
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/utils/functional.py", line 293, in arity_greater
    argspec = getfullargspec(fun)
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/vine/five.py", line 357, in getfullargspec
    s = _getargspec(fun)
  File "/usr/lib/python2.7/inspect.py", line 816, in getargspec
    raise TypeError('{!r} is not a Python function'.format(func))
TypeError: <functools.partial object at 0x7ff86fe95c58> is not a Python function

  exc, exc_info.traceback)))
[2018-01-31 11:40:20,771: ERROR/MainProcess] Pool callback raised exception: TypeError('<functools.partial object at 0x7ff86fe95c58> is not a Python function',)
Traceback (most recent call last):
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/billiard/pool.py", line 1747, in safe_apply_callback
    fun(*args, **kwargs)
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/worker/request.py", line 366, in on_failure
    self.id, exc, request=self, store_result=self.store_errors,
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/backends/base.py", line 160, in mark_as_failure
    self._call_task_errbacks(request, exc, traceback)
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/backends/base.py", line 166, in _call_task_errbacks
    if arity_greater(errback.type.__header__, 1):
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/celery/utils/functional.py", line 293, in arity_greater
    argspec = getfullargspec(fun)
  File "/localscripts/celery-test/venv/local/lib/python2.7/site-packages/vine/five.py", line 357, in getfullargspec
    s = _getargspec(fun)
  File "/usr/lib/python2.7/inspect.py", line 816, in getargspec
    raise TypeError('{!r} is not a Python function'.format(func))
TypeError: <functools.partial object at 0x7ff86fe95c58> is not a Python function

```

@auvipy could you reopen this task?
could you plz also try upgrading the dependencies to check? like kombu etc
Tested with kombu and billiard versions build from master
```
Processing ./kombu-4.1.0-py2.py3-none-any.whl
Requirement already satisfied: amqp<3.0,>=2.1.4 in /localscripts/celery-test/venv/lib/python2.7/site-packages (from kombu==4.1.0)
Requirement already satisfied: vine>=1.1.3 in /localscripts/celery-test/venv/lib/python2.7/site-packages (from amqp<3.0,>=2.1.4->kombu==4.1.0)
Installing collected packages: kombu
Successfully installed kombu-4.1.0
```

```
Processing ./billiard-3.5.0.3-cp27-none-linux_x86_64.whl
Installing collected packages: billiard
Successfully installed billiard-3.5.0.3
```

Bug still exists.
Hmm I just noticed that I didn't test with vine 1.1.4 while the issue seems to be related to it!
I'll check tomorrow morning.
Nope. False hope.
Tested with vine 1.1.4 too, got the same error.
I think I got it...

```diff
diff --git a/celery/backends/base.py b/celery/backends/base.py
index fc49105..3725452 100644
--- a/celery/backends/base.py
+++ b/celery/backends/base.py
@@ -12,6 +12,7 @@ import sys
 import time
 from collections import namedtuple
 from datetime import timedelta
+from functools import partial
 from weakref import WeakValueDictionary
 
 from billiard.einfo import ExceptionInfo
@@ -163,7 +164,10 @@ class Backend(object):
         old_signature = []
         for errback in request.errbacks:
             errback = self.app.signature(errback)
-            if arity_greater(errback.type.__header__, 1):
+            if (
+                type(errback.type.__header__) is not partial
+                and arity_greater(errback.type.__header__, 1)
+            ):
                 errback(request, exc, traceback)
             else:
                 old_signature.append(errback)
```

This patch fixes my problem although it looks like a workaround and I don't know how to fix it properly...

@auvipy any ideas?
Please use isinstance instead of type() is ...
So just to sum up, the problem was introduced with following commit:
https://github.com/celery/celery/commit/ae3f36f7dadde455098963ecdada12b043fdfe41 (see also https://github.com/celery/celery/pull/2538#issuecomment-227865509)

Unfortunately in my case I need a bound task as link error because I want to use retry mechanism during error handling - in distributed systems it's common that different components are not 100% available.

Is there any straightforward way to use retries machinery without bound tasks?

Resolving patch:
diff --git a/celery/backends/base.py b/celery/backends/base.py
--- a/celery/backends/base.py
+++ b/celery/backends/base.py
@@ -12,6 +12,7 @@
 import time
 from collections import namedtuple
 from datetime import timedelta
+from functools import partial
 from weakref import WeakValueDictionary
 
 from billiard.einfo import ExceptionInfo
@@ -163,7 +164,12 @@ def _call_task_errbacks(self, request, exc, traceback):
         old_signature = []
         for errback in request.errbacks:
             errback = self.app.signature(errback)
-            if arity_greater(errback.type.__header__, 1):
+            if (
+                # workaround to support tasks with bind=True executed as
+                # link errors. Otherwise retries can't be used
+                not isinstance(errback.type.__header__, partial) and
+                arity_greater(errback.type.__header__, 1)
+            ):
                 errback(request, exc, traceback)
             else:
                 old_signature.append(errback)


Test patch:
diff --git a/t/unit/backends/test_base.py b/t/unit/backends/test_base.py
--- a/t/unit/backends/test_base.py
+++ b/t/unit/backends/test_base.py
@@ -218,6 +218,17 @@ class test_BaseBackend_dict:
     def setup(self):
         self.b = DictBackend(app=self.app)
 
+        @self.app.task(shared=False, bind=True)
+        def bound_errback(self, result):
+            pass
+
+        @self.app.task(shared=False)
+        def errback(arg1, arg2):
+            errback.last_result = arg1 + arg2
+
+        self.bound_errback = bound_errback
+        self.errback = errback
+
     def test_delete_group(self):
         self.b.delete_group('can-delete')
         assert 'can-delete' not in self.b._data
@@ -303,6 +314,28 @@ def test_mark_as_done__chord(self):
         b.mark_as_done('id', 10, request=request)
         b.on_chord_part_return.assert_called_with(request, states.SUCCESS, 10)
 
+    def test_mark_as_failure__bound_errback(self):
+        b = BaseBackend(app=self.app)
+        b._store_result = Mock()
+        request = Mock(name='request')
+        request.errbacks = [
+            self.bound_errback.subtask(args=[1], immutable=True)]
+        exc = KeyError()
+        group = self.patching('celery.backends.base.group')
+        b.mark_as_failure('id', exc, request=request)
+        group.assert_called_with(request.errbacks, app=self.app)
+        group.return_value.apply_async.assert_called_with(
+            (request.id, ), parent_id=request.id, root_id=request.root_id)
+
+    def test_mark_as_failure__errback(self):
+        b = BaseBackend(app=self.app)
+        b._store_result = Mock()
+        request = Mock(name='request')
+        request.errbacks = [self.errback.subtask(args=[2, 3], immutable=True)]
+        exc = KeyError()
+        b.mark_as_failure('id', exc, request=request)
+        assert self.errback.last_result == 5
+
     def test_mark_as_failure__chord(self):
         b = BaseBackend(app=self.app)
         b._store_result = Mock()