Repository: celery/celery

Issue:
SQLAlchemy's ProgrammingError exception causes a traceback when AsyncResult.ready() is called
## Checklist

- [ ] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
- [ ] I have verified that the issue exists against the `master` branch of Celery.

Checked with 4.2.1

## Steps to reproduce

Simple reproducer (please note I used dummy values but I hit this in production with real world values):

**tasks.py**
```python
from celery import Celery

from sqlalchemy.exc import ProgrammingError

celery = Celery(...)

@celery.task(name='fail')
def fail(param):
    raise ProgrammingError(Exception(), 'params', Exception('exception message'))
```

**test.py**
```python
import time
from celery import Celery

import sqlalchemy   # this import must be here, otherwise it behaves differently and works

celery = Celery(...)

if __name__ == '__main__':
    time.sleep(5)

    for dom in ('example.com', 'example2.com'):
        result = celery.send_task('fail', args=(dom,))
        time.sleep(10)
        if result.ready() and result.successful():  # .ready() fails
            print(result.get())
```

**Result**
```
celery_test_tasks_1  | [2018-09-18 13:16:45,295: ERROR/ForkPoolWorker-1] Task fail[610284d4-e7dc-47a0-9ab0-efcec8609599] raised unexpected: ProgrammingError('(builtins.Exception) exception message',)
celery_test_tasks_1  | Traceback (most recent call last):
celery_test_tasks_1  |   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 382, in trace_task
celery_test_tasks_1  |     R = retval = fun(*args, **kwargs)
celery_test_tasks_1  |   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 641, in __protected_call__
celery_test_tasks_1  |     return self.run(*args, **kwargs)
celery_test_tasks_1  |   File "/home/user/tasks.py", line 18, in fail
celery_test_tasks_1  |     raise ProgrammingError(Exception(), 'params', Exception('exception message'))
celery_test_tasks_1  | sqlalchemy.exc.ProgrammingError: (builtins.Exception) exception message [SQL: Exception()] [parameters: 'params'] (Background on this error at: http://sqlalche.me/e/f405)
```
```
celery_run_test_1    | Traceback (most recent call last):
celery_run_test_1    |   File "test.py", line 22, in <module>
celery_run_test_1    |     if result.ready():
celery_run_test_1    |   File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 311, in ready
celery_run_test_1    |     return self.state in self.backend.READY_STATES
celery_run_test_1    |   File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 471, in state
celery_run_test_1    |     return self._get_task_meta()['status']
celery_run_test_1    |   File "/usr/local/lib/python3.6/site-packages/celery/result.py", line 410, in _get_task_meta
celery_run_test_1    |     return self._maybe_set_cache(self.backend.get_task_meta(self.id))
celery_run_test_1    |   File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 365, in get_task_meta
celery_run_test_1    |     meta = self._get_task_meta_for(task_id)
celery_run_test_1    |   File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 680, in _get_task_meta_for
celery_run_test_1    |     return self.decode_result(meta)
celery_run_test_1    |   File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 284, in decode_result
celery_run_test_1    |     return self.meta_from_decoded(self.decode(payload))
celery_run_test_1    |   File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 280, in meta_from_decoded
celery_run_test_1    |     meta['result'] = self.exception_to_python(meta['result'])
celery_run_test_1    |   File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 260, in exception_to_python
celery_run_test_1    |     exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
celery_run_test_1    | TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig'
```

## Expected behavior

Do not fail when the received exception cannot be reconstructed. Use a dummy exception.

## Actual behavior

Fails because repr() of sqlalchemy's `ProgrammingError` cannot be used to reconstruct exception.
```python
e = ProgrammingError(Exception(), 'params', Exception('exception message'))
>>> repr(e)
"ProgrammingError('(builtins.Exception) exception message',)"
>>> ProgrammingError('(builtins.Exception) exception message',)
TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig'
````


Discussion:
Was there ever a fix for this? I am running into this problem.
I'm hitting this too. Is there a workaround we can use while we wait for a fix?
@rclecha @oskarsinger can you try the following:

```python
return_value = result.get(propagate=False)
if result.failed():
  ...
```
So I wonder if https://github.com/celery/celery/commit/33713dbf69cbd05b59a55077c137d256d652524b is fixing this? #4860
Nah, that doesn't do the trick.
So to summarize:

- [Some SQLAlchemy exceptions](https://github.com/zzzeek/sqlalchemy/blob/dc48ac54893491c5ecd64a868883a22769376e9a/lib/sqlalchemy/exc.py#L462-L478) can't be instantiated again by simply using the `args` attribute but require the original exception that it wraps and some other arguments. In fact the args attribute is set with a human-readable representation of the passed arguments.

- When Celery is configured to use the JSON serializer the exceptions are [prepared to be stored as JSON](https://github.com/celery/celery/blob/e257646136e6fae73186d7385317f4e20cd36130/celery/backends/base.py#L244-L251) which means using the [`args` attribute](https://github.com/zzzeek/sqlalchemy/blob/dc48ac54893491c5ecd64a868883a22769376e9a/lib/sqlalchemy/exc.py#L471-L472) of the exception instance, which is a human readable representation of the exception but doesn't contain the arguments needed for deserialization.
- When a task is throwing an SQLAlchemy exception, it's successfully serialized for storage but misses the [required arguments for deserialization](https://github.com/celery/celery/blob/e257646136e6fae73186d7385317f4e20cd36130/celery/backends/base.py#L269-L270).

I'm mostly writing this down so other people can find this information as well as I'm not sure how this has worked before. Fact is, the exception behavior has been like this in SQLAlchemy [for a while now](https://github.com/zzzeek/sqlalchemy/commit/2d8b5bb4f36e5624f25b170391fe42d3bfbeb623), so my guess is that either a) this never worked with the JSON serializer b) or has been subtly introduced recently in Celery. Can anyone sanity check that?
Yeah, still the case. This is kind of a shame I can’t keep track of failed tasks natively.
This issue also affects me:

```
Traceback (most recent call last):
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/app/builtins.py", line 69, in unlock_chord
    ready = deps.ready()
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/result.py", line 619, in ready
    return all(result.ready() for result in self.results)
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/result.py", line 619, in <genexpr>
    return all(result.ready() for result in self.results)
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/result.py", line 313, in ready
    return self.state in self.backend.READY_STATES
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/result.py", line 473, in state
    return self._get_task_meta()['status']
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/result.py", line 412, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/backends/base.py", line 386, in get_task_meta
    meta = self._get_task_meta_for(task_id)
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
    return fun(*args, **kwargs)
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 130, in _get_task_meta_for
    return self.meta_from_decoded(task.to_dict())
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/backends/base.py", line 301, in meta_from_decoded
    meta['result'] = self.exception_to_python(meta['result'])
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/backends/base.py", line 281, in exception_to_python
    exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/app/builtins.py", line 72, in unlock_chord
    exc=exc, countdown=interval, max_retries=max_retries,
  File "/opt/app/.venv/lib/python3.6/site-packages/celery/app/task.py", line 725, in retry
    raise ret
celery.exceptions.Retry: Retry in 1s: TypeError("__init__() missing 2 required positional arguments: 'params' and 'orig'",)
```
We're also hitting this. 
Although not the best solution, would anyone perhaps like to try the following patch:

```
--- a/celery/backends/base.py
+++ b/celery/backends/base.py
@@ -278,7 +278,10 @@ class Backend(object):
                         cls = create_exception_cls(exc_type,
                                                    celery.exceptions.__name__)
                 exc_msg = exc['exc_message']
-                exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
+                try:
+                    exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
+                except Exception:  # noqa
+                    exc = Exception(repr(exc))
             if self.serializer in EXCEPTION_ABLE_CODECS:
                 exc = get_pickled_exception(exc)
         return exc
```
@georgepsarakis The patch is working for now. Thanks
@georgepsarakis That seems to work quite well, any plans to submit this as pull request?

JFYI: I'm experiencing same issue with filelock module [Timeout exception](https://github.com/benediktschmitt/py-filelock/blob/71a5e02664a34c48792f4716f0761dc3bd23a4c4/filelock.py#L85-L101).
@nijel thanks for the feedback, glad it works! Contributions are welcome, so if anyone that has already applied the patch and works for their deployment, can open a PR with the patch and a simple test case, please feel free to mention me so I can review as soon as possible.

Resolving patch:
diff --git a/celery/backends/base.py b/celery/backends/base.py
--- a/celery/backends/base.py
+++ b/celery/backends/base.py
@@ -278,7 +278,10 @@ def exception_to_python(self, exc):
                         cls = create_exception_cls(exc_type,
                                                    celery.exceptions.__name__)
                 exc_msg = exc['exc_message']
-                exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
+                try:
+                    exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
+                except Exception as err:  # noqa
+                    exc = Exception('{}({})'.format(cls, exc_msg))
             if self.serializer in EXCEPTION_ABLE_CODECS:
                 exc = get_pickled_exception(exc)
         return exc


Test patch:
diff --git a/t/unit/backends/test_base.py b/t/unit/backends/test_base.py
--- a/t/unit/backends/test_base.py
+++ b/t/unit/backends/test_base.py
@@ -8,6 +8,7 @@
 from case import ANY, Mock, call, patch, skip
 from kombu.serialization import prepare_accept_content
 
+import celery
 from celery import chord, group, signature, states, uuid
 from celery.app.task import Context, Task
 from celery.backends.base import (BaseBackend, DisabledBackend,
@@ -29,6 +30,12 @@ def __init__(self, *args, **kwargs):
         self.args = args
 
 
+class paramexception(Exception):
+
+    def __init__(self, param):
+        self.param = param
+
+
 if sys.version_info[0] == 3 or getattr(sys, 'pypy_version_info', None):
     Oldstyle = None
 else:
@@ -456,6 +463,17 @@ def test_exception_to_python_when_attribute_exception(self):
         result_exc = b.exception_to_python(test_exception)
         assert str(result_exc) == 'Raise Custom Message'
 
+    def test_exception_to_python_when_type_error(self):
+        b = BaseBackend(app=self.app)
+        celery.TestParamException = paramexception
+        test_exception = {'exc_type': 'TestParamException',
+                          'exc_module': 'celery',
+                          'exc_message': []}
+
+        result_exc = b.exception_to_python(test_exception)
+        del celery.TestParamException
+        assert str(result_exc) == "<class 't.unit.backends.test_base.paramexception'>([])"
+
     def test_wait_for__on_interval(self):
         self.patching('time.sleep')
         b = BaseBackend(app=self.app)