Repository: celery/celery

Issue:
Exception Serialization error? Tasks raising exceptions stuck in "PENDING" state
## Checklist

- [x] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.

## Steps to reproduce
Still working on a standalone repro, but what we see in production is this traceback from a task making requests module POSTs:
```
Client caught exception in _post(). Unable to retry.
Traceback (most recent call last):
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), sel
f.timeout, **extra_kw)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
    raise err
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the 
above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/va
r/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
    self, "F
ailed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f695118a4a8>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/celery/venv/lib
/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/var/lib/celery/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, 
error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='sjc1.solvedirect.com', port=443): Max retries exceeded with url: /ws/rest/oauth/token?grant_type=client_credentials (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f695118a4a8>: Failed to establish a new connection: [Errno 110] Connection timed out
',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/celery/venv/lib/python3.6/site-packages/cisco_tac/ciscotac.py", line 67, in _make_post_request
    result = requests.request('POST', url, json=payload, headers=header, verify=True, auth=auth)
  File "/var/lib/celery/venv/lib/python3.6
/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adap
ter.send(request, **kwargs)
  File "/var/lib/celery/venv/lib/python3.6/site-packages/requests/adapters.py", line 513, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='sjc1.solvedirect.com', port=443): Max retries exceeded with url: /ws/rest/oauth/token?grant_type=client_credentials (Caused by NewConnecti
onError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f695118a4a8>: Failed to establish a new connection: [Errno 110] Connection timed out',))
```


## Expected behavior

The exception should fail the task, and bubble up in the result.traceback.

## Actual behavior
No event is fired, and no tombstone record is recorded.  The task is stuck in 'PENDING' using the python library, and 'STARTED' state in flower (since it got a started event).


Discussion:
(net-task) johnar@netdev1-westus2:~/scripts-tools/project/NetTask$ celery -A net_task report

software -> celery:4.2.0rc2 (windowlicker) kombu:4.1.0 py:3.6.3
            billiard:3.5.0.3 redis:2.10.6
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:://_task.extendedredisbackend%2Bredis//:@localhost:6379/1

ENV_FILE_PATH: '/etc/celery/conf.d/celery'
REQUIRED_VARS: ['CELERY_BROKER_HOST',
 'CELERY_RESULT_BACKEND_HOST']
RFC5424TaskFormatter: <class 'net_task.log.RFC5424TaskFormatter'>
accept_content: ['json']
after_setup_logger: <Signal: after_setup_logger providing_args={'format', 'loglevel', 'logfile', 'colorize', 'logger'}>
after_setup_task_logger: <Signal: after_setup_task_logger providing_args={'format', 'loglevel', 'logfile', 'colorize', 'logger'}>
broker_url: 'redis://localhost:6379/0'
dump: ('/usr/bin/python -c "import os, json;print '
 'json.dumps(dict(os.environ))"')
...
pipe: <subprocess.Popen object at 0x7fe118351ba8>
result_backend: '://_task.extendedredisbackend%2Bredis//:@localhost:6379/1'
result_compression: 'gzip'
result_serializer: 'json'
setup_logging: <function setup_logging at 0x7fe1151438c8>
source: 'source /etc/celery/conf.d/celery'
sp: <module 'subprocess' from '/usr/lib/python3.6/subprocess.py'>
sys: <module 'sys' (built-in)>
task_always_eager: False
task_default_queue: 'default'
task_send_sent_event: True
task_serializer: 'json'
urllib: <module 'urllib' from '/usr/lib/python3.6/urllib/__init__.py'>
urlsafe_password: '********'
worker_hijack_root_logger: True
worker_log_color: False
worker_redirect_stdouts: False
worker_send_task_events: True
This looks like an exception serialization error.  trying to repro the problem, we caught this nasty traceback on the worker:

https://pastebin.com/F4m8gzie
Possibly related -- kombu catches all connection errors defined by the provider (redis in my case):
https://github.com/celery/kombu/issues/802

It seems like tasks should not be able to bubble up exceptions which will break kombu....
Acutally the original issue may be different than those Kombu issues... the original exception seems to get stuck in the worker, not make it out to kombu.

Looking into the worker's serialization prechecks -- it seems to test for whether the exception is Pickleable, but not whether it's json-able.

Resolving patch:
diff --git a/celery/backends/base.py b/celery/backends/base.py
--- a/celery/backends/base.py
+++ b/celery/backends/base.py
@@ -33,6 +33,7 @@
 from celery.utils.functional import LRUCache, arity_greater
 from celery.utils.log import get_logger
 from celery.utils.serialization import (create_exception_cls,
+                                        ensure_serializable,
                                         get_pickleable_exception,
                                         get_pickled_exception)
 
@@ -236,7 +237,7 @@ def prepare_exception(self, exc, serializer=None):
         if serializer in EXCEPTION_ABLE_CODECS:
             return get_pickleable_exception(exc)
         return {'exc_type': type(exc).__name__,
-                'exc_message': exc.args,
+                'exc_message': ensure_serializable(exc.args, self.encode),
                 'exc_module': type(exc).__module__}
 
     def exception_to_python(self, exc):
diff --git a/celery/utils/serialization.py b/celery/utils/serialization.py
--- a/celery/utils/serialization.py
+++ b/celery/utils/serialization.py
@@ -56,6 +56,8 @@ def find_pickleable_exception(exc, loads=pickle.loads,
 
     Arguments:
         exc (BaseException): An exception instance.
+        loads: decoder to use.
+        dumps: encoder to use
 
     Returns:
         Exception: Nearest pickleable parent exception class
@@ -84,6 +86,26 @@ def create_exception_cls(name, module, parent=None):
     return subclass_exception(name, parent, module)
 
 
+def ensure_serializable(items, encoder):
+    """Ensure items will serialize.
+
+    For a given list of arbitrary objects, return the object
+    or a string representation, safe for serialization.
+
+    Arguments:
+        items (Iterable[Any]): Objects to serialize.
+        encoder (Callable): Callable function to serialize with.
+    """
+    safe_exc_args = []
+    for arg in items:
+        try:
+            encoder(arg)
+            safe_exc_args.append(arg)
+        except Exception:  # pylint: disable=broad-except
+            safe_exc_args.append(safe_repr(arg))
+    return tuple(safe_exc_args)
+
+
 @python_2_unicode_compatible
 class UnpickleableExceptionWrapper(Exception):
     """Wraps unpickleable exceptions.
@@ -116,13 +138,7 @@ class UnpickleableExceptionWrapper(Exception):
     exc_args = None
 
     def __init__(self, exc_module, exc_cls_name, exc_args, text=None):
-        safe_exc_args = []
-        for arg in exc_args:
-            try:
-                pickle.dumps(arg)
-                safe_exc_args.append(arg)
-            except Exception:  # pylint: disable=broad-except
-                safe_exc_args.append(safe_repr(arg))
+        safe_exc_args = ensure_serializable(exc_args, pickle.dumps)
         self.exc_module = exc_module
         self.exc_cls_name = exc_cls_name
         self.exc_args = safe_exc_args


Test patch:
diff --git a/t/unit/backends/test_base.py b/t/unit/backends/test_base.py
--- a/t/unit/backends/test_base.py
+++ b/t/unit/backends/test_base.py
@@ -145,6 +145,17 @@ def test_unpickleable(self):
         y = self.b.exception_to_python(x)
         assert isinstance(y, KeyError)
 
+    def test_json_exception_arguments(self):
+        self.b.serializer = 'json'
+        x = self.b.prepare_exception(Exception(object))
+        assert x == {
+            'exc_message': serialization.ensure_serializable(
+                (object,), self.b.encode),
+            'exc_type': Exception.__name__,
+            'exc_module': Exception.__module__}
+        y = self.b.exception_to_python(x)
+        assert isinstance(y, Exception)
+
     def test_impossible(self):
         self.b.serializer = 'pickle'
         x = self.b.prepare_exception(Impossible())
diff --git a/t/unit/utils/test_serialization.py b/t/unit/utils/test_serialization.py
--- a/t/unit/utils/test_serialization.py
+++ b/t/unit/utils/test_serialization.py
@@ -1,14 +1,17 @@
 from __future__ import absolute_import, unicode_literals
 
+import json
+import pickle
 import sys
 from datetime import date, datetime, time, timedelta
 
 import pytest
 import pytz
-from case import Mock, mock
+from case import Mock, mock, skip
 from kombu import Queue
 
 from celery.utils.serialization import (UnpickleableExceptionWrapper,
+                                        ensure_serializable,
                                         get_pickleable_etype, jsonify)
 
 
@@ -25,6 +28,23 @@ def test_no_cpickle(self):
             sys.modules['celery.utils.serialization'] = prev
 
 
+class test_ensure_serializable:
+
+    @skip.unless_python3()
+    def test_json_py3(self):
+        assert (1, "<class 'object'>") == \
+            ensure_serializable([1, object], encoder=json.dumps)
+
+    @skip.if_python3()
+    def test_json_py2(self):
+        assert (1, "<type 'object'>") == \
+            ensure_serializable([1, object], encoder=json.dumps)
+
+    def test_pickle(self):
+        assert (1, object) == \
+            ensure_serializable((1, object), encoder=pickle.dumps)
+
+
 class test_UnpickleExceptionWrapper:
 
     def test_init(self):