Repository: celery/celery

Issue:
Result: Deserializing exceptions for unknown classes causes KeyError
## Checklist

- [x] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.

## Steps to reproduce

Write a task that throws an exception from a custom module.
Run the task
Get the task result.state from an interpreter that does not have the custom module installed.

## Expected behavior
Task result should be returned.

## Actual behavior
KeyError on deserializing the Exception.

My original; traceback is from flower:
```
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:      Traceback (most recent call last):
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:        File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1541, in _execute
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:          result = method(*self.path_args, **self.path_kwargs)
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:        File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 2949, in wrapper
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:          return method(self, *args, **kwargs)
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:        File "/usr/local/lib/python3.5/dist-packages/flower/api/tasks.py", line 314, in get
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:          response = {'task-id': taskid, 'state': result.state}
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:        File "/usr/local/lib/python3.5/dist-packages/celery/result.py", line 471, in state
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:          return self._get_task_meta()['status']
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:        File "/usr/local/lib/python3.5/dist-packages/celery/result.py", line 410, in _get_task_meta
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:          return self._maybe_set_cache(self.backend.get_task_meta(self.id))
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:        File "/usr/local/lib/python3.5/dist-packages/celery/backends/base.py", line 359, in get_task_meta
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:          meta = self._get_task_meta_for(task_id)
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:        File "/usr/local/lib/python3.5/dist-packages/celery/backends/base.py", line 674, in _get_task_meta_for
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:          return self.decode_result(meta)
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:        File "/usr/local/lib/python3.5/dist-packages/celery/backends/base.py", line 278, in decode_result
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:          return self.meta_from_decoded(self.decode(payload))
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:        File "/usr/local/lib/python3.5/dist-packages/celery/backends/base.py", line 274, in meta_from_decoded
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:          meta['result'] = self.exception_to_python(meta['result'])
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:        File "/usr/local/lib/python3.5/dist-packages/celery/backends/base.py", line 252, in exception_to_python
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:          cls = getattr(sys.modules[exc_module], exc_type)
Jun 21 01:23:24 netdocker1-eastus2 daemon INFO 94e57cb12059[92630]:      KeyError: 'net_devices2.exceptions'
```

```
celery -A net_task report

software -> celery:4.2.0rc2 (windowlicker) kombu:4.1.0 py:3.6.3
            billiard:3.5.0.3 redis:2.10.6
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:://_task.extendedredisbackend%2Bredis//:@localhost:6379/1

ENV_FILE_PATH: '/etc/celery/conf.d/celery'
REQUIRED_VARS: ['CELERY_BROKER_HOST',
 'CELERY_RESULT_BACKEND_HOST']
RFC5424Formatter: <class 'syslog_rfc5424_formatter.RFC5424Formatter'>
accept_content: ['json']
after_setup_logger: <Signal: after_setup_logger providing_args={'logfile', 'logger', 'format', 'colorize', 'loglevel'}>
after_setup_task_logger: <Signal: after_setup_task_logger providing_args={'logfile', 'logger', 'format', 'colorize', 'loglevel'}>
broker_url: 'redis://localhost:6379/0'
dump: ('/usr/bin/python -c "import os, json;print '
 'json.dumps(dict(os.environ))"')
env: {
##redacted
}
pipe: <subprocess.Popen object at 0x7f1bd1dd8ac8>
result_backend: '://_task.extendedredisbackend%2Bredis//:@localhost:6379/1'
result_compression: 'gzip'
result_serializer: 'json'
setup_logging: <function setup_logging at 0x7f1bcff64400>
source: 'source /etc/celery/conf.d/celery'
sp: <module 'subprocess' from '/usr/lib/python3.6/subprocess.py'>
sys: <module 'sys' (built-in)>
task_always_eager: False
task_default_queue: 'default'
task_send_sent_event: True
task_serializer: 'json'
urllib: <module 'urllib' from '/usr/lib/python3.6/urllib/__init__.py'>
urlsafe_password: '********'
worker_log_color: False
worker_redirect_stdouts: False
worker_send_task_events: True
```



Discussion:
Looks like this issue was introduced here:
https://github.com/celery/celery/commit/bd347f7565f3a72c8cfb686ea0bfe38cfa76e09b

Resolving patch:
diff --git a/celery/backends/base.py b/celery/backends/base.py
--- a/celery/backends/base.py
+++ b/celery/backends/base.py
@@ -21,6 +21,7 @@
 from kombu.utils.encoding import bytes_to_str, ensure_bytes, from_utf8
 from kombu.utils.url import maybe_sanitize_url
 
+import celery.exceptions
 from celery import current_app, group, maybe_signature, states
 from celery._state import get_current_task
 from celery.exceptions import (ChordError, ImproperlyConfigured,
@@ -249,7 +250,11 @@ def exception_to_python(self, exc):
                 else:
                     exc_module = from_utf8(exc_module)
                     exc_type = from_utf8(exc['exc_type'])
-                    cls = getattr(sys.modules[exc_module], exc_type)
+                    try:
+                        cls = getattr(sys.modules[exc_module], exc_type)
+                    except KeyError:
+                        cls = create_exception_cls(exc_type,
+                                                   celery.exceptions.__name__)
                 exc_msg = exc['exc_message']
                 exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
             if self.serializer in EXCEPTION_ABLE_CODECS:


Test patch: