Repository: google/turbinia

Issue:
Serialization failing for Kombu messages
It seems like this happens when there are unrelated task failures, so perhaps there is something being set that is not serializable by kombu/json that is added during task failure.

```
2021-02-27 01:50:05 [INFO] Trying last ditch attempt to close result
2021-02-27 01:50:05 [INFO] Task Result was auto-closed from task executor on f170ddbbab0a likely due to previous failures.  Previous status: [PartitionEnumerationTask Task failed with exception: [Evidence RawDisk:/evidence/test-small.img:/evidence/test-small.img being processed by Task PartitionEnumerationTask requires Evidence to be in state ATTACHED, but earlier pre-processors may have failed.  Current state is [MOUNTED: False, ATTACHED: False, DECOMPRESSED: False]. See previous logs for more information.]]
2021-02-27 01:50:05 [INFO] Task Result was auto-closed from task executor on f170ddbbab0a likely due to previous failures.  Previous status: [PartitionEnumerationTask Task failed with exception: [Evidence RawDisk:/evidence/test-small.img:/evidence/test-small.img being processed by Task PartitionEnumerationTask requires Evidence to be in state ATTACHED, but earlier pre-processors may have failed.  Current state is [MOUNTED: False, ATTACHED: False, DECOMPRESSED: False]. See previous logs for more information.]]
2021-02-27 01:50:05 [INFO] Starting post-processor for evidence /evidence/test-small.img
2021-02-27 01:50:05 [DEBUG] Evidence state: [MOUNTED: False, ATTACHED: False, DECOMPRESSED: False]
2021-02-27 01:50:05 [DEBUG] Not copying source file /evidence/1614390605-ef6523d455ed40b1933de0226e08f798-PartitionEnumerationTask/worker-log.txt already in output dir /evidence/1614390605-ef6523d455ed40b1933de0226e08f798-PartitionEnumerationTask
2021-02-27 01:50:05 [DEBUG] Result close successful. Status is [Task Result was auto-closed from task executor on f170ddbbab0a likely due to previous failures.  Previous status: [PartitionEnumerationTask Task failed with exception: [Evidence RawDisk:/evidence/test-small.img:/evidence/test-small.img being processed by Task PartitionEnumerationTask requires Evidence to be in state ATTACHED, but earlier pre-processors may have failed.  Current state is [MOUNTED: False, ATTACHED: False, DECOMPRESSED: False]. See previous logs for more information.]]]
2021-02-27 01:50:05 [DEBUG] Checking TurbiniaTaskResult for serializability
2021-02-27 01:50:05 [INFO] Result check: Successful
[2021-02-27 01:50:05,280: INFO/MainProcess] Lock 139817451534264 released on /evidence/turbinia-worker.lock
2021-02-27 01:50:05 [DEBUG] Returning original result object 6601587c9d6a4bd092c0399f8752ae4c after task execution
[2021-02-27 01:50:05,283: ERROR/MainProcess] Task task_runner[5ef00ecf-066b-4ba7-bbea-6e3c90a063ed] raised unexpected: EncodeError(TypeError("Object of type 'type' is not JSON serializable",),)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.6/dist-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.6/dist-packages/kombu/utils/json.py", line 70, in dumps
    **dict(default_kwargs, **kwargs))
  File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.6/dist-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'type' is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 480, in trace_task
    uuid, retval, task_request, publish_result,
  File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 158, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 443, in store_result
    request=request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 865, in _store_result
    self._set_with_state(self.get_key_for_task(task_id), self.encode(meta), state)
  File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 332, in encode
    _, _, payload = self._encode(data)
  File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 336, in _encode
    return dumps(data, serializer=self.serializer)
  File "/usr/local/lib/python3.6/dist-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/dist-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/usr/local/lib/python3.6/dist-packages/vine/five.py", line 194, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/dist-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.6/dist-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.6/dist-packages/kombu/utils/json.py", line 70, in dumps
    **dict(default_kwargs, **kwargs))
  File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.6/dist-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
kombu.exceptions.EncodeError: Object of type 'type' is not JSON serializable
```


Discussion:

Resolving patch:
diff --git a/turbinia/workers/__init__.py b/turbinia/workers/__init__.py
--- a/turbinia/workers/__init__.py
+++ b/turbinia/workers/__init__.py
@@ -20,6 +20,7 @@
 from datetime import datetime, timedelta
 from enum import IntEnum
 import getpass
+import json
 import logging
 import os
 import pickle
@@ -315,8 +316,8 @@ def set_error(self, error, traceback_):
         error: Short string describing the error.
         traceback_: Traceback of the error.
     """
-    self.error['error'] = error
-    self.error['traceback'] = traceback_
+    self.error['error'] = str(error)
+    self.error['traceback'] = str(traceback_)
 
   def serialize(self):
     """Creates serialized result object.
@@ -690,13 +691,21 @@ def validate_result(self, result):
 
     if isinstance(result, TurbiniaTaskResult):
       try:
-        log.debug('Checking TurbiniaTaskResult for serializability')
+        log.debug('Checking TurbiniaTaskResult for pickle serializability')
         pickle.dumps(result.serialize())
       except (TypeError, pickle.PicklingError) as exception:
         message = (
             'Error pickling TurbiniaTaskResult object. Returning a new result '
             'with the pickling error, and all previous result data will be '
             'lost. Pickle Error: {0!s}'.format(exception))
+      try:
+        log.debug('Checking TurbiniaTaskResult for JSON serializability')
+        json.dumps(result.serialize())
+      except (TypeError) as exception:
+        message = (
+            'Error JSON serializing TurbiniaTaskResult object. Returning a new '
+            'result with the JSON error, and all previous result data will '
+            'be lost. JSON Error: {0!s}'.format(exception))
     else:
       message = (
           'Task returned type [{0!s}] instead of TurbiniaTaskResult.').format(


Test patch: