Repository: celery/celery

Issue:
Unable to save pickled objects with couchbase as result backend
Hi it seems like when I attempt to process groups of chords, the couchbase result backend is consistently failing to unlock the chord when reading from the db:

`celery.chord_unlock[e3139ae5-a67d-4f0c-8c54-73b1e19433d2] retry: Retry in 1s: ValueFormatError()`

This behavior does not occur with the redis result backend, i can switch between them and see that the error unlocking only occurs on couchbase.

## Steps to reproduce
Attempt to process a chord with couchbase backend using pickle serialization.

## Expected behavior
Chords process correctly, and resulting data is fed to the next task

## Actual behavior
Celery is unable to unlock the chord from the result backend

## Celery project info: 
```
celery -A ipaassteprunner report

software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:2.7.10
            billiard:3.5.0.3 py-amqp:2.2.2
platform -> system:Darwin arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:couchbase://isadmin:**@localhost:8091/tasks

task_serializer: 'pickle'
result_serializer: 'pickle'
dbconfig: <ipaascommon.ipaas_config.DatabaseConfig object at 0x10fbbfe10>
db_pass: u'********'
IpaasConfig: <class 'ipaascommon.ipaas_config.IpaasConfig'>
imports:
    ('ipaassteprunner.tasks',)
worker_redirect_stdouts: False
DatabaseConfig: u'********'
db_port: '8091'
ipaas_constants: <module 'ipaascommon.ipaas_constants' from '/Library/Python/2.7/site-packages/ipaascommon/ipaas_constants.pyc'>
enable_utc: True
db_user: 'isadmin'
db_host: 'localhost'
result_backend: u'couchbase://isadmin:********@localhost:8091/tasks'
result_expires: 3600
iconfig: <ipaascommon.ipaas_config.IpaasConfig object at 0x10fbbfd90>
broker_url: u'amqp://guest:********@localhost:5672//'
task_bucket: 'tasks'
accept_content: ['pickle']
```
### Additional Debug output
```
[2017-12-13 15:39:57,860: INFO/MainProcess] Received task: celery.chord_unlock[e3139ae5-a67d-4f0c-8c54-73b1e19433d2]  ETA:[2017-12-13 20:39:58.853535+00:00] 
[2017-12-13 15:39:57,861: DEBUG/MainProcess] basic.qos: prefetch_count->27
[2017-12-13 15:39:58,859: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x10b410b90> (args:('celery.chord_unlock', 'e3139ae5-a67d-4f0c-8c54-73b1e19433d2', {'origin': 'gen53678@silo2460', 'lang': 'py', 'task': 'celery.chord_unlock', 'group': None, 'root_id': '0acd3e0d-7532-445c-8916-b5fc8a6395ab', u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'celery', u'exchange': u''}, 'expires': None, u'correlation_id': 'e3139ae5-a67d-4f0c-8c54-73b1e19433d2', 'retries': 311, 'timelimit': [None, None], 'argsrepr': "('90c64bef-21ba-42f9-be75-fdd724375a7a', {'chord_size': 2, 'task': 'ipaassteprunner.tasks.transfer_data', 'subtask_type': None, 'kwargs': {}, 'args': (), 'options': {'chord_size': None, 'chain': [...], 'task_id': '9c6b5e1c-2089-4db7-9590-117aeaf782c7', 'root_id': '0acd3e0d-7532-445c-8916-b5fc8a6395ab', 'parent_id': 'c27c9565-19a6-4683-8180-60f0c25007e9', 'reply_to': '0a58093c-6fdd-3458-9a34-7d5e094ac6a8'}, 'immutable': False})", 'eta': '2017-12-13T20:39:58.853535+00:00', 'parent_id': 'c27c9565-19a6-4683-8180-60f0c25007e9', u'reply_to':... kwargs:{})
[2017-12-13 15:40:00,061: DEBUG/MainProcess] basic.qos: prefetch_count->26
[2017-12-13 15:40:00,065: DEBUG/MainProcess] Task accepted: celery.chord_unlock[e3139ae5-a67d-4f0c-8c54-73b1e19433d2] pid:53679
[2017-12-13 15:40:00,076: INFO/ForkPoolWorker-6] Task celery.chord_unlock[e3139ae5-a67d-4f0c-8c54-73b1e19433d2] retry: Retry in 1s: ValueFormatError()
```

### Stack trace from chord unlocking failure
```python
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/celery/app/trace.py", line 374, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/celery/app/trace.py", line 629, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/celery/app/builtins.py", line 75, in unlock_chord
    raise self.retry(countdown=interval, max_retries=max_retries)
  File "/Library/Python/2.7/site-packages/celery/app/task.py", line 689, in retry
    raise ret
Retry: Retry in 1s
```


Discussion:
Some more info after digging into the library a little bit: The exception message that occurs (for some reason it's not written out to console) looks like this: 

`<Couldn't decode as UTF-8, Results=1, inner_cause='utf8' codec can't decode byte 0x80 in position 1: invalid start byte, C Source=(src/convert.c,65)>`
Any thoughts as to a possible work around, or root cause?
Finally had some time to go back and figure out what was wrong here (as opposed to just using Redis). For anyone else experiencing trouble with couchbase as a result backend:

It seems the "cant decode byte" error comes about when trying to use the pickle serialization only, as the pickle serialized strings do not conform to the data structure that couchbase is expecting.

Once I switched to json serialization, the error went away, but of course I encountered the issue of my complex python objects not being json serializable.

As a workaround to enable the use of pickled objects in the couchbase result_backend, I wrote a custom encoder that first pickles my objects to strings, then encodes the resulting string in json, decoding works in the opposite way:

```python
import json, pickle
from kombu.serialization import register

def dumps(obj):
    obj_serialized = pickle.dumps(obj)
    return json.dumps(obj_serialized)


def loads(obj):
    obj_pickled = json.loads(obj)
    obj_deserialized = pickle.loads(obj_pickled)
    return obj_deserialized

register('cb_pickle', dumps, loads, content_type='application/x-cbpickle',
         content_encoding='utf-8')

result_serializer = 'cb_pickle'
accept_content = ['cb_pickle']
```

Using those methods for encoding/decoding, couchbase is able to store all my complex python objects and is a functional result backend.

@dfresh613 nice work in locating the root cause! Do you think that `base64` encoding would be equivalent? I think it may be faster, and is widely used to avoid encoding issues.

I wonder what is the best approach here. I don't think that it is possible to use a different serializer just for this backend specifically, at least not in a straightforward way.

We could change the Pickle serializer functions (see [here](https://github.com/celery/kombu/blob/a600ab87d9c32d23396f1171486541ce0b6d937d/kombu/serialization.py#L340) and [here](https://github.com/celery/kombu/blob/a600ab87d9c32d23396f1171486541ce0b6d937d/kombu/serialization.py#L349)), but it would have to be done in a backwards compatible manner, probably something like:

```python
import pickle
import base64

def dumps(obj):
    return base64.b64encode(pickle.dumps(obj))

def loads(obj):
    try:
        obj = base64.b64decode(obj)
    except TypeError:
        pass
    return pickle.loads(obj)
```

I found a benchmark [here](https://github.com/non/benchmarks-1) stating that base64 is indeed faster compared to JSON (builtin).
Thanks @georgepsarakis for the suggestion. I took your advice and am now base64 encoding the results, and everything is still working.

I'd be more than happy to change the serializer functions if that's what should be done. Though, since this doesn't seem to be an issue with many other result backends, do we really want to do the base64 encoding by default, adding potential overhead to every result encoding?

What about a separate celery built-in serialization type like "safe_pickle" that users can choose to implement if they need for situations like this?
Read through a bit more of the couchbase documentation, and it seems like this may best be fixed in the couchbase backend code itself, instead of a serializer change.

It seems couchbase allows you to specify the formatting with the fmt kwarg. Its default formatting seems to be json, which makes sense as to why I was encountering the problem (although celery was encoding in pickle, it was still trying to save as a json string by default in couchbase).

[Here](http://docs.couchbase.com/sdk-api/couchbase-python-client-1.2.3/api/couchbase.html#couchbase.connection.Connection.default_format) is the link to the couchbase Connection API which talks about the default formatting.

[Here](https://github.com/celery/celery/blob/master/celery/backends/couchbase.py#L106) is where celery is currently saving the result backend with no formatting specified (defaults to json)

The other formatting options available to use (including pickle) are available [Here](http://docs.couchbase.com/sdk-api/couchbase-python-client-1.2.3/api/couchbase.html#format-info) 

I think the best fix here is to ~~change the celery backend code such that it asseses the current result serializer being used, and changes the fmt kwarg when running Connection.set() so that it matches the actual serialization celery is using.~~ 

~~Just use the FMT_AUTO format, and let couchbase identify the formatting.~~

Just use FMT_UTF8, since celery is already serializing the results as pickle/json, and specifying any other format will just make couchbase re-serialize again, and run an additional json.dumps/pickle.dumps 

What do you think?
@dfresh613 that seems as a far better solution for the time being, nice finding. There might be value in changing the serializer in a future major release.
FYI - Encountered some issues using FMT_UTF8 format in couchbase, it was still having some issues attempting to store python pickled objects.

The formatting i found which consistently works for both json and pickle  serialization is the FMT_AUTO.


Resolving patch:
diff --git a/celery/backends/couchbase.py b/celery/backends/couchbase.py
--- a/celery/backends/couchbase.py
+++ b/celery/backends/couchbase.py
@@ -19,6 +19,7 @@
     from couchbase import Couchbase
     from couchbase.connection import Connection
     from couchbase.exceptions import NotFoundError
+    from couchbase import FMT_AUTO
 except ImportError:
     Couchbase = Connection = NotFoundError = None   # noqa
 
@@ -106,7 +107,7 @@ def get(self, key):
             return None
 
     def set(self, key, value):
-        self.connection.set(key, value, ttl=self.expires)
+        self.connection.set(key, value, ttl=self.expires, format=FMT_AUTO)
 
     def mget(self, keys):
         return [self.get(key) for key in keys]


Test patch: