ask/celery — Issue #503: Cache key error (MemcachedStringEncodingError)
Author: macro
Created: 2011-10-13 16:30:34+00:00
State: open

Kombu only decodes a message if it's not unicode (https://github.com/ask/kombu/blob/master/kombu/serialization.py#L139) This causes the task_id to become unicode even if it was originally a byte string.

This results in an error from the memcached library when store_result is called since the task_id is used to build the result cache key. The following is the error from python-memcached, pylibmc raises a TypeError.

```python [2011-10-11 19:39:20,246: WARNING/PoolWorker-1] /py27-env/lib/python2.7/site-packages/celery/worker/job.py:113: UserWarning: Exception outside body: : Keys must be str()'s, not unicode. Convert your unicode strings using mystring.encode(charset)! Traceback (most recent call last):
File "/py27-env/lib/python2.7/site-packages/celery/worker/job.py", line 107, in execute_safe
return self.execute(*args, **kwargs)
File "/py27-env/lib/python2.7/site-packages/celery/worker/job.py", line 125, in execute
return super(WorkerTaskTrace, self).execute()
File "/py27-env/lib/python2.7/site-packages/celery/execute/trace.py", line 79, in execute
retval = self._trace()
File "/py27-env/lib/python2.7/site-packages/celery/execute/trace.py", line 93, in _trace
r = handler(trace.retval, trace.exc_type, trace.tb, trace.strtb)
File "/py27-env/lib/python2.7/site-packages/celery/worker/job.py", line 153, in handle_failure
exc = self.task.backend.mark_as_failure(self.task_id, exc, strtb)
File "/py27-env/lib/python2.7/site-packages/celery/backends/base.py", line 64, in mark_as_failure
traceback=traceback)
File "/py27-env/lib/python2.7/site-packages/celery/backends/base.py", line 199, in store_result
return self._store_result(task_id, result, status, traceback, **kwargs)
File "/py27-env/lib/python2.7/site-packages/celery/backends/base.py", line 342, in _store_result
self.set(self.get_key_for_task(task_id), pickle.dumps(meta))
File "/py27-env/lib/python2.7/site-packages/celery/backends/cache.py", line 88, in set
return self.client.set(key, value, self.expires)
File "/py27-env/lib/python2.7/site-packages/python_memcached-1.45-py2.7.egg/memcache.py", line 515, in set
return self._set("set", key, val, time, min_compress_len)
File "/py27-env/lib/python2.7/site-packages/python_memcached-1.45-py2.7.egg/memcache.py", line 718, in _set
self.check_key(key)
File "/py27-env/lib/python2.7/site-packages/python_memcached-1.45-py2.7.egg/memcache.py", line 931, in check_key
"Keys must be str()'s, not unicode. Convert your unicode "
MemcachedStringEncodingError: {'exc_message': "Keys must be str()'s, not unicode. Convert your unicode strings using mystring.encode(charset)!", 'exc_type': 'MemcachedStringEncodingError'}
```

Celery should always encode the key value to ensure it's accepted by memcached.