zodb/relstorage — pull request #2 by ossmkitty (2013-03-07 18:09:08+00:00)
Title: Unpack error with newer releases of persistent

We recently started testing with newer versions of ZODB3 and RelStorage, specifically ZODB3 3.11.0a1 together with the current release of RelStorage 1.5.1. The updated ZODB3 release brings in the newly-independent release of persistent (currently 4.0.6), and together there seems to be a small incompatibility resulting in the following errors at commit time:

> File 'python2.7/site-packages/transaction-1.4.1-py2.7.egg/transaction/_transaction.py', line 394 in _commitResources
> rm.tpc_vote(self)
> File 'python2.7/site-packages/ZODB-4.0.0a4-py2.7.egg/ZODB/Connection.py', line 781 in tpc_vote
> s = vote(transaction)
> File 'python2.7/site-packages/relstorage/storage.py', line 789 in tpc_vote
> return self._vote()
> File 'python2.7/site-packages/relstorage/storage.py', line 823 in _vote
> self._prepare_tid()
> File 'python2.7/site-packages/relstorage/storage.py', line 705 in _prepare_tid
> tid_int = u64(tid)
> File 'python2.7/site-packages/ZODB-4.0.0a4-py2.7.egg/ZODB/utils.py', line 82 in u64
> return unpack(">Q", v)[0]
> error: unpack requires a string argument of length 8

In `relstorage.storage.RelStorage._prepare_tid`, a `persistent.timestamp.TimeStamp` is used to get the `tid` value. Instead of using `TimeStamp.raw()`, however, RelStorage uses `repr(TimeStamp)`.

Prior to persistent 4.0.5, `repr(TimeStamp)` was equivalent to `TimeStamp.raw()`, but now `repr(TimeStamp)` actually returns a platform-specific representation, using the appropriate escaping (commit https://github.com/zopefoundation/persistent/commit/e692af8281466fa309aae9273864039dcb287383 for the C version). The net result is that instead of `repr(TimeStamp)` returning an 8-byte string like b'\x01\x01\x01\x01\x01\x01\x01\x01' it returns a 25-byte string like b"'\\x01\\x01\\x01\\x01\\x01\\x01\\x01\\x01'" (on python2.7) and RelStorage's assumption is broken, leading to the above error.

I only spotted two uses of this assumption in RelStrorage, the above-mentioned `_prepare_tid`, plus `pack`. The attached simple patch to change those places to use `raw` makes our own internal tests (python2.7, MySQL) pass.

RelStorage's setup.py requires ZODB3 >= 3.7.0. Based on what's at http://svn.zope.org/ZODB3/branches/3.7/src/persistent/, it looks like the `TimeStamp.raw` method has been available since at least ZODB 3.7, so it looks like this should be safe to use on all supported versions (but I haven't tried to test that).