Package: zodbpickle
zodbpickleREADMEThis package presents a uniform pickling interface for ZODB:Under Python2, this package forks both Python 2.7’spickleandcPicklemodules, adding support for theprotocol 3opcodes.
It also provides a new subclass ofbytes,zodbpickle.binary,
which Python2 applications can use to pickle binary values such that
they will be unpickled asbytesunder Py3k.Under Py3k, this package forks thepicklemodule (and the supporting
C extension) from both Python 3.2 and Python 3.3.  The fork add support
for thenoloadoperations used by ZODB.Cautionzodbpicklerelies on Python’spicklemodule.
Thepicklemodule is not intended to be secure against erroneous or
maliciously constructed data. Never unpickle data received from an
untrusted or unauthenticated source as arbitrary code might be executed.Also seehttps://docs.python.org/3.6/library/pickle.htmlGeneral UsageTo get compatibility between Python 2 and 3 pickling, replace:import pickleby:from zodbpickle import pickleThis provides compatibility, but has the effect that you get the fast implementation
in Python 3, while Python 2 uses the slow version.To get a more deterministic choice of the implementation, use one of:from zodbpickle import fastpickle # always C
from zodbpickle import slowpickle # always PythonBoth modules can co-exist which is helpful for comparison.But there is a bit more to consider, so please read on!Loading/Storing Python 2 StringsIn all their wisdom, the Python developers have decided that Python 2strinstances should be loaded as Python 3strobjects (i.e. unicode
strings). Patches were proposed in Pythonissue 6784but were never
applied. This code base contains those patches.Example 1: Loading Python 2 pickles on Python 3$ python2
>>> import pickle
>>> pickle.dumps('\xff', protocol=0)
"S'\\xff'\np0\n."
>>> pickle.dumps('\xff', protocol=1)
'U\x01\xffq\x00.'
>>> pickle.dumps('\xff', protocol=2)
'\x80\x02U\x01\xffq\x00.'

$ python3
>>> from zodbpickle import pickle
>>> pickle.loads(b"S'\\xff'\np0\n.", encoding='bytes')
b'\xff'
>>> pickle.loads(b'U\x01\xffq\x00.', encoding='bytes')
b'\xff'
>>> pickle.loads(b'\x80\x02U\x01\xffq\x00.', encoding='bytes')
b'\xff'Example 2: Loading Python 3 pickles on Python 2$ python3
>>> from zodbpickle import pickle
>>> pickle.dumps(b"\xff", protocol=0)
b'c_codecs\nencode\np0\n(V\xff\np1\nVlatin1\np2\ntp3\nRp4\n.'
>>> pickle.dumps(b"\xff", protocol=1)
b'c_codecs\nencode\nq\x00(X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02tq\x03Rq\x04.'
>>> pickle.dumps(b"\xff", protocol=2)
b'\x80\x02c_codecs\nencode\nq\x00X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02\x86q\x03Rq\x04.'

$ python2
>>> import pickle
>>> pickle.loads('c_codecs\nencode\np0\n(V\xff\np1\nVlatin1\np2\ntp3\nRp4\n.')
'\xff'
>>> pickle.loads('c_codecs\nencode\nq\x00(X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02tq\x03Rq\x04.')
'\xff'
>>> pickle.loads('\x80\x02c_codecs\nencode\nq\x00X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02\x86q\x03Rq\x04.')
'\xff'Example 3: everything breaks down$ python2
>>> class Foo(object):
...     def __init__(self):
...         self.x = 'hello'
...
>>> import pickle
>>> pickle.dumps(Foo(), protocol=0)
"ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb."
>>> pickle.dumps(Foo(), protocol=1)
'ccopy_reg\n_reconstructor\nq\x00(c__main__\nFoo\nq\x01c__builtin__\nobject\nq\x02Ntq\x03Rq\x04}q\x05U\x01xq\x06U\x05helloq\x07sb.'
>>> pickle.dumps(Foo(), protocol=2)
'\x80\x02c__main__\nFoo\nq\x00)\x81q\x01}q\x02U\x01xq\x03U\x05helloq\x04sb.'

$ python3
>>> from zodbpickle import pickle
>>> class Foo(object): pass
...
>>> foo = pickle.loads("ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb.", encoding='bytes')
>>> foo.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Foo' object has no attribute 'x'wait what?>>> foo.__dict__
{b'x': b'hello'}oooh.  So we useencoding='ASCII'(the default) anderrors='bytes'and
hope it works:>>> foo = pickle.loads("ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb.", errors='bytes')
>>> foo.x
'hello'falling back to bytes if necessary>>> pickle.loads(b'\x80\x02U\x01\xffq\x00.', errors='bytes')
b'\xff'Support fornoload()The ZODB usescPickle’snoload()method to retrieve all persistent
references from a pickle without loading any objects. This feature was removed
from Python 3’s pickle. Unfortuantely, this unnecessarily fills the pickle
cache.This module provides anoload()method again.Changelog3.2 (2024-02-16)Add preliminary support for Python 3.13 as of 3.13a3.3.1 (2023-10-05)Add support for Python 3.12.3.0.1 (2023-03-28)FixNameErrorin.fastpickleand.slowpickle.3.0 (2023-03-24)Build Linux binary wheels for Python 3.11.Add preliminary support for Python 3.12a5.Drop support for Python 2.7, 3.5, 3.6.Drop support for deprecatedpython setup.py test.2.6 (2022-11-17)Add support for building arm64 wheels on macOS.2.5 (2022-11-03)Add support for the final Python 3.11 release.2.4 (2022-09-15)Add support for Python 3.11 (as of 3.11.0b3).Disable unsafe math optimizations in C code.  Seepull request 73.2.3 (2022-04-22)Add support for Python 3.11 (as of 3.11.0a7).2.2.0 (2021-09-29)Add support for Python 3.10.2.1.0 (2021-09-24)Add support for Python 3.9.2.0.0 (2019-11-13)CPython 2: Makezodbpickle.binaryobjects smaller and untracked
by the garbage collector. Now they behave more like the native bytes
object. Just like it, and just like on Python 3, they cannot have
arbitrary attributes or be weakly referenced. Seeissue 53.1.1 (2019-11-09)Add support for Python 3.8.Drop support for Python 3.4.1.0.4 (2019-06-12)Fix pickle corruption under certain conditions. Seepull request 47.1.0.3 (2018-12-18)Fix a bug: zodbpickle.slowpickle assigned_PicklertoUnpickler.1.0.2 (2018-08-10)Add support for Python 3.7.1.0.1 (2018-05-16)Fix a memory leak in pickle protocol 3 under Python 2. Seeissue 36.1.0 (2018-02-09)Add a warning to the readme not to use untrusted pickles.Drop support for Python 3.3.0.7.0 (2017-09-22)Drop support for Python 2.6 and 3.2.Add support for Jython 2.7.Add support for Python 3.5 and 3.6.0.6.0 (2015-04-02)Restore thenoloadbehaviour from Python 2.6 and provide thenoloadmethod on the non-C-accelerated unpicklers under PyPy and
Python 2.Add support for PyPy, PyPy3, and Python 3.4.0.5.2 (2013-08-17)Import accelerator fromourextension module under Py3k.
Seehttps://github.com/zopefoundation/zodbpickle/issues/6,https://github.com/zopefoundation/zodbpickle/issues/7.Fix unpickler’sload_short_binstringacross supported platforms.0.5.1 (2013-07-06)Update all code and tests to Python 2.6.8, 2.7.5, 3.2.5, 3.3.2 .Add the moduleszodbpickle.fastpickleandzodbpickle.slowpickle.
This provides a version-independent choice of the C or Python
implementation.Fix a minor bug on OS X0.5.0 (2013-06-14)Removed support for thebytes_as_stringsarguments to pickling APIs:
the pickles created when that argument was true might not be unpickled
without passingencoding='bytes', which ZODB couldn’t reliably enforce.
On Py3k, ZODB will be usingprotocol=3pickles anyway.0.4.4 (2013-06-07)Add protocol 3 opcodes to the C version of thenoload()dispatcher.0.4.3 (2013-06-07)Packaging error:  remove spurious-ASIDEfile from sdist.0.4.2 (2013-06-07)Fix NameError in pure-Python version ofUnpickler.noload_appends.Fix NameError in pure-Python version ofUnpickler.noload_setitems.0.4.1 (2013-04-29)Fix typo in Python2 version ofzodbpickle.picklemodule.0.4 (2013-04-28)Support the common pickle module interface for Python 2.6, 2.7, 3.2, and 3.3.Split the Python implementations / tests into Python2- and Py3k-specific
variants.Added a fork of the Python 2.7_pickle.c, for use under Python2.
The fork adds support for the Py3kprotocol 3opcodes.Added a custombinarytype for use in Python2 apps.
Derived frombytes, thebinarytype allows Python2 apps to pickle
binary data using opcodes which will cause it to be unpickled asbyteson Py3k.  Under Py3k, thebinarytype is just an alias forbytes.0.3 (2013-03-18)Addednoloadcode to Python 3.2 version ofUnpickler.  As with
the Python 3.3 version, this code remains untested.Addedbytes_as_stringsoption to the Python 3.2 version ofPickler,dump, anddumps.0.2 (2013-03-05)Addedbytes_as_stringsoption toPickler,dump, anddumps.Incomplete support for Python 3.2:Move_pickle.c->_pickle_33.c.Clone Python 3.2.3’s_pickle.c->_pickle_32.cand apply the
same patch.Choose between them at build time based onsys.version_info.Disable some tests of 3.3-only features.Missing: implementation ofnoload()in_pickle_32.c.Missing: implementation ofbytes_as_strings=Truein_pickle_32.c.0.1.0 (2013-02-27)Initial release of Python 3.3’s pickle with the patches of Pythonissue 6784applied.Added support forerrors="bytes".