Package: pickle-compat
pickle-compatPython 2/3 compatibility layer for PickleTL;DRTo make your pickle forward- and backward-compatible between Python versions, use this:pip install pickle-compatThen monkey-patch your pickle library with this:importpickle_compatpickle_compat.patch()From this point, you can safely assume that what's pickled withpickle.dumps()in Python 2 can be converted back to the real object in Python 3 withpickle.loads(), and vise versa. However, note that it doesn't play well with cPickle, future.moves.pickle, or six.moves.cPickle, you need to use plain "import pickle" instead.If you want to roll back the patch, use:pickle_compat.unpatch()Problem StatementYou were always aware of how pickle is unsafe, hard to debug, and how backward-incompatibility issues may bite you if you decide to update the version. You also heard that you should never use the pickle in a multi-language environment because it's Python-specific.You knew it all, but you considered it "good enough" for your case. You worked on a monolith application, and pickle provides a serialization mechanism that works out of the box for anything you can create from your Python code.Until came the time to migrate to Python 3. Anxious, you postponed it for your big legacy app for as long as you could, but there's no way you can delay it even further. This was when you realized that Python 2 and Python 3 are not two versions of the same language, but actuallytwo different languagesthat happen to share some code constructs.OK, now all of a sudden, you came up with a multi-language environment, where you need to read the pickle content, serialized by Python 2, from your code in Python 3. If you're making gradual migration, the opposite is also true.First frustrationsThings work out of the box only for the most straightforward cases.$python2-c'import pickle; print pickle.dumps("Hello world")'|python3-c'import pickle, sys; print(repr(pickle.load(sys.stdin.buffer)))''Hello world'All of a sudden, things start to get broken in the most unexpected places. For example, Python 3 fails to unpickle Python 2's datetime, spitting the scariest issue of any Python developer, a UnicodeDecodeError.$python2-c'import pickle, datetime; print pickle.dumps(datetime.datetime.utcnow())'|python3-c'import pickle, sys; print(repr(pickle.load(sys.stdin.buffer)))'Traceback(mostrecentcalllast):File"<string>",line1,in<module>
UnicodeDecodeError:'ascii'codeccan'tdecodebyte0xe4inposition1:ordinalnotinrange(128)Let's follow the rabbit to learn a bit more about the pickle, just enough to make it work for Python 2 and Python 3. At this point, I'm not sure how to make a smooth transition from where you are to where I wanted us to be, so I start throwing random facts at you in the hope that they build a more or less consistent picture in your head.Protocol versionsPickle has several so-called "protocols," or formats in which the file can be written. You can optionally define the protocol version in thepickle.dumps(). The default format in Python 2.7 is 0 (also known as ASCII format), but it can read and write in formats 1 and 2 as well. Formats 1 and 2 are not ASCII-safe, but they are more compact and faster.>>>pickle.dumps("hello")"S'hello'\np0\n.">>>pickle.dumps("hello",protocol=1)'U\x05helloq\x00.'>>>pickle.dumps("hello",protocol=2)'\x80\x02U\x05helloq\x00.'In Python 3, Guido introduced a new version of the protocol, intentionally make it backward-incompatible with Python 2.7.See the commit. The comment around theDEFAULT_PROTOCOLconstant warns, "We intentionally write a protocol that Python 2.x cannot read; there are too many issues with that."The main takeaway from us is that if we want to have a backward- and forward-compatible code, we can only use protocols that both Python 2 and Python 3 understand: from 0 to 2 inclusive.Pickle format and pickletoolsModulepickletoolscalls itself an "Executable documentation" for the pickle module. I highly recommend we open the source code and read an extensive introduction, starting with the words "A pickle is a program for a virtual pickle machine." Another useful feature of pickletools is that it provides a readable representation of the pickle stack.$python2>>>importpickle,pickletools>>>pickletools.dis(pickle.dumps("hello"))0:SSTRING'hello'9:pPUT012:.STOPhighestprotocolamongopcodes=0Here the main takeaway is that data in a pickle are represented in the format of the "opcode - data," where opcode decides, roughly speaking, the type of the following element. The list of opcodes is quite extensive and is always growing. You can find themhereStrings and bytesLet's find out how text and bytes are represented in Python 2 and Python 3 and what the differences are between them. We'll use Pickle version 2 for comparison. There's no surprise that Python 2 encodes strings and bytes asBINSTRINGand Unicode objects asBINUNICODE.$python2>>>importpickle,pickletools>>>pickletools.dis(pickle.dumps("foo",protocol=2))0:\x80PROTO22:USHORT_BINSTRING'foo'7:qBINPUT09:.STOPhighestprotocolamongopcodes=2>>>pickletools.dis(pickle.dumps(b"foo",protocol=2))0:\x80PROTO22:USHORT_BINSTRING'foo'7:qBINPUT09:.STOPhighestprotocolamongopcodes=2>>>pickletools.dis(pickle.dumps(u"foo",protocol=2))0:\x80PROTO22:XBINUNICODEu'foo'10:qBINPUT012:.STOPhighestprotocolamongopcodes=2On the contrary, Python 3 doesn't want to deal with "strings" as the name is ambiguous and prefers to deal withBINBYTESandBINUNICODE. I will show how it's encoded in protocol 3 that doesn't mean to be compatible with Python 2.$python3>>>importpickle,pickletools>>>pickletools.dis(pickle.dumps(b"foo",protocol=3))0:\x80PROTO32:CSHORT_BINBYTESb'foo'7:qBINPUT09:.STOPhighestprotocolamongopcodes=3>>>pickletools.dis(pickle.dumps(u"foo",protocol=3))0:\x80PROTO32:XBINUNICODE'foo'10:qBINPUT012:.STOPhighestprotocolamongopcodes=2Here come two questions:How Python 3 encode bytes in protocol 2? Note that the second protocol knows nothing aboutBINBYTES?How Python 3 decodes theBINSTRINGtype, provided that it's a Python 2 type and it's ambiguous?Answering the first question is easy. The pickler introduces a backward-compatible hack.$python3>>>pickletools.dis(pickle.dumps(b'foo',protocol=2))0:\x80PROTO22:cGLOBAL'_codecs encode'18:qBINPUT020:XBINUNICODE'foo'28:qBINPUT130:XBINUNICODE'latin1'41:qBINPUT243:\x86TUPLE244:qBINPUT346:RREDUCE47:qBINPUT449:.STOPhighestprotocolamongopcodes=2Converting back to Python, it saves the byte sequence to a Unicode object, puts it to the stack, and tells the unpickler to execute the following command:import_codecs_codecs.encode(u"foo","latin1")A side note. I did not know, but apparently, you can convert safely to Unicode and back any byte sequence.$python3>>>importos>>>s=os.urandom(100000)>>>s==s.decode('latin1').encode('latin1')TrueIt also works for Python 2, so we shouldn't care much about the backward compatibility.Now, how Python 3 decodesBINSTRINGopcodes? From the first example, we can see that a string in Python 2 is now a Unicode object in Python 3. In other words, the pickler tries to convert bytes to Unicode.$python2-c'import pickle; print pickle.dumps("Hello world")'|python3-c'import pickle, sys; print(repr(pickle.load(sys.stdin.buffer)))''Hello world'At this point, you probably ask yourself what encoding does it use? Fortunately, the answer is right there, inthe documentation. Python 3 introduced a parameter "encoding" that defaults to ASCII.The encoding and errors tell pickle how to decode 8-bit string instances pickled by Python 2; these default to ‘ASCII’ and ‘strict’, respectively. The encoding can be ‘bytes’ to read these 8-bit string instances as bytes objects. Using encoding='latin1' is required for unpickling NumPy arrays and instances of datetime, date and time pickled by Python 2.If you wonder what's wrong with datetime, here's how its output looks like in Python 2.$python2>>>importpickle,pickletools,datetime>>>pickletools.dis(pickle.dumps(datetime.datetime.utcnow(),protocol=2))0:\x80PROTO22:cGLOBAL'datetime datetime'21:qBINPUT023:USHORT_BINSTRING'\x07\xe4\x05\x1a\x0f\x01\x16\x00\x96\x10'35:qBINPUT137:\x85TUPLE138:qBINPUT240:RREDUCE41:qBINPUT343:.STOPhighestprotocolamongopcodes=2Here comes yet another surprise for me: datetime constructor can accept a byte sequence to initialize its internal state, and pickle takes advantage of this.>>>importdatetime>>>datetime.datetime(b'\x07\xe4\x05\x1a\x0f\x01\x16\x00\x96\x10')datetime.datetime(2020,5,26,15,1,22,38416)Setting the encoding to "latin1" seems to work.python2-c'import pickle, datetime; print pickle.dumps(datetime.datetime.utcnow())'|python3-c'import pickle, sys; print(repr(pickle.load(sys.stdin.buffer, encoding="latin1")))'datetime.datetime(2020,5,26,15,19,6,275120)The main takeaway is that strings in Python 2 are converted to Unicode objects in Python 3, and you can control the encoding.Non-latin strings in Python 2What if you have non-ASCII content, represented as an old string and not a Unicode object? If you pickle it in Python 2 and unpickle it back in Python 3, you are in trouble.The byte string doesn't have any information about the encoding. In Python 2, you probably implicitly supposed that it's a UTF-8, but when you convert it back to Python 3 with Unpickle, it appears as encoded with latin1.python2-c'import pickle; print pickle.dumps("©")'|python3-c'import pickle, sys; print(repr(pickle.load(sys.stdin.buffer, encoding="latin1")))''Â©'To workaround, you need to use UTF-8, which will work for this case.python2-c'import pickle; print pickle.dumps("©")'|python3-c'import pickle, sys; print(repr(pickle.load(sys.stdin.buffer, encoding="utf8")))''©'Unfortunately, it will not work for datetimes and other binary strings that don't represent a valid UTF-8 sequence.Unpickling with "bytes" encodingWell, we were so close to the victory, and we're back to square one. What we're going to do? Fortunately, there's a documented escape hatch, the "bytes" encoding. This encoding looks precisely the way we need it. It doesn't try to outsmart you and convert bytes to something that looks like a string. Instead, it returns bytes as bytes objects. Even better than "latin1"!python2-c'import pickle; print pickle.dumps("©")'|python3-c'import pickle, sys; print(repr(pickle.load(sys.stdin.buffer, encoding="bytes")))'b'\xc2\xa9'Datetime objects also work. Is this a victory? Not so fast.Unpickling with "bytes" encoding. The string constantsI never bothered annotating my string constants that play the role of the "string constants" with au""prefix. It's ugly and redundant, and in most cases, when my strings don't contain any non-ASCII symbols, the migration works just fine.It's not the case, though, when pickle comes into play. For example, consider the function which takes two arguments and an operation name as a string:defapply_operation(a,b,op):ifop=="ADD":returna+belifop=="SUB":returna-belse:raiseValueError("Unknown operation")Somewhere else, I let theoppass through the pickle-unpickle pipeline so that Python 2 would convert it to a binary string, and Python 3 would unpickle it exactly as is. In my case, this could be a caching library or a queue processor. Now, I pass myopas a binary object to the function, and because"ADD" != b"ADD", it will always fail with an "Unknown operation" exception.The most common bytes vs. Unicode problems look like these. Case one:>>>"foo"==b"foo"FalseCase two:>>>{"key":"value"}[b"key"]Traceback(mostrecentcalllast):File"<stdin>",line1,in<module>KeyError:b'key'One solution is to make all string constants behave like Unicode objects. You can convert them explicitly from""tou""or addfrom __future__ import unicode_literals. Both solutions don't look quite elegant. Fortunately, when you finally migrate to Python 3, you can delete both of them, and you can do this automatically. I prefer the "future" solution because it generates a smaller diff.futurize--stage1--unicode-literals--write--nobackupspath/to/codeUnpickling with "bytes" encoding. Objects with attributesIt is, by far, not the worst case. To make things even more complicated, let's try to serializefoo.foo.# file: foo.pyclassFoo(object):a='UNSET'b='UNSET'def__init__(self):self.a=1self.b=2def__repr__(self):return'Foo(%s,%s)'%(self.a,self.b)foo=Foo()As long as we use the default settings, we're good.$python2-c'import pickle, foo; print pickle.dumps(foo.foo)'|python3-c'import pickle, sys; print(repr(pickle.load(sys.stdin.buffer)))'Foo(1,2)But if we pass "bytes" as an argument, all of a sudden, something goes wrong.python2-c'import pickle, foo; print pickle.dumps(foo.foo)'|python3-c'import pickle, sys; print(repr(pickle.load(sys.stdin.buffer, encoding="bytes")))'Foo(UNSET,UNSET)We lost the attributes ofaandb. Where do they go? The samepickletool.dis()helps us to find the answer:$python2>>>importpickle,pickletools,foo>>>pickletools.dis(pickle.dumps(foo.foo,protocol=2))0:\x80PROTO22:cGLOBAL'foo Foo'11:qBINPUT013:)EMPTY_TUPLE14:\x81NEWOBJ15:qBINPUT117:}EMPTY_DICT18:qBINPUT220:(MARK21:USHORT_BINSTRING'a'24:qBINPUT326:KBININT1128:USHORT_BINSTRING'b'31:qBINPUT433:KBININT1235:uSETITEMS(MARKat20)36:bBUILD37:.STOPhighestprotocolamongopcodes=2The pickle loader doesn't call__init__. Instead, it creates a new empty "dummy" object of the classFooand populates its state by updating the__dict__. If this would be Python, we could write it like this:obj=object.__new__(foo.Foo)obj.__dict__={"a":1,"b":2}I think now you understand what went wrong. Because of thebytesencoding, we did not convert b"a" and b"b" to their "python3-string" representations. You can put anything to an object's dict, but only the keys that are strings are represented as "proper object attributes."The next command shows the contents of the__dict__of an object and proves that we were right?python2-c'import pickle, foo; print pickle.dumps(foo.foo)'|python3-c'import pickle, sys; print(pickle.load(sys.stdin.buffer, encoding="bytes").__dict__)'{b'a':1,b'b':2}OK, we can't useASCII,latin1,utf8as an encoding, and now we learned that we couldn't usebytes? It looks like a dead-end. Or you can get to your last resort, dirty and evil, monkey-patching. The previous version of the pickle-compat used this approach, but we eventually decided to get away without it in favor of "latin1", due to way too many corner cases.Get back to unpickling with "latin1" encoding. Be careful with non-ASCII stringsSo, as we learned, the only practical unpickling option is to automatically decode Python 2 str to Python 3 str using "latin1" as an encoding. As we discussed earlier, though, we need to be very careful with byte strings implicitly encoded with UTF-8. I will only provide some examples where you can accidentally come across them. This way or another, you need to fix everything outlined below, regardless of whether you plan to deal with Pickle cases.CodeThis returns str:# coding: utf-8copy="©"Use this instead:# coding: utf-8from__future__importunicode_literalscopy="©"FilesThis returns an str:open('test.txt').read()Use the variant that returns unicode and works correctly in Python 2 and Python 3:importioio.open('test.txt','rt',encoding='utf8').read()RedisThis returns an str:redis.Redis().get("foo")Usedecode_responsesinstead:redis.Redis(decode_responses=True).get("foo")CSV readerThis returns str objects:>>>importcsv>>>list(csv.reader(open("foo.csv")))[['a','b'],['c','d']]Installbackports.csvand open files in text modes instead:>>>frombackportsimportcsv>>>importio>>>list(csv.reader(io.open("foo.csv","rt",encoding="utf8")))[[u'a',u'b'],[u'c',u'd']]RequestsThis returns an str:requests.get("https://example.com").contentUse the variant that returns unicode:requests.get("https://example.com").textOld-style classesWe are almost there, except for one thing: old-style classes. As you know, in Python 3, everything subclasses objects, while in Python 2, unless you explicitly inherit your class from it, the top-level class will be "type". It is considered outdated, but it's still used in different places of the standard library, waiting to ruin your life in the most unexpected moment.This time we talk about forward-compatibility and want to make sure that anything that is pickled in Python 3 can be successfully unpicked in Python 2.Let's take an object that is an old-style class in Python 2.python3-c'import pickle, smtplib, sys; sys.stdout.buffer.write(pickle.dumps(smtplib.SMTP(), protocol=2))'|python2-c'import pickle, sys; print pickle.load(sys.stdin)'Traceback(mostrecentcalllast):File"<string>",line1,in<module>File"2.7.15/lib/python2.7/pickle.py",line1384,inloadreturnUnpickler(file).load()File"2.7.15/lib/python2.7/pickle.py",line864,inloaddispatch[key](self)File"2.7.15/lib/python2.7/pickle.py",line1089,inload_newobjobj=cls.__new__(cls,*args)AttributeError:classSMTPhasnoattribute'__new__'The approach is similar to the old one: find out how unpickler loads new objects and then patch it to see if the class is old. The Python 2 implementation liveshere.Note that the protocol version 0 doesn't contain a NEWOBJ opcode and uses a set of workarounds to make it work, so this approach will only work for version 2 of the protocol.cPickle, future, and six movesHere is a word of warning. The patcher doesn't fix cPickle of Python 2 and _pickle of Python 3. The latter is an undocumented module imported by Python 3's pickle, if possible.The way we solved the problem for ourselves at Doist is by importing "pickle" everywhere. It works slower on Python 2, but that only serves as an extra incentive to finish the migration faster. You can usefuturizefrom the "future" package to make it automatically, and it will convert all occurrences ofimport cPickletoimport pickle.If you chose a different migration strategy, with "moves," this can become cumbersome because you can import cPickle unknowingly. More specifically, this will import cPickle implementation under the hood:from future.moves import pickleThe same goes for this:from six.moves import cPickleThe main takeaway is that this patcher will not as expected if you use cPickle, future.moves.pickle or six.moves.cPickle.Putting it all togetherWhat we learnedThe default version of the protocol has to be 2, both for Python 2 and Python 3.We must use the "latin1" encoding in the pickle for Python 3.We must be careful with plain strings, represented non-ASCII objects.We must patch Unpickler in Python 2 to correctly unpickle instances of old-style classes.Also, we learned some of the internals of pickle and learned how to use pickletools. Finally, we wrapped everything with apickle_compatlibrary that monkey-patches the standard pickle module.