Package: anyencoder
Here’s a little library that makes it easy to perform dynamic dispatch
for multiple object serializers.OverviewFeaturesDeveloped on Python 3.7 (and requires 3.7+, sorry not sorry.)Tested-ish with ~90% code coverage.You can create as many custom encoders as you want (as long as the
number of encoders you want is 128 or less.)Types are associated with encoders via a registry or object
attribute inspection.Getting StartedInstall the package:pip install anyencoderEncode a list:>>>importanyencoder>>>letters=['a','b','c']>>>anyencoder.encode(letters)b'\x05\x80\x00\x00\x01\x80\x04\x95\x11\x00\x00\x00\x00\x00\x00\x00]\x94(\x8c\x01a\x94\x8c\x01b\x94\x8c\x01c\x94e.'Absent other parameters or method calls, the default encoder is used
– probablypickle. I realize this isn’t terribly useful. Let’s dig
deeper.TypesBuiltin TypesInstantiateDynamicEncoderand register aTypeTagspecifying that
list should be serialized usingmsgpack:>>>fromanyencoderimportDynamicEncoder,TypeTag>>>type_tag=TypeTag(type_=list,evaluator=lambda_:'msgpack')>>>letters=['a','b','c']>>>encoder=DynamicEncoder()>>>encoder.load_encoder_plugins()>>>encoder.register(type_tag)>>>encoder.encode(letters)b'\x05\x83\x00\x00\x01\x93\xa1a\xa1b\xa1c'Types are associated with an evaluator. The evaluator is called
against the object being serialized. This can be used to inspect
the object and choose the encoding scheme dynamically:>>>fromanyencoderimportDynamicEncoder,TypeTag>>>defi_care_about_keys(obj):..."""
...     If all the keys in the dictionary are strings, I want
...     to store the dictionary as msgpack. Otherwise, I want to
...     store it as bson. For some reason.
...     """...ifall(map(lambdax:isinstance(x,str),obj.keys())):...return'msgpack'...else:...return'bson'...>>>dict_tag=TypeTag(dict,i_care_about_keys)>>>str_dict=dict(a=1,b=2,c=3)>>>int_dict={1:'a',2:'b',3:'c'}>>>encoder=DynamicEncoder()>>>encoder.load_encoder_plugins()>>>encoder.register(dict_tag)>>>encoder.encode(str_dict)b'\x05\x83\x00\x00\x01\x83\xa1a\x01\xa1b\x02\xa1c\x03'>>>encoder.encode(int_dict)b'\x05\x88\x00\x00\x01\x00\x00\x00\x021\x00\x02\x00\x00\x00a\x00\x022\x00\x02\x00\x00\x00b\x00\x023\x00\x02\x00\x00\x00c\x00\x00'Custom TypesClasses can implement a method to specify how they should be
serialized. The method should return the name of the desired encoder:>>>fromanyencoderimportDynamicEncoder>>>classMyClass:...z=False......def_encoder_id(self):...ifself.z:...return'cloudpickle'...else:...return'dill'>>>my_cls=MyClass()...withDynamicEncoder()asencoder:...with_z_false=encoder.encode(my_cls)...my_cls.z=True...with_z_true=encoder.encode(my_cls)...>>>with_z_falseb'\x05\x81\x00\x00\x01\x80\x04\x95\xa8\x00\x00\x00\x00\x00\x00\x00\x8c\ndill._dill\x94\x8c\x0c_create_type\x94\x93\x94(h\x00\x8c\n_load_type\x94\x93\x94\x8c\tClassType\x94\x85\x94R\x94\x8c\x07MyClass\x94h\x04\x8c\x06object\x94\x85\x94R\x94\x85\x94}\x94(\x8c\n__module__\x94\x8c\x08__main__\x94\x8c\x01z\x94\x89\x8c\x07__doc__\x94N\x8c\r__slotnames__\x94]\x94ut\x94R\x94)\x81\x94}\x94h\x10\x89sb.'>>>with_z_trueb'\x05\x82\x00\x00\x01\x80\x04\x95\xb8\x00\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x19_rehydrate_skeleton_class\x94\x93\x94(\x8c\x08builtins\x94\x8c\x04type\x94\x93\x94\x8c\x07MyClass\x94h\x03\x8c\x06object\x94\x93\x94\x85\x94}\x94\x8c\x07__doc__\x94Ns\x87\x94R\x94}\x94(\x8c\n__module__\x94\x8c\x08__main__\x94\x8c\x01z\x94\x89\x8c\r__slotnames__\x94]\x94utR)\x81\x94}\x94h\x11\x88sb.'This doesn’t have to be a method; an attribute namedencoder_idwill also work.If that sounds like too much work for you, try theencode_withdecorator:>>>fromanyencoderimportDynamicEncoder,encode_with>>>@encode_with('dill')...classMyClass:...pass......my_cls=MyClass()...withDynamicEncoder()asencoder:...encoded=encoder.encode(my_cls)...>>>encodedb'\x05\x81\x00\x00\x01\x80\x04\x95\xb1\x00\x00\x00\x00\x00\x00\x00\x8c\ndill._dill\x94\x8c\x0c_create_type\x94\x93\x94(h\x00\x8c\n_load_type\x94\x93\x94\x8c\tClassType\x94\x85\x94R\x94\x8c\x07MyClass\x94h\x04\x8c\x06object\x94\x85\x94R\x94\x85\x94}\x94(\x8c\n__module__\x94\x8c\x08__main__\x94\x8c\x07__doc__\x94N\x8c\x0b_encoder_id\x94\x8c\x04dill\x94\x8c\r__slotnames__\x94]\x94ut\x94R\x94)\x81\x94.'Rather than implementing methods, classes can be registered like any
other type:>>>fromanyencoderimportDynamicEncoder,TypeTag>>>defevaluate_class(obj):...return'cloudpickle'ifobj.zelse'dill'...>>>classMyClass:...z=False...>>>type_tag=TypeTag(MyClass,evaluate_class)>>>my_cls=MyClass()>>>encoder=DynamicEncoder()>>>encoder.load_encoder_plugins()>>>encoder.register(type_tag)>>>encoder.encode(my_cls)b'\x05\x81\x00\x00\x01\x80\x04\x95\xa8\x00\x00\x00\x00\x00\x00\x00\x8c\ndill._dill < SNIP >>>>my_cls.z=True>>>encoder.encode(my_cls)b'\x05\x82\x00\x00\x01\x80\x04\x95\xb8\x00\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle < SNIP >EncodersBuiltin EncodersSeveral pre-built encoders are included:bsonbzip2cloudpickledillgzipjsonmsgpackorjsonpicklestrbyteujsonzlibCustom EncodersCustom encoders can be defined and registered for use. To create
a custom encoder, subclassAbstractEncoder:>>>fromanyencoderimportDynamicEncoder,TypeTag,AbstractEncoder,EncoderTag>>>classStrToUtf16(AbstractEncoder):...encoder_id=10......defencode(self,obj):...returnobj.encode('utf-16')......defdecode(self,data):...returndata.decode('utf-16')...>>>my_encoder=StrToUtf16()>>>encoder_tag=EncoderTag('str-to-utf-16',my_encoder)>>>encoder.register(encoder_tag)>>>encoder.register(type_tag)>>>encoder.encode('hello world')b'\x05\n\x00\x00\x01\xff\xfeh\x00e\x00l\x00l\x00o\x00\x00w\x00o\x00r\x00l\x00d\x00'NoteBy now you may have noticed that there’s some extra data included
in these outputs. More on that later.Considerations for Custom EncodersThey must subclassAbstractEncoderand overrideAbstractEncoder.encodeandAbstractEncoder.decode.Theencodemethod must return astrorbytesobject.Encoders must have a uniqueencoder_id. This should be
an integer0 <= encoder_id <= 127. If you find you need more
than 128 custom encoders, well, that’s just crazy talk.Encoders must be added to the registry and named by being
wrapped in aEncoderTagobject.Proxying EncodersTheAbstractEncoderclass has a built-in proxy pattern which can
be utilized to build a proxy ‘stack’ of encoders in order to perform
logging, inspection, and multi-step object manipulation:>>>fromanyencoderimportDynamicEncoder,EncoderTag,TypeTag>>>fromanyencoder.plugins.zlibimportZlibEncoder>>>fromanyencoder.plugins.strbyteimportStrByteEncoder>>>fromanyencoder.plugins.ujsonimportUJsonEncoder>>>zlib=ZlibEncoder()>>>strbyte=StrByteEncoder(proxy_to=zlib)>>>json_zlib=UJsonEncoder(encoder_id=1,proxy_to=strbyte)>>>encoder_tag=EncoderTag('json-zlib',json_zlib)>>>type_tag=TypeTag(dict,lambda_:'json-zlib')>>>data=dict(a=1,b=2,c=3)>>>withDynamicEncoder()asencoder:...encoder.register([encoder_tag,type_tag])...result=encoder.encode(data)...>>>resultb'\x05\x01\x00\x00\x01x\x9c\xabVJT\xb22\xd4QJR\xb22\xd2QJV\xb22\xae\x05\x00-=\x04\x87'Considerations for Proxying EncodersWhen building a proxy stack, theencoder_idis only relevant for
the bottom (first) encoder in the stack. The proxy stack counts as
a single encoder, and the first encoder in the stack needs a uniqueencoder_id. Theencoder_idcan be passed as an argument to
facilitate easily re-using existing classes in proxy stacks.A proxy ‘stack’ is itself registered as a unique encoder with a
uniqueencoder_id. Think of the whole stack as a single
encoder. As with other encoders, a proxy stack’sencodemethod must return eitherbytesorstrdata. However,
individual encoders in the stack needn’t do anything to manipulate
data at all, as long as the stacks’sencodemethod provides
data anddecodemethod can do something with that data.This allows you to do other useful things with indivudal encoders
in the stack, such as implementing callbacks, logging, heuristics,
object inspection, etc…Encoder Plugin LoadingSeveral pre-baked encoder plugins are included, and are loaded by theload_encoder_pluginsmethod. This method is called automatically
whenDynamicEncoder’s context manager is invoked:>>>frompprintimportpprint>>>fromanyencoderimportDynamicEncoder>>>withDynamicEncoder()asencoder:...types,encoders=encoder.registry.dump()...>>>pprint(encoders)[EncoderTag(name='bson',encoder=BSONEncoder(encode_kwargs={},decode_kwargs={},encoder_id=136,proxy_to=None)),EncoderTag(name='bzip2',encoder=Bzip2Encoder(encode_kwargs={},decode_kwargs={},encoder_id=137,proxy_to=None)),EncoderTag(name='cloudpickle',encoder=CloudPickleEncoder(encode_kwargs={},decode_kwargs={},encoder_id=130,proxy_to=None)),EncoderTag(name='dill',encoder=DillEncoder(encode_kwargs={'protocol':4},decode_kwargs={},encoder_id=129,proxy_to=None)),EncoderTag(name='gzip',encoder=GzipEncoder(encode_kwargs={},decode_kwargs={},encoder_id=144,proxy_to=None)),EncoderTag(name='json',encoder=JSONEncoder(encode_kwargs={},decode_kwargs={},encoder_id=133,proxy_to=None)),EncoderTag(name='msgpack',encoder=MessagePackEncoder(encode_kwargs={'use_bin_type':True},decode_kwargs={'raw':False},encoder_id=131,proxy_to=None)),EncoderTag(name='orjson',encoder=OrJsonEncoder(encode_kwargs={},decode_kwargs={},encoder_id=134,proxy_to=None)),EncoderTag(name='pickle',encoder=PickleEncoder(encode_kwargs={'protocol':4},decode_kwargs={},encoder_id=128,proxy_to=None)),EncoderTag(name='strbyte',encoder=StrByteEncoder(encode_kwargs={},decode_kwargs={},encoder_id=132,proxy_to=None)),EncoderTag(name='ujson',encoder=UJsonEncoder(encode_kwargs={},decode_kwargs={},encoder_id=135,proxy_to=None)),EncoderTag(name='zlib',encoder=ZlibEncoder(encode_kwargs={},decode_kwargs={},encoder_id=145,proxy_to=None))]NoteSeveral of the plugins require third-party libraries in order to
function.How It WorksLabelsAfter object encoding,anyencoderprepends a label to the data.
At decode time, the label is removed and read in order to determine
how to decode the data.For binary data, the label is 5 bytes in length:label_len|encoder_id|version_major|version_minor|version_microFor text data, the label is a small JSON dictionary.WarningBecause the data is modified to include the label, itmustbe decoded
withanyencoderin order to extract the label. Serializing an
object withanyencoderand then trying to decode the result with
the concrete serializer isguaranteedto fail.Encoder IDsBecauseencoder_idis limited to a single byte, it must be a
value between0and255. Values128through255are
reserved for the library, and therefore you should choose avaluewhere0 <= value <= 127when choosing theencoder_idfor a
custom encoder.