Package: binfootprint
binfootprint - unique serialization of python objectsWhy unique serializationWhen caching computationally expansive function calls, the input arguments (*args, **kwargs)
serve as key to look up the result of the function.
To perform efficient lookups these keys (often a large number of nested python objects) needs to be hashable.
Since python's build-in hash function is randomly seeded (and applies to a few data types only) it is not
suited for persistent caching.
Alternatively, standard hash functions, as provided by thehashlib library, can be used.
As they relay on  byte sequences as input, python objects need to be converted to such a sequence first.
Surely, python's pickle module provides such a serialization which, for our purpose, has the drawback that
the byte sequence is not guaranteed to be unique (e.g., a dictionary can be stored as different byte sequences,
as the order of the (key, value) pairs is irrelevant).The binfootprint module fills that gap.
It guarantees that a particular python object will have a unique binary representation which
can serve as input for any hash function.Quick startbinfootprint.dump(data)generate a unique binary representation
(binary footprint) ofdata.b=binfootprint.dump(['hallo',42])Its output can serve as suitable input for a hash function.hashlib.sha256(b).hexdigest()binfootprint.load(data)reconstructs the original python object.ob=binfootprint.load(b)Numpy array can be serialized.a=numpy.asarray([0,2.3,4])b=binfootprint.dump(a)Classes which implement__getstate__(pickle interface) or__bfkey__can be
serialized too.classPoint:def__init__(self,x,y):self.x=xself.y=ydef__getstate__(self):return[self.x,self.y]ob=Point(4,-2)b=binfootprint.dump(ob)If__bfkey__is implemented, it is used over__getstate__.New since version 1.2.0:functools.partialobjects can now be serialized too. This allows to cache a function which takes afunctools.partialas argument.defgaussian(x,a,sigma,x0):returna*math.exp(-(x-x0)**2/2/sigma**2)@binfootprint.util.ShelveCacheDec()defquad(f,x_min,x_max,dx):r=0x=x_minwhilex<x_max:r+=f(x)x+=dxreturndx*rg=functools.partial(gaussian,a=1,sigma=1,x0=-2.34)quad(g,x_min=-10,x_max=10,dx=0.001)cache decoratorUtilizing the unique binary representation of python objects, a persistent
cache for quite general functions is provided by theShelveCacheclass.
The decoratorShelveCacheDecmakes it really easy to use:@binfootprint.ShelveCacheDec(path='.cache')defarea(p):returnp.x*p.yparameter base classTo conveniently organize a set of parameters suitable as key for caching
you can subclassABCParameter. Why should you do that?The__bfkey__method ofABCParameterignores parameters that areNone.
This allows to extend your function interface without loosing access to
cached results from earlier stages.You can add informative information to the__non_key__member which
are not included in the binary representation of the parameter class.classParam(bf.ABCParameter):__slots__=["x","y","__non_key__"]def__init__(self,x,y,msg=""):super().__init__()self.x=xself.y=yself.__non_key__=dict()self.__non_key__["msg"]=msgWhich data types can be serializedPython'sfundamental data typesare supportedintegerfloat (64bit)complex (128bit)stringsbyte arraysspecial build-in constants:True,False,Noneas well as theirnested combinationby means of thenative data structurestuplelistdictionarynamedtuple.In addition, the following types are supported:numpyndarray:
The serialization makes use of numpy'sformat.write_array()function using version 1.0.functools.partialobjects (new since version 1.2.0)Furthermore, any class that implements__getstate__(python's pickle interface)can be serialized as well, given that the returned data from__getstate__can be serializedand the returned data is notNoneDistinction between objects is realized by adding the class name and the name of the module which defines
the class to the binary data.
This in turn allows to also reconstruct the original object by means of the__setstate__method.In case the__getstate__method is not suitable, you can implement__bfkey__which should return the necessary data to distinguish different objects.
The spirit of__bfkey__is very similar to that of__getstate__, although it is meant
for serialization only, and to for reconstruction the original object.Note that, if__bfkey__is implemented it will be used, regardless of__getstate__.Note: dumping older version is not supported anymore. If backwards compatibility is needed check out older
code from git. If needed converters should/will be written.be carefull with functionsSince a function objects seem to implement__getstate__which, however, returnsNone,
dumping a function will fail.Whether this makes sense, can be discussed.Implementing your own callable ore usingpartioalobjects can circumvent this.Installationpipinstall the latest version using pippip install binfootprintpoetryUsing poetry allows you to include this package in your project as a dependency.gitcheck out the code from githubgit clone https://github.com/richard-hartmann/binfootprint.gitdependenciespython3numpyHow to use the binfootprint moduledata serializationGenerating the binary footprint is done using thedump(obj)method.very simpleimportbinfootprintasbfbf.dump(['hallo',42])more compleximporthashlibimportbinfootprintasbfSIGMA_Z=0x34data={'Færøerne':{'area':(1399,'km^2'),'population':54000},SIGMA_Z:[[-1,0],[0,1]],'usefulness':None}b=bf.dump(data)print("MD5 check sum:",hashlib.md5(b).hexdigest())reconstruct serialized dataAlthough the primary focus of this module is the binary representation,
for reasons of convenience or debugging it might be useful restore the original
python object from the binary data.
Calling theload(bin_data)function achieves that task.importbinfootprintasbfdata=['hallo',42]b=bf.dump(data)data_prime=bf.load(b)print(data_prime)python objects -__getstate__Since__getstate__is assumed to uniquely represent the state of an
object by means of the returned data, it can be used to generate a unique binary
representation.importbinfootprintasbfclassPoint:def__init__(self,x,y):self.x=xself.y=ydef__getstate__(self):return[self.x,self.y]def__setstate__(self,state):self.x=state[0]self.y=state[1]ob=Point(4,-2)b=bf.dump(ob)Since__setstate__is implemented as well, the original object can be
reconstructed.ob_prime=bf.load(b)print("type:",type(ob_prime))# type: <class '__main__.Point'>print("member x:",ob_prime.x)# member x: 4print("member y:",ob_prime.y)# member y: -2implement__bfkey__if__getstate__is not suitedIn case__getstate__returns data which is not sufficient to uniquely label
an object or if the data cannot be serialized by the binaryfootprint module,
the method__bfkey__should be implemented.
It is expected to return serializable data which uniquely identifies the state
of the object.
Note that, if__bfkey__is present,__getstate__is ignored.Importantly, when deserializing the binary data from an object implementing__bfkey__, the python objectis not returned, since there is no__setstate__equivalent. Instead, the class name, the name of the module defining
the class and the data returned by__bfkey__is recovered.
This should not pose a problem, since the main focus of the binfootprint module is
the unique binary serialization of an object.
To ensure deserialization use python'spicklemodule.classPoint2(Point):def__bfkey__(self):return{'x':self.x,'y':self.y}ob=Point2(5,3)b=bf.dump(ob)ob_prime=bf.load(b)print("load on bfkey:",ob_prime)# load on bfkey: ('Point2', '__main__', {'x': 5, 'y': 3})numpy ndarraysNumpy'sndarrayare supported by relaying on numpy's binary serialization
usingformat.write_array().importbinfootprintasbfimportnumpyasnpa=np.asarray([0,1,1,0])b1=bf.dump(a)As expected, changing the shape or data type yield a different binary representationa2=a1.reshape(2,2)b2=bf.dump(a2)a3=np.asarray(a1,dtype=np.complex128)b3=bf.dump(a3)print("            MD5 of int array :",hashlib.md5(b1).hexdigest())# 949bfba1237c48007a066398f744a161print("MD5 of int array shape (2,2) :",hashlib.md5(b2).hexdigest())# e9049a19f82c6f282d65466a72360cd8print("        MD5 of complex array :",hashlib.md5(b3).hexdigest())# 2274ea54925d88ec4d53853050e55a82cachingWith the binaryfootprint module, caching function calls is straight forward.
An implementation of such a cache using python'sshelvefor persistent storage
is provided by theShelveCacheDecclass.@binfootprint.ShelveCacheDec()defarea(p):print(" * f(p(x={},y={})) called".format(p.x,p.y))returnp.x*p.yIt is safe to use theShelveCacheDecwith the same data location (path)on different functions, since the name of the function and the name of the
module defining the function determined the name of the underlying database.In addition to caching the decorator extends the function signature by the
kwarg_cache_flagwhich modifies the caching behavior as follows:_cache_flag = 'no_cache': Simple call offncwith no caching._cache_flag = 'update': Callfncand update the cache with recent return value._cache_flag = 'has_key': ReturnTrueif the call has already been cached, otherwiseFalse._cache_flag = 'cache_only': Raises aKeyErrorif the result has not been cached yet.p=Point(10,10)print("first call results in")print(area(p))# * f(p(x=10,y=10)) called# 100print("second call results in")print(area(p))# 100p=Point(10,11)print("f(p(10, 11)) is in cache?")print(area(p,_cache_flag='has_key'))# Falsepitfallsints and floatsSince the binary representation between ints and floats is different,1and1.0will be treated as different things.
This means that the cached value of a function call with an argument being1is
not found when passing1.0as argument.
Although the result of the function will most likely be the same.
Obviously, the same holds true for numpy array of differentdtype.Parameter classTha abstract base classABCParameterallows to conveniently manage a set
of parameters.Relevant parameters, explicitly specified as data member via__slots__mechanism, are returned by__bfkey__method (see above).
Their order in the__slots__definition is irrelevant.Importantly, class members are included only if they are notNone.
In this way a parameter class definition can be extended while still being
able to reproduce the binary footprint of an older class definition.If present, the class member__non_key__has a special meaning.
It is not included in the parameter-values list returned by__bfkey__.
It is expected to be dictionary-like and allows storing
additional / informative information.
This is also reflected by the string representation of the class.classParam(binfootprint.ABCParameter):__slots__=["x","y","__non_key__"]def__init__(self,x,y,msg=""):super().__init__()self.x=xself.y=yself.__non_key__=dict()self.__non_key__['msg']=msgp=Param(3,4.5)bfp=binfootprint.dump(p)print("{}\nhas hex hash value{}...".format(p,binfootprint.hash_hex_from_bin_data(bfp)[:6]))# x : 3# y : 4.5# --- extra info ---# msg :# has hex hash value 38dbe8...p=Param(3,4.5,msg="I told you, don't use x=3!")bfp=binfootprint.dump(p)print("{}\nhas hex hash value{}...".format(p,binfootprint.hash_hex_from_bin_data(bfp)[:6]))# x : 3# y : 4.5# --- extra info ---# msg : I told you, don't use x=3!# has hex hash value 38dbe8...