Package: cachable
cachingLibrary for easily caching computations that take a long time to run, e.g., training Keras models.InstallationOptionally, create and/or activate a virtual environment to install this package in. E.g.,conda create -n example_env python=3
conda activate example_envClone the caching repo to a location of your choice:git clone https://github.com/klasleino/caching.gitChange into the repo directory and install using pip:cd caching
pip install -e .UsageCaching utility is provided viadecorators.
The primary decorator is theCachabledecorator found in thecachablepackage.
This decorator is used on a function (either member or non-member) to specify that the value returned by the function should be cached.
If the function is called again with the same parameters, the result will be loaded from the cache rather than running the function again.For example, consider the following code:fromcachableimportCachable@Cachable(directory='cache')deff(a,b,c):returndict(a=a,b=b,c=c)# Call 1 (body of `f` is run).res=f(1,2,3)# Call 2 (loaded from cache).res=f(1,2,3)# Call 3 (body of `f` is run).res=f(4,5,6)In the first call, the function,f, will be run, and the result will be saved in a file in a directory called "cache".
The file name of the cached result will contain the parameters,a,b, andcand the values passed to them in call 1.
In the second call, the function will not be run, because an existing file with the given parameters exists; instead, the result will be loaded from the cached file.
Finally, in the third call, the parameters are different, thus the function will be run again, and a new cache entry made.Cached ObjectsObjects returned byCachablefunctions are wrapped as acachable.cached_objects.CachedObject.CachedObjectss can mostly be used the same as their non-wrapped counterparts, but the original object returned by the non-decorated function can be obtained from theobjfield of theCachedObject.
TheCachedObjectalso keeps track of the parameters used to create the object.Default ParametersParameters taking their default values will not be included in the name of the cached file.
This keeps file names shorter, and allows backwards compatibility with previously cached files, provided the default value is chosen appropriately.Non-primitive ArgumentsAny argument with a canonical string representation can be passed as an argument to aCachablefunction.
The argument,arg, is taken to be an unseen value ifstr(arg)does not match the string representation of any previous values passed.
This may present a problem for generic objects, as two objects may be the same, whilestr(obj1) != str(obj2).Any instance ofCachedObjecthas a canonical string representation, soCachedObjects can be given as parameters to other cached objects.For other custom objects, theCachableParamdecorator in thecachablepackage can be used.
This gives the decorated object a canonical string representation based on the arguments passed to the object's__init__function.
For example, a custom object can be used as a parameter to aCachablefunction using the following code:fromcachableimportCachable,CachableParam@CachableParam()classP(object):def__init__(a,b,c):self.a=aself.b=bself.c=c@Cachable(directory='cache')deff(p):returndict(a=p.a,b=p.b,c=p.c)# Call 1 (body of `f` is run).res=f(P(1,2,3))# Call 2 (loaded from cache).res=f(P(1,2,3))Note that the behavior of the caching may not be correct if theCachableParamis modified after it is constructed; therefore this should only be used on objects that are treated as immutable.Hidden ParametersIf aCachablefunction takes a parameter that for whatever reason should not affect the cached file name, the parameter can be hidden by beginning its name with an underscore, e.g.,fromcachableimportCachable@Cachable(directory='cache')deff(a,b,c,_msg):print(_msg)returndict(a=a,b=b,c=c)# Call 1 (body of `f` is run).res=f(1,2,3,'hello')# Call 2 (loaded from cache).res=f(1,2,3,'world')RefreshingIf for whatever reason, it is desired for the body of aCachablefunction to be rerun with a set of parameters previously used, a special parameter called_refreshcan be used, e.g.,fromcachableimportCachable@Cachable(directory='cache')deff(a,b,c):returndict(a=a,b=b,c=c)# Call 1 (body of `f` is run).res=f(1,2,3)# Call 2 (body of `f` is run).res=f(1,2,3,_refresh=True)LoadersBy default, the results ofCachablefunctions are saved using python'spickleutility.
If another format is preferred, an alternative loader can be used.
Several default implementations of alternative loaders can be found in thecachable.loaderspackage.
For example, to cache a Keras model, the following could be used to save the resulting model as anh5file using Keras'model.save():fromcachableimportCachablefromcachable.loadersimportKerasModelLoader@Cachable(name='my_cached_model',directory='cache',loader=KerasModelLoader())defcreate_and_train_model(**model_hyperparameters):# Create and train a model...returnmodel