Package: cacheables
CacheablesCacheables is a Python package that makes it easy to cache function outputs.Cacheables is well suited to building efficient data workflows, because:functions will only recompute if their inputs have changed.everything is versioned: the functions, the inputs and the outputs.the cache is reused between different processes/executions (stored onDiskCacheby default).cached outputs are readable since you choose the file format (PickleSerializeris just a default).InstallpipinstallcacheablesBasic Example@cacheableis the decorator that makes a function cacheable.# basic_example.pyfromcacheablesimportcacheablefromtimeimportsleep@cacheabledeffoo(text:str)->int:sleep(1)# simulate a long running functionreturnlen(text)if__name__=="__main__":foo("hello")withfoo.enable_cache():foo("world")# python basic_example.py  # 2 seconds# python basic_example.py  # 1 seconds (foo("world") used cache)When the cache is enabled on a function, the following happens:aninput_keywill be calculated from the function argumentsif theinput_keyexists in the cachethe output will be loaded from the cacheusingcache.readand thenserializer.deserializeand the output will be returnedif theinput_keydoesn't exist in the cachethe original function will execute to get an outputthe output will be dumped in the cacheusingserializer.serializeand thencache.writeand the output will be returnedStandard ExampleCacheables is well suited to building efficient data workflows.As a simple example, let's assume we have a data workflow that processes a
string by removing the vowels, reversing the output, and then finally
concatenating that output with the original string. We'll assume that two of
these steps are computationally expensive (remove_vowelsandconcatenate),
so we decorate them with@cacheable.After running the workflow twice (showing that the cached results are used), we
modify the workflow by removing thereversestep. Onlyconcatenateis run on
the third workflow execution, which is much more efficient than running the
whole workflow (includingremove_vowels) again.# standard_example.pyfromcacheablesimportcacheable,enable_all_cachesfromtimeimportsleep@cacheabledefremove_vowels(text:str)->str:sleep(1)# simulate a long running functionreturn''.join([charforcharintextifcharnotin"aeiou"])defreverse(text:str)->str:returntext[::-1]@cacheabledefconcatenate(reversed_text:str,text:str)->str:sleep(1)# simulate a long running functionreturn(reversed_text+text)defrun_workflow(text:str)->int:t=remove_vowels(text)t=reverse(t)output=concatenate(t,text)returnoutputif__name__=="__main__":enable_all_caches()run_workflow("cache this")# 2 secondsrun_workflow("cache this")# 0 secondsdefrun_workflow(text:str)->int:t=remove_vowels(text)# t = reverse(t)  # removedoutput=concatenate(t,text)returnoutputrun_workflow("cache this")# 1 second# python standard_example.py  # 5 seconds# python standard_example.py  # 0 seconds (current and previous are still both cached)Advanced ExampleCacheables has many other features, a few of which are shown below.# advanced_example.pyfromcacheablesimportcacheable,enable_all_caches,enable_loggingfromcacheables.cachesimportDiskCachefromcacheables.serializersimportJsonSerializerfromtimeimportsleep@cacheable(function_id="example",cache=DiskCache(base_path="~/.cache"),serializer=JsonSerializer(),exclude_args_fn=lambdae:ein["verbose"])deffoo(text:str,verbose:bool=False)->int:sleep(1)# simulate a long running functionreturnlen(text)if__name__=="__main__":enable_all_caches()enable_logging()foo("cache this")# 1 secondsfoo("cache this",verbose=True)# 0 seconds# manually write output to cacheinput_id=foo.get_input_id("and cache that")foo.dump_output(14,input_id)foo("and cache that")# 0 seconds# manually read output from cacheinput_id=foo.get_input_id("cache this")foo.load_output(input_id)# 0 seconds# show output path in cachefoo.get_output_path(input_id)# ~/.cache/functions/example/inputs/cf5b2ab47064bd0e/aab3238922bcc25a.json# only use certain outputs in cache, recompute otherswithfoo.enable_cache(filter=lambdaoutput:output<=10):foo("cache this")# 0 secondsfoo("and cache that")# 1 seconds# overwrite cachewithfoo.enable_cache(read=False,write=True):foo("cache this")# 1 seconds# python advanced_example.py  # 3 seconds# python advanced_example.py  # 2 seconds (first foo("cache this") used cache)PickleSerializer & DiskCacheWhen you use@cacheablewithout any argument,PickleSerializerandDiskCachewill be used by default.After executing a function likefoo("hello")with the cache enabled, you can
expect to see the following files on disk:<cwd>/.cacheables
└── functions
    └── <function_id>
        └── inputs
            └── <input_id>
                ├── <output_id>.pickle
                └── metadata.jsonfunction_idAnfunction_iduniquely identifies a function. Unless specified using thefunction_idargument tocacheable, thefunction_idwill take the following form:module.submodule:foo.input_idAninput_iduniquely identifies a set of inputs to a function. We assume that changes to the inputs of a function will result in a change to the output of the function. Under the hood, eachinput_idis created by first hashing each individual input argument (which is itself cached!) and then hashing all of the argument hashes into a single hash.output_idAnoutput_iduniquely identifies an output to a function. Similar to theinput_id, it is a hash of the function's output.Other DocumentationSee theofficial documentationfor more details.Start by wrapping your function with the@cacheabledecorator.@cacheabledeffoo(text:str)->int:sleep(10)# simulate a long running functionreturnlen(text)Customization is possible by passing in arguments to the decorator.@cacheable(function_id="example",cache=DiskCache(base_path="~/.cache"),serializer=JsonSerializer(),exclude_args_fn=lambdae:ein["verbose"])deffoo(text:str,verbose:bool=False)->int:sleep(10)# simulate a long running functionreturnlen(text)See the@cacheabledocstring for more details.CachingUsefoo.enable_cache()to enable the cache on a single function orenable_all_cachesto enable the cache on all functions.@cacheabledeffoobar(text:str)->int:sleep(10)# simulate another long running functionreturnlen(text)foo.clear_cache()foo("hello")# returns after 10 secondsfoo("hello")# returns after 10 secondsfoo.enable_cache()foo("hello")# returns after 10 seconds (writes to cache)foo("hello")# returns immediately (reads from cache)foobar("hello")# returns after 10 secondsfoobar("hello")# returns after 10 secondsenable_all_caches()foobar("hello")# returns after 10 seconds (writes to cache)foobar("hello")# returns immediately (reads from cache)You can also use both of these as context managers, if you only want to enable
the cache temporarily within a certain scope.foo.clear_cache()foobar.clear_cache()foo("hello")# returns after 10 secondsfoo("hello")# returns after 10 secondswithfoo.enable_cache():foo("hello")# returns after 10 seconds (writes to cache)foo("hello")# returns immediately (reads from cache)foo("hello")# returns after 10 secondswithfoo.enable_cache(),bar.enable_cache():foo("hello")# returns immediately (reads from cache)foobar("hello")# returns after 10 seconds (writes to cache)foobar("hello")# returns immediately (reads from cache)foo("hello")# returns after 10 secondsfoobar("hello")# returns after 10 secondswithenable_all_caches():foo("hello")# returns immediately (reads from cache)foobar("hello")# returns immediately (reads from cache)foo("hello")# returns after 10 secondsfoobar("hello")# returns after 10 secondsCache SettingWhen a cacheable function is called afterenable_cache, the cache will be
read from and written too. Sometimes you might need to leave the results in the
cache untouched, or even overwrite the results in the cache. You can do this by
specifying thereadandwritearguments.foo.enable_cache(read=False,write=True)foo("hello")# foo called, and result added to cachefoo("hello")# foo called, and result re-added to cacheYou have three levels of cache settings:Function: controlled byfoo.enable_cache/foo.disable_cacheGlobal: controlled byenable_all_caches/disable_all_cachesEnvironment: controlled byCACHEABLES_ENABLED/CACHEABLES_DISABLEDWhen nothing is explicitly enabled/disabled (i.e. default), the cache will be disabled so that the cacheable function runs without any caching. Whenanylevel is explicitly set to disabled, the cache will be disabled, regardless of the other level settings (even if they are explicitly set to enabled).Output loadOften you just want to load a result from the cache, but not execute it.
You can do this by using theload_outputmethod.input_id=foo.get_input_id("hello")output=foo.load_output(input_id)# will error if result is not in cacheOutput dumpSome more advanced use-cases might want to manually write results to the cache (e.g. batched processing). You can do this by using thedump_outputmethod.input_id=foo.get_input_id("hello")output=foo.dump_output(5,input_id)Developmentpoetryinstall
poetryruntasktest# pytestpoetryruntaskformat# blackpoetryruntasklint# ruffUse pre-commit to automatically format and lint before each commit.pre-commitinstall