Package: doorget
doorgetPython package which memoizes functions and supports data dependencies across them.
With thecachedecorator the memoization of your code is low touch.MemoizationDefinitionIn computing, memoization or memoisation is an optimization technique used primarily to speed up computer programs by storing the results of expensive function calls to pure functions and returning the cached result when the same inputs occur again. Memoization has also been used in other contexts, such as in simple mutually recursive descent parsing.WikipediaExamplefromdoorgetimportcacheimportpandaaspd@cachedeffetch(name:str)->pd.DataFrame:pass# do itfoo=fetch('foo')# The function is called# get itfoo_again=fetch('foo')# The previous returned data is read from the cache and the function is not called.# do itbar=fetch('bar')# The function is called because the input is not known yetStorage modesThe package propose you 3 built in storage modes and the ability to proivde your customized storages.Memory: The fastest mode.Disk: The unlimited mode.IdentityThe tracker mode for datadependenciesandcascades.Custom: Your best mode.MemoryThe memory storage use the RAM memory to memoize data.This is the default storage mode. It is the fastest way to retreive  data from an existing input.
But this mode has 2 caveats. The first caveat is that the data is cached within your current process only. Once your process ends all cached data are lost. It avoids you sharing the cached data between processes neither. The second caveat is the limited memory, constrained by your hardware to a couple of Giga Bytes maximum.fromdoorgetimportcache,StorageModeimportpandaaspd@cache# Memory by defaultdeffoo(name:str)->pd.DataFrame:pass@cache(mode=StorageMode.Memory)defbar(name:str)->pd.DataFrame:passDiskThe disk storage use the disk to memoize data.This is the most common usage of the memoization. It is slower than theMemorymode because the data is read from
the disk but it solves the 2 caveats. The built inDiskmode usesparquetformat if a pandasDataFrameis returned,pickleotherwise.If no cache folder is specified, a global folder is used instead. When the global folder is prefered, a sub folder is created by function from its name and module to guaranty an unique storage location.The global folder can change at any time with the functionsetup_disk_storage.fromdoorgetimportcache,StorageMode,setup_disk_storageimportpandaaspd@cache(mode=StorageMode.Disk)# Use the global folderdeffetch_default(name:str)->pd.DataFrame:pass@cache(cache_folder='./my_custom/folder/bar')# Overrides the global folderdeffetch_custom(name:str)->pd.DataFrame:pass# do itfetch_default('foo')# do itfetch_custom('bar')setup_disk_storage('./my_default/folder')# do itfetch_default('foo')# Function is called because the cache folder changed# get itfetch_custom('bar')# The function is not called because the custom folder is unchangedCustomThe custom storage allows you to provide you own cache where the data is memoized.When you specify this mode you have to specify thestorageargument. This argument takes any kind of storage which could fits better your use case, for performances, infrastructure, sharing, ... purpose.Your custom storage imlplementation must inherit from theStorageclass.fromtypingimportAny,ListfromdataclassesimportdataclassfromdoorgetimportCacheKey@dataclassclassStorage:name:str# Core functionsdefcontains(self,key:CacheKey)->bool:passdeffetch(self,key:CacheKey)->Any:passdefstore(self,key:CacheKey,data:Any)->None:pass# Cache management helpersdefclear(self)->None:passdefremove(self,key:CacheKey)->bool:passdefkeys(self)->List[CacheKey]:passData dependencyThe added value from a simple memoized package is the carrying of data dependencies.When a complex object, like a pandasDataFrame, is passed as an argument, a simple memoization won't work.
Withdoorgetwhen a complex object is passed as an argument, it is not used directly to build the key, but substitued with its own memoized key.fromdoorgetimportcacheimportpandaaspd@cachedeffetch(name:str)->pd.DataFrame:pass@cachedefsummarize(by:str,df:pd.DataFrame)->pd.DataFrame:passdf=fetch('foo')# key for df: <fetch('foo')>summary=summarize('date',df)# key for summary: <summarize(<fetch('foo')>, 'daily')>Under the hood a memoized function keep a track of any object reference (Identity in python) returned by attaching its
memoized key. If the function returns aTuple, the contained items are tracked as well.Cascade data identitySometime it is faster to transform a data than memoizing it. If this transformed data is used as an argument for another memoized function, the data dependency is lost. To avoid breaking a dependency, you can isolate your
data transformation within a memoized function with theIdentitystorage mode. It will carry no additional storage
than what your program is currently using. Under the hood the data graph is tracked by their references orIdentityin python.fromdoorgetimportcache,StorageModeimportpandaaspd@cache(mode=StorageMode.Identity)deftransform(x:pd.DataFrame)->pd.DataFrame:returnx.copy()df=fetch('foo')# key for df: <fetch('foo')>df_copy=transform(df)# key for df_copy: <transform(<fetch('foo')>)>summary=summarize('date',df_copy)# key for summary: <summarize(<transform(<fetch('foo')>)>, 'daily')>Administrate your cachesThe package provides you some backdoors to administrate your caches and more precisely your storages.list_memory_storages()list_disk_storages()list_custom_storages()Each memoized function has its owned storage list. From a storage you can list all the contained keys,remove some items or clear them all. A function call, is stored in aCacheKeyobject. ACacheKeyobject
contains the function name with its arguments combination as aTuple. If the key has data depedencies,
the dependencies are nested by usingTuplerecursively.