Package: perscache
perscacheAn easy to use decorator for persistent memoization: likefunctools.lrucache, but results persist between runs and can be stored in any format to any storage.fromperscacheimportCachecache=Cache()@cachedefget_data():...ContentsUse casesFeaturesGetting startedMake your own serialization and storage back-endsAPI ReferenceUse casesCache the results of a function that uses a lot of resources: runs for a long time, consumes a lot of traffic, uses up paid API calls etc.Speed up retreival of data that doesn't change often.Inspect the results of a decorated function while debugging.Build a simple data store.FeaturesCachingEasy to swap out the cache configuration when switching environments.Async functions supported.Time-to-live (TTL) support - automatically invalidate cache entries after a certain time.Automatic cache invalidation when the decorated function arguments or code have been changed.You can ignore changes in certain arguments of the decorated function.Serialization and storageVarious serialization formats: JSON, YAML, pickle, Parquet, CSV etc.Various storage backends: local disk, GCS (Google Cloud Storage) and others to be implemented soon (S3,Azure Blob Storage).Serialization and storage are separated into two different classes, so that you can mix various serialization formats and storage back-ends as you like - JSON to local storage, Pickle to AWS, Parquet to Google Cloud Storage etc.You can set default serialization format and storage backend and then change them on a per-function basis.You can easily add new serialization formats and storage back-ends.Local storage is file-based, so you can use human-readable serialization (JSON, YAML, CSV) and inspect cached results.Automatic cleanup: least recently used results can be removed from storage when the total storage size exceeds a given threshold.Getting startedInstallationpipinstallperscacheThere are also some optional dependencies you need to install to use some of the features:yaml- for YAML serializationpyarrow- for Parquet serializationpandas- for CSV serializationgcsfs- for GCS storageBasic usagefromperscacheimportCachecache=Cache()counter=0@cachedefget_data():print("Fetching data...")globalcountercounter+=1return"abc"print(get_data())# the function is called# Fetching data...# abcprint(get_data())# the cache is used# abcprint(counter)# the function was called only once# 1Changing parameters or the code of the function invalidates the cache@cachedefget_data(key):print("The function has been called...")returnkeyprint(get_data("abc"))# the function has been called# The function has been called...# abcprint(get_data("fgh"))# the function has been called again# The function has been called...# fghprint(get_data("abc"))# using the cache# abc@cachedefget_data(key):print("This function has been changed...")returnkeyprint(get_data("abc"))# the function has been called again# This function has been changed...# abcNOTE:perscachehashes the function arguments, its code and the name of the class of the serializer, so that the cache is invalidated when any of these change. You can use theignore_argsparameter toignore changes in certain argumentsof the decorated function.However, if you change the code of the serializer, the cache is not invalidated. (This is because we cannot hash the code of the serializer when in was made with the factory functionmake_serializer. SeeMake your own serialization and storage back-endsfor more details.)Setting the expiry time of the cacheimportdatetimeasdt@cache(ttl=dt.timedelta(days=1))defget_data():"""This function will be cached for 1 dayand called again after this period expires."""...Ignoring certain argumentsBy specifying the arguments that should be ignored, you can still use the cache even in the values of these arguments have changed.@cache(ignore="ignore_this")defget_data(key,ignore_this):print("The function has been called...")returnkeyprint(get_data("abc","ignore_1"))# the function has been called# The function has been called...# abc# using the cache although the the second argument is differentprint(get_data("abc","ignore_2"))# abcChanging the default serialization format and storage backend# set up serialization format and storage backendcache=Cache(serializer=JSONSerializer(),storage=GoogleCloudStorage("/bucket/folder"))...# change the default serialization format@cache(serialization=PickleSerializer())defget_data(key):...Alternating cache settings depending on the environmentimportosfromperscacheimportCache,NoCachefromperscache.storageimportLocalFileStorageifos.environ.get["DEBUG"]:cache=NoCache()# turn off caching in debug modeelse:cache=(GoogleCloudStorage("/bucket/folder")ifos.environ.get["GOOGLE_PROJECT_NAME"]# if running in the cloudelseLocalFileStorage())@cachedeffunction():...Inspecting cached resultsUse human-readable serialization (JSONSerializer,YAMLSerializer,CSVSerializer) and a file storage (LocalFileStorage,GoogleCloudStorage) to inspect cached results.Automatic cleanupWhen usingLocalFileStorage(max_size=...)orGoogleCloudStorage(max_size=...), the least recently used cache entries are automatically removed to keep the total cache size with themax_sizelimit.Make your own serialization and storage backendsSerializersAlthough you can use the standardCloudPickleSerializer()for almost any type of data, sometimes you want to inspect the results of a decorated function by lookin into the cache files. This requires the data to be serialized in a human-readable format. But the included human-readable serializers (JSONSerializer(),YAMLSerializer(),CSVSerializer()) sometimes cannot process complex objects.To see which serializers are compatible with which data types, see thecompatibility.pyfile.That's when making your own serializer comes in handy.To do this, you should:Derive your own serialization classe from the abstractSerializerclass and override the abstract methods. You should also provide theextensionclass variable that specifies the file extension.Use your class with theCacheclass.fromperscache.serializersimportSerializerclassMySerializer(Serializer):extension="data"defdumps(self,data:Any)->bytes:...defloads(self,data:bytes)->Any:...cache=Cache(serializer=MySerializer())Serializer factoryYou can also use theperscache.serializers.make_serializer()function to create a serializer for a given data type.importpyrogramfromperscache.serializersimportmake_serializerPyrogramSerializer=make_serializer("PyrogramSerializer","pyro",dumps_fn=lambdadata:str(data).encode("utf-8"),loads_fn=lambdadata:eval(data.decode("utf-8")),)cache=Cache(serializer=PyrogramSerializer())@cacheasyncdefsome_pyrogram_func()->pyrogram.Message:...Storage back-endsMaking a custom storage backed is similar:classMyStorage(Storage):defread(self,path,deadline:datetime.datetime)->bytes:"""Read the file at the given path and return its contents as bytes.If the file does not exist, raise FileNotFoundError. If the file isolder than the given deadline, raise CacheExpired."""...defwrite(self,path,data:bytes)->None:"""Write the file at the given path."""...cache=Cache(storage=MyStorage())You can also derive your storage class fromperscache.storage.FileStorageif you are building a filesystem-based storage back-end. Refer to thestorage.pyfile for more information.