Package: checkpointing
checkpointingPersistent cache for Python functions.DocumentationUser ManualIntroductioncheckpointingprovides a decorator which allows you to cache the return value of apure function[1],
by default as a pickle file on the disk.
When the function is called later with the same arguments, it automatically skips the function execution,
retrieves the cached value and return.For example,fromcheckpointingimportcheckpoint@checkpoint()defcalc(a,b):print(f"calc is running for{a},{b}")returna+bif__name__=="__main__":result=calc(1,2)print(f"result:{result}")Run this script, and the output will becalc is running for 1, 2
result: 3Now the return value has been cached, and if you rerun this script, the output will beresult: 3The execution ofcalcis skipped, but the result value is retrieved from the disk and returned as expected.However, if the function call context has changed, the function will be re-executed and return the new value.
For example,if it is passed with different arguments, e.g.calc(1, 3),calcwould rerun and return4if the code logic has changed, e.g.return a - b,calcwould rerun and return-1Thecheckpointhas a built-in wise strategy to decide when it needs or doesn't need to re-execute the function.
More details are discussed inBehavior on Code Change.
This is also the main advantage ofcheckpointingcompared to other similar packages,
seeComparison with similar packages.!!! attention
However, there are some cases where the checkpoint cannot correctly make the rerun decision.
Please read through theCaveatspage and avoid those patterns.Although the package focuses on persisting the cache across different executions,
it also works if you call the same function multiple times within one execution.Use casesThe built-incheckpointis designed for projects thatruns in a local development environmentinvolves repeatedly executing long-runningpure functions[1]
on the same set of argumentsare somewhat "experimental", so it involves a lot of code changes back and forthFor example, such use cases are very common in the preliminary stage of machine learning projects.InstallationThis package is available onPyPI, and can be installed withpip.$pipinstallcheckpointingBasic usageCreate a checkpointImport thecheckpointfrom this package and use it as the decorator of a function
(notice the()aftercheckpoint)fromcheckpointingimportcheckpoint@checkpoint()deffoo():return0After that,foowill be automatically cached, skipped,
or re-executed as described previously.
You can callfooin the same way as you normally would.Configure the checkpointCache directoryBy default, the results are saved as pickle files in./.checkpointing/,
if you want to store them elsewhere, you can do@checkpoint(directory="other_dir")Behavior on internal errorDuring the execution, there could be unexpected errors within the checkpoint.
When this happens, the default behavior is to give you a warning,
and just rerun the function without the caching stuff.
This ensures that your code won't fail because of using this package.
However, you can change this behavior with theon_erroroption.@checkpoint(on_error="raise")This will terminate the function call and raise the internal error.@checkpoint(on_error="ignore")This will rerun the function when an internal error occurs without raising any warning.Pickle protocolThe function return value will be saved with the built-inpicklemodule.
We useprotocol 5by default for all Python versions,
in favor of its ability to efficiently handle large data.
However, if you want to change the protocol, you could use thecache_pickle_protocoloption.importpickle@checkpoint(cache_pickle_protocol=pickle.DEFAULT_PROTOCOL)Global settingBy modifying a global dictionary, you can change the configurations for all checkpoints.fromcheckpointingimportdefaultsimportpickledefaults["cache.filesystem.directory"]="other_dir"defaults["checkpoint.on_error"]="ignore"defaults["cache.pickle_protocol"]=pickle.DEFAULT_PROTOCOLPlease set this at the top level of your module/script, before you create anycheckpoint.Further customizationIf you want more flexibility, such as storing the cache not as a pickle file,
or ignore/consider some additional aspects of the function call context,
please seeExtending the Checkpointfor details.Force rerun a checkpointYou can force rerun a checkpointed function withfoo.rerun(arg)wherefoois the decorated function.
This would be equivalent to directly invokingfoo(arg).
The return value of this rerun will be cached to the disk and overwrite the previous one, if it exists.This is useful if some factors that would affect the function return value have changed,
butcheckpointfailed to capture this difference, as described in theCaveats.Usage notesPlease be aware thatSince the function will be skipped if it was cached before, user shouldn't mutate an argument in the function body
(as required by the definition of pure function)If the project involves randomness, it's the user's responsibility to set the random seed or random state,
such that the arguments and reference global variables of the cached function are identicalThe built-in strategy to determine if a function needs to be re-executed is imperfect.
Please seeCaveats,
and avoid those cases when the rerun condition cannot be correctly determined.Footnotes[1]: We take the alternative definition of the "pure function", meaning that it only has property 2:
"the function has no side effects (no mutation of local static variables, non-local variables,
mutable reference arguments or input/output streams)".
We do allow the return value to vary due to changes in non-local variables and other factors,
as it's often the case in project development.