Package: xetcache
XetCacheXetCache provides persistent caching of long running functions or
jupyter notebook cells.The cache can be stored on local disk together with your notebooks and
code (and using LFS, or the git-xet extension to store it with your git
repo). Or alternatively, it can be fully managed by theXetHubservice to easily share the cache with your
collaborators.Table of ContentsInstallSetupSetup With LocalSetup with Git StorageSetup With XetHubAuthenticationCommand LineEnvironment VariableIn PythonUsageUsage For Jupyter NotebooksUsage For Function CachingUsage For Function Call
CachingLicenseInstallpip install xetcacheOr to install from source from GitHub:pip install git+https://github.com/xetdata/xetcache.gitSetupSetup With LocalNo additional set up needed. See Usage below.Setup with Git StorageIf using LFS, you can just directly commit and push the cache files in
thexmemofolder.However, if using GitHub we recommend the use ofXetHub’s extensions to
GitHubfor performance and the ability to
lazily fetch cached objects.For instance, a repository with the XetHub extension will allow you to
lazy clone the repository withgit xet clone --lazy [repo]which will avoid fetching any large cached objects until they are
actually needed.Setup With XetHubThe use of the fully managed XetHub service provides more powerful data
deduplication capabilities that allows similar objects to be stored or
loaded, without needing to upload or download everything. We can also
deploy caches near your compute workloads to accelerate dataloading by
over 10x.AuthenticationSignup onXetHuband obtain a
username and access token. You should write this down.There are three ways to authenticate with XetHub:Command Linexetlogin-e<email>-u<username>-p<personal_access_token>Xet login will write authentication information to~/.xetconfigEnvironment VariableEnvironment variables may be sometimes more convenient:exportXET_USER_EMAIL=<email>exportXET_USER_NAME=<username>exportXET_USER_TOKEN=<personal_access_token>In PythonFinally if in a notebook environment, or a non-persistent environment,
we also provide a method to authenticate directly from Python. Note that
this must be the first thing you run before any other operation:importpyxetpyxet.login(<username>,<personal_access_token>,<email>)UsageOptional: to cache on XetHub, you need to run:import xetcache
xetcache.set_xet_project([give a project name here])Usage For Jupyter NotebooksFor Jupyter notebooks, run the following command to load the extensionimportxetcacheAfter which adding the following line to the top of a cell%%xetmemoinput=v1,v2output=v3,v4will cache the specified output variables (v3,v4 here) each time it is
called. If called later with the same input values for v1,v2, the cached
value is returned and not reevaluated. The cache is persistent across
Python runs.By default, the output will only be cached if the cell takes longer the
3 seconds to run. “always=True” can be added to the xetmemo arguments to
ignore the runime and to always cache:%%xetmemo input=v1,v2 output=v3,v4 always=TrueNote that inputs can be anything picklable including functions.A key parameter can be added to group the stored objects together.
Objects stored with one key will not be retrievable with a different key%%xetmemoinput=v1,v2output=v3,v4always=Truekey=experiment1Usage For Function CachingTo cache the output of a function:fromxetcacheimportxetmemo@xetmemodefslowfunction(arg1,arg2):...# Stores with a key@xetmemo(key="hello")defslowfunction(arg1,arg2):...By default, the output will only be cached if the cell takes longer the
3 seconds to run. “always=True” can be added to the xetmemo arguments to
ignore the runtime and to always cache:# This will always cache irrespective of runtime@xetmemo(always=True)defslowfunction(arg1,arg2):...Usage For Function Call CachingTo cache a function call:defslowfn(x):..dostuff..# caches the call to slowfn with argument xxeteval(slowfn,x)# Stores with a keyxeteval("key",slowfn,x)By default, the output will only be cached if the cell takes longer the
3 seconds to run.xeteval_alwayscan be used instead to ignore the
runtime and to always cache:# Store even if function is quick to runxeteval_always(quickfn,x)# Store with a key and to always store even the function is quick to runxeteval_always("key",quickfn,x)LicenseBSD 3