Package: graphchain
GraphchainWhat is graphchain?Graphchain is likejoblib.Memoryfor dask graphs.Dask graph computationsare cached to a local or remote location of your choice, specified by aPyFilesystem FS URL.When you change your dask graph (by changing a computation's implementation or its inputs), graphchain will take care to only recompute the minimum number of computations necessary to fetch the result. This allows you to iterate quickly over your graph without spending time on recomputing previously computed keys.Source:xkcd.com/1205/The main difference between graphchain and joblib.Memory is that in graphchain a computation's materialised inputs arenotserialised and hashed (which can be very expensive when the inputs are large objects such as pandas DataFrames). Instead, a chain of hashes (hence the name graphchain) of the computation object and its dependencies (which are also computation objects) is used to identify the cache file.Additionally, the result of a computation is only cached if it is estimated that loading that computation from cache will save time compared to simply computing the computation. The decision on whether to cache depends on the characteristics of the cache location, which are different when caching to the local filesystem compared to caching to S3 for example.Usage by exampleBasic usageInstall graphchain with pip to get started:pipinstallgraphchainTo demonstrate how graphchain can save you time, let's first create a simple dask graph that (1) creates a few pandas DataFrames, (2) runs a relatively heavy operation on these DataFrames, and (3) summarises the results.importdaskimportgraphchainimportpandasaspddefcreate_dataframe(num_rows,num_cols):print("Creating DataFrame...")returnpd.DataFrame(data=[range(num_cols)]*num_rows)defexpensive_computation(df,num_quantiles):print("Running expensive computation on DataFrame...")returndf.quantile(q=[i/num_quantilesforiinrange(num_quantiles)])defsummarize_dataframes(*dfs):print("Summing DataFrames...")returnsum(df.sum().sum()fordfindfs)dsk={"df_a":(create_dataframe,10_000,1000),"df_b":(create_dataframe,10_000,1000),"df_c":(expensive_computation,"df_a",2048),"df_d":(expensive_computation,"df_b",2048),"result":(summarize_dataframes,"df_c","df_d")}Usingdask.getto fetch the"result"key takes about 6 seconds:>>>%timedask.get(dsk,"result")CreatingDataFrame...RunningexpensivecomputationonDataFrame...CreatingDataFrame...RunningexpensivecomputationonDataFrame...SummingDataFrames...CPUtimes:user7.39s,sys:686ms,total:8.08sWalltime:6.19sOn the other hand, usinggraphchain.getfor the first time to fetch'result'takes only 4 seconds:>>>%timegraphchain.get(dsk,"result")CreatingDataFrame...RunningexpensivecomputationonDataFrame...SummingDataFrames...CPUtimes:user4.7s,sys:519ms,total:5.22sWalltime:4.04sThe reasongraphchain.getis faster thandask.getis because it can loaddf_banddf_dfrom cache afterdf_aanddf_chave been computed and cached. Note that graphchain will only cache the result of a computation if loading that computation from cache is estimated to be faster than simply running the computation.Runninggraphchain.geta second time to fetch"result"will be almost instant since this time the result itself is also available from cache:>>>%timegraphchain.get(dsk,"result")CPUtimes:user4.79ms,sys:1.79ms,total:6.58msWalltime:5.34msNow let's say we want to change how the result is summarised from a sum to an average:defsummarize_dataframes(*dfs):print("Averaging DataFrames...")returnsum(df.mean().mean()fordfindfs)/len(dfs)If we then ask graphchain to fetch"result", it will detect that onlysummarize_dataframeshas changed and therefore only recompute this function with inputs loaded from cache:>>>%timegraphchain.get(dsk,"result")AveragingDataFrames...CPUtimes:user123ms,sys:37.2ms,total:160msWalltime:86.6msStoring the graphchain cache remotelyGraphchain's cache is by default./__graphchain_cache__, but you can ask graphchain to use a cache at anyPyFilesystem FS URLsuch ass3://mybucket/__graphchain_cache__:graphchain.get(dsk,"result",location="s3://mybucket/__graphchain_cache__")Excluding keys from being cachedIn some cases you may not want a key to be cached. To avoid writing certain keys to the graphchain cache, you can use theskip_keysargument:graphchain.get(dsk,"result",skip_keys=["result"])Using graphchain with dask.delayedAlternatively, you can use graphchain together with dask.delayed for easier dask graph creation:importdaskimportpandasaspd@dask.delayeddefcreate_dataframe(num_rows,num_cols):print("Creating DataFrame...")returnpd.DataFrame(data=[range(num_cols)]*num_rows)@dask.delayeddefexpensive_computation(df,num_quantiles):print("Running expensive computation on DataFrame...")returndf.quantile(q=[i/num_quantilesforiinrange(num_quantiles)])@dask.delayeddefsummarize_dataframes(*dfs):print("Summing DataFrames...")returnsum(df.sum().sum()fordfindfs)df_a=create_dataframe(num_rows=10_000,num_cols=1000)df_b=create_dataframe(num_rows=10_000,num_cols=1000)df_c=expensive_computation(df_a,num_quantiles=2048)df_d=expensive_computation(df_b,num_quantiles=2048)result=summarize_dataframes(df_c,df_d)After which you can computeresultby setting thedelayed_optimizemethod tographchain.optimize:importgraphchainfromfunctoolsimportpartialoptimize_s3=partial(graphchain.optimize,location="s3://mybucket/__graphchain_cache__/")withdask.config.set(scheduler="sync",delayed_optimize=optimize_s3):print(result.compute())Using a custom a serializer/deserializerBy default graphchain will cache dask computations withjoblib.dumpand LZ4 compression. However, you may also supply a customserializeanddeserializefunction that writes and reads computations to and from aPyFilesystem filesystem, respectively. For example, the following snippet shows how to serialize dask DataFrames withdask.dataframe.to_parquet, while other objects are serialized with joblib:importdask.dataframeimportgraphchainimportfs.osfsimportjoblibimportosfromfunctoolsimportpartialfromtypingimportAnydefcustom_serialize(obj:Any,fs:fs.osfs.OSFS,key:str)->None:"""Serialize dask DataFrames with to_parquet, and other objects with joblib.dump."""ifisinstance(obj,dask.dataframe.DataFrame):obj.to_parquet(os.path.join(fs.root_path,"parquet",key))else:withfs.open(f"{key}.joblib","wb")asfid:joblib.dump(obj,fid)defcustom_deserialize(fs:fs.osfs.OSFS,key:str)->Any:"""Deserialize dask DataFrames with read_parquet, and other objects with joblib.load."""iffs.exists(f"{key}.joblib"):withfs.open(f"{key}.joblib","rb")asfid:returnjoblib.load(fid)else:returndask.dataframe.read_parquet(os.path.join(fs.root_path,"parquet",key))optimize_parquet=partial(graphchain.optimize,location="./__graphchain_cache__/custom/",serialize=custom_serialize,deserialize=custom_deserialize)withdask.config.set(scheduler="sync",delayed_optimize=optimize_parquet):print(result.compute())ContributingSetup: once per deviceGenerate an SSH keyandadd the SSH key to your GitHub account.Configure SSH to automatically load your SSH keys:cat<< EOF >> ~/.ssh/configHost *AddKeysToAgent yesIgnoreUnknown UseKeychainUseKeychain yesEOFInstall Docker Desktop.EnableUse Docker Compose V2in Docker Desktop's preferences window.Linux only:Configure Docker and Docker Compose to use the BuildKit build system. On macOS and Windows, BuildKit is enabled by default in Docker Desktop.Export your user's user id and group id so thatfiles created in the Dev Container are owned by your user:cat<< EOF >> ~/.bashrcexport UID=$(id --user)export GID=$(id --group)EOFInstall VS CodeandVS Code's Remote-Containers extension. Alternatively, installPyCharm.Optional:Install aNerd Fontsuch asFiraCode Nerd Fontwithbrew tap homebrew/cask-fonts && brew install --cask font-fira-code-nerd-fontandconfigure VS Codeorconfigure PyCharmto use'FiraCode Nerd Font'.Setup: once per projectClone this repository.Start aDev Containerin your preferred development environment:VS Code: open the cloned repository and runCtrl/⌘+⇧+P→Remote-Containers: Reopen in Container.PyCharm: open the cloned repository andconfigure Docker Compose as a remote interpreter.Terminal: open the cloned repository and rundocker compose run --rm devto start an interactive Dev Container.DevelopingThis project follows theConventional Commitsstandard to automateSemantic VersioningandKeep A ChangelogwithCommitizen.Runpoefrom within the development environment to print a list ofPoe the Poettasks available to run on this project.Runpoetry add {package}from within the development environment to install a run time dependency and add it topyproject.tomlandpoetry.lock.Runpoetry remove {package}from within the development environment to uninstall a run time dependency and remove it frompyproject.tomlandpoetry.lock.Runpoetry updatefrom within the development environment to upgrade all dependencies to the latest versions allowed bypyproject.toml.Runcz bumpto bump the package's version, update theCHANGELOG.md, and create a git tag.Developed by RadixRadixis a Belgium-based Machine Learning company.Our vision is to make technology work for and with us. We believe that if technology is used in a creative way, jobs become more fulfilling, people become the best version of themselves, and companies grow.