Package: merkl
MerkL - ML pipelines in pure Python with great cachingMerkL is a tool for creating ML pipelines in pure Python that are useful for
development and experimentation, but also easy to deploy to production without
modifications. Results are cached using Merkle hashes of the code and inputs as
keys, greatly simplifying caching and reducing the need for a feature store.NOTE: this project is not production ready and maybe never will be. At this
point I'm the only user. If you're interested in talking about it I can be
reached atme@martindbp.com.Installationpip install merklTour and examplesIn MerkL, pipelines are built using functions decorated with thetaskdecorator. When a task is called, the function
body is not exectuted immediately, but insteadFutureobjects are returned in place of real outputs. These can then be
passed on to other tasks:pipeline1.pymerkl dot pipeline1.my_pipeline 42 | dot -Tpng | displayfrommerklimporttask@taskdeftask1(input_value):return2*input_value@taskdeftask2(input_value):returninput_value**2defmy_pipeline(input_value:int):val=task1(input_value)final_val=task2(val)returnfinal_valmerkl run pipeline1.my_pipeline 427056No function body is executed before.eval()is called on a Future, or it is returned by a pipeline executed by theruncommand.
Instead a graph is built and each Future is assigned a hash that uniquely identifies its future value. If the code or input values change, then the
output Future hashes also change. These hashes are then used to find cached results.As seen in the table above, one can use themerkl runcommand to run a pipline, and themerkl dotcommand to create
a dot file for visualizing the computational graph. The graph can be rendered and displayed by piping the dot file
through thedotandvisualizeprograms (note that these require that graphviz and imagemagick are installed).Arguments can be passed to the pipeline through theruncommand. Seeclizefor more information on how to pass parameters from the command line.By default, all computed outputs are cached, unless the--no-cacheoption is supplied. If we run the pipeline a second time, we can see that the hashes turn green, which indicates that the values are already in the cache. Changing the code intask2, for example by changing the2to a3, we can see that the output value is
not cached anymore:First runSecond runModify task2Here's a slightly more realistic pipeline consisting of a model training and evaluation task:pipeline2.pymerkl dot pipeline2.train_eval  | dot -Tpng | displayfrommerklimporttask,read_future@taskdeftrain(data,iterations):return'trained model'@taskdefevaluate(model,data):return99.3deftrain_eval():train_data=read_future('train.csv')test_data=read_future('test.csv')model=train(train_data,iterations=100)model>'model.bin'score=evaluate(model,test_data)returnscore,modelmerkl run pipeline2.train_eval(99.3, 'trained model')HashingIn order to compute the recursive Merkle hashes, the content (md5) hash of the
input files are needed. MerkL hashes these on demand as they are needed, but
stores these hashes in the.merkl/cache.sqlite3database. If the timestamp of
the file ever changes, the file is hashed again.Files created by MerkL tasks or pipelines are also tracked in the database so that MerkL knows if a file needs to be updated or not.Multiple outsThe number of outputs that you want to track separately has to be determinable at DAG "build" time,
therefore it cannot be dependent on some value computed inside a task.By default, MerkL tries to guess the number of outs by looking at the return statements in the AST (Abstract Syntax
Tree) using Python's inspection capabilities. If the return statement is a Tuple or Dict literal, the values are split
into separate Futures:Tuple multiple outsTuple single outDict multiple outs@taskdefmy_task():return1,2,3print(my_task())@taskdefmy_task():values=1,2,3returnvaluesprint(my_task())@taskdefmy_task():return{'key1':1,'key2':2,'key3':3,}print(my_task())STDOUT(
    <Future: b7a152bd>,
    <Future: 3612309d>,
    <Future: 541053ee>,
)<Future: bc930b90>{
    'key1': <Future: 7d7f02e7>,
    'key2': <Future: 09f5f892>,
    'key3': <Future: aaba8931>,
}In some cases we want the number of outs to be dynamic and depend on the input to the function. In this case you can
supply a callable to theoutstask parameter with the same signature as the task function, and return the number of
outs:outs_lambda.pymerkl dot outs_lambda.pipeline  | dot -Tpng | displayfrommerklimporttask@task(outs=lambdadata,k:k)defsplit(data,k):ksize=len(data)//kreturn[data[i*ksize:(i+1)*ksize]foriinrange(k)]defpipeline():returnsplit([1,2,3,4,5,6],k=2)merkl run outs_lambda.pipeline([1, 2, 3], [4, 5, 6])If all else fails, you can setoutsto any positive integer or an iterable of output dictionary keys.Pipe syntaxFor convenience, you can optionally chain tasks using the|operator, if the tasks have a single input and output. A
future can also be "piped" directly to a file path using the>operator:# Originalwrite_future(train(preprocess(read_future('train.csv'))),'model.bin')# With pipe syntaxread_future('train.csv')|preprocess|train>'model.bin'PipelinesPipeline functions can optionally be decorated with thepipelinedecorator. The decorator provides caching of the
graph construction (as opposed to only the evaluation). This is useful when for example there is an inference stage
which depends on a training stage which preprocesses thousands of images. Even though theresultof the training is
cached, theconstructionof the graph may become slow.@taskdefpreprocess(image):...@task(outs=lambdaimage,k:k)defaugment(image,k):...@taskdeftrain(Batch tasksSome functions have a batch version that has a more efficient implementation than the single item version. In this case, we
might want to track the outputs from the batch taskas ifthey were produced by the single item version, such that
outputs between them will be cached. In this case you can use thebatchdecorator:@batch(embed_word)defembed_words(words):...returnembedded_wordsIn some cases, you might only have a batch implementation, but you want each output to be treated individually. In that
case you can also use thebatchdecorator but without an argument:@batchdefembed_words(words):...returnembedded_wordsThe difference here is that identical inputs will have the same output Future hash, which is not true for a regular
task. You can tell the difference in these two graphs:12outs=embed_sentences_batch(['my sentence','my sentence'])assertouts[0].hash==outs[1].hashouts=embed_sentences_task(['my sentence','my sentence'])assertouts[0].hash!=outs[1].hashHashMode and dependenciesWhen the hash of a task function is determined, there are three differentHashModes:FUNCTION,MODULEandFIND_DEPS. The defaultHashModefor bothtask,batchandpipelineisFIND_DEPS.FUNCTION: Uses the code of the task function for hashingMODULE: Uses the whole module fileFIND_DEPS: Finds dependencies such as other functions or nonlocal variables used within the function that are
defined in the same module. If a dep is ataskthen its dependencies are added in turn.Deps can be added manually and could be a module, function, task, bytes, string or any JSON-serializable object:@task(deps=['my string'])defmy_task():passAs a convenience, you can use thepip_version()function to easily add a library's version string as a hash key:@task(deps=[pip_version('numpy')])defmy_task():passFiles and directories can be added as dependencies using theFileRefandDirRefclasses:frommerklimporttask,FileRef@task(deps=[FileRef('my_file.txt')])defmy_task():passRefs added as dependencies will contribute the file content hash to the task.Filesystem IOTBDTask versionsTBDThe CacheTBDWrapping shell commandsTBDDev NotesReleasing a new version:Bump version in setup.pyCommitgit tag $VERSIONmake release. This bundles the package and uploads it using twineNote that twine is required:pip install twine