Package: memori
memoriA python library for creating memoized data and code for neuroimaging pipelinesTable of ContentsInstallationCommand-Line UsagememoripathmanPython UsageTheStageObjectThePipelineObjectStage Aliases and Complex PipelinesHashing external functionsPath ManagementInstallationTo install, usepip:pip install memoriCommand-Line Usagememorican be used to memoize the running of command-line scripts. It is designed
to check the inputs and sha256 integrity of the calling script and determines whether the running of that calling script should be run or not. It accomplishes
this through 3 checks:Check against the stored cache that input arguments are all the same.Check that the sha256 hash of the calling script (and dependents) are the same.(Optional) Check that the desired outputs match the hashes stored in the cache.If at least one of these conditions is not met,memoriwill re-run the script.memoriThe main command-line script to usememoriis simply calledmemorion the command-line:memori-h[anycommand/scripthere]The above command will let you view the help of the memori script.To call memori on a script simply addmemoribefore the command you want
to call. For example:# wrapping echo in memorimemoriecho"this echo command has been wrapped in memori"This will call theechocommand withmemori. To cache the running of the
command, you need to specify the-d/--hash_outputflag:# same command but with cached runmemori-d/path/to/cacheecho1# the first call will print 1memori-d/path/to/cacheecho1# running this a second time will not print anything to the screen# since the inputs/command is the same, so execution is skipped!Since memori determines if a calling script has changed through hashing, you
may want to determine script execution if the calling script depends on another
script. This can occur if calling script 1 calls script 2 and changes are made
to script 2. This can be accomplished through the-c/--dependentsflag.# script execution of script1.sh is now sensitive to changes in script2.shmemori-cscript2.sh-d/path/to/cachescript1.sharg0arg1...If we are expecting certain files to be written from a calling script,
we can informmemoriof their existence through the-o/--outputsflag.memoriwill re-run the calling script if the files are missing/modified.memori-o/path/to/an/expected/output-d/path/to/cachescript.sharg0arg1...The-k/--killflag can be used to kill the parent process, if the calling
script returns an error code. This can be useful to halt a parent script if
execution has failed.Use the--verboseflag for under the hood logging info!pathmanpathmanis a script that allows for the convenient management of file
path manipulations.pathman-hTo view the full help.Python Usagememoriuses a directed acyclic graph (DAG) approach to constructing pipelines.
Nodes of the the graph represent a "logical unit of processing" (up to the user
to define) that can be encomposed in a function. The edges of the
graph transfers data between these nodes to create a pipeline.
To represent thismemoriemploys the use of theStageandPipelineobjects.TheStageobjectAStageis a wrapper around a python function and is the conceptual equivalent
of a node of our graph. AStageobject can take input/output from/to otherStageobjects, but can also be run in isolation. Here is an example of aStagewrapped
around a python function:# our example functiondeftest_function(a,b,c):# Do some stuffd=a+be=b+c# and return stuffreturnd,eWe can wrap this function in aStageobject and run it:frommemoriimportStage# any values a function returns need to be labeled with the `stage_outputs` parametermy_test_stage=Stage(test_function,stage_outputs=["d","e"])# we can run this stage with the run method and store the resultsresult=my_test_stage.run(1,2,3)# result will return a dictionary containing: {"d": 3, "e": 5}# running it again with different parametersresult=my_test_stage.run(2,3,4)# result will return a dictionary containing: {"d": 5, "e": 7}Now lets write a 2nd function that can take input from ourtest_function. Note that
the input arguments for this function should match the key names of the stage outputs
for thetest_function.# new test function with input arguments matching previous stage# function stage_output namesdeftest_function2(d,e):returnd+e# and wrap this in a Stagemy_test_stage2=Stage(test_function2,stage_outputs=["f"])# to run this we just merely need to **results (kwarg unpacking) to pass information# from my_test_stage to my_test_stage2result2=my_test_stage2.run(**results)# result2 will return a dictionary containing: {"f": 12}# or running the entire pipeline from the beginningresult2=my_test_stage2.run(**my_test_stage.run(1,2,3))# result2 will return a dictionary containing: {"f": 8}# The previous two lines is the equivalent to runningtest_function2(**test_function(1,2,3))We can create static values in ourStageobject that ignores inputs from other stages
that are passed into therunmethod.# Stage will take the same params as test_function# and use them as static valuesmy_test_stage=Stage(test_function,stage_outputs=["d","e"],a=1,b=2,c=3)# when we run the stage, we will see that it does not change with the input (2, 3, 4)result=my_test_stage.run(2,3,4)# result will return a dictionary containing: {"d": 3, "e": 5}# if static values weren't used this should return {"d": 5, "e": 7}Now we know how to wrap the functions we write into aStageobject, but what benefit
does this provide? The main feature ofmemoriis tomemoizethe inputs to each
stage and recall the outputs if they are the same. This can enable long running
functions to be skipped if the results are going to be the same!# To enable memoization feature, we need to add the hash_output# parameter when constructing a Stage object. hash_output is# just some directory to where the memoization files can be# written to.my_test_stage=Stage(test_function,stage_output=["d","e"],hash_output="/test/directory")# run the stagemy_test_stage.run(1,2,3)This will write 3 files:test_function.inputs,test_function.stage, andtest_function.outputsat the location: /test/directory
These 3 files record the important states of the Stage for memoization, after it has
been run.The.stagefile contains information about the function that was run.
It contains some rudimentary static analysis to check whether and code
wrapped by a Stage has changed in a way that will affect the result. If it has
detected this, it will rerun the stage. Note that this file contains binary data
is mostly non-human readable (unlike the.inputsand.outputsfiles).The.inputsand.outputsfiles contain information about the inputs and outputs of the stage. These files are simply JSON files and upon opening them in a text editor you should see the following:test_function.inputs{"a":1,"b":2,"c":3}test_function.outputs{"d":3,"e":5}memorichecks the.inputsfile on each run to determine if the stage needs to be run (assuming it has also passed the.stagefile check). If the stage is skipped, the.outputsfile is used to load the results into the stage.By default,memoriuses the name of the function as the name for the hash files. If you
would like to use a different name for these files, you can set the name of the Stage object with
thestage_nameparameter in the constructor:# Stage with a custom stage nameStage(...stage_name="my_stage_name"...)When passing path/file strings betweenStageobjects,memorihas a special behavior: if it
determines the string to be a valid file on the disk, it will hash it with the SHA256
algorithm. For files, this gives memoization results that can reflect changes in data integrity:# now we specify the input and output to be files on the diskfile0="/Some/file/path"file1="/Some/second/file/path"# define our simple test_function that outputs a file pathdeftest_function3(f0):# always return file1returnfile1# Now we wrap it in a stagemy_test_stage3=Stage(test_function3,stage_outputs=["file1"],hash_output="/test/directory")# and run the stage with file0 as the inputresults3=my_test_stage3.run(file0)Now if you examine thetest_function3.inputsandtest_function3.outputsyou will see the following:test_function3.inputs{"file0":{"file":"/Some/file/path","hash":"f0e4c2f76c58916ec258f246851bea091d14d4247a2fc3e18694461b1816e13b"}}test_function3.outputs{"file1":{"file":"/Some/second/file/path","hash":"f91c3b6b3ec826aca3dfaf46d47a32cc627d2ba92e2d63d945fbd98b87b2b002"}}As shown abovememorireplaces a valid file path with a dictionary entry containing the"file"and"hash"keys. Valid files are compared by hash values rather than path/filename ensuring data integrity.NOTE: Since"file"and"hash"are keywords used to hash valid files. These are reserved keywords that should NOT be used when returning an output from a stage using a dictionary. Doing so could lead to catastrophic results!CAUTION:memoriuses JSON to memoize and pass information
betweenStageobjects. This means that the inputs/outputs of your function MUST be JSON
serializable or you will get a serialization error. You can
also get data conversion effects if you don't use the proper
data types. For example, python always converts a Tuple to a
List when serializing a dictionary to JSON. This will lead to
hash check fail each time you run the Stage! Since whenever memori loads the stage
output data from the.outputsfile, the Tuple in the code will never match against
list it was converted to in the JSON. So take care to
use only JSON compatible data types (This means None, integers, floats,
strings, bools, lists, and dictionaries are the only valid
input/output data types inmemori).For data that is not JSON serializable, the typical workaround is to save it to a file
and pass the file location between theStageobjects. This also allows you to take
advantage of the SHA256 file hashing features ofmemori.ThePipelineobjectWhat happens when you have more complex pipelines? Maybe you have aStagethat needs to provide input to two differentStageobjects.This is where thePipelineobject comes in. APipelineis a collection of Stage
objects with their input/output connections defined. APipelineobject represents
the conceptual DAG that was mentioned above.frommemoriimportStage,Pipeline# create some stages (see the last section on Stages for details)stage0=Stage(someparamsgohere...)stage1=Stage(someparamsgohere...)stage2=Stage(someparamsgohere...)stage3=Stage(someparamsgohere...)# Now we create a Pipeline object, a pipeline takes a definition list during construction# the definition list is a list of tuples specifying the connection between stages## The "start" keyword is a special instruction that the Pipeline object can read# it specifies that a particular stage has not precedent Stage and should be a Stage# that is run first in the Pipeline.p=Pipeline([("start",stage0),# stage0 takes no input from other stages, so it should run first(stage0,stage1),# stage0 passes it's output to stage1(stage0,stage2),# and also to stage2((stage1,stage2),stage3)# stage3 needs inputs from stage1 and stage2, so we use a# special tuple-in-tuple so that it can get outputs from both# NOTE: if stage1 and stage2 have stage_outputs with the same# name, the last stage (right-most) stage will have precedence# for it's output])# we can run the Pipeline with the run method, and get it's resultresult=p.run(someinputparametershere...)Running the pipeline has the effect of invoking the run method
of eachStageobject individually, and passing the result of the stage onto the
next stage as defined by thePipelinedefinition passed in duringPipelineinitialization.Stage Aliases and Complex PipelinesWhen building a complicated pipleine, sometimes the functions that you write
will have input argument names that are different from thestage_outputnames
that you have defined in aStage. Consider the following example:deftest_function(a,b):returna+bdeftest_function2(c):# this might represent some complicated processingc+=1returncdeftest_function3(d):# this might be another function with some more complocated processingd+=2returndNow let's say I want to pass the result oftest_functionto bothtest_function2andtest_function3. This presents an issue becausetest_function2andtest_function3have
different input argument names. So if I define thestage_outputof the wrappedtest_functionto bestage_outputs=["c"]this won't work fortest_function3and if I define it to bestage_outputs=["d"]it won't work fortest_function2.One way of solving this issue would be to rewrite thetest_function2andtest_function3functions to have the same argument name, this may not always be possible (particularly when
wrapping a function call from a third-party library). Another option would be to wrap the
call of eithertest_function2ortest_function3to take in the same input. For example:# this is necessary hashing external function calls# more about the hashable wrapper in the next sectionfrommemoriimporthashable# we wrap the call of test_function3deftest_function3_mod(c):returnhashable(test_function3)(c)Now when we create theStagefor each function,test_function2andtest_function3_modnow have the same input argument names and can take in input fromtest_function.While this solution works (and indeed this was how it used to be done),memoriprovides a more
convienent solution through Stage aliases. Aliases can map the name of one of the stage outputs to
another name. When creating aStageobject, you can define this through thealiasesparameter.# We wrap test_function in a Stage, and specify an alias from d -> ctest_stage=Stage(test_function,stage_outputs=["c"],aliases=["d":"c"])# Now I can construct stages around test_function2 and test_function3 without# writing extra codetest_stage2=Stage(test_function2,stage_outputs=["e"])test_stage3=Stage(test_function3,stage_outputs=["f"])# now definte the pipelinemy_pipeline=Pipeline([("start",test_stage),(test_stage,test_stage2),(test_stage,test_stage3),# because we mapped d -> c, memori know where to pass the result to])Stage aliases reduces the need for extra boilerplate code, and adding on an extra
stage the feeds fromtest_stageis as simple as adding another alias.Hashing external functionsIn the last section, we saw the use of the hashable wrapper when trying to wrap a
function call in another function. But what does it actually do? Consider the
following example:deftest_function(a,b)c=a+bd=test_function2(c)returnddeftest_function2(c)returnc+1stage0=Stage(test_function,stage_outputs=["d"],hash_output="test")result=stage0.run(1,2)# this will return the result {"d": 4}Now, what if we change the code of test_function to:# change up test_function!deftest_function(a,b)c=a+b+1d=test_function2(c)returndRebuilding the stage on this function and invoking therunmethod it will cause the.stagehash to mismatch (since the function signature is different with the added+ 1in the code), and the function will rerun instead of loading from cache
(this should return the result{"d": 5}).So the function hashing feature of memori works! but what happens when we modifytest_function2and rerun our stage.# will memori see this change?deftest_function2(c):returnc+2Rerunning the stage with the updatedtest_function2, you will see that after invokingrun, theStageobject simply loads the result from the.outputfile and ignores
the difference in the updatedtest_function2(this will still return{"d": 5}rather
than{"d": 6}.This occurs becausememorifunction hashing only occurs one call deep. Meaning that
only the instructions of the wrapped callable are the only thing that is hashed. Function calls inside a function are simply recorded as constants, meaning that only
the nametest_function2is memoized, not the actual instructions!To correct this issue,memoriprovides thehashablewrapper. This wrapper marks
a function so that memori knows to try and hash it.# wrap test_funtion2 in hashabledeftest_function(a,b)c=a+b+1d=hashable(test_function2)(c)returndAlternatively, you can add the hashable wrapper a decorator.# this is the same as calling hashable(test_function2)# but makes everything transparent@hashabledeftest_function2(c)returnc+1This allows you to simply calltest_function2without worrying about calling
the hashable wrapper each time.Path Managementmemorialso provides a path management utility calledPathManager. It
is useful for manipulating file paths as well as suffixes and extensions.
If is derived from aPathobject from thepathliblibrary, and so can use any of the
parent methods as well.Here are a few useful examples:frommemoriimportPathManagerasPathMan# a string to a path I want PathManager to managemy_file_path_pm=PathMan("/my/path/to/a/file.ext.ext2")# get only the file prefixprefix=my_file_path_pm.get_prefix()# prefix contains "file"# get the path and file prefixpath_and_prefix=my_file_path_pm.get_path_and_prefix()# path_and_prefix contains "/my/path/to/a/file"# change path of the file, keeping the filename the samerepathed=my_file_path_pm.repath("/new/path")# repathed contains "/new/path/file.ext.ext2"# append a suffix (following the BIDS standard, suffixes should always have _)suffixed=my_file_path_pm.append_suffix("_newsuffix")# suffixed contains "/my/path/to/a/file_newsuffix.ext.ext2"# replace last suffixreplaced=suffixed.replace_suffix("_newsuffix2")# replaced contains "/my/path/to/a/file_newsuffix2.ext.ext2"# delete last suffixdeleted=replaced.delete_suffix()# deleted contains "/my/path/to/a/file.ext.ext2"# methods can be chained togetherchained=my_file_path_pm.repath("/new").append_suffix("_test").get_path_and_prefix()# chained contains /new/file_test# return as a stringmy_file_path=my_file_path_pm.path# /new/file_test