Package: awflow
Reproducible research and reusable acyclic workflows in Python. Execute code on HPC systems as if you executed them on your machine!MotivationWould you like fully reproducible research or reusable workflows that seamlessly run on HPC clusters?
Tired of writing and managing large Slurm submission scripts? Do you have comment out large parts of your pipeline whenever its results have been generated? Hate YAML?
Don't waste your precious time!awflowallows you to directly describe complex pipelines in Python, that run on your personal computer and large HPC clusters.importglobimportnumpyasnpimportosfromawflowimportafter,ensure,job,schedulen=10000tasks=10@ensure(lambdai:os.path.exists(f'pi-{i}.npy'))@job(cpus='4',memory='4GB',array=tasks)defestimate(i:int):print(f'Executing task{i+1}/{tasks}.')x=np.random.random(n)y=np.random.random(n)pi_estimate=(x**2+y**2<=1)np.save(f'pi-{i}.npy',pi_estimate)@after(estimate)@ensure(lambda:os.path.exists('pi.npy'))@job(cpus='4')defmerge():files=glob.glob('pi-*.npy')stack=np.vstack([np.load(f)forfinfiles])pi_estimate=stack.sum()/(n*tasks)*4print('π ≅',pi_estimate)np.save('pi.npy',pi_estimate)merge.prune()# Prune jobs whose postconditions have been satisfiedschedule(merge,backend='local')# Executes merge and its dependenciesExecuting this Python program (python examples/pi.py --backend slurm) on a Slurm HPC cluster will launch the following jobs.1803299       all    merge username PD       0:00      1 (Dependency)
     1803298_[6-9]       all estimate username PD       0:00      1 (Resources)
         1803298_3       all estimate username  R       0:01      1 compute-xx
         1803298_4       all estimate username  R       0:01      1 compute-xx
         1803298_5       all estimate username  R       0:01      1 compute-xxThe following example shows how workflow graphs can be dynamically allocated:fromawflowimportafter,job,schedule,terminal_nodes@job(cpus='2',memory='4GB',array=5)defgenerate(i:int):print(f'Generating data block{i}.')@after(generate)@job(cpus='1',memory='2GB',array=5)defpostprocess(i:int):print(f'Postprocessing data block{i}.')defdo_experiment(parameter):r"""This method allocates a `fit` and `make_plot` jobbased on the specified parameter."""@after(postprocess)@job(name=f'fit_{parameter}')# By default, the name is equal to the function namedeffit():print(f'Fit{parameter}.')@after(fit)@job(name=f'plt_{parameter}')# Simplifies the identification of the logfiledefmake_plot():print(f'Plot{parameter}.')# Programmatically build workflowforparameterin[0.1,0.2,0.3,0.4,0.5]:do_experiment(parameter)leafs=terminal_nodes(generate,prune=True)# Find terminal nodes of workflow graphschedule(*leafs,backend='local')Check theexamplesdirectory to explore the functionality.UsageTODOAvailable backendsCurrently,awflow.scheduleonly supports alocalandslurmbackend.InstallationTheawflowpackage is available onPyPi, which means it is installable viapip.you@local:~ $pipinstallawflowIf you would like the latest features, you can install it using this Git repository.you@local:~ $pipinstallgit+https://github.com/JoeriHermans/awflowIf you would like to run the examples as well, be sure to install theoptionalexample dependencies.you@local:~ $pipinstall'awflow[examples]'Roadmap and TODOShouldschedulereturn metadata of jobs and workflow?Check for cyclic dependencies.More examples and documentation.Utilities to cleanup generated metadata and crashed jobs for the Slurm backend.Can jobs submit jobs on both local and Slurm backend?ContributingSeeCONTRIBUTING.md.LicenseAs described in theLICENSEfile.