Package: dawgz
Directed Acyclic Workflow Graph SchedulingWould you like fully reproducible and reusable experiments that run on HPC clusters as seamlessly as on your machine? Do you have to comment out large parts of your pipelines whenever something failed? Tired of writing and submittingSlurmscripts? Thendawgzis made for you!Thedawgzpackage provides a lightweight and intuitive Python interface to declare jobs along with their dependencies, requirements, settings, etc. A single line of code is then needed to execute automatically all or part of the workflow, while complying to the dependencies. Importantly,dawgzcan also hand over the execution to resource management backends likeSlurm, which enables to execute the same workflow on your machine and HPC clusters.InstallationThedawgzpackage is available onPyPi, which means it is installable viapip.pip install dawgzAlternatively, if you need the latest features, you can install it usingpip install git+https://github.com/francois-rozet/dawgzGetting startedIndawgz, a job is a Python function decorated by@dawgz.job. This decorator allows to define the job's parameters, like its name, whether it is a job array, the resources it needs, etc. The job's dependencies are declared with the@dawgz.afterdecorator. At last, thedawgz.schedulefunction takes care of scheduling the jobs and their dependencies, with a selected backend. For more information, check out theinterfaceand theexamples.Follows a small example demonstrating how one could usedawgzto calculateπ(very roughly) using theMonte Carlo method. We define two jobs:generateandestimate. The former is ajob array, meaning that it is executed concurrently for all values ofi = 0up totasks - 1. It also defines apostconditionensuring that the filepi_{i}.npyexists after the job's completion. The jobestimatehasgenerateas dependency, meaning it should only start aftergeneratesucceeded.importglobimportnumpyasnpimportosfromdawgzimportjob,after,ensure,schedulesamples=10000tasks=5@ensure(lambdai:os.path.exists(f'pi_{i}.npy'))@job(array=tasks,cpus=1,ram='2GB',time='5:00')defgenerate(i:int):print(f'Task{i+1}/{tasks}')x=np.random.random(samples)y=np.random.random(samples)within_circle=x**2+y**2<=1np.save(f'pi_{i}.npy',within_circle)@after(generate)@job(cpus=2,ram='4GB',time='15:00')defestimate():files=glob.glob('pi_*.npy')stack=np.vstack([np.load(f)forfinfiles])pi_estimate=stack.mean()*4print(f'π ≈{pi_estimate}')schedule(estimate,name='pi.py',backend='async')Running this script with the'async'backend displays$ python examples/pi.py
Task 1 / 5
Task 2 / 5
Task 3 / 5
Task 4 / 5
Task 5 / 5
π ≈ 3.141865Alternatively, on a Slurm HPC cluster, changing the backend to'slurm'results in the following job queue.$ squeue -u username
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           1868832       all estimate username PD       0:00      1 (Dependency)
     1868831_[2-4]       all generate username PD       0:00      1 (Resources)
         1868831_0       all generate username  R       0:01      1 node-x
         1868831_1       all generate username  R       0:01      1 node-yIn addition to the Python interface,dawgzprovides a simple command-line interface (CLI) to list the scheduled workflows, the jobs of a workflow or the output(s) of a job.$ dawgz
    Name    ID        Date                 Backend      Jobs    Errors
--  ------  --------  -------------------  ---------  ------  --------
 0  pi.py   8094aa20  2022-02-28 16:37:58  async           2         0
 1  pi.py   9cc409fd  2022-02-28 16:38:33  slurm           2         0
$ dawgz 1
    Name                ID  State
--  -------------  -------  -------
 0  generate[0-4]  1868831  MIXED
 1  estimate       1868832  PENDING
$ dawgz 1 0
    Name         State      Output
--  -----------  ---------  ----------
 0  generate[0]  COMPLETED  Task 1 / 5
 1  generate[1]  COMPLETED  Task 2 / 5
 2  generate[2]  RUNNING
 3  generate[3]  RUNNING
 4  generate[4]  PENDINGInterfaceDecoratorsThe package provides four decorators:@dawgz.jobregisters a function as a job, with its settings (name, array, resources, ...). It should always be the first (lowest) decorator. In the following example,ais a job with the name'A'and a time limit of one hour.@job(name='A',time='01:00:00')defa():All keyword arguments other thannameandarrayare passed as settings to the scheduler. For example, with theslurmbackend, the following would lead to a job array of 64 tasks, with a maximum of 3 simultaneous tasks running exclusively on'tesla'or'quadro'partitions.@job(array=64,maxsim=3,partition='tesla,quadro')Importantly, a job isshipped with its context, meaning that modifying global variables after it has been created does not affect its execution.@dawgz.afteradds one or more dependencies to a job. By default, the job waits for its dependencies to complete with success. The desired status can be set to'success'(default),'failure'or'any'. In the following example,bwaits forato complete with'failure'.@after(a,status='failure')@jobdefb():@dawgz.waitfordeclares whether the job has to wait for'all'(default) or'any'of its dependencies to be satisfied before starting. In the following example,cwaits for eitheraorbto complete (with success).@after(a,b)@waitfor('any')@jobdefc():@dawgz.ensureadds apostconditionto a job, i.e. a condition that must beTrueafter the execution of the job. Not satisfying all postconditions after execution results in anAssertionErrorat runtime. In the following example,densures that the filelog.txtexists.@ensure(lambda:os.path.exists('log.txt'))@jobdefd():Traditionally, postconditions are onlynecessaryindicators that a task completed with success. Indawgz, they are considered both necessary andsufficientindicators. Therefore, postconditions can be used to detect jobs that have already been executed and prune them out of the workflow. To do so, setprune=Trueindawgz.schedule.BackendsCurrently,dawgz.schedulesupports three backends:async,dummyandslurm.asyncwaits asynchronously for dependencies to complete before executing each job. The jobs are executed by the current Python interpreter.dummyis equivalent toasync, but instead of executing the jobs, prints their name before and after a short (random) sleep time. The main use ofdummyis debugging.slurmsubmits the jobs to the Slurm workload manager by automatically generatingsbatchsubmission scripts.