Package: jor
jor(JOb Runner) keeps track of jobs. Jobs are specified
in a simple format and can then be run collectively with a
simple command, mindful of which outputs already exist.
Jobs can optionally be run in acondaenvironment or in aSingularitycontainer, and can be run either locally or
submitted to a compute node withslurm.joris particularly usefulif a large number of different jobs needs to be runfor parameter sweepsin an HPC environment:jorcan submit jobs withslurmfor reproducible research results:jorcan run jobs in a Singularity containerUsageTo usejorrequires 3 preparatory steps:Write a simple Python wrapper around the actual computations, specifying the job’s inputs and outputsCollect all the jobs in a todo-listWrite up some runtime-arguments in a config fileThese steps are described in detail below.Once these preparations are done, all jobs can be run withjorrunand the status of the jobs specified in the todo-list can be inspected withjorstatusPreparatory steps1. The Job-wrapperThis is the most complex of the 3 steps. It requires to implement 4-5 functions that specifythe parameters for each jobthe output folder for each jobthe name of the output file for each jobthe actual computation for each jobif applicable, how to concatenate all outputs of this job into a single fileNoteAll these steps would be necessary no matter ifjoris used to perform
the jobs’ computations or not, the only difference is that, to usejorthey
need to be specified in a particular way. Thusjordoesn’t lead to much
code overhead.Here is a template:importpathlibimportjorclassJobs(jor.JobsBase):# slurm optionsname='job'time='0-23:59:59'mem='5G'cpus_per_task=1def__init__(self,n=3,path_prefix='.'):# init base classsuper().__init__(path_prefix=path_prefix)# store job-specific keyword argumentsself.n=int(n)# assemble a list of jobsself._mk_jobs()def_mk_jobs(self):"""Generates the attribute ``self._jobs``

        ``self._jobs`` is a list of dictionaries, containing one dictionary
        for each jobs to be run. Each of these dictionaries specifies the
        parameters for each individual job.
        """self._jobs=[dict(index=i)foriinrange(self.n)]def_get_output_folder(self):"""Return output folder for jobs

        Output folder is the ``path_prefix``, followed by the name of a subfolder
        encoding the arguments given in the constructor.
        """output_folder=pathlib.Path(self.path_prefix)/f'example{self.n}'returnstr(output_folder)def_get_output_fname(self,index):"""Return output file name for a given job

        The particular job is specified by the arguments to this function,
        which match the keys of the dictionaries in ``self._jobs`` (cf.
        :func:`self._mk_jobs`).
        """outfname=f'ind{index}.txt'returnoutfnamedefexecute(self,i):"""This function performs the actual work

        Parameters
        ----------
        i : int between 0 and number of jobs (``len(self._jobs)``)
            indicating the index in ``self._jobs`` from which to take
            the dictionary with this job's parameter values
        """myargs=self._jobs[i]output_path=self._get_output_path(**myargs)# do the work and write outcomes to file ``output_path``withopen(output_path,'wt')asf:f.write(str(myargs)+'\n')defcollect(self):"""Concatenates all outputs into a common output

        This function is optional and can be implemented if desired for
        the particular job. It is called by running ``jor collect`` on the
        command line.
        """passNoteThe wrapper needs to be a Python file containaing a classJobs, derived fromjor.JobsBaseThe indicatedslurmoptions are defaults inherited fromjor.JobsBase, i.e. they only need to be specified if a different value is desiredTo adapt this to a specific application:Adapt the constructor to take job-specific argumentsReimplement_mk_jobsReimplement_get_output_folderReimplement_get_output_fnameReimplementexecuteIf applicable, reimplementcollect2. The todo-listThe file containing the todo-list is aYAMLfile namedtodo.yamlby default.
It has the following format:jobs:-jobmodule:./jobs_example.pyjobargs:n=3-jobmodule:./jobs_example.pyjobargs:n=4There can be an arbitrary number of jobs specified in this file.NoteThe file needs to start withjobs:Each job is specified by 2 lines, 1 starting with- jobmodule:the other with an indentedjobargs:.The argument forjobmodule:is the name (or path to) the Python file containing the wrapper code.The argument tojobargs:is a comma-separated list of keyword-arguments for the constructor of theJobsclass in the wrapper file. It needs to be valid Python and can be empty if no keyword arguments are necessary.3. The config fileThe config file needs to be namedjor.cfgand needs to
reside in the working directory from whichjoris called.
It has the following format:[global]path-prefix=outputoverwrite-output=False[run]todo-list=todo.yaml[submit]scheduler=localpartition=daysif=condaenv=[collect]missing-output=ignoreThe configuration options have the following meaning:Configuration optionsKeywordAllowed valuesMeaningpath-prefixfile-system pathsthe job-wrapper should receivepath_prefixas a keyword argument in theJobsconstructor, and should prefix all internally generated output-paths with the value ofpath-prefixoverwrite-outputTrueorFalseifFalsejorwill check which outputs already exist and only run jobs that result in the remaining outputstodo-lista filenamefile name containing todo-list, by default this istodo.yaml, there’s probably no reason to change thisschedulerlocalorslurmiflocaljobs will be run in order locally; ifslurmone job-array will be submitted viaslurm’ssbatchcommand per entry in the todo-listpartitiona validslurmpartition (queue) nameignored ifscheduler = localsifeither empty or path to a Singularity containerif not empty all jobs will be run in this containercondaenveither empty of name of acondaenvironmentif not empty, thiscondaenvironment will be activated before running each jobmissing-outputignoreorraisein case the job-wrapper implements acollectmethod to concatenate outputs, this specifies how missing files are handled: ifignoremissing outputs will be ignored, ifraisemissing outputs will cause concatenation to abortNoteAll configuration-options injor.cfgcan be
overwritten in the command-line call tojor.ExampleAn example is provided in theexamplessubfolder. The filejobs_example.pycontains the code shownabove. Likewise thetodo.yamlandjor.cfgfiles from above can be found there. Callingjorrunreturns[jor]Submittingjob:./jobs_example.py[jor]Submittingjob:./jobs_example.pyand inspecting theoutputfolderls-Routputshows that all output files are present:example3example4output/example3:ind0.txtind1.txtind2.txtoutput/example4:ind0.txtind1.txtind2.txtind3.txt