Package: decimate
NAMEdecimate - a fault-tolerant SLURM scheduler extensionSYNOPSISdbatch [ Slurm options ] [ --check <user_script> ][ --max-retry=<number of restart> ]script [args...]DESCRIPTIONDevelopped by the KAUST Supercomputing Laboratory (KSL),decimate is a SLURM extension written in python designed to handledependent jobs more easely and efficiently.Decimate transparently adds parameters to SLURM sbatch commandto check the correctness of jobs and automaticallyreschedules jobs found faulty.Using Decimate on Shaheen II, one can submit, run, monitor orterminate a workflow composed of dependent jobs. If asked,thanks to standardized or customized messages, the user will beinformed by mail of the progress of its workflow on the system.In case of failure of one part of tne workflow, decimateautomatically detects the failure, signals it to the user andlaunches the misbehaving part after having fixed the jobdependency. By default if the same failure happens threeconsecutive times, decimate cancels the whole workfow removingall the depending jobs from the scheduling. In a next version,decimate will allow the automatic restarting of the workflowonce the problem causing its failure has been cured.decimate also allows the user to define his own mail alertsthat can be sent at any point of the workflow through a call toa python method. This feature will also be available from bashin a next version.Some customized checking functions can also be designed by theuser. Their purpose is to validate if a step of the workflowwas succesful or not. It could involved checking for thepresence of some result files, grepping some error or successmessages in them, computing ratio or checksum... Theseintermediate results can be easely transmitted to decimatevalidating or not the correctness of any step. They can also beforwarded by mail to the user where as the workflow isexecuting.USEAt this moment, jobs only need to be submitted through thedbatchcommand that accepts exactely the same parameters as theoriginal SLURM sbatch command plus the new parameters--check=SCRIPT_FILEwhere SCRIPT_FILE  is a pythonor shell scriptto check if results are ok.--max-retry=MAX_RETRYnumber of time a step can fail and berestarted automatically before failing thewhole workflow  (3 per default)sslog tails out the decimate logging file attached to thecurrent directory, tracking all the jobs that were launchedwith dbatch from this directory.sstatus gives the current status of the workflow excecutingin the current directory.Decimate is still in a beta phase and under test with some ofour KSL users. More documentations will be provided once thestabilized and fully tested version is made available by theend of June 2018.If interested in testing decimate or contributing, please senda mail to help@hpc.kaust.edu.saAUTHORWritten by Samuel Kortas (samuel.kortas (at) kaust.edu.sa)REPORTING BUGSReport decimate bugs to help@hpc.kaust.edu.saCOPYRIGHTCopyright (c) 2017, KAUST Supercomputing LaboratoryAll rights reserved.Redistribution and use in source and binary forms, with or withoutmodification, are permitted provided that the following conditions are met:* Redistributions of source code must retain the above copyright notice, thislist of conditions and the following disclaimer.* Redistributions in binary form must reproduce the above copyright notice,this list of conditions and the following disclaimer in the documentationand/or other materials provided with the distribution.THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THEIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AREDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLEFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIALDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS ORSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVERCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USEOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.SEE ALSOdecimate official documentation pages:<http://http://decimate.readthedocs.io>KAUST Supercomputing Laboratory: <http://hpc.kaust.edu.sa/>