Package: covalent-slurm-plugin
Covalent Slurm PluginCovalent is a Pythonic workflow tool used to execute tasks on advanced computing hardware. This executor plugin interfaces Covalent with HPC systems managed bySlurm. For workflows to be deployable, users must have SSH access to the Slurm login node, writable storage space on the remote filesystem, and permissions to submit jobs to Slurm.InstallationTo use this plugin with Covalent, simply install it usingpip:pip install covalent-slurm-pluginOn the remote system, the Python version in the environment you plan to use must match that used when dispatching the calculations. Additionally, the remote system's Python environment must have the basecovalent packageinstalled (e.g.pip install covalent).UsageThe following shows an example of a Covalentconfigurationthat is modified to support Slurm:[executors.slurm]username = "user"address = "login.cluster.org"ssh_key_file = "/home/user/.ssh/id_rsa"remote_workdir = "/scratch/user"cache_dir = "/tmp/covalent"[executors.slurm.options]nodes = 1ntasks = 4cpus-per-task = 8constraint = "gpu"gpus = 4qos = "regular"[executors.slurm.srun_options]cpu_bind = "cores"gpus = 4gpu-bind = "single:1"The first stanza describes default connection parameters for a user who can connect to the Slurm login node using, for example:ssh -i /home/user/.ssh/id_rsa user@login.cluster.orgThe second and third stanzas describe default parameters for#SBATCHdirectives and default parameters passed directly tosrun, respectively.This example generates a script containing the following preamble:#!/bin/bash#SBATCH--nodes=1#SBATCH--ntasks=4#SBATCH--cpus-per-task=8#SBATCH--constraint=gpu#SBATCH--gpus=4#SBATCH--qos=regularand subsequent workflow submission with:srun --cpu_bind=cores --gpus=4 --gpu-bind=single:1To use the configuration settings, an electron’s executor must be specified with a string argument, in this case:importcovalentasct@ct.electron(executor="slurm")defmy_task(x,y):returnx+yAlternatively, passing aSlurmExecutorinstance enables custom behavior scoped to specific tasks. Here, the executor'sprerun_commandsandpostrun_commandsparameters can be used to list shell commands to be executed before and after submitting the workflow. These may include any additionalsruncommands apart from workflow submission. Commands can also be nested inside the submission call tosrunby using thesrun_appendparameter.More complex jobs can be crafted by using these optional parameters. For example, the instance below runs a job that accesses CPU and GPU resources on a single node, while profiling GPU usage viansysand issuing complementary commands that pause/resume the central hardware counter.executor=ct.executor.SlurmExecutor(remote_workdir="/scratch/user/experiment1",options={"qos":"regular","time":"01:30:00","nodes":1,"constraint":"gpu",},prerun_commands=["module load package/1.2.3","srun --ntasks-per-node 1 dcgmi profile --pause"],srun_options={"n":4,"c":8,"cpu-bind":"cores","G":4,"gpu-bind":"single:1"},srun_append="nsys profile --stats=true -t cuda --gpu-metrics-device=all",postrun_commands=["srun --ntasks-per-node 1 dcgmi profile --resume",])@ct.electron(executor=executor)defmy_custom_task(x,y):returnx+yHere the corresponding submit script contains the following commands:module load package/1.2.3srun --ntasks-per-node 1 dcgmi profile --pausesrun -n 4 -c 8 --cpu-bind=cores -G 4 --gpu-bind=single:1 \nsys profile --stats=true -t cuda --gpu-metrics-device=all \python /scratch/user/experiment1/workflow_script.pysrun --ntasks-per-node 1 dcgmi profile --resumeRelease NotesRelease notes are available in theChangelog.CitationPlease use the following citation in any publications:W. J. Cunningham, S. K. Radha, F. Hasan, J. Kanem, S. W. Neagle, and S. Sanand.Covalent.Zenodo, 2022.https://doi.org/10.5281/zenodo.5903364LicenseCovalent is licensed under the Apache License 2.0. See theLICENSEfile or contact thesupport teamfor more details.