Package: minislurm
Opposed toslurm, minislurm is a single node workload manager.
It is intended for repeated program execution with different parameters on a single machine
(e.g. physical process simulation with different boundary conditions).
Different processes should be put in differentsystemdservice files so
appropriate resource restrictions may be applied.InstallationTo install simply issue the commandpip3install--userminislurmConfigurationFull configuration file is[SERVER]SOCKET=/tmp/minislurm.socketTIMEZONE_OFFSET=+3MAX_PARALLEL=4QUEUE_SIZE=100UPDATE_TIME=1LOG_LEVEL=INFOCALLBACK=[PROGRAM]COMMAND=sleep {}TIMEOUT=1h 30mKILL_TIMEOUT=10mSERVERsection contains configuration related to server itselfSOCKETis aUNIX socketfile
location. Use a descriptive name in a writable folder (e.g./tmp/minislurm_openfoam.socket).TIMEZONE_OFFSETis used for displaying time with specified offsetMAX_PARALLELcontrols how many processes may run simultaneouslyQUEUE_SIZEspecifies job queue size. New processes cannot be added is all queue slots are
occupied and all jobs are either running or waiting to be executed.UPDATE_TIMEcontrols how frequently processes are probed for status. There's not much point
in changing this value.LOG_LEVELsets the logging level. When running server using systemd, log file may be
examined usingjournalctl --user -u minislurm@<instance>.servicecommand, where<instance>is a server instance name (for details read below).CALLBACKmay be used to run a callback command with argumentsjob_name,job_id,job_statuswhen job finishes execution. One example of such a callback is aDBUSnotification.CALLBACK=dbus-send --session --type=method_call --dest=org.freedesktop.Notifications / org.freedesktop.Notifications.Notify string:'' uint32:0 string:'' string:MiniSlurm string:"Job {} ID {} stopped with status {}" array:string:"" dict:string:string:'' int32:5000Configuration presented above spawns DBUS notification for 5 seconds after job is complete.PROGRAMsection configures spawned processesCOMMANDis a command to be spawned by server. It usesPythonstr.formatsyntax to supply
command arguments.
In simpler words, each curly brace pair{}will be substituted with the arguments, specified
byminislurm_clientprogram (uselist --commandcommand to inspect a command template of a
server instance).TIMEOUTdetermines how much time is given to the process to finish. It this time is
exceeded, process will be terminated. May be overriden by a user.TIMEOUT_KILLdetermines much time is given to the process to terminate (to save data,
cleanup etc.). It this time is exceeded, process will be killed.Each configuration may be overriden by environment variables with nameMINISLURM_<SECTION>_<CONFIG>(e.g.MINISLURM_SERVER_MAX_PARALLEL,MINISLURM_PROGRAM_COMMAND).SystemdSystemd template file <minislurm@.service> may be used to start server instances and control
resources.
It should be placed inside~/.config/systemd/userfolder to be used as a local user.
This configuration assumes that minislurm configuration files are placed in users$HOMEdirectory
and named.minislurm_<instance_name>.ini.
For example, for the configuration file~/.minislurm_openfoam.iniserver instance may be start
with commandsystemctl --user start minislurm@openfoam.service.
Note, thatSOCKETconfiguration in~/.minislurm_openfoam.inishould be adjusted to use different
name in order to avoid instance collisions.
Minislurm service instance may be started using commandsystemctlstart--userminislurm@openfoamAdjustingCPUQuotaandMemoryMaxlimits should be done on per-instance basis.
After starting the service create a drop-in override by issuing the commandsystemctledit--userminislurm@openfoamIn the opened text file add lines[Service]MemoryMax=10GCPUQuota=800%This particular configuration will limit memory usage to 10Gb and allow using up to 8 CPU threads.Enable service to start minislurm service automatically on system startupsystemctlenable--userminislurm@openfoamNote that running server as root is extremely dangerous. Instead, create a dedicated user and group for global
minislurm instance.Job submissionSocket selectionminislurm_clientcommand is used to submit jobs to server.
Firstly, client should know server socket location.
It may be supplied directly usingsocketargument or be read from configuration file pointed to byconfigargument.Examples:Connect to socket at specific locationminislurm_clientsocket/tmp/minislurm.socketlist--allRead socket location from configuration fileminislurm_clientconfig~/.minislurm_test.inilist--allIt may be handy to define shell aliases for server instancesaliasminislurm_openfoam="minislurm_client config ~/.minislurm_openfoam.ini"This allows quick access to specific server instanceminislurm_openfoamlist--allAdd jobJob submission syntaxminislurm_client(socket<socket>|config<config>)add[--path=<path>--name=<name>--stdout=<stdout>--stderr=<stderr>--timeout=<timeout>]--<args>...Mandatory mutually exclusive options<socket>and<config>are explained in a section above.To submit job user must at least supply a list of arguments<args>to fill a command template.
Use quotes""and''to group space separated words together.
For example, supplying command templateecho There are {} apples in the {}with argumentsthirty two basketwould expand asecho There are thirty apples in the two.
When wrapping word group in quotes"thirty two" basketexpansion resultThere are thirty two apples in the basketmakes much more sense.Other options are:<path>is a path to run program from. Defaults to the directory, from which call was made.<name>is a name of a process or a process group. Multiple processes may share the name, which
may be used remove/pause/continue them all.<timeout>overrides globalTIMEOUTsetting for job cancellation.<stdout>and<stderr>specify files to which write program's stdout and stderr streams.There's another version of the add commandminislurm_client(socket<socket>|config<config>)add<base_name>[--path=<path>--timeout=<timeout>]--<args>...In this shortcut version<base_name>will be used as a<name>of a job;
stdout and stderr files will be called<base_name>.outand<base_name>.err.Examples:minislurm_client config ~/.minislurm_test.ini add --stdout /tmp/1.out --name $USER --timeout "1m 1second" -- 'thirty two' basketminislurm_client socket /tmp/minislurm_test.socket add take1 -- arg1 arg2minislurm_client config ~/.minislurm_test.ini add --stdout /tmp/1.out --name $USER --timeout "1m 1second" -- 'thirty two' basketRemove/pause/continue jobsRemove, pause and continue commands remove, pause and continue specified job respectively.
Their syntax is similar.minislurm_client(socket<socket>|config<config>)rm(--all|--id=<id>|--name<name>)minislurm_client(socket<socket>|config<config>)pause(--all|--id=<id>|--name<name>)minislurm_client(socket<socket>|config<config>)continue(--all|--id=<id>|--name=<name>)Mandatory mutually exclusive options<socket>and<config>are explained in a section above.Option--alldoes required action for all jobs in queue.
Note that if job is paused while waiting for the execution, it will get a new ID when continued.<id>and<name>arguments allow selecting job by id or name respectively.
This arguments allow usingregexto select multiple
jobs.
Strings are matched partially from the beginning of the string.
For example, selector1would match all IDs or names beginning with1.
If you want to match string exactly, terminate selector with$character.Examples:Remove all jobsminislurm_client config ~/.minislurm_test.ini rm --allStop jobs with IDs ending with 1, 2, 3 or 4minislurm_client config ~/.minislurm_test.ini stop --id '.*[1234]$'Continue execution of jobs with name containing stringunitand number1maybe separated by
non-numeric character; matching is case insensitiveminislurm_client config ~/.minislurm_test.ini continue --name "(?i).*unit[^\d]?1"List jobsJob list syntaxminislurm_client(socket<socket>|config<config>)list(--all|--command|--ids|--names|--id=<id>|--name=<name>)Mandatory mutually exclusive options<socket>and<config>are explained in a section above.Option--alllists all jobs in queue.<id>and<name>arguments allow selecting job by id or name respectively using regex selectors.
Options--idsand--nameswill list all IDs and unique names in queue.Examples:List all jobsminislurm_client config ~/.minislurm_test.ini list --allList jobs with IDs ending with 1, 2, 3 or 4minislurm_client config ~/.minislurm_test.ini list --id '.*[1234]$'List jobs with name containing stringunitand number1maybe separated by
non-numeric character; matching is case insensitiveminislurm_client config ~/.minislurm_test.ini list --name "(?i).*unit[^\d]?1"Job statusTable of possible job statesStateDescriptionQUEUEDJob is waiting to be executedRUNNINGJob is runningCOMPLETEDJob is completedFAILEDJob is completed with non-zero exit statusTERMINATINGServer is terminating a jobTERMINATEDJob is terminatedKILLEDJob exceeded termination time and was killedPAUSEDJob was running and now its execution is pausedHELDJob was waiting and now its execution is deferredSetup exampleSimulations usingDolfinXFEM library might be quite resource heavy so it makes sense to
manage simulation jobs and machine resources using systemd and minislurm.Firstly, we copy sample systemd service fileminislurm@.serviceto the~.config/systemd/user/directory.
Assuming thatdolfinxC++ library files are located in/opt/dolfinx/usr/directory,
and the python virtual environment is in/opt/dolfinx/dolfinx_env, we override
environment variables for our service instancesystemctledit--userminislurm@dolfinx.serviceAnd set required envvars and limits for CPU and Memory[Service]Environment=PETSC_DIR=/usr/lib/petscdir/petsc-complexEnvironment=SLEPC_DIR=/usr/lib/slepcdir/slepc-complexEnvironment=PETSC_ARCH=linux-gnu-complex-64Environment=LD_LIBRARY_PATH=/opt/dolfinx/usr/libEnvironment=PKG_CONFIG_PATH=/opt/dolfinx/usr/lib/pkgconfigEnvironment=VIRTUAL_ENV=/opt/dolfinx/dolfinx_envEnvironment=PATH=/opt/dolfinx/usr/bin:/opt/dolfinx/dolfinx_env/bin:/usr/local/bin:/usr/bin:/binEnvironment=PYTHONPATH=/usr/lib/petscdir/petsc-complex/lib/python3/dist-packages:/usr/lib/slepcdir/slepc-complex/lib/python3/dist-packages:/opt/dolfinx/dolfinx_env/lib/python3.9/site-packagesMemoryMax=10GCPUQuota=400%Simply closing the file editor to apply these settings.Next, we copyconfig.ini.samplefile to ~/.minislurm_dolfinx.ini and adjusting it[SERVER]SOCKET=/tmp/minislurm_dolfinx.socket# server socketTIMEZONE_OFFSET=+3# timezone offsetMAX_PARALLEL=1# number of running processesQUEUE_SIZE=100# queue sizeUPDATE_TIME=1# queue update period in secondsLOG_LEVEL=INFO# server log level[PROGRAM]COMMAND=python3 {}# command arguments in curly braces are set by clientTIMEOUT=2h# execution timeout. awailable units are s,m,h,d,wKILL_TIMEOUT=10m# soft stop timeout. awailable units are s,m,h,d,wNow out service is ready to be startedsystemctlstart--userminislurm@dolfinx.serviceOptionally enabling service autostartsystemctlenable--userminislurm@dolfinx.serviceFor convenience adding command alias to the~/.profilefilealiasminislurm_dolfinx="minislurm_client config ~/.minislurm_dolfinx.ini"That is it.
Now adding a dolfinx script to job queue simply by typingminislurm_dolfinx add testrun -- script.py