neulab/tldr huggingface dataset:
Gres=<list> Specifies a comma delimited list of generic consumable resources. The format of each entry on the list is "name[:count[*cpu]]".  The name is that of the consumable resource.  The count is the number of those resources with a default value of 1.  The specified resources will be allocated to the job on each node allocated unless "*cpu" is appended, in which case the resources will be allocated on a per cpu basis.  The available generic consumable resources is configurable by the system administrator.  A list of available generic consumable resources will be printed and the command will exit if the option argument is "help".  Examples of use include "Gres=gpus:2*cpu,disk=40G" and "Gres=help".
JobId=<job_list> Identify the job(s) to be updated.  The job_list may be a comma separated list of job IDs.  Either JobId or JobName is required.
Licenses=<name> Specification of licenses (or other resources available on all nodes of the cluster) as described in salloc/sbatch/srun man pages.
MailType=<types> Set the mail event types. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL (equivalent to BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage out and teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent of time limit), TIME_LIMIT_80 (reached 80 percent of time limit), TIME_LIMIT_50 (reached 50 percent of time limit) and ARRAY_TASKS (send emails for each array task). Multiple type values may be specified in a comma separated list. Unless the ARRAY_TASKS option is specified, mail notifications on job BEGIN, END and FAIL apply to a job array as a whole rather than generating individual email messages for each task in the job array.
MailUser=<name> Set the user to receive email notification of state changes. A blank string will set the mail user to the default which is the submitting user.
MinCPUsNode=<count> Set the job's minimum number of CPUs per node to the specified value.
MinMemoryCPU=<megabytes> Set the job's minimum real memory required per allocated CPU to the specified value. Either MinMemoryCPU or MinMemoryNode may be set, but not both.
MinMemoryNode=<megabytes> Set the job's minimum real memory required per node to the specified value.  Either MinMemoryCPU or MinMemoryNode may be set, but not both.
MinTmpDiskNode=<megabytes> Set the job's minimum temporary disk space required per node to the specified value.  Only the Slurm administrator or root can change this parameter.
TimeMin=<timespec> Change TimeMin value which specifies the minimum time limit minutes of the job.
JobName=<name> Identify the name of jobs to be modified or set the job's name to the specified value.  When used to identify jobs to be modified, all jobs belonging to all users are modified unless the UserID option is used to identify a specific user.  Either JobId or JobName is required.
Name[=<name>] See JobName.
Nice[=<adjustment>] Update the job with an adjusted scheduling priority within Slurm. With no adjustment value the scheduling priority is decreased by 100. A negative nice value increases the priority, otherwise decreases it. The adjustment range is +/- 2147483645. Only privileged users can specify a negative adjustment.
NodeList=<nodes> Change the nodes allocated to a running job to shrink its size. The specified list of nodes must be a subset of the nodes currently allocated to the job. Multiple node names may be specified using simple node range expressions (e.g. "lx[10-20]"). After a job's allocation is reduced, subsequent srun commands must explicitly specify node and task counts which are valid for the new allocation.
NumCPUs=<min_count>[-<max_count>] Set the job's minimum and optionally maximum count of CPUs to be allocated.
NumNodes=<min_count>[-<max_count>] Set the job's minimum and optionally maximum count of nodes to be allocated.  If the job is already running, use this to specify a node count less than currently allocated and resources previously allocated to the job will be relinquished. After a job's allocation is reduced, subsequent srun commands must explicitly specify node and task counts which are valid for the new allocation. Also see the NodeList parameter above. This is the same than ReqNodes.
NumTasks=<count> Set the job's count of required tasks to the specified value. This is the same than ReqProcs.
OverSubscribe=<yes|no> Set the job's ability to share compute resources (i.e. individual CPUs) with other jobs. Possible values are "YES" and "NO".  This option can only be changed for pending jobs.
Partition=<name> Set the job's partition to the specified value.
Priority=<number> Set the job's priority to the specified value.  Note that a job priority of zero prevents the job from ever being scheduled.  By setting a job's priority to zero it is held.  Set the priority to a non-zero value to permit it to run.  Explicitly setting a job's priority clears any previously set nice value and removes the priority/multifactor plugin's ability to manage a job's priority. In order to restore the priority/multifactor plugin's ability to manage a job's priority, hold and then release the job.  Only the Slurm administrator or root can increase job's priority.
QOS=<name> Set the job's QOS (Quality Of Service) to the specified value. Value may be cleared with blank data value, "QOS=".
Reboot=<yes|no> Set the job's flag that specifies whether to force the allocated nodes to reboot before starting the job. This is only supported with some system configurations and therefore it could be silently ignored.
ReqCores=<count> Change the job's requested Cores count.
ReqNodeList=<nodes> Set the job's list of required node. Multiple node names may be specified using simple node range expressions (e.g. "lx[10-20]"). Value may be cleared with blank data value, "ReqNodeList=".
ReqNodes=<min_count>[-<max_count>] See NumNodes.
ReqProcs=<count> See NumTasks.
ReqSockets=<count> Change the job's requested socket count.
ReqThreads=<count> Change the job's requested threads count.
Requeue=<0|1> Stipulates whether a job should be requeued after a node failure: 0 for no, 1 for yes.
ReservationName=<name> Set the job's reservation to the specified value.  Value may be cleared with blank data value, "ReservationName=".
ResetAccrueTime Reset the job's accrue time value to 0 meaning it will lose any time previously accrued for priority.  Helpful if you have a large queue of jobs already in the queue and want to start limiting how many jobs can accrue time without waiting for the queue to flush out.
SiteFactor=<account> Specify the job's admin priority factor in the range of +/-2147483645.  Only privileged users can modify the value.
StdOut=<filepath> Set the batch job's stdout file path.
Shared=<yes|no> See OverSubscribe option above.
StartTime=<time_spec> Set the job's earliest initiation time.  It accepts times of the form HH:MM:SS to run a job at a specific time of day (seconds are optional).  (If that time is already past, the next day is assumed.)  You may also specify midnight, noon, fika (3 PM) or teatime (4 PM) and you can have a time-of-day suffixed with AM or PM for running in the morning or the evening.  You can also say what day the job will be run, by specifying a date of the form MMDDYY or MM/DD/YY or MM.DD.YY, or a date and time as YYYY-MM-DD[THH:MM[:SS]].  You can also give times like now + count time-units, where the time-units can be seconds (default), minutes, hours, days, or weeks and you can tell Slurm to run the job today with the keyword today and to run the job tomorrow with the keyword tomorrow.
Notes on date/time specifications: - although the 'seconds' field of the HH:MM:SS time specification is allowed by the code, note that the poll time of the Slurm scheduler is not precise enough to guarantee dispatch of the job on the exact second.  The job will be eligible to start on the next poll following the specified time. The exact poll interval depends on the Slurm scheduler (e.g., 60 seconds with the default sched/builtin). - if no time (HH:MM:SS) is specified, the default is (00:00:00). - if a date is specified without a year (e.g., MM/DD) then the current year is assumed, unless the combination of MM/DD and HH:MM:SS has already passed for that year, in which case the next year is used.
Switches=<count>[@<max-time-to-wait>] When a tree topology is used, this defines the maximum count of switches desired for the job allocation. If Slurm finds an allocation containing more switches than the count specified, the job remain pending until it either finds an allocation with desired switch count or the time limit expires. By default there is no switch count limit and no time limit delay. Set the count to zero in order to clean any previously set count (disabling the limit).  The job's maximum time delay may be limited by the system administrator using the SchedulerParameters configuration parameter with the max_switch_wait parameter option.  Also see wait-for-switch.
wait-for-switch=<seconds> Change max time to wait for a switch <seconds> secs.
TasksPerNode=<count> Change the job's requested TasksPerNode.
ThreadSpec=<count> Number of threads to reserve per node for system use.  The job will be charged for these threads, but be unable to use them. Will be reported as "*" if not constrained.
TimeLimit=<time> The job's time limit.  Output format is [days-]hours:minutes:seconds or "UNLIMITED".  Input format (for update command) set is minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes or days-hours:minutes:seconds.  Time resolution is one minute and second values are rounded up to the next minute.  If changing the time limit of a job, either specify a new time limit value or precede the time and equal sign with a "+" or "-" to increment or decrement the current time limit (e.g. "TimeLimit+=30"). In order to increment or decrement the current time limit, the JobId specification must precede the TimeLimit specification.  Note that incrementing or decrementing the time limit for a job array is only allowed before the job array has been split into more than one job record.  Only the Slurm administrator or root can increase job's TimeLimit.
UserID=<UID or name> Used with the JobName option to identify jobs to be modified. Either a user name or numeric ID (UID), may be specified.
WCKey=<key> Set the job's workload characterization key to the specified value.
WorkDir=<directory_name> Set the job's working directory to the specified value. Note that this may only be set for jobs in the PENDING state, and that jobs may fail to launch if they rely on relative paths to the originally submitted WorkDir.
SPECIFICATIONS FOR SHOW COMMAND, JOBS
The "show" command, when used with the "job" or "job <jobid>" entity displays detailed information about a job or jobs.  Much of this information may be modified using the "update job" command as described above.  However, the following fields displayed by the show job command are read-only and cannot be modified:
AllocNode:Sid Local node and system id making the resource allocation.
BatchFlag Jobs submitted using the sbatch command have BatchFlag set to 1. Jobs submitted using other commands have BatchFlag set to 0.
ExitCode=<exit>:<sig> Exit status reported for the job by the wait() function.  The first number is the exit code, typically as set by the exit() function.  The second number of the signal that caused the process to terminate if it was terminated by a signal.
GroupId The group under which the job was submitted.
JobState The current state of the job.
NodeListIndices The NodeIndices expose the internal indices into the node table associated with the node(s) allocated to the job.
NtasksPerN:B:S:C= <tasks_per_node>:<tasks_per_baseboard>:<tasks_per_socket>:<tasks_per_core> Specifies the number of tasks to be started per hardware component (node, baseboard, socket and core).  Unconstrained values may be shown as "0" or "*".
PreemptEligibleTime Time the job becomes eligible for preemption. Modified by PreemptExemptTime, either from the global option in slurm.conf or the job QOS. This is hidden if the job has not started or if PreemptMode=OFF.
PreemptTime Time at which job was signaled that it was selected for preemption.  (Meaningful only for PreemptMode=CANCEL and the partition or QOS with which the job is associated has a GraceTime value designated.) This is hidden if the job has not started or if PreemptMode=OFF.
PreSusTime Time the job ran prior to last suspend.
Reason The reason job is not running: e.g., waiting "Resources".
ReqB:S:C:T= <baseboard_count>:<socket_per_baseboard_count>:<core_per_socket_count>:<thread_per_core_count> Specifies the count of various hardware components requested by the job.  Unconstrained values may be shown as "0" or "*".
SecsPreSuspend=<seconds> If the job is suspended, this is the run time accumulated by the job (in seconds) prior to being suspended.
Socks/Node=<count> Count of desired sockets per node
SubmitTime The time  and  date stamp (in localtime) the job was submitted. The format of the output is identical to that of the EndTime field.
NOTE: If a job is requeued, the submit time is reset.  To obtain the original submit time it is necessary to use the "sacct -j <job_id[.<step_id>]" command also designating the -D or --duplicate option to display all duplicate entries for a job.
SuspendTime Time the job was last suspended or resumed.
NOTE on information displayed for various job states: When you submit a request for the "show job" function the scontrol process makes an RPC request call to slurmctld with a REQUEST_JOB_INFO message type.  If the state of the job is PENDING, then it returns some detail information such as: min_nodes, min_procs, cpus_per_task, etc. If the state is other than PENDING the code assumes that it is in a further state such as RUNNING, COMPLETE, etc. In these cases the code explicitly returns zero for these values. These values are meaningless once the job resources have been allocated and the job has started.
SPECIFICATIONS FOR UPDATE COMMAND, STEPS
StepId=<job_id>[.<step_id>] Identify the step to be updated.  If the job_id is given, but no step_id is specified then all steps of the identified job will be modified.  This specification is required.
CompFile=<completion file> Update a step with information about a step's completion.  Can be useful if step statistics aren't directly available through a jobacct_gather plugin.  The file is a space-delimited file with format for Version 1 is as follows
1 34461 0 2 0 3 1361906011 1361906015 1 1 3368 13357 /bin/sleep A B     C D E F G          H          I J K    L     M
Field Descriptions:
A file version B ALPS apid C inblocks D outblocks E exit status F number of allocated CPUs G start time H end time I utime J stime K maxrss L uid M command name
TimeLimit=<time> The job's time limit.  Output format is [days-]hours:minutes:seconds or "UNLIMITED".  Input format (for update command) set is minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes or days-hours:minutes:seconds.  Time resolution is one minute and second values are rounded up to the next minute.  If changing the time limit of a step, either specify a new time limit value or precede the time with a "+" or "-" to increment or decrement the current time limit (e.g. "TimeLimit=+30"). In order to increment or decrement the current time limit, the StepId specification must precede the TimeLimit specification.
SPECIFICATIONS FOR UPDATE COMMAND, NODES
NodeName=<name> Identify the node(s) to be updated. Multiple node names may be specified using simple node range expressions (e.g. "lx[10-20]"). This specification is required.
ActiveFeatures=<features> Identify the feature(s) currently active on the specified node. Any previously active feature specification will be overwritten with the new value.  Also see AvailableFeatures.  Typically ActiveFeatures will be identical to AvailableFeatures; however ActiveFeatures may be configured as a subset of the AvailableFeatures. For example, a node may be booted in multiple configurations. In that case, all possible configurations may be identified as AvailableFeatures, while ActiveFeatures would identify the current node configuration.
AvailableFeatures=<features> Identify the feature(s) available on the specified node.  Any previously defined available feature specification will be overwritten with the new value.  AvailableFeatures assigned via scontrol will only persist across the restart of the slurmctld daemon with the -R option and state files preserved or slurmctld's receipt of a SIGHUP.  Update slurm.conf with any changes meant to be persistent across normal restarts of slurmctld or the execution of scontrol reconfig.
Note: Available features being removed via scontrol must not be active (i.e. remove them from ActiveFeatures first).
Comment=<comment> Arbitrary descriptive string.  Use quotes to enclose a comment having more than one word
CpuBind=<node> Specify the task binding mode to be used by default for this node. Supported options include: "none", "board", "socket", "ldom" (NUMA), "core", "thread" and "off" (remove previous binding mode).
Gres=<gres> Identify generic resources to be associated with the specified node.  Any previously defined generic resources will be overwritten with the new value.  Specifications for multiple generic resources should be comma separated.  Each resource specification consists of a name followed by an optional colon with a numeric value (default value is one) (e.g. "Gres=bandwidth:10000").  Modification of GRES count associated with specific files (e.g. GPUs) is not allowed other than to set their count on a node to zero.  In order to change the GRES count to another value, modify your slurm.conf and gres.conf files and restart daemons.  If GRES as associated with specific sockets, that information will be reported For example if all 4 GPUs on a node are all associated with socket zero, then "Gres=gpu:4(S:0)". If associated with sockets 0 and 1 then "Gres=gpu:4(S:0-1)".  The information of which specific GPUs are associated with specific GPUs is not reported, but only available by parsing the gres.conf file.  Generic resources assigned via scontrol will only persist across the restart of the slurmctld daemon with the -R option and state files preserved or slurmctld's receipt of a SIGHUP.  Update slurm.conf with any changes meant to be persistent across normal restarts of slurmctld or the execution of scontrol reconfig.
NodeAddr=<node address> Name that a node should be referred to in establishing a communications path.  This name will be used as an argument to the getaddrinfo() function for identification. If a node range expression is used to designate multiple nodes, they must exactly match the entries in the NodeName (e.g. "NodeName=lx[0-7] NodeAddr=elx[0-7]"). NodeAddr may also contain IP addresses.
NodeHostname=<node hostname> Typically this would be the string that "/bin/hostname -s" returns.  It may also be the fully qualified domain name as returned by "/bin/hostname -f" (e.g. "foo1.bar.com"), or any valid domain name associated with the host through the host database (/etc/hosts) or DNS, depending on the resolver settings. Note that if the short form of the hostname is not used, it may prevent use of hostlist expressions (the numeric portion in brackets must be at the end of the string). A node range expression can be used to specify a set of nodes. If an expression is used, the number of nodes identified by NodeHostname must be identical to the number of nodes identified by NodeName.
Reason=<reason> Identify the reason the node is in a "DOWN", "DRAINED", "DRAINING", "FAILING" or "FAIL" state.  Use quotes to enclose a reason having more than one word.
Weight=<weight> Identify weight to be associated with specified nodes. This allows dynamic changes to weight associated with nodes, which will be used for the subsequent node allocation decisions.  Weight assigned via scontrol will only persist across the restart of the slurmctld daemon with the -R option and state files preserved or slurmctld's receipt of a SIGHUP.  Update slurm.conf with any changes meant to be persistent across normal restarts of slurmctld or the execution of scontrol reconfig.
SPECIFICATIONS FOR SHOW COMMAND, NODES
The meaning of the energy information is as follows:
CurrentWatts The instantaneous power consumption of the node at the time of the last node energy accounting sample, in watts.
LowestJoules The energy consumed by the node between the last time it was powered on and the last time it was registered by slurmd, in joules.
ConsumedJoules The energy consumed by the node between the last time it was registered by the slurmd daemon and the last node energy accounting sample, in joules.
If the reported value is "n/s" (not supported), the node does not support the configured AcctGatherEnergyType plugin. If the reported value is zero, energy accounting for nodes is disabled.
The meaning of the external sensors information is as follows:
ExtSensorsJoules The energy consumed by the node between the last time it was powered on and the last external sensors plugin node sample, in joules.
ExtSensorsWatts The instantaneous power consumption of the node at the time of the last external sensors plugin node sample, in watts.
ExtSensorsTemp The temperature of the node at the time of the last external sensors plugin node sample, in celsius.
If the reported value is "n/s" (not supported), the node does not support the configured ExtSensorsType plugin.
The meaning of the resource specialization information is as follows:
CPUSpecList The list of Slurm abstract CPU IDs on this node reserved for exclusive use by the Slurm compute node daemons (slurmd, slurmstepd).
MemSpecLimit The combined memory limit, in megabytes, on this node for the Slurm compute node daemons (slurmd, slurmstepd).
The meaning of the memory information is as follows:
RealMemory The total memory, in MB, on the node.
AllocMem The total memory, in MB, currently allocated by jobs on the node.
FreeMem The total memory, in MB, currently free on the node as reported by the OS.
SPECIFICATIONS FOR UPDATE COMMAND, FRONTEND
FrontendName=<name> Identify the front end node to be updated. This specification is required.
Reason=<reason> Identify the reason the node is in a "DOWN" or "DRAIN" state.  Use quotes to enclose a reason having more than one word.
State=<state> Identify the state to be assigned to the front end node. Possible values are "DOWN", "DRAIN" or "RESUME".  If you want to remove a front end node from service, you typically want to set its state to "DRAIN".  "RESUME" is not an actual node state, but will return a "DRAINED", "DRAINING", or "DOWN" front end node to service, either "IDLE" or "ALLOCATED" state as appropriate.  Setting a front end node "DOWN" will cause all running and suspended jobs on that node to be terminated.
SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS, PARTITIONS
AllowGroups=<name> Identify the user groups which may use this partition.  Multiple groups may be specified in a comma separated list.  To permit all groups to use the partition specify "AllowGroups=ALL".
AllocNodes=<name> Comma separated list of nodes from which users can execute jobs in the partition.  Node names may be specified using the node range expression syntax described above.  The default value is "ALL".
Alternate=<partition name> Alternate partition to be used if the state of this partition is "DRAIN" or "INACTIVE."  The value "NONE" will clear a previously set alternate partition.
CpuBind=<node> Specify the task binding mode to be used by default for this partition.  Supported options include: "none", "board", "socket", "ldom" (NUMA), "core", "thread" and "off" (remove previous binding mode).
Default=<yes|no> Specify if this partition is to be used by jobs which do not explicitly identify a partition to use.  Possible output values are "YES" and "NO".  In order to change the default partition of a running system, use the scontrol update command and set Default=yes for the partition that you want to become the new default.
DefaultTime=<time> Run time limit used for jobs that don't specify a value. If not set then MaxTime will be used.  Format is the same as for MaxTime.
DefMemPerCPU=<MB> Set the default memory to be allocated per CPU for jobs in this partition.  The memory size is specified in megabytes.
DefMemPerNode=<MB> Set the default memory to be allocated per node for jobs in this partition.  The memory size is specified in megabytes.
DisableRootJobs=<yes|no> Specify if jobs can be executed as user root.  Possible values are "YES" and "NO".
GraceTime=<seconds> Specifies, in units of seconds, the preemption grace time to be extended to a job which has been selected for preemption.  The default value is zero, no preemption grace time is allowed on this partition or qos.  (Meaningful only for PreemptMode=CANCEL)
Hidden=<yes|no> Specify if the partition and its jobs should be hidden from view. Hidden partitions will by default not be reported by Slurm APIs or commands.  Possible values are "YES" and "NO".
JobDefaults=<specs> Specify job default values using a comma delimited list of "key=value" pairs.  Supported keys include
DefCpuPerGPU  Default number of CPUs per allocated GPU.
DefMemPerGPU  Default memory limit (in megabytes) per allocated GPU.
MaxMemPerCPU=<MB> Set the maximum memory to be allocated per CPU for jobs in this partition.  The memory size is specified in megabytes.
MaxMemPerCNode=<MB> Set the maximum memory to be allocated per node for jobs in this partition.  The memory size is specified in megabytes.
MaxNodes=<count> Set the maximum number of nodes which will be allocated to any single job in the partition. Specify a number, "INFINITE" or "UNLIMITED".  Changing the MaxNodes of a partition has no effect upon jobs that have already begun execution.
MaxTime=<time> The maximum run time for jobs.  Output format is [days-]hours:minutes:seconds or "UNLIMITED".  Input format (for update command) is minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes or days-hours:minutes:seconds.  Time resolution is one minute and second values are rounded up to the next minute.  Changing the MaxTime of a partition has no effect upon jobs that have already begun execution.
MinNodes=<count> Set the minimum number of nodes which will be allocated to any single job in the partition.  Changing the MinNodes of a partition has no effect upon jobs that have already begun execution. Increasing this value may prevent pending jobs from starting, even if they were submitted without -N/--nodes specification.  If you do get in that situation, updating the MinNodes value of a pending job using the scontrol command will allow that job to be scheduled.
Nodes=<name> Identify the node(s) to be associated with this partition. Multiple node names may be specified using simple node range expressions (e.g. "lx[10-20]").  Note that jobs may only be associated with one partition at any time.  Specify a blank data value to remove all nodes from a partition: "Nodes=".  Changing the Nodes in a partition has no effect upon jobs that have already begun execution.
OverTimeLimit=<count> Number of minutes by which a job can exceed its time limit before being canceled.  The configured job time limit is treated as a soft limit.  Adding OverTimeLimit to the soft limit provides a hard limit, at which point the job is canceled.  This is particularly useful for backfill scheduling, which bases upon each job's soft time limit.  A partition-specific OverTimeLimit will override any global OverTimeLimit value.  If not specified, the global OverTimeLimit value will take precedence.  May not exceed 65533 minutes.  An input value of "UNLIMITED" will clear any previously configured partition-specific OverTimeLimit value.
OverSubscribe=<yes|no|exclusive|force>[:<job_count>] Specify if compute resources (i.e. individual CPUs) in this partition can be shared by multiple jobs.  Possible values are "YES", "NO", "EXCLUSIVE" and "FORCE".  An optional job count specifies how many jobs can be allocated to use each resource.
PartitionName=<name> Identify the partition to be updated. This specification is required.
PreemptMode=<mode> Reset the mechanism used to preempt jobs in this partition if PreemptType is configured to preempt/partition_prio. The default preemption mechanism is specified by the cluster-wide PreemptMode configuration parameter.  Possible values are "OFF", "CANCEL", "REQUEUE" and "SUSPEND".
Priority=<count> Jobs submitted to a higher priority partition will be dispatched before pending jobs in lower priority partitions and if possible they will preempt running jobs from lower priority partitions. Note that a partition's priority takes precedence over a job's priority.  The value may not exceed 65533.
PriorityJobFactor=<count> Partition factor used by priority/multifactor plugin in calculating job priority.  The value may not exceed 65533.  Also see PriorityTier.
PriorityTier=<count> Jobs submitted to a partition with a higher priority tier value will be dispatched before pending jobs in partition with lower priority tier value and,  if  possible,  they  will  preempt running jobs from partitions with lower priority tier values. Note that a partition's priority tier takes precedence over a job's priority.  The value may not exceed 65533.  Also see PriorityJobFactor.
QOS=<QOSname|blank to remove> Set the partition QOS with a QOS name or to remove the Partition QOS leave the option blank.
RootOnly=<yes|no> Specify if only allocation requests initiated by user root will be satisfied.  This can be used to restrict control of the partition to some meta-scheduler.  Possible values are "YES" and "NO".
ReqResv=<yes|no> Specify if only allocation requests designating a reservation will be satisfied.  This is used to restrict partition usage to be allowed only within a reservation.  Possible values are "YES" and "NO".
Shared=<yes|no|exclusive|force>[:<job_count>] Renamed to OverSubscribe, see option descriptions above.
State=<up|down|drain|inactive> Specify if jobs can be allocated nodes or queued in this partition.  Possible values are "UP", "DOWN", "DRAIN" and "INACTIVE".
UP        Designates that new jobs may queued on the partition, and that jobs may be allocated nodes and run from the partition.
DOWN      Designates that new jobs may be queued on the partition, but queued jobs may not be allocated nodes and run from the partition. Jobs already running on the partition continue to run. The jobs must be explicitly canceled to force their termination.
DRAIN     Designates that no new jobs may be queued on the partition (job submission requests will be denied with an error message), but jobs already queued on the partition may be allocated nodes and run.  See also the "Alternate" partition specification.
INACTIVE  Designates that no new jobs may be queued on the partition, and jobs already queued may not be allocated nodes and run.  See also the "Alternate" partition specification.
TRESBillingWeights=<TRES Billing Weights> TRESBillingWeights is used to define the billing weights of each TRES type that will be used in calculating the usage of a job. The calculated usage is used when calculating fairshare and when enforcing the TRES billing limit on jobs.  Updates affect new jobs and not existing jobs.  See the slurm.conf man page for more information.
SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS, RESERVATIONS
Reservation=<name> Identify the name of the reservation to be created, updated, or deleted.  This parameter is required for update and is the only parameter for delete.  For create, if you do not want to give a reservation name, use "scontrol create res ..." and a name will be created automatically.
Accounts=<account list> List of accounts permitted to use the reserved nodes, for example "Accounts=physcode1,physcode2".  A user in any of the accounts may use the reserved nodes.  A new reservation must specify Users or Groups and/or Accounts.  If both Users/Groups and Accounts are specified, a job must match both in order to use the reservation. Accounts can also be denied access to reservations by preceding all of the account names with '-'. Alternately precede the equal sign with '-'.  For example, "Accounts=-physcode1,-physcode2" or "Accounts-=physcode1,physcode2" will permit any account except physcode1 and physcode2 to use the reservation.  You can add or remove individual accounts from an existing reservation by using the update command and adding a '+' or '-' sign before the '=' sign.  If accounts are denied access to a reservation (account name preceded by a '-'), then all other accounts are implicitly allowed to use the reservation and it is not possible to also explicitly specify allowed accounts.
BurstBuffer=<buffer_spec>[,<buffer_spec>,...] Specification of burst buffer resources which are to be reserved. "buffer_spec" consists of four elements: [plugin:][type:]#[units] "plugin" is the burst buffer plugin name, currently either "datawarp" or "generic".  If no plugin is specified, the reservation applies to all configured burst buffer plugins. "type" specifies a Cray generic burst buffer resource, for example "nodes".  if "type" is not specified, the number is a measure of storage space.  The "units" may be "N" (nodes), "K|KiB", "M|MiB", "G|GiB", "T|TiB", "P|PiB" (for powers of 1024) and "KB", "MB", "GB", "TB", "PB" (for powers of 1000).  The default units are bytes for reservations of storage space.  For example "BurstBuffer=datawarp:2TB" (reserve 2TB of storage plus 3 nodes from the Cray plugin) or "BurstBuffer=100GB" (reserve 100 GB of storage from all configured burst buffer plugins).  Jobs using this reservation are not restricted to these burst buffer resources, but may use these reserved resources plus any which are generally available.  NOTE: Usually Slurm interprets KB, MB, GB, TB, PB, TB units as powers of 1024, but for Burst Buffers size specifications Slurm supports both IEC/SI formats.  This is because the CRAY API for managing DataWarps supports both formats.
CoreCnt=<num> This option is only supported when SelectType=select/cons_res or select/cons_tres. Identify number of cores to be reserved.  If NodeCnt is used without the FIRST_CORES flag, this is the total number of cores to reserve where cores per node is CoreCnt/NodeCnt.  If a nodelist is used, or if NodeCnt is used with the FIRST_CORES flag, this should be an array of core numbers by node: Nodes=node[1-5] CoreCnt=2,2,3,3,4 or flags=FIRST_CORES NodeCnt=5 CoreCnt=1,2,1,3,2.
Licenses=<license> Specification of licenses (or other resources available on all nodes of the cluster) which are to be reserved.  License names can be followed by a colon and count (the default count is one). Multiple license names should be comma separated (e.g. "Licenses=foo:4,bar").  A new reservation must specify one or more resource to be included: NodeCnt, Nodes and/or Licenses.  If a reservation includes Licenses, but no NodeCnt or Nodes, then the option Flags=LICENSE_ONLY must also be specified.  Jobs using this reservation are not restricted to these licenses, but may use these reserved licenses plus any which are generally available.
MaxStartDelay[=<timespec>] Change MaxStartDelay value which specifies the maximum time an eligible job not requesting this reservation can delay a job requesting it. Default is none.  Valid formats are minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes, days-hours:minutes:seconds.  Time resolution is one minute and second values are rounded up to the next minute. Output format is always [days-]hours:minutes:seconds.
NodeCnt=<num>[,num,...] Identify number of nodes to be reserved. The number can include a suffix of "k" or "K", in which case the number specified is multiplied by 1024.  A new reservation must specify one or more resource to be included: NodeCnt, Nodes and/or Licenses.
Nodes=<name> Identify the node(s) to be reserved. Multiple node names may be specified using simple node range expressions (e.g. "Nodes=lx[10-20]").  Specify a blank data value to remove all nodes from a reservation: "Nodes=".  A new reservation must specify one or more resource to be included: NodeCnt, Nodes and/or Licenses. A specification of "ALL" will reserve all nodes. Set Flags=PART_NODES and PartitionName= in order for changes in the nodes associated with a partition to also be reflected in the nodes associated with a reservation.
StartTime=<time_spec> The start time for the reservation.  A new reservation must specify a start time.  It accepts times of the form HH:MM:SS for a specific time of day (seconds are optional).  (If that time is already past, the next day is assumed.)  You may also specify midnight, noon, fika (3 PM) or teatime (4 PM) and you can have a time-of-day suffixed with AM or PM for running in the morning or the evening.  You can also say what day the job will be run, by specifying a date of the form MMDDYY or MM/DD/YY or MM.DD.YY, or a date and time as YYYY-MM-DD[THH:MM[:SS]].  You can also give times like now + count time-units, where the time-units can be seconds (default), minutes, hours, days, or weeks and you can tell Slurm to run the job today with the keyword today and to run the job tomorrow with the keyword tomorrow. You cannot update the StartTime of a reservation in ACTIVE state.
EndTime=<time_spec> The end time for the reservation.  A new reservation must specify an end time or a duration.  Valid formats are the same as for StartTime.
Duration=<time> The length of a reservation.  A new reservation must specify an end time or a duration.  Valid formats are minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes, days-hours:minutes:seconds, or UNLIMITED. Time resolution is one minute and second values are rounded up to the next minute. Output format is always [days-]hours:minutes:seconds.
PartitionName=<name> Partition used to reserve nodes from. This will attempt to allocate all nodes in the specified partition unless you request fewer resources than are available with CoreCnt, NodeCnt or TRES. Jobs will be allowed to use this reservation even if running in a different partition. There only needs to be overlapping nodes from that different partition and the nodes used in the reservation.
Flags=<flags> Flags associated with the reservation.  You can add or remove individual flags from an existing reservation by adding a '+' or '-' sign before the '=' sign.  For example: Flags-=DAILY (NOTE: this shortcut is not supported for all flags).  Currently supported flags include:
ANY_NODES     This is a reservation for burst buffers and/or licenses only and not compute nodes.  If this flag is set, a job using this reservation may use the associated burst buffers and/or licenses plus any compute nodes.  If this flag is not set, a job using this reservation may use only the nodes and licenses associated with the reservation.
DAILY         Repeat the reservation at the same time every day.
FLEX          Permit jobs requesting the reservation to begin prior to the reservation's start time, end after the reservation's end time, and use any resources inside and/or outside of the reservation regardless of any constraints possibly set in the reservation. A typical use case is to prevent jobs not explicitly requesting the reservation from using those reserved resources rather than forcing jobs requesting the reservation to use those resources in the time frame reserved. Another use case could be to always have a particular number of nodes with a specific feature reserved for a specific account so users in this account may use this nodes plus possibly other nodes without this feature.
FIRST_CORES   Use the lowest numbered cores on a node only. Flag removal with '-=' is not supported.
IGNORE_JOBS   Ignore currently running jobs when creating the reservation.  This can be especially useful when reserving all nodes in the system for maintenance.
LICENSE_ONLY  See ANY_NODES.
MAINT         Maintenance mode, receives special accounting treatment.  This reservation is permitted to use resources that are already in another reservation.
MAGNETIC      This flag allows jobs to be considered for this reservation even if they didn't request it.
NO_HOLD_JOBS_AFTER By default, when a reservation ends the reservation request will be removed from any pending jobs submitted to the reservation and will be put into a held state.  Use this flag to let jobs run outside of the reservation after the reservation is gone. Flag removal with '-=' is not supported.
OVERLAP       This reservation can be allocated resources that are already in another reservation. Flag removal with '-=' is not supported.
PART_NODES    This flag can be used to reserve all nodes within the specified partition.  PartitionName and Nodes=ALL must be specified with this flag.
PURGE_COMP[=<timespec>] Purge the reservation if it is ever idle for timespec (no jobs associated with it).  If timespec isn't given then 5 minutes is the default.  Valid timespec formats are minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes, days-hours:minutes:seconds. Time resolution is one minute and second values are rounded up to the next minute. Output format is always [days-]hours:minutes:seconds.
REPLACE       Nodes which are DOWN, DRAINED, or allocated to jobs are automatically replenished using idle resources. This option can be used to maintain a constant number of idle resources available for pending jobs (subject to availability of idle resources).  This should be used with the NodeCnt reservation option; do not identify specific nodes to be included in the reservation. Flag removal with '-=' is not supported.
NOTE: Removing a node from the cluster while in a reservation with the REPLACE flag will not cause it to be replaced.
REPLACE_DOWN  Nodes which are DOWN or DRAINED are automatically replenished using idle resources.  This option can be used to maintain a constant sized pool of resources available for pending jobs (subject to availability of idle resources).  This should be used with the NodeCnt reservation option; do not identify specific nodes to be included in the reservation. Flag removal with '-=' is not supported.
NOTE: Removing a node from the cluster while in a reservation with the REPLACE_DOWN flag will not cause it to be replaced.
SPEC_NODES    Reservation is for specific nodes (output only).
STATIC_ALLOC  Make it so after the nodes are selected for a reservation they don't change.  Without this option when nodes are selected for a reservation and one goes down the reservation will select a new node to fill the spot.
TIME_FLOAT    The reservation start time is relative to the current time and moves forward through time (e.g. a StartTime=now+10minutes will always be 10 minutes in the future). Repeating (e.g. DAILY) floating reservations are not supported. Flag cannot be added to or removed from an existing reservation.
WEEKDAY       Repeat the reservation at the same time on every weekday (Monday, Tuesday, Wednesday, Thursday and Friday).
WEEKEND       Repeat the reservation at the same time on every weekend day (Saturday and Sunday).
WEEKLY        Repeat the reservation at the same time every week.
Features=<features> Set the reservation's required node features. Multiple values may be "&" separated if all features are required (AND operation) or separated by "|" if any of the specified features are required (OR operation).  Parenthesis are also supported for features to be ANDed together with counts of nodes having the specified features. For example "Features=[(knl&a2a&flat)*4&haswell*2]" indicates the advanced reservation should include 4 nodes with ALL of the features "knl", "a2a", and "flat" plus 2 nodes with the feature "haswell".
Value may be cleared with blank data value, "Features=".
Groups=<group list> List of groups permitted to use the reserved nodes, for example "Group=bio,chem".  A new reservation must specify Users or Groups and/or Accounts.  If both Users/Groups and Accounts are specified, a job must match both in order to use the reservation.  Unlike users groups do not allow denied access to reservations.  You can add or remove individual groups from an existing reservation by using the update command and adding a '+' or '-' sign before the '=' sign.  NOTE: Groups and Users are mutually exclusive in reservations, if you want to switch between the 2 you must update the reservation with a group='' or user='' and fill in the opposite with the appropriate setting.
Skip   Used on a reoccurring reservation, skip to the next reservation iteration.  NOTE: Only available for update.
Users=<user list> List of users permitted to use the reserved nodes, for example "User=jones1,smith2".  A new reservation must specify Users or Groups and/or Accounts.  If both Users/Groups and Accounts are specified, a job must match both in order to use the reservation. Users can also be denied access to reservations by preceding all of the user names with '-'. Alternately precede the equal sign with '-'.  For example, "User=-jones1,-smith2" or "User-=jones1,smith2" will permit any user except jones1 and smith2 to use the reservation.  You can add or remove individual users from an existing reservation by using the update command and adding a '+' or '-' sign before the '=' sign.  If users are denied access to a reservation (user name preceded by a '-'), then all other users are implicitly allowed to use the reservation and it is not possible to also explicitly specify allowed users.  NOTE: Groups and Users are mutually exclusive in reservations, if you want to switch between the 2 you must update the reservation with a group='' or user='' and fill in the opposite with the appropriate setting.
TRES=<tres_spec> Comma-separated list of TRES required for the reservation. Current supported TRES types with reservations are: CPU, Node, License and BB. CPU and Node follow the same format as CoreCnt and NodeCnt parameters respectively.  License names can be followed by an equal '=' and a count:
License/<name1>=<count1>[,License/<name2>=<count2>,...]
BurstBuffer can be specified in a similar way as BurstBuffer parameter. The only difference is that colon symbol ':' should be replaced by an equal '=' in order to follow the TRES format.
Some examples of TRES valid specifications:
TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3
TRES=node=5k,license/iop1=2
As specified in CoreCnt, if a nodelist is specified, cpu can be an array of core numbers by node: nodes=compute[1-3] TRES=cpu=2,2,1,bb/cray=4,license/iop1=2
Please note that CPU, Node, License and BB can override CoreCnt, NodeCnt, Licenses and BurstBuffer parameters respectively.  Also CPU represents CoreCnt, in a reservation and will be adjusted if you have threads per core on your nodes.
Note that a reservation that contains nodes or cores is associated with one partition, and can't span resources over multiple partitions.  The only exception from this is when the reservation is created with explicitly requested nodes.
Executing scontrol sends a remote procedure call to slurmctld. If enough calls from scontrol or other Slurm client commands that send remote procedure calls to the slurmctld daemon come in at once, it can result in a degradation of performance of the slurmctld daemon, possibly resulting in a denial of service.
Do not run scontrol or other Slurm client commands that send remote procedure calls to slurmctld from loops in shell scripts or other programs. Ensure that programs limit calls to scontrol to the minimum necessary for the information you are trying to gather.
Some scontrol options may be set via environment variables. These environment variables, along with their corresponding options, are listed below. (Note: Commandline options will always override these settings.)
SCONTROL_ALL        -a, --all
SCONTROL_FEDERATION --federation
SCONTROL_FUTURE     -F, --future
SCONTROL_LOCAL      --local
SCONTROL_SIBLING    --sibling
SLURM_BITSTR_LEN    Specifies the string length to be used for holding a job array's task ID expression.  The default value is 64 bytes.  A value of 0 will print the full expression with any length required.  Larger values may adversely impact the application performance.
SLURM_CLUSTERS      Same as --clusters
SLURM_CONF_OUT      When running 'write config', the location of the Slurm configuration file to be written.
SLURM_TOPO_LEN      Specify the maximum size of the line when printing Topology. If not set, the default value is unlimited.
When using SlurmDBD, users who have an AdminLevel defined (Operator or Admin) and users who are account coordinators are given the authority to view and modify jobs, reservations, nodes, etc., as defined in the following table - regardless of whether a PrivateData restriction has been defined in the slurm.conf file.