neulab/tldr huggingface dataset:
reconfigure Reconfigures the SlurmDBD if running with one.
remove <ENTITY> where <SPECS> Delete the specified entities.  Identical to the delete command.
show <ENTITY> [<SPECS>] Display information about the specified entity.  By default, all entries are displayed, you can narrow results by specifying SPECS in your query.  Identical to the list command.
shutdown Shutdown the server.
version Display the version number of sacctmgr.
NOTE: All commands listed below can be used in the interactive mode, but NOT on the initial command line.
exit   Terminate sacctmgr interactive mode.  Identical to the quit command.
quiet  Print no messages other than error messages.
quit   Terminate the execution of sacctmgr interactive mode.  Identical to the exit command.
verbose Enable detailed logging.  This includes time-stamps on data structures, record counts, etc.  This is an independent command with no options meant for use in interactive mode.
!!     Repeat the last command.
account A bank account, typically specified at job submit time using the --account= option.  These may be arranged in a hierarchical fashion, for example accounts 'chemistry' and 'physics' may be children of the account 'science'.  The hierarchy may have an arbitrary depth.
association The entity used to group information consisting of four parameters: account, cluster, partition (optional), and user. Used only with the list or show command.  Add, modify, and delete should be done to a user, account or cluster entity.  This will in turn update the underlying associations.
cluster The ClusterName parameter in the slurm.conf configuration file, used to differentiate accounts on different machines.
configuration Used only with the list or show command to report current system configuration.
coordinator A special privileged user, usually an account manager, that can add users or sub-accounts to the account they are coordinator over.  This should be a trusted person since they can change limits on account and user associations, as well as cancel, requeue or reassign accounts of jobs inside their realm.
event  Events like downed or draining nodes on clusters.
federation A group of clusters that work together to schedule jobs.
job    Used to modify specific fields of a job: Derived Exit Code, the Comment String, or wckey.
problem Use with show or list to display entity problems.
qos    Quality of Service.
reservation A collection of resources set apart for use by a particular account, user or group of users for a given period of time.
resource Software resources for the system. Those are software licenses shared among clusters.
RunawayJobs Used only with the list or show command to report current jobs that have been orphaned on the local cluster and are now runaway. If there are jobs in this state it will also give you an option to "fix" them.  NOTE: You must have an AdminLevel of at least Operator to perform this.
stats  Used with list or show command to view server statistics.  Accepts optional argument of ave_time or total_time to sort on those fields. By default, sorts on increasing RPC count field.
transaction List of transactions that have occurred during a given time period.
tres   Used with list or show command to view a list of Trackable RESources configured on the system.
user   The login name. Usernames are case-insensitive (forced to lowercase) unless the PreserveCaseUser option has been set in the SlurmDBD configuration file.
wckeys Workload  Characterization  Key. An arbitrary  string  for grouping orthogonal accounts.
NOTE: The group limits (GrpJobs, GrpTRES, etc.) are tested when a job is being considered for being allocated resources.  If starting a job would cause any of its group limit to be exceeded, that job will not be considered for scheduling even if that job might preempt other jobs which would release sufficient group resources for the pending job to be initiated.
DefaultQOS=<default qos> The default QOS this association and its children should have. This is overridden if set directly on a user.  To clear a previously set value use the modify command with a new value of -1.
Fairshare=<fairshare number | parent> Number used in conjunction with other accounts to determine job priority. Can also be the string parent, when used on a user this means that the parent association is used for fairshare.  If Fairshare=parent is set on an account, that account's children will be effectively reparented for fairshare calculations to the first parent of their parent that is not Fairshare=parent.  Limits remain the same, only its fairshare value is affected.  To clear a previously set value use the modify command with a new value of -1.
GrpTRESMins=<TRES=max TRES minutes,...> The total number of TRES minutes that can possibly be used by past, present and future jobs running from this association and its children.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.
NOTE: This limit is not enforced if set on the root association of a cluster.  So even though it may appear in sacctmgr output, it will not be enforced.
ALSO NOTE: This limit only applies when using the Priority Multifactor plugin.  The time is decayed using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf.  When this limit is reached all associated jobs running will be killed and all future jobs submitted with associations in the group will be delayed until they are able to run inside the limit.
GrpTRESRunMins=<TRES=max TRES run minutes,...> Used to limit the combined total number of TRES minutes used by all jobs running with this association and its children.  This takes into consideration time limit of running jobs and consumes it, if the limit is reached no new jobs are started until other jobs finish to allow time to free up.
GrpTRES=<TRES=max TRES,...> Maximum number of TRES running jobs are able to be allocated in aggregate for this association and all associations which are children of this association.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.
NOTE: This limit only applies fully when using the Select Consumable Resource plugin.
GrpJobs=<max jobs> Maximum number of running jobs in aggregate for this association and all associations which are children of this association.  To clear a previously set value use the modify command with a new value of -1.
GrpJobsAccrue=<max jobs> Maximum number of pending jobs in aggregate able to accrue age priority for this association and all associations which are children of this association.  To clear a previously set value use the modify command with a new value of -1.
GrpSubmitJobs=<max jobs> Maximum number of jobs which can be in a pending or running state at any time in aggregate for this association and all associations which are children of this association.  To clear a previously set value use the modify command with a new value of -1.
NOTE: This setting shows up in the sacctmgr output as GrpSubmit.
GrpWall=<max wall> Maximum wall clock time running jobs are able to be allocated in aggregate for this association and all associations which are children of this association.  To clear a previously set value use the modify command with a new value of -1.
MaxTRESMinsPerJob=<max TRES minutes> Maximum number of TRES minutes each job is able to use in this association.  This is overridden if set directly on a user. Default is the cluster's limit.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.
NOTE: This setting shows up in the sacctmgr output as MaxTRESMins.
MaxTRESPerJob=<max TRES> Maximum number of TRES each job is able to use in this association.  This is overridden if set directly on a user. Default is the cluster's limit.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.
NOTE: This setting shows up in the sacctmgr output as MaxTRES.
NOTE: This limit only applies fully when using cons_res or cons_tres select type plugins.
MaxJobs=<max jobs> Maximum number of jobs each user is allowed to run at one time in this association.  This is overridden if set directly on a user. Default is the cluster's limit.  To clear a previously set value use the modify command with a new value of -1.
MaxJobsAccrue=<max jobs> Maximum number of pending jobs able to accrue age priority at any given time for the given association.  This is overridden if set directly on a user.  Default is the cluster's limit.  To clear a previously set value use the modify command with a new value of -1.
MaxSubmitJobs=<max jobs> Maximum number of jobs which can this association can have in a pending or running state at any time.  Default is the cluster's limit.  To clear a previously set value use the modify command with a new value of -1.
NOTE: This setting shows up in the sacctmgr output as MaxSubmit.
MaxWallDurationPerJob=<max wall> Maximum wall clock time each job is able to use in this association.  This is overridden if set directly on a user. Default is the cluster's limit.  <max wall> format is <min> or <min>:<sec> or <hr>:<min>:<sec> or <days>-<hr>:<min>:<sec> or <days>-<hr>.  The value is recorded in minutes with rounding as needed.  To clear a previously set value use the modify command with a new value of -1.
NOTE: Changing this value will have no effect on any running or pending job.
NOTE: This setting shows up in the sacctmgr output as MaxWall.
Priority What priority will be added to a job's priority when using this association.  This is overridden if set directly on a user. Default is the cluster's limit.  To clear a previously set value use the modify command with a new value of -1.
QosLevel<operator><comma separated list of qos names> Specify the default Quality of Service's that jobs are able to run at for this association.  To get a list of valid QOS's use 'sacctmgr list qos'.  This value will override its parents value and push down to its children as the new default.  Setting a QosLevel to '' (two single quotes with nothing between them) restores its default setting.  You can also use the operator += and -= to add or remove certain QOS's from a QOS list.
Valid <operator> values include:
=    Set QosLevel to the specified value. Note: the QOS that can be used at a given account in the hierarchy are inherited by the children of that account.  By assigning QOS with the = sign only the assigned QOS can be used by the account and its children.
+=   Add the specified <qos> value to the current QosLevel. The account will have access to this QOS and the other previously assigned to it.
-=   Remove the specified <qos> value from the current QosLevel.
See the EXAMPLES section below.
Cluster=<cluster> Specific cluster to add account to.  Default is all in system.
Description=<description> An arbitrary string describing an account.
Name=<name> The name of a bank account.  Note the name must be unique and can not be represent different bank accounts at different points in the account hierarchy.
Organization=<org> Organization to which the account belongs.
Parent=<parent> Parent account of this account. Default is the root account, a top level account.
RawUsage=<value> This allows an administrator to reset the raw usage accrued to an account.  The only value currently supported is 0 (zero).  This is a settable specification only - it cannot be used as a filter to list accounts.
WithAssoc Display all associations for this account.
WithCoord Display all coordinators for this account.
WithDeleted Display information with previously deleted data.
NOTE: If using the WithAssoc option you can also query against association specific information to view only certain associations this account may have.  These extra options can be found in the SPECIFICATIONS FOR ASSOCIATIONS section.  You can also use the general specifications list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.
LIST/SHOW ACCOUNT FORMAT OPTIONS Account The name of a bank account.
Description An arbitrary string describing an account.
Organization Organization to which the account belongs.
Coordinators List of users that are a coordinator of the account. (Only filled in when using the WithCoordinator option.)
NOTE: If using the WithAssoc option you can also view the information about the various associations the account may have on all the clusters in the system. The association information can be filtered.  Note that all the accounts in the database will always be shown as filter only takes effect over the association data. The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.
Clusters=<comma separated list of cluster names> List the associations of the cluster(s).
Accounts=<comma separated list of account names> List the associations of the account(s).
Users=<comma separated list of user names> List the associations of the user(s).
Partition=<comma separated list of partition names> List the associations of the partition(s).
NOTE: You can also use the general specifications list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.
Other options unique for listing associations:
OnlyDefaults Display only associations that are default associations
Tree   Display account names in a hierarchical fashion.
WithSubAccounts Display information with subaccounts.  Only really valuable when used with the account= option.  This will display all the subaccount associations along with the accounts listed in the option.
WOLimits Display information without limit information. This is for a smaller default format of "Cluster,Account,User,Partition".
WOPInfo Display information without parent information (i.e. parent id, and parent account name). This option also implicitly sets the WOPLimits option.
WOPLimits Display information without hierarchical parent limits (i.e. will only display limits where they are set instead of propagating them from the parent).
LIST/SHOW ASSOCIATION FORMAT OPTIONS Account The name of a bank account in the association.
Cluster The name of a cluster in the association.
DefaultQOS The QOS the association will use by default if it as access to it in the QOS list mentioned below.
Fairshare Number used in conjunction with other accounts to determine job priority. Can also be the string parent, when used on a user this means that the parent association is used for fairshare.  If Fairshare=parent is set on an account, that account's children will be effectively reparented for fairshare calculations to the first parent of their parent that is not Fairshare=parent.  Limits remain the same, only its fairshare value is affected.
GrpTRESMins The total number of TRES minutes that can possibly be used by past, present and future jobs running from this association and its children.
GrpTRESRunMins Used to limit the combined total number of TRES minutes used by all jobs running with this association and its children.  This takes into consideration time limit of running jobs and consumes it, if the limit is reached no new jobs are started until other jobs finish to allow time to free up.
GrpTRES Maximum number of TRES running jobs are able to be allocated in aggregate for this association and all associations which are children of this association.
GrpJobs Maximum number of running jobs in aggregate for this association and all associations which are children of this association.
GrpJobsAccrue Maximum number of pending jobs in aggregate able to accrue age priority for this association and all associations which are children of this association.
GrpSubmitJobs Maximum number of jobs which can be in a pending or running state at any time in aggregate for this association and all associations which are children of this association.
GrpWall Maximum wall clock time running jobs are able to be allocated in aggregate for this association and all associations which are children of this association.
ID     The id of the association.
LFT    Associations are kept in a hierarchy: this is the left most spot in the hierarchy.  When used with the RGT variable, all associations with a LFT inside this LFT and before the RGT are children of this association.
MaxTRESPerJob Maximum number of TRES each job is able to use.
MaxTRESMinsPerJob Maximum number of TRES minutes each job is able to use.
MaxTRESPerNode Maximum number of TRES each node in a job allocation can use.
MaxJobs Maximum number of jobs each user is allowed to run at one time.
MaxJobsAccrue Maximum number of pending jobs able to accrue age priority at any given time.
MaxSubmitJobs Maximum number of jobs pending or running state at any time.
MaxWallDurationPerJob Maximum wall clock time each job is able to use.
Qos Valid QOS' for this association.
QosRaw QOS' ID.
ParentID The association id of the parent of this association.
ParentName The account name of the parent of this association.
Partition The name of a partition in the association.
Priority What priority will be added to a job's priority when using this association.
WithRawQOSLevel Display QosLevel in an unevaluated raw format, consisting of a comma separated list of QOS names prepended with '' (nothing), '+' or '-' for the association. QOS names without +/- prepended were assigned (ie, sacctmgr modify ... set QosLevel=qos_name) for the entity listed or on one of its parents in the hierarchy. QOS names with +/- prepended indicate the QOS was added/filtered (ie, sacctmgr modify ... set QosLevel=[+-]qos_name) for the entity listed or on one of its parents in the hierarchy. Including WOPLimits will show exactly where each QOS was assigned, added or filtered in the hierarchy.
RGT    Associations are kept in a hierarchy: this is the right most spot in the hierarchy.  When used with the LFT variable, all associations with a LFT inside this RGT and after the LFT are children of this association.
User   The name of a user in the association.
Classification=<classification> Type of machine, current classifications are capability, capacity and capapacity.
Features=<comma separated list of feature names> Features that are specific to the cluster. Federated jobs can be directed to clusters that contain the job requested features.
Federation=<federation> The federation that this cluster should be a member of. A cluster can only be a member of one federation at a time.
FedState=<state> The state of the cluster in the federation. Valid states are:
ACTIVE Cluster will actively accept and schedule federated jobs.
INACTIVE Cluster will not schedule or accept any jobs.
DRAIN  Cluster will not accept any new jobs and will let existing federated jobs complete.
DRAIN+REMOVE Cluster will not accept any new jobs and will remove itself from the federation once all federated jobs have completed. When removed from the federation, the cluster will accept jobs as a non-federated cluster.
Name=<name> The name of a cluster.  This should be equal to the ClusterName parameter in the slurm.conf configuration file for some Slurm-managed cluster.
RPC=<rpc list> Comma separated list of numeric RPC values.
WithFed Appends federation related columns to default format options (e.g. Federation,ID,Features,FedState).
WOLimits Display information without limit information. This is for a smaller default format of Cluster,ControlHost,ControlPort,RPC
LIST/SHOW CLUSTER FORMAT OPTIONS Classification Type of machine, i.e. capability, capacity or capapacity.
Cluster The name of the cluster.
ControlHost When a slurmctld registers with the database the ip address of the controller is placed here.
ControlPort When a slurmctld registers with the database the port the controller is listening on is placed here.
Features The list of features on the cluster (if any).
Federation The name of the federation this cluster is a member of (if any).
FedState The state of the cluster in the federation (if a member of one).
FedStateRaw Numeric value of the name of the FedState.
Flags  Attributes possessed by the cluster. Current flags include Cray, External and MultipleSlurmd.
External clusters are registration only clusters. A slurmctld can designate an external slurmdbd with the AccountingStorageExternalHost slurm.conf option. This allows a slurmctld to register to an external slurmdbd so that clusters attached to the external slurmdbd can communicate with the external cluster with Slurm commands.
ID     The ID assigned to the cluster when a member of a federation. This ID uniquely identifies the cluster and its jobs in the federation.
NodeCount The current count of nodes associated with the cluster.
NodeNames The current Nodes associated with the cluster.
PluginIDSelect The numeric value of the select plugin the cluster is using.
RPC    When a slurmctld registers with the database the rpc version the controller is running is placed here.
TRES   Trackable RESources (Billing, BB (Burst buffer), CPU, Energy, GRES, License, Memory, and Node) this cluster is accounting for.
NOTE: You can also view the information about the root association for the cluster.  The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.
Account=<comma separated list of account names> Account name to add this user as a coordinator to.
Names=<comma separated list of user names> Names of coordinators.
NOTE: To list coordinators use the WithCoordinator options with list account or list user.
All_Clusters Get information on all cluster shortcut.
All_Time Get time period for all time shortcut.
Clusters=<comma separated list of cluster names> List the events of the cluster(s).  Default is the cluster where the command was run.
End=<OPT> Period ending of events. Default is now.
Valid time formats are...
HH:MM[:SS] [AM|PM] MMDD[YY] or MM/DD[/YY] or MM.DD[.YY] MM/DD[/YY]-HH:MM[:SS] YYYY-MM-DD[THH:MM[:SS]] now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
Event=<OPT> Specific events to look for, valid options are Cluster or Node, default is both.
MaxTRES=<OPT> Max number of TRES affected by an event.
MinTRES=<OPT> Min number of TRES affected by an event.
Nodes=<comma separated list of node names> Node names affected by an event.
Reason=<comma separated list of reasons> Reason an event happened.
Start=<OPT> Period start of events.  Default is 00:00:00 of previous day, unless states are given with the States= spec events.  If this is the case the default behavior is to return events currently in the states specified.
States=<comma separated list of states> State of a node in a node event.  If this is set, the event type is set automatically to Node.
User=<comma separated list of users> Query against users who set the event.  If this is set, the event type is set automatically to Node since only user slurm can perform a cluster event.
LIST/SHOW EVENT FORMAT OPTIONS Cluster The name of the cluster event happened on.
ClusterNodes The hostlist of nodes on a cluster in a cluster event.
Duration Time period the event was around for.
End    Period when event ended.
Event  Name of the event.
EventRaw Numeric value of the name of the event.
NodeName The node affected by the event.  In a cluster event, this is blank.
Reason The reason an event happened.
Start  Period when event started.
State  On a node event this is the formatted state of the node during the event.
StateRaw On a node event this is the numeric value of the state of the node during the event.
TRES   Number of TRES involved with the event.
User   On a node event this is the user who caused the event to happen.
Clusters[+|-]=<comma separated list of cluster names> List of clusters to add/remove to a federation. A blank value (e.g. clusters=) will remove all federations for the federation. NOTE: a cluster can only be a member of one federation.
Name=<name> The name of the federation.
Tree   Display federations in a hierarchical fashion.
LIST/SHOW FEDERATION FORMAT OPTIONS Features The list of features on the cluster.
Federation The name of the federation.
Cluster Name of the cluster that is a member of the federation.
FedState The state of the cluster in the federation.
Index  The index of the cluster in the federation.
Comment=<comment> The job's comment string when the AccountingStoreJobComment parameter in the slurm.conf file is set (or defaults) to YES.  The user can only modify the comment string of their own job.
Cluster=<cluster_list> List of clusters to alter jobs on, defaults to local cluster.
DerivedExitCode=<derived_exit_code> The derived exit code can be modified after a job completes based on the user's judgment of whether the job succeeded or failed. The user can only modify the derived exit code of their own job.
EndTime Jobs must end before this time to be modified. Format output is, YYYY-MM-DDTHH:MM:SS, unless changed through the SLURM_TIME_FORMAT environment variable.
JobID=<jobid_list> The id of the job to change.  Not needed if altering multiple jobs using wckey specification.
NewWCKey=<newwckey> Use to rename a wckey on job(s) in the accounting database
StartTime Jobs must start at or after this time to be modified in the same format as EndTime.
User=<user_list> Used to specify the jobs of users jobs to alter.
WCKey=<wckey_list> Used to specify the wckeys to alter.
The DerivedExitCode, Comment and WCKey fields are the only fields of a job record in the database that can be modified after job completion.
LIST/SHOW JOB FORMAT OPTIONS The sacct command is the exclusive command to display job records from the Slurm database.
NOTE: The group limits (GrpJobs, GrpNodes, etc.) are tested when a job is being considered for being allocated resources.  If starting a job would cause any of its group limit to be exceeded, that job will not be considered for scheduling even if that job might preempt other jobs which would release sufficient group resources for the pending job to be initiated.
Flags  Used by the slurmctld to override or enforce certain characteristics. Valid options are
DenyOnLimit If set, jobs using this QOS will be rejected at submission time if they do not conform to the QOS 'Max' limits.  Group limits will also be treated like 'Max' limits as well and will be denied if they go over. By default jobs that go over these limits will pend until they conform.  This currently only applies to QOS and Association limits.
EnforceUsageThreshold If set, and the QOS also has a UsageThreshold, any jobs submitted with this QOS that fall below the UsageThreshold will be held until their Fairshare Usage goes above the Threshold.
NoDecay If set, this QOS will not have its GrpTRESMins, GrpWall and UsageRaw decayed by the slurm.conf PriorityDecayHalfLife or PriorityUsageResetPeriod settings.  This allows a QOS to provide aggregate limits that, once consumed, will not be replenished automatically.  Such a QOS will act as a time- limited quota of resources for an association that has access to it.  Account/user usage will still be decayed for associations using the QOS.  The QOS GrpTRESMins and GrpWall limits can be increased or the QOS RawUsage value reset to 0 (zero) to again allow jobs submitted with this QOS to be queued (if DenyOnLimit is set) or run (pending with QOSGrp{TRES}MinutesLimit or QOSGrpWallLimit reasons, where {TRES} is some type of trackable resource).
NoReserve If this flag is set and backfill scheduling is used, jobs using this QOS will not reserve resources in the backfill schedule's map of resources allocated through time. This flag is intended for use with a QOS that may be preempted by jobs associated with all other QOS (e.g use with a "standby" QOS). If this flag is used with a QOS which can not be preempted by all other QOS, it could result in starvation of larger jobs.
PartitionMaxNodes If set jobs using this QOS will be able to override the requested partition's MaxNodes limit.
PartitionMinNodes If set jobs using this QOS will be able to override the requested partition's MinNodes limit.
OverPartQOS If set jobs using this QOS will be able to override any limits used by the requested partition's QOS limits.
PartitionTimeLimit If set jobs using this QOS will be able to override the requested partition's TimeLimit.
RequiresReservation If set jobs using this QOS must designate a reservation when submitting a job.  This option can be useful in restricting usage of a QOS that may have greater preemptive capability or additional resources to be allowed only within a reservation.
UsageFactorSafe If set, and AccountingStorageEnforce includes Safe, jobs will only be able to run if the job can run to completion with the UsageFactor applied.
GraceTime Preemption grace time to be extended to a job which has been selected for preemption.
GrpTRESMins The total number of TRES minutes that can possibly be used by past, present and future jobs running from this QOS.
GrpTRESRunMins Used to limit the combined total number of TRES minutes used by all jobs running with this QOS.  This takes into consideration time limit of running jobs and consumes it, if the limit is reached no new jobs are started until other jobs finish to allow time to free up.
GrpTRES Maximum number of TRES running jobs are able to be allocated in aggregate for this QOS.
GrpJobs Maximum number of running jobs in aggregate for this QOS.
GrpJobsAccrue Maximum number of pending jobs in aggregate able to accrue age priority for this QOS.
GrpSubmitJobs Maximum number of jobs which can be in a pending or running state at any time in aggregate for this QOS.
GrpWall Maximum wall clock time running jobs are able to be allocated in aggregate for this QOS. If this limit is reached submission requests will be denied and the running jobs will be killed.
ID     The id of the QOS.
MaxJobsAccruePerAccount Maximum number of pending jobs an account (or subacct) can have accruing age priority at any given time.
MaxJobsAccruePerUser Maximum number of pending jobs a user can have accruing age priority at any given time.
MaxJobsPerAccount Maximum number of jobs each account is allowed to run at one time.
MaxJobsPerUser Maximum number of jobs each user is allowed to run at one time.
MaxSubmitJobsPerAccount Maximum number of jobs pending or running state at any time per account.
MaxSubmitJobsPerUser Maximum number of jobs pending or running state at any time per user.
MaxTRESPerAccount Maximum number of TRES each account is able to use.
MaxTRESPerUser Maximum number of TRES each user is able to use.
MinPrioThreshold Minimum priority required to reserve resources when scheduling.
MinTRESPerJob Minimum number of TRES each job running under this QOS must request.  Otherwise the job will pend until modified.
NOTE: This setting shows up in the sacctmgr output as MinTRES.
Name   Name of the QOS.
Preempt Other QOS' this QOS can preempt.
NOTE: The Priority of a QOS is NOT related to QOS preemption, only Preempt is used to define which QOS can preempt others.
PreemptExemptTime Specifies a minimum run time for jobs of this QOS before they are considered for preemption. This QOS option takes precedence over the global PreemptExemptTime. Setting to -1 disables the option, allowing another QOS or the global option to take effect. Setting to 0 indicates no minimum run time and supersedes the lower priority QOS (see OverPartQOS) and/or the global option in slurm.conf.
PreemptMode Mechanism used to preempt jobs or enable gang scheduling for this QOS when the cluster PreemptType is set to preempt/qos.  This QOS-specific PreemptMode will override the cluster-wide PreemptMode for this QOS. Unsetting the QOS specific PreemptMode, by specifying "OFF", "" or "Cluster", makes it use the default cluster-wide PreemptMode. See the description of the cluster-wide PreemptMode parameter for further details of the available modes.
Priority What priority will be added to a job's priority when using this QOS.
NOTE: The Priority of a QOS is NOT related to QOS preemption, see Preempt instead.
RawUsage=<value> This allows an administrator to reset the raw usage accrued to a QOS.  The only value currently supported is 0 (zero).  This is a settable specification only - it cannot be used as a filter to list accounts.
UsageFactor Usage factor when running with this QOS. See below for more details.
UsageThreshold A float representing the lowest fairshare of an association allowable to run a job.  If an association falls below this threshold and has pending jobs or submits new jobs those jobs will be held until the usage goes back above the threshold.  Use sshare to see current shares on the system.
LIST/SHOW QOS FORMAT OPTIONS Description An arbitrary string describing a QOS.
GraceTime Preemption grace time to be extended to a job which has been selected for preemption in the format of hh:mm:ss.  The default value is zero, no preemption grace time is allowed on this partition.  NOTE: This value is only meaningful for QOS PreemptMode=CANCEL.
GrpTRESMins The total number of TRES minutes that can possibly be used by past, present and future jobs running from this QOS.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.  NOTE: This limit only applies when using the Priority Multifactor plugin.  The time is decayed using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf.  When this limit is reached all associated jobs running will be killed and all future jobs submitted with this QOS will be delayed until they are able to run inside the limit.
GrpTRES Maximum number of TRES running jobs are able to be allocated in aggregate for this QOS.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.
GrpJobs Maximum number of running jobs in aggregate for this QOS.  To clear a previously set value use the modify command with a new value of -1.
GrpJobsAccrue Maximum number of pending jobs in aggregate able to accrue age priority for this QOS.  To clear a previously set value use the modify command with a new value of -1.
GrpSubmitJobs Maximum number of jobs which can be in a pending or running state at any time in aggregate for this QOS.  To clear a previously set value use the modify command with a new value of -1.
GrpWall Maximum wall clock time running jobs are able to be allocated in aggregate for this QOS.  To clear a previously set value use the modify command with a new value of -1.  NOTE: This limit only applies when using the Priority Multifactor plugin.  The time is decayed using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf.  When this limit is reached all associated jobs running will be killed and all future jobs submitted with this QOS will be delayed until they are able to run inside the limit.
MaxTRESMinsPerJob Maximum number of TRES minutes each job is able to use.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.
MaxTRESPerAccount Maximum number of TRES each account is able to use.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.
MaxTRESPerJob Maximum number of TRES each job is able to use.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.
MaxTRESPerNode Maximum number of TRES each node in a job allocation can use.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.
MaxTRESPerUser Maximum number of TRES each user is able to use.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.
MaxJobsPerAccount Maximum number of jobs each account is allowed to run at one time. To clear a previously set value use the modify command with a new value of -1.
MaxJobsPerUser Maximum number of jobs each user is allowed to run at one time. To clear a previously set value use the modify command with a new value of -1.
MaxSubmitJobsPerAccount Maximum number of jobs pending or running state at any time per account.  To clear a previously set value use the modify command with a new value of -1.
MaxSubmitJobsPerUser Maximum number of jobs pending or running state at any time per user.  To clear a previously set value use the modify command with a new value of -1.
MaxWallDurationPerJob Maximum wall clock time each job is able to use.  <max wall> format is <min> or <min>:<sec> or <hr>:<min>:<sec> or <days>-<hr>:<min>:<sec> or <days>-<hr>.  The value is recorded in minutes with rounding as needed.  To clear a previously set value use the modify command with a new value of -1.
MinPrioThreshold Minimum priority required to reserve resources when scheduling. To clear a previously set value use the modify command with a new value of -1.
MinTRES Minimum number of TRES each job running under this QOS must request.  Otherwise the job will pend until modified.  To clear a previously set value use the modify command with a new value of -1 for each TRES id.
Name   Name of the QOS.  Needed for creation.
Preempt Other QOS' this QOS can preempt.  Setting a Preempt to '' (two single quotes with nothing between them) restores its default setting.  You can also use the operator += and -= to add or remove certain QOS's from a QOS list.
PreemptMode Mechanism used to preempt jobs of this QOS if the clusters PreemptType is configured to preempt/qos.  The default preemption mechanism is specified by the cluster-wide PreemptMode configuration parameter.  Possible values are "Cluster" (meaning use cluster default), "Cancel", and "Requeue".  This option is not compatible with PreemptMode=OFF or PreemptMode=SUSPEND (i.e. preempted jobs must be removed from the resources).
Priority What priority will be added to a job's priority when using this QOS.  To clear a previously set value use the modify command with a new value of -1.
UsageFactor A float that is factored into a jobâ€™s TRES usage (e.g. RawUsage, TRESMins, TRESRunMins). For example, if the usagefactor was 2, for every TRESBillingUnit second a job ran it would count for 2. If the usagefactor was .5, every second would only count for half of the time. A setting of 0 would add no timed usage from the job.
The usage factor only applies to the job's QOS and not the partition QOS.
If the UsageFactorSafe flag is set and AccountingStorageEnforce includes Safe, jobs will only be able to run if the job can run to completion with the UsageFactor applied.
If the UsageFactorSafe flag is not set and AccountingStorageEnforce includes Safe, a job will be able to be scheduled without the UsageFactor applied and will be able to run without being killed due to limits.
If the UsageFactorSafe flag is not set and AccountingStorageEnforce does not include Safe, a job will be able to be scheduled without the UsageFactor applied and could be killed due to limits.
See AccountingStorageEnforce in slurm.conf man page.
Default is 1. To clear a previously set value use the modify command with a new value of -1.
Clusters=<comma separated list of cluster names> List the reservations of the cluster(s).  Default is the cluster where the command was run.
End=<OPT> Period ending of reservations. Default is now.
ID=<OPT> Comma separated list of reservation ids.
Names=<OPT> Comma separated list of reservation names.
Nodes=<comma separated list of node names> Node names where reservation ran.
Start=<OPT> Period start of reservations.  Default is 00:00:00 of current day.
LIST/SHOW RESERVATION FORMAT OPTIONS Associations The id's of the associations able to run in the reservation.
Cluster Name of cluster reservation was on.
End    End time of reservation.
Flags  Flags on the reservation.
ID     Reservation ID.
Name   Name of this reservation.
NodeNames List of nodes in the reservation.
Start  Start time of reservation.
TRES   List of TRES in the reservation.
UnusedWall Wall clock time in seconds unused by any job. A job's allocated usage is its run time multiplied by the ratio of its CPUs to the total number of CPUs in the reservation. For example, a job using all the CPUs in the reservation running for 1 minute would reduce unused_wall by 1 minute.
Clusters=<name list> Comma separated list of cluster names on which specified resources are to be available.  If no names are designated then the clusters already allowed to use this resource will be altered.
Count=<OPT> Number of software resources of a specific name configured on the system being controlled by a resource manager.
Descriptions= A brief description of the resource.
Flags=<OPT> Flags that identify specific attributes of the system resource. At this time no flags have been defined.
ServerType=<OPT> The type of a software resource manager providing the licenses. For example FlexNext Publisher Flexlm license server or Reprise License Manager RLM.
Names=<OPT> Comma separated list of the name of a resource configured on the system being controlled by a resource manager.  If this resource is seen on the slurmctld its name will be name@server to distinguish it from local resources defined in a slurm.conf.
PercentAllowed=<percent allowed> Percentage of a specific resource that can be used on specified cluster.
Server=<OPT> The name of the server serving up the resource.  Default is 'slurmdb' indicating the licenses are being served by the database.
Type=<OPT> The type of the resource represented by this record.  Currently the only valid type is License.
WithClusters Display the clusters percentage of resources.  If a resource hasn't been given to a cluster the resource will not be displayed with this flag.
NOTE:  Resource is used to define each resource configured on a system available for usage by Slurm clusters.
LIST/SHOW RESOURCE FORMAT OPTIONS Cluster Name of cluster resource is given to.
Count  The count of a specific resource configured on the system globally.
Allocated The percent of licenses allocated to a cluster.
Description Description of the resource.
ServerType The type of the server controlling the licenses.
Name   Name of this resource.
Server Server serving up the resource.
Type   Type of resource this record represents.
LIST/SHOW RUNAWAYJOB FORMAT OPTIONS Cluster Name of cluster job ran on.
ID     Id of the job.
Name   Name of the job.
Partition Partition job ran on.
State  Current State of the job in the database.
TimeStart Time job started running.
TimeEnd Current recorded time of the end of the job.
Accounts=<comma separated list of account names> Only print out the transactions affecting specified accounts.
Action=<Specific action the list will display>
Actor=<Specific name the list will display> Only display transactions done by a certain person.
Clusters=<comma separated list of cluster names> Only print out the transactions affecting specified clusters.
End=<Date and time of last transaction to return> Return all transactions before this Date and time.  Default is now.
Start=<Date and time of first transaction to return> Return all transactions after this Date and time.  Default is epoch.
Valid time formats for End and Start are...
Users=<comma separated list of user names> Only print out the transactions affecting specified users.
WithAssoc Get information about which associations were affected by the transactions.
LIST/SHOW TRANSACTIONS FORMAT OPTIONS Action Displays the type of Action that took place.
Actor  Displays the Actor to generate a transaction.
Info   Displays details of the transaction.
TimeStamp Displays when the transaction occurred.
Where  Displays details of the constraints for the transaction.
NOTE: If using the WithAssoc option you can also view the information about the various associations the transaction affected.  The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.
Account=<account> Account name to add this user to.
AdminLevel=<level> Admin level of user.  Valid levels are None, Operator, and Admin.
Cluster=<cluster> Specific cluster to add user to the account on.  Default is all in system.
DefaultAccount=<account> Identify the default bank account name to be used for a job if none is specified at submission time.
DefaultWCKey=<defaultwckey> Identify the default Workload Characterization Key.
Name=<name> Name of user.
NewName=<newname> Use to rename a user in the accounting database
Partition=<name> Partition name.
RawUsage=<value> This allows an administrator to reset the raw usage accrued to a user.  The only value currently supported is 0 (zero).  This is a settable specification only - it cannot be used as a filter to list users.
WCKeys=<wckeys> Workload Characterization Key values.
WithAssoc Display all associations for this user.
WithCoord Display all accounts a user is coordinator for.
NOTE: If using the WithAssoc option you can also query against association specific information to view only certain associations this user may have.  These extra options can be found in the SPECIFICATIONS FOR ASSOCIATIONS section.  You can also use the general specifications list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.
LIST/SHOW USER FORMAT OPTIONS AdminLevel Admin level of user.
DefaultAccount The user's default account.
User   The name of a user.
NOTE: If using the WithAssoc option you can also view the information about the various associations the user may have on all the clusters in the system. The association information can be filtered.  Note that all the users in the database will always be shown as filter only takes effect over the association data. The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.
LIST/SHOW WCKey WCKey  Workload Characterization Key.
Cluster Specific cluster for the WCKey.
User   The name of a user for the WCKey.
NOTE: If using the WithAssoc option you can also view the information about the various associations the user may have on all the clusters in the system.  The Association format fields are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.
LIST/SHOW TRES Name   The name of the trackable resource.  This option is required for TRES types BB (Burst buffer), GRES, and License.  Types CPU, Energy, Memory, and Node do not have Names.  For example if GRES is the type then name is the denomination of the GRES itself e.g. GPU.
ID     The identification number of the trackable resource as it appears in the database.
Type   The type of the trackable resource. Current types are BB (Burst buffer), CPU, Energy, GRES, License, Memory, and Node.
TRES information Trackable RESources (TRES) are used in many QOS or Association limits. When setting the limits they are comma separated list.  Each TRES has a different limit, i.e. GrpTRESMins=cpu=10,mem=20 would make 2 different limits 1 for 10 cpu minutes and 1 for 20 MB memory minutes.  This is the case for each limit that deals with TRES.  To remove the limit -1 is used i.e. GrpTRESMins=cpu=-1 would remove only the cpu TRES limit.
NOTE: When dealing with Memory as a TRES all limits are in MB.
NOTE: The Billing TRES is calculated from a partition's TRESBillingWeights. It is temporarily calculated during scheduling for each partition to enforce billing TRES limits. The final Billing TRES is calculated after the job has been allocated resources. The final number can be seen in scontrol show jobs and sacct output.
When using the format option for listing various fields you can put a %NUMBER afterwards to specify how many characters should be printed.
e.g. format=name%30 will print 30 characters of field name right justified.  A -30 will print 30 characters left justified.
sacctmgr has the capability to load and dump Slurm association data to and from a file.  This method can easily add a new cluster or copy an existing cluster's associations into a new cluster with similar accounts. Each file contains Slurm association data for a single cluster.  Comments can be put into the file with the # character.  Each line of information must begin with one of the four titles; Cluster, Parent, Account or User. Following the title is a space, dash, space, entity value, then specifications. Specifications are colon separated. If any variable, such as an Organization name, has a space in it, surround the name with single or double quotes.
To create a file of associations you can run sacctmgr dump tux file=tux.cfg
To load a previously created file you can run sacctmgr load file=tux.cfg
sacctmgr dump/load must be run as a Slurm administrator or root. If using sacctmgr load on a database without any associations, it must be run as root (because there aren't any users in the database yet).
Other options for load are: clean     - delete what was already there and start from scratch with this information. Cluster=  - specify a different name for the cluster than that which is in the file.
Since the associations in the system follow a hierarchy, so does the file.  Anything that is a parent needs to be defined before any children. The only exception is the understood 'root' account.  This is always a default for any cluster and does not need to be defined.