SSS Job Object Specification

Draft Release Version 3.0.1

29 MAR 2004

Scott Jackson, PNNL

David Jackson, Ames Lab

Brett Bode, Ames Lab

[]{#__RefHeading___Toc68321106 .anchor}Scalable Systems Software Job
Object Specification

Status of this Memo

This document describes the job object to be used by Scalable Systems
Software compliant components. It is envisioned for this specification
to be used in conjunction with the SSSRMAP protocol with the job object
passed in the Data field of Requests and Responses. Queries can be
issued to a job-cognizant component in the form of modified XPATH
expressions to the Get field to extract specific information from the
job object as described in the SSSRMAP protocol.

**Abstract**

This document describes the syntax and structure of the SSS job object.
A job model is described that is flexible enough to support the
specification of very simple jobs as well as jobs with elaborate and
complex specification requirements in a way that avoids complex
structures and syntax when it is not needed. The basic assumption is
that a solitary job specification should be usable for all phases of the
job lifecycle and can be used at submission, queuing, staging,
reservations, quotations, execution, charging, accounting, etc. This job
specification provides support for multi-step jobs, as well as jobs with
disparate task descriptions. It takes into account the aspects necessary
to operate within a grid or meta-scheduled environment where the job
might be executed at one (or more) destination among a variety of
different administrative domains and resource management systems.

# Table of Contents {#table-of-contents .unnumbered}

[Scalable Systems Software Job Object Specification
1](#__RefHeading___Toc68321106)

[Table of Contents 1](#table-of-contents)

[1. Introduction 3](#introduction)

[1.1 Goals 3](#goals)

[1.2 Non-Goals 3](#non-goals)

[1.3 Examples 4](#examples)

[1.3.1 Very Simple Example 4](#very-simple-example)

[1.3.2 Moderate Example 4](#moderate-example)

[1.3.3 Elaborate Example 5](#elaborate-example)

[2. Conventions used in this document
7](#conventions-used-in-this-document)

[2.1 Keywords 7](#keywords)

[2.2 Table Column Interpretations 7](#table-column-interpretations)

[2.3 Element Syntax Cardinality 8](#element-syntax-cardinality)

[3. The Job Model 8](#the-job-model)

[4. JobGroup Element 10](#jobgroup-element)

[4.1 JobGroup Properties 10](#jobgroup-properties)

[4.1.1 Simple JobGroup Properties 11](#simple-jobgroup-properties)

[4.1.2 Job 11](#job)

[4.1.3 JobDefaults 11](#jobdefaults)

[5. Job and JobDefaults Element 11](#job-and-jobdefaults-element)

[5.1 Job Properties 12](#job-properties)

[5.1.1 Simple Job Properties 12](#simple-job-properties)

[5.1.2 Credentials 16](#credentials)

[5.1.3 Environment Element 16](#environment-element)

[5.1.3.1 Variable Element 17](#variable-element)

[5.1.4 NodeList Element 17](#nodelist-element)

[5.1.4.1 Node Element 17](#node-element)

[5.1.5 TaskDistribution Element 18](#taskdistribution-element)

[5.1.6 Dependency Element 18](#dependency-element)

[5.1.7 Consumable Resources 19](#consumable-resources)

[5.1.8 Resource Element 20](#resource-element)

[5.1.9 NodeProperties Element 20](#nodeproperties-element)

[5.1.9.1 Node Properties 21](#node-properties)

[5.1.10 Extension Element 21](#extension-element)

[5.1.11 TaskGroup 22](#taskgroup)

[5.1.12 TaskGroupDefaults 22](#taskgroupdefaults)

[6. TaskGroup and TaskGroupDefaults Element
22](#taskgroup-and-taskgroupdefaults-element)

[6.1 TaskGroup Properties 23](#taskgroup-properties)

[6.1.1 Simple TaskGroup Properties 23](#simple-taskgroup-properties)

[6.1.2 Task 23](#task)

[6.1.3 TaskDefaults 24](#taskdefaults)

[7. Task and TaskDefaults Element 24](#task-and-taskdefaults-element)

[7.1 Task Properties 24](#task-properties)

[7.1.1 Simple Task Properties 24](#simple-task-properties)

[8. Property Categories 25](#property-categories)

[8.1 Requested Element 25](#requested-element)

[8.2 Utilized Element 27](#utilized-element)

[8.3 Dedicated Element 28](#dedicated-element)

[9. AwarenessPolicy Attribute 29](#awarenesspolicy-attribute)

[10. References 30](#references)

[Appendix A 31](#appendix-a)

[Units of Measure Abbreviations 31](#units-of-measure-abbreviations)

# Introduction

This specification proposes a standard XML representation for a job
object for use by the various components in the SSS Resource Management
System. This object will be used in multiple contexts and by multiple
components. It is anticipated that this object will be passed via the
Data Element of SSSRMAP Requests and Responses.

## Goals

There are several goals motivating the design of this representation.

It needs to be inherently flexible. We recognize we will not be able to
exhaustively include the ever-changing job properties and capabilities
that constantly arise.

The same job object should be used at all stages of its lifecycle. This
object will be used at job submission, queuing, scheduling, charging and
accounting, hence it needs to distinguish between requested, utilized
and dedicated properties.

Its design must take into account the properties and structure required
to function in a meta or grid environment. It needs to include the
capability to support local mapping of properties, global namespaces,
etc.

The equivalent of multi-step jobs must be supported. Each step (job) can
have multiple logical task descriptions.

Many potential users of the specification will not be prepared to
implement the complex portions or fine-granularity that others need.
There needs to be a way to allow the more complicated structure to be
added as needed while leaving more straightforward cases simple.

There needs to be guidance for how to understand a given job object when
higher order features are not supported by an implementation, and which
parts are required, recommended and optional for implementors to
implement.

It needs to support composite resources.

It should include the ability to specify preferences or fuzzy
requirements.

## Non-Goals

The following topics are outside the scope of the job object

This specification does not attempt to specify namespaces or even naming
conventions for most property values.

## Examples

### Very Simple Example

This example shows that a very simple and direct representation can be
used when more complex features of this specification are not needed.

\<Job\>

\<JobId\>PBS.1234.0\</JobId\>

\<JobState\>Idle\</JobState\>

\<UserId\>scottmo\</UserId\>

\<Executable\>/bin/hostname\</Executable\>

\<Processors\>16\</Processors\>

\<WallDuration\>3600\</WallDuration\>

\</Job\>

### Moderate Example

This example shows a single job of medium complexity that makes use of
features such as required versus utilized properties.

\<Job\>

\<JobId\>PBS.1234.0\</JobId\>

\<JobName\>Heavy Water\</JobName\>

\<ProjectId\>nwchemdev\</ProjectId\>

\<UserId\>peterk\</UserId\>

\<Application\>NWChem\</Application\>

\<Executable\>/usr/local/nwchem/bin/nwchem\</Executable\>

\<Arguments\>-input basis.in\</Arguments\>

\<InitialWorkingDirectory\>/home/peterk\</InitialWorkingDirectory\>

\<MachineName\>Colony\</MachineName\>

\<QualityOfService\>BottomFeeder\</QualityOfService\>

\<Queue\>batch_normal\</Queue\>

\<JobState\>Completed\</JobState\>

\<StartTime\>1051557713\</StartTime\>

\<EndTime\>1051558868\</EndTime\>

\<Charge\>25410\</Charge\>

\<Requested\>

> \<Processors op="ge"\>12\</Processors\>
>
> \<Memory op="ge" units="GB"\>2\</Memory\>
>
> \<WallDuration\>3600\</WallDuration\>

\</Requested\>

\<Utilized\>

\<Processors\>16\</Processors\>

\<Memory metric="Average" units="GB"\>1.89\</Memory\>

> \<WallDuration\>1155\</WallDuration\>

\</Utilized\>

\<Environment\>

\<Variable name="PATH"\>/usr/bin:/home/peterk\</Variable\>

\</Environment\>

\</Job\>

### Elaborate Example

This example uses a job group to encapsulate a multi-step job. It shows
that complex capabilities are expressible through this specification
when such capabilities are needed. Not all components will care about
this level of detail and components that use this specification can
include or discard the information that is relevant to their function.
Superfluous information can be ignored by the component or filtered out
(by XSLT for example).

\<JobGroup\>

\<JobGroupId\>fr15n05.1234\</JobGroupId\>

\<JobGroupState\>Active\</JobGroupState\>

\<JobGroupName\>ShuttleTakeoff\</JobGroupName\>

\<JobDefaults\>

\<StagedTime\>1051557859\</StagedTime\>

\<SubmitHost\>asteroid.lbl.gov\</SubmitHost\>

\<SubmissionTime\>1051556734\</SubmissionTime\>

\<ProjectId\>GrandChallenge18\</ProjectId\>

\<GlobalUserId\>C=US,O=LBNL,CN=Keith Jackson\</GlobalUserId\>

\<UserId\>keith\</UserId\>

\<Environment\>

\<Variable name="LD_LIBRARY_PATH"\>/usr/lib\</Variable\>

\<Variable name="PATH"\>/usr/bin:\~/bin:\</Variable\>

\<Environment\>

\</JobDefaults\>

\<Job\>

\<JobId\>fr15n05.1234.0\</JobId\>

\<JobName\>Launch Vector Initialization\</JobName\>

\<Executable\>/usr/local/gridphys/bin/lvcalc\</Executable\>

\<Queue\>batch\</Queue\>

\<JobState\>Completed\</JobState\>

\<MachineName\>SMP2.emsl.pnl.gov\</MachineName\>

\<StartTime\>1051557713\</StartTime\>

\<EndTime\>1051558868\</EndTime\>

\<QuoteId\>http://www.pnl.gov/SMP2#654321\</QuoteId\>

\<Charge units="USD"\>12.75\</Charge\>

\<Requested\>

\<WallDuration\>3600\</WallDuration\>

\<Processors\>2\</Processors\>

\<Memory\>1024\</Memory\>

\</Requested\>

\</Utilized\>

\<WallDuration\>1155\</WallDuration\>

\<Processors consumptionRate="0.78"\>2\</Processors\>

\<Memory metric="max"\>975\</Memory\>

\</Utilized\>

\<TaskGroup\>

\<TaskCount\>2\</TaskCount\>

\<TaskDistribution type="TasksPerNode"\>1\</TaskDistribution\>

\<Task\>

\<Node\>node1\</Node\>

\<ProcessId\>99353\</ProcessId\>

\</Task\>

\<Task\>

\<Node\>node12\</Node\>

\<ProcessId\>80209\</ProcessId\>

\</Task\>

\</TaskGroup\>

\</Job\>

\<Job\>

\<JobId\>fr15n05.1234.1\</JobId\>

\<JobName\>3-Phase Ascension\</JobName\>

\<Queue\>batch_normal\</Queue\>

\<JobState\>Idle\</JobState\>

\<MachineName\>Colony.emsl.pnl.gov\</MachineName\>

\<Priority\>1032847\</Priority\>

\<Hold\>System\</Hold\>

\<StatusMessage\>Insufficient funds to start job\</StatusMessage\>

\<Requested\>

\<WallDuration\>43200\</WallDuration\>

\</Requested\>

\<TaskGroup\>

\<TaskCount\>1\</TaskCount\>

\<TaskName\>Master\</TaskName\>

\<Executable\>/usr/local/bin/stage-coordinator\</Executable\>

\<Memory\>2048\<Memory\>

\<Resource name="License" type="ESSL2"\>1\</Resource\>

\<NodeProperties\>

\<Feature\>Jumbo-Frame\</Feature\>

\</NodeProperties\>

\</TaskGroup\>

\<TaskGroup\>

\<TaskName\>Slave\</TaskName\>

\<TaskDistribution type="Rule"\>RoundRobin\</TaskDistribution\>

\<Executable\>/usr/local/bin/stage-slave\</Executable\>

\<NodeCount\>4\</NodeCount\>

\<Requested\>

\<Processors group="-1"\>12\</Processors\>

\<Processors conj="or" group="1"\>16\</Processors\>

\<Memory\>512\</Memory\>

\<NodeProperties\>

\<Name op="match"\>fr15n.\*\</Name\>

\</NodeProperties\>

\</Requested\>

\</TaskGroup\>

\</Job\>

\</JobGroup\>

# Conventions used in this document

## Keywords

The keywords "MUST", "MUST NOT", "REQUIIRED", "SHALL", "SHALL NOT",
"SHOULD", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to
be interpreted as described in RFC2119.

## Table Column Interpretations

In the property tables, the columns are interpreted to have the
following meanings:

Element Name: Name of the XML element (xsd:element) see \[DATATYPES\]

Type: Data type defined by xsd (XML Schema Definition) as:

String xsd:string (a finite length sequence of printable characters)

Integer xsd:integer (a signed finite length sequence of decimal digits)

Float xsd:float (single-precision 32-bit floating point)

Boolean xsd:boolean (consists of the literals "true" or "false")

DateTime xsd:int (a 32-bit unsigned long in GMT seconds since the EPOCH)

> Duration xsd:int (a 32-bit unsigned long measured in seconds)

Description: Brief description of the meaning of the property

Appearance: This column indicates whether the given property has to
appear within the parent element. It assumes the following meanings:

MUST This property is REQUIRED when the parent is specified.

SHOULD This property is RECOMMENDED when the parent is specified.

MAY This property is OPTIONAL when the parent is specified.

Compliance: The column indicates whether a compliant implementation has
to support the given property.

MUST A compliant implementation MUST support this property.

SHOULD A compliant implementation SHOULD support this property.

MAY A compliant implementation MAY support this property.

Categories: Some properties may be categorized into one of several
categories. Letters in this column indicate that the given property can
be classified in the the following property categories.

R This property can be encompassed in a Requested element.

U This property can be encompassed in a Utilized element.

D This property can be encompassed in a Dedicated element.

## Element Syntax Cardinality

The cardinality of elements in the element syntax sections may make use
of regular expression wildcards with the following meanings:

\* Zero or more occurrences

\+ One or more occurrences

? Zero or one occurrences

The absence of one of these symbols implies exactly one occurrence.

# The Job Model

The primary object within the job model is a job. A job can be thought
of as a single schedulable entity and will be the object normally seen
in job queues.

Jobs with dependencies on other jobs may be submitted in a job group.
Jobs within a job group form a DAG (directed acyclic graph) where the
nodes are jobs and the edges represent dependencies on the status of
previous jobs. A job group will consist of at least one job. A job group
can optionally specify job defaults which are a set of job properties to
be assumed by all jobs within the job group unless overridden within the
job.

A job may consist of multiple tasks, which are the finest grained work
unit and represent an endpoint for executing a given process instance.
For example, a job that requests 3 nodes and 4 processors will have 4
tasks, two on one node and one on each of two nodes. Tasks may be
grouped into task groups, which are logical aggregations of tasks and
their common properties. Submit filters, prologs, epilogs, notification
scripts, etc. run once only for each job. Whereas task groups function
as logical descriptions of tasks and their properties, they also
describe the number of such tasks and the nodes that they run on. As an
example, a master task group (consisting of a single task) might ask for
a node with a MATLAB license, 2GB of memory and an internet connected
network adapter while a slave task group (consisting of 12 tasks) could
be targeted for nodes with more CPU bandwidth \-- all within the same
job and utilizing a common MPI ring. Tasks (and hence taskgroups) can
have different executables or environments, specify different consumable
resources or node properties. A job, therefore, may specify one or more
task group. A job that does not specify an explicit task group is
considered as having a single implicit task group. A job can optionally
specify task group defaults which are a set of task group properties to
be assumed by all task groups within the job unless overridden within a
task group.

A task group may specify one or more tasks. A task group that does not
specify an explicit task is considered as having a single implicit task.
A task group can optionally specify task defaults which are a set of
task properties to be assumed by all tasks within the task group unless
overridden within a task.

# JobGroup Element

A JobGroup is an optional element used to aggregate one or more
interdependent jobs. Some resource managers allow for the submission of
job groups (multi-step jobs) and allow for queries on the status of an
entire job group.

-   A compliant implementation MAY support this element.

-   A JobGroup MUST specify one or more JobGroup Properties.

-   A JobGroup MUST contain one or more Jobs.

-   A JobGroup MAY contain zero or more JobsDefaults.

The following illustrates the syntax of this element:

\<JobGroup\>

\<!\-- JobGroup Properties \--\>+

\<Job/\>+

\<JobDefaults/\>?

\</Job\>

## JobGroup Properties

JobGroup Properties are properties that always apply to the job group as
a whole. These include the job group id, jobs and job defaults, and
other simple optional job properties.

### Simple JobGroup Properties

Simple (unstructured) job group properties are enumerated in Table 1.

Table 1 Simple JobGroup Properties

  --------------- ---------- -------------- ------------ ------------ ------------
  Element Name    Type       Description    Appearance   Compliance   Categories

  CreationTime    DateTime   Date and time  MAY          MAY          
                             that the job                             
                             group was                                
                             instantiated                             

  Description     String     Description of MAY          MAY          
                             the job group                            

  JobGroupId      String     Job group      MUST         MUST         
                             identifier                               

  JobGroupName    String     Name of the    MAY          SHOULD       
                             job group                                

  JobGroupState   String     State of the   MAY          SHOULD       
                             job as a                                 
                             whole. Valid                             
                             states may                               
                             include                                  
                             "NotQueued",                             
                             "Unstarted",                             
                             "Active",                                
                             "Completed".                             
  --------------- ---------- -------------- ------------ ------------ ------------

### Job

A job group MUST contain one or more jobs.

See the next section for element details.

### JobDefaults

A job group MAY contain zero or one job defaults.

See the next section for element details.

# Job and JobDefaults Element

The Job and JobDefaults elements are of the same structure. A Job
element encapsulates a job and may be expressed as a standalone object.
A JobDefaults element may only appear within a JobGroup and represents
the defaults to be taken by all jobs within the job group. Job
properties in Job elements override any properties found in a sibling
JobDefaults element.

-   A compliant implementation MUST support the Job element.

-   A compliant implementation MAY support the JobDefaults element only
    if it supports the JobGroup element.

-   A job MUST specify one or more Job Properties.

-   One or more TaskGroup elements MAY appear at this level.

-   Zero or one TaskGroupDefaults elements MAY appear at this level.

The following illustrates the syntax of this element:

\<Job\>

\<!\-- Job Properties \--\>+

\<TaskGroup/\>\*

\<TaskGroupDefaults/\>?

\</Job\>

## Job Properties

Job Properties are properties that apply to a particular job or as
default properties to all jobs. These properties include the job id, job
credentials, task groups, task group defaults, and other simple optional
job properties.

### Simple Job Properties

Simple (unstructured) job properties are enumerated in Table 2.

Table 2 Simple Job Properties

  ------------------------- ---------- --------------- ------------ ------------ ------------
  Element Name              Type       Description     Appearance   Compliance   Categories

  Application               String     Type of         MAY          MAY          
                                       application                               
                                       such as                                   
                                       "Gaussian" or                             
                                       "Nwchem".                                 

  Arguments                 String     The arguments   MAY          SHOULD       
                                       for the                                   
                                       executable.                               

  EligibleTime              DateTime   Date and time   MAY          MAY          
                                       that a job                                
                                       became eligible                           
                                       to run.                                   

  EndTime                   DateTime   Date and time   MAY          MUST         RU
                                       that a job                                
                                       ended                                     
                                       (independent of                           
                                       success or                                
                                       failure).                                 

  Executable                String     Executable.     MAY          MUST         
                                       This may be an                            
                                       absolute or                               
                                       relative path                             
                                       or a URI.\*                               

  GlobalJobId               String     Globally unique MAY          SHOULD       
                                       job identifier                            
                                       (possibly in                              
                                       the form of a                             
                                       URI).                                     

  Hold                      String     Hold(s) on the  MAY          SHOULD       
                                       job if any.                               

  InitialWorkingDirectory   String     Initial working MAY          SHOULD       
                                       directory                                 

  JobId                     String     A local job     MUST         MUST         
                                       identifier                                
                                       assigned to the                           
                                       job by the                                
                                       local resource                            
                                       manager.                                  

  JobName                   String     Name of the job MAY          SHOULD       

  JobState                  String     State of the    MAY          MUST         
                                       sjob. Valid                               
                                       states may                                
                                       include "Idle",                           
                                       "Hold",                                   
                                       "Running",                                
                                       "Suspended",                              
                                       "Completed".                              

  MachineName               String     Name of the     MAY          MUST         RU
                                       system or                                 
                                       cluster that                              
                                       the job runs                              
                                       on.                                       

  NodeCount                 Integer    Number of nodes MAY          MUST         RU
                                       used by the                               
                                       job.                                      

  Priority                  Integer    Current queue   MAY          SHOULD       
                                       priority (or                              
                                       rank) for the                             
                                       job.                                      

  QualityOfService          String     Name of the     MAY          SHOULD       RU
                                       Quality of                                
                                       Service (QOS).                            

  Queue                     String     Name of the     MAY          SHOULD       RU
                                       Queue (or                                 
                                       class) that the                           
                                       job job runs                              
                                       in.                                       

  QuoteId                   String     Identifier for  MAY          MAY          
                                       a guaranteed                              
                                       charge rate                               
                                       quote obtained                            
                                       by the job.                               

  ReservationTime           DateTime   Date and time   MAY          MAY          
                                       that a                                    
                                       reservation was                           
                                       placed for the                            
                                       job.                                      

  StagedTime                DateTime   Date and time   MAY          MAY          
                                       that a job was                            
                                       staged to the                             
                                       local resource                            
                                       management                                
                                       system.                                   

  StartCount                Integer    Number of times MAY          MAY          
                                       the scheduler                             
                                       tried to start                            
                                       the job.                                  

  StartTime                 DateTime   Date and time   MAY          MUST         RU
                                       that the job                              
                                       started.                                  

  StatusMessage             String     Natural         MAY          SHOULD       
                                       language                                  
                                       message that                              
                                       can be used to                            
                                       provide detail                            
                                       on why a job                              
                                       failed, isn't                             
                                       running, etc.                             

  SubmissionTime            DateTime   Date and time   MAY          SHOULD       
                                       that a job was                            
                                       submitted.                                

  SubmitHost                String     FQDN of host    MAY          SHOULD       
                                       where the job                             
                                       was submitted                             
                                       from.                                     

  SuspendDuration           Integer    Number of       MAY          MAY          
                                       seconds the job                           
                                       was in the                                
                                       "Suspended"                               
                                       state.                                    

  TimeCategory              String     This allows the MAY          MAY          
                                       specification                             
                                       of shifts like                            
                                       "PrimeTime" for                           
                                       charging                                  
                                       purposes.                                 

  Type                      String     Type of job.    MAY          MAY          
                                       Meaning of this                           
                                       extension                                 
                                       property is                               
                                       context                                   
                                       specific.                                 

  WallDuration              Duration   Number of       SHOULD       MUST         RU
                                       seconds the job                           
                                       was in the                                
                                       "Running"                                 
                                       state.                                    
  ------------------------- ---------- --------------- ------------ ------------ ------------

\* The Executable may be a script or a binary executable. If it is
already on the target system it may be referenced by an absolute or
relative pathname (relative to InitialWorkingDirectory). If it is passed
along with the job in a File object (see SSSRMAP), it can be referenced
by an absolute or relative URI. An absolute URI would specify a URL
where the file can be downloaded (like with wget). A relative URI is
specified by preceding an identifier by a pound sign as in
\<Executable\>#Script\</Executable\> and will be found in a File object
included along with the Job object with the Script as an identifier as
in \<File id="Script"\>echo hello world\</File\>.

### Credentials

Credentials are a special group of job properties that relate to some
kind of authenticated token or id and are also distinct in the fact that
they can be categorized in both requested and utilized forms.

Credential job properties are enumerated in Table 3.

Table 3 Credential Job Properties

  --------------- --------- ------------- ------------ ------------ ------------
  Element Name    Type      Description   Appearance   Compliance   Categories

  ProjectId       String    Name of the   MAY          SHOULD       RU
                            Project or                              
                            Charge                                  
                            Account                                 

  GlobalUserId    String    Globally      MAY          SHOULD       RU
                            unique user                             
                            identifier.                             
                            This may be                             
                            an X.509 DN                             
                            for example.                            

  GroupId         String    Name of the   MAY          MAY          RU
                            local group                             
                            id.                                     

  UserId          String    Name of the   MAY          MUST         RU
                            local userid                            
                            for the job.                            
  --------------- --------- ------------- ------------ ------------ ------------

### Environment Element

The Environment element is used to encapsulate environment variables.

-   A compliant implementation SHOULD support this element.

-   An Environment element MAY appear zero or one times within a given
    set of Job (or TaskGroup) Properties.

-   An Environment element MUST contain one or more Variable elements.

The following illustrates the syntax of this element:

\<Environment\>

\<Variable/\>+

\</Environment\>

#### Variable Element

The Variable element represents an environment variable with its name
and value.

This element MUST have a name attribute that is of type String. A
compliant implementation MUST support this attribute if this element is
supported.

The character content of this element is the value of the environment
variable.

The following is an example of a Variable element:

\<Variable name="PATH"\>/usr/bin:/home/sssdemo\</Variable\>

### NodeList Element

The NodeList element is used to aggregate nodes.

-   A compliant implementation SHOULD support this element.

-   This element MAY appear zero or one times within a given set of Job
    Properties.

-   This element MUST contain one or more Node elements.

-   This element MAY be categorized as a requested or utilized property
    by being encompassed by the appropriate element.

The following illustrates the syntax of this element:

\<NodeList\>

\<Node/\>+

\</NodeList\>

#### Node Element

The Node element represents a node.

-   This element is of type String.

The following is an example of a Node element:

\<Node\>node1\</Node\>

### TaskDistribution Element

The TaskDistribution element describes how tasks are to be mapped to
nodes.

This may be a rule name, a task per node ratio or an arbitrary geometry.

-   A compliant implementation SHOULD support this element.

-   This element MAY appear zero or one times within a given set of Job
    (or TaskGroup) Properties.

-   This element is of type String.

-   This element MAY have a type attribute of type String that provides
    a hint as to the type of mapping guidance provided. It may have
    values including "Rule", "TasksPerNode", "ProcessorsPerTask" or
    "Geometry". A compliant implementation MAY support this attribute if
    this element is supported.

-   It is possible when specifying a task distribution along with
    Processors, NodeCount or TaskCount to encounter a contradiction. The
    components are responsible for resolving conflicting requirements.

The following are three examples of a TaskDistribution element:

\<TaskDistribution type="TasksPerNode"\>2\</TaskDistribution\>

\<TaskDistribution type="Rule"\>RoundRobin\</TaskDistribution\>

\<TaskDistribution type="Geometry"\>{1,4}{2}{3,5}\</TaskDistribution\>

### Dependency Element

The Dependency element allows jobs to have execution dependencies on the
status of previous jobs. In a job group (multi-step job), some jobs may
delay execution until the failure or success of other jobs creating in
general a Directed Acyclic Graph relationship between the jobs. This
content of this element is of type String and represents the JobId that
the job is dependent upon. A job may have more than one dependencies so
this element may appear more than once in a given job scope. A compliant
implementation SHOULD support this element if job groups are supported.
This element MAY have a type attribute that is of type String that
indicates what basis is used to determine the execution of the current
job in relation to the specified job. The default basis (if the type
attribute is omitted) is to allow the current job to run if the
specified job completes successfully (this is specified explicitly with
a value of "OnSuccess"). The type attribute MAY be present and have
values including "OnSuccess" and "OnFailure". A compliant implementation
MUST support this attribute if this element is supported.

The following is an example of a Dependency element:

\<Dependency type="OnSuccess"\>PBS.1234.0\</Dependency\>

### Consumable Resources

Consumable Resources are a special group of properties that can have
additional attributes and can be used in multiple contexts. In general a
consumable resource is a resource that can be consumed in a measurable
quantity.

-   A consumable resource MAY have a units attribute that is of type
    String that specifies the units by which it is being measured. If
    this attribute is omitted, a default unit is implied. A compliant
    implementation MAY support this attribute if the element is
    supported.

-   A consumable resource MAY have a metric attribute that is of type
    String that specifies the type of measurement being described. For
    example, the measurement may be a Total, an Average, a Min or a Max.
    A compliant implementation MAY support this attribute if the element
    is supported.

-   A consumable resource MAY have a wallDuration attribute of type
    Duration that indicates the amount of time for which that resource
    was used. This need only be specified if the resource was used for a
    different amount of time than the wallDuration for the job. A
    compliant implementation MAY support this attribute if the element
    is supported.

-   A consumable resource MAY have a consumptionRate attribute of type
    Float that indicates the average percentage that a resource was used
    over its wallDuration. For example, an overbooked SMP running 100
    jobs across 32 processors may wish to scale the usage and charge by
    the average fraction of processor usage actually delivered. A
    compliant implementation MAY support this attribute if the element
    is supported.

A list of simple consumable resources is listed in Table 4.

Table 4 Simple Consumable Resources

  ----------------- ---------- ------------- ------------ ------------ ------------
  Element Name      Type       Description   Appearance   Compliance   Categories

  Disk              Float      Amount of     MAY          SHOULD       RUD
                               disk.                                   

  Memory            Float      Amount of     MAY          SHOULD       RUD
                               memory.                                 

  Network           Float      Amount of     MAY          MAY          RUD
                               network.                                

  Processors        Integer    Number of     MAY          MUST         RUD
                               processors.                             

  Swap              Float      Amount of     MAY          MAY          RUD
                               virtual                                 
                               memory.                                 
  ----------------- ---------- ------------- ------------ ------------ ------------

The following are two examples for specifying a consumable resource:

\<Memory metric="Max" units="GB"\>483\</Memory\>

\<Processors wallDuration="1234"
consumptionRate="0.63"\>4\</Processors\>

### Resource Element

In addition to the consumable resources enumerated in the above table,
an extensible consumable resource is defined by the Resource element.

-   A compliant implementation SHOULD support this element.

-   This element MAY appear zero or more times within a given set of job
    (or task group) properties.

-   Like the other consumable resources, this property MAY be
    categorized as a requested, utilized or dedicated property by being
    encompassed in the appropriate element.

-   This element is of type Float.

-   It shares the other same properties and attributes as the other
    consumable resources but it requires an additional name (and
    optional type) attribute to describe it.

-   This element MUST have a name attribute of type String that
    indicates the type of consumable resource being measured. A
    compliant implementation MUST support this attribute if the element
    is supported.

-   This element MAY have a type attribute of type String that
    distinguishes it within a general resource class. A compliant
    implementation SHOULD support this attribute if the element is
    supported.

The following are two examples for specifying a Resource element:

\<Resource name="License" type="MATLAB"\>1\</Resource\>

\<Resource name="Telescope" type="Zoom2000" wallDuration="750"
metric="KX"\>10\</Resource\>

### NodeProperties Element

The NodeProperties element is used to encapsulate node properties
required of nodes selected for the job (or task group) to run on.

-   A compliant implementation MAY support this element.

-   This element MAY appear zero or one times within a given set of Job
    (or TaskGroup) Properties.

-   This element MUST contain one or more Node Properties.

The following illustrates the syntax of this element:

\<NodeProperties\>

\<!\-- Node Properties \--\>+

\</NodeProperties\>

#### Node Properties

Node Properties allow a distinction to be made between the resources or
properties you desire for a task, and the resources or properties you
want the node than the task runs on to have. For example, you may want
your task to only run on nodes that have a certain number of cpu's,
though you will not be using all of them and are still willing to share
them with other jobs.

##### Consumable Resources

Any of the Consumable Resources described above may be included as node
properties.

##### Other Optional Simple Node Properties

Other simple (unstructured) node properties are enumerated in Table 5.

Table 5 Optional Simple Node Properties

  ----------------- ---------- ------------- ------------ ------------ ------------
  Element Name      Type       Description   Appearance   Compliance   Categories

  Feature           String     Arbitrary     MAY          SHOULD       
                               named feature                           
                               of the node.                            

  Name              String     Node name or  MAY          MAY          
                               pattern.                                
  ----------------- ---------- ------------- ------------ ------------ ------------

### Extension Element

The Extension element provides a means to pass extensible properties
with the job object.

Some applications may find it easier to deal with a named extension
property than discover and handle elements for which they do not
understand or anticipate by name.

-   A compliant implementation MAY support this element.

-   This element MUST have a name attribute that is of type String and
    represents the name of the extension property. A compliant
    implementation MUST support this attribute if this element is
    supported.

-   This element MAY have a type attribute that is of type String and
    provides a hint about the context within which the property should
    be understood. A compliant implementation SHOULD support this
    attribute if this element is supported.

-   The character content of this element is of type String and is the
    value of the extension property.

The following is an example of an Extension element:

\<Extension type="Scheduler" name="Restartable"\>true\</Extension\>

### TaskGroup

A job MAY specify one or more task groups.

See the next section for element details.

### TaskGroupDefaults

A job MAY specify zero or more task group defaults.

See the next section for element details.

# TaskGroup and TaskGroupDefaults Element

The TaskGroup and TaskGroupDefaults elements are of the same structure.
A TaskGroup element aggregates tasks. A TaskGroupDefaults element may
only appear within a Job (or JobDefaults) and represents the defaults to
be taken by all task groups within the job. Task group properties in
TaskGroup elements override any properties found in a sibling
TaskGroupDefaults element.

-   A compliant implementation MAY support the TaskGroup element.

-   A compliant implementation MAY support the TaskGroupDefaults
    element.

-   A task group MUST specify one or more TaskGroup Properties.

-   One or more Task elements MAY appear at this level.

-   Zero or one TaskDefaults elements MAY appear at this level.

The following illustrates the syntax of this element:

\<TaskGroup\>

\<!\-- TaskGroup Properties \--\>+

\<!\-- Job Properties \--\>\*

\<Task\>+

\<TaskDefaults\>?

\</TaskGroup\>

## TaskGroup Properties

TaskGroup Properties are properties that apply to a particular task
group or as default properties to encompassed task groups. These
properties include the task group id, its tasks, task defaults, and
other simple task group properties.

### Simple TaskGroup Properties

Simple (unstructured) task group properties are enumerated in Table 6.

Table 6 Simple TaskGroup Properties

  -------------------- --------- ------------- ------------ ------------ ------------
  Element Name         Type      Description   Appearance   Compliance   Categories

  SessionId            Integer   Session id    MAY          MAY          
                                 for the task                            
                                 group or job.                           

  TaskCount            Integer   Number of     MAY          MUST         
                                 tasks in this                           
                                 taskgroup                               

  TaskGroupId          String    A task group  MAY          MAY          
                                 identifier                              
                                 unique within                           
                                 the job.                                

  TaskGroupName        String    A task group  MAY          SHOULD       
                                 name (such as                           
                                 "Master").                              
  -------------------- --------- ------------- ------------ ------------ ------------

### Task

A task group MAY specify zero or more tasks.

See the next section for element details.

### TaskDefaults

A task group MAY specify zero or more task defaults.

See the next section for element details.

# Task and TaskDefaults Element

The Task and TaskDefaults elements are of the same structure. A Task
element contains information specific to a task (like the process id or
the host it ran on). A TaskDefaults element may only appear within a
TaskGroup (or TaskGroupDefaults) element and represents the defaults to
be taken by all tasks within the task group. Task properties in Task
elements override any properties found in a sibling TaskDefaults
element.

-   A compliant implementation MAY support the TaskGroup element.

-   A compliant implementation MAY support the TaskGroupDefaults
    element.

-   A task group MUST specify one or more TaskGroup Properties.

-   One or more Task elements MAY appear at this level.

-   Zero or one TaskDefaults elements MAY appear at this level.

The following illustrates the syntax of this element:

\<Task\>

\<!\-- Task Properties \--\>+

\<!\-- Job Properties \--\>\*

\</Task\>

## Task Properties

Task Properties are properties that apply to a particular task or as
default properties to encompassed tasks. These properties include the
task id and other task properties.

### Simple Task Properties

Simple (unstructured) task properties are enumerated in Table 7.

Table 7 Simple Task Properties

  -------------------- --------- ------------- ------------ ------------ ------------
  Element Name         Type      Description   Appearance   Compliance   Categories

  Node                 String    Name of the   MAY          MUST         
                                 node this                               
                                 task ran on.                            

  ProcessId            Integer   Parent or     MAY          MAY          
                                 group process                           
                                 id for the                              
                                 task.                                   

  TaskId               String    A task        MAY          MAY          
                                 identifier                              
                                 unique within                           
                                 the                                     
                                 taskgroup.                              
  -------------------- --------- ------------- ------------ ------------ ------------

# Property Categories

Certain properties need to be classified as being in a particular
category. This is done when it is necessary to distinguish between a
property that is requested versus a property that was utilized or
dedicated. When no such distinction is necessary, it is recommended that
the property not be enveloped in one of these elements. In general, a
property should be enveloped in a category element only if it is
expected that the property will need to be attributed to more than one
property category, or if it needs to make use of some of the special
attributes inherited from the category.

## Requested Element

A requested property reflects properties as they were requested. A
disparity might occur between the requested value and the values
utilized or dedicated if a preference was expressed, if multiple options
were specified, or if ranges or pattern matching was specified.

-   A compliant implementation SHOULD support this element.

The following illustrates the syntax of this element:

\<Requested\>

\<!\-- Requested Properties \--\>+

\</Requested\>

The following describes the attributes and elements for the example
above:

/Requested

This element is used to encapsulate requested properties.

/Requested/\<Requested Property\>

Requested properties appear at this level.

Requested Properties inherit some additional attributes.

-   A requested property MAY have an op attribute of type String that
    indicates a conditional operation on the value. A compliant
    implementation SHOULD support this attribute. Valid values for the
    op attribute include "eq" meaning equals (which is the default),
    "ne" meaning not equal, "lt" meaning less than, "gt" meaning greater
    than, "le" meaning less than or equal to, "ge" meaning greater than
    or equal to, "match" which implies the value is a pattern to be
    matched.

-   A requested property MAY have a conj attribute of type String that
    indicates a conjunctive relationship with the previous element. A
    compliant implementation MAY support this attribute. Valid values
    for the conj attribute include "and" (which is the default), "or",
    "nand" meaning and not, and "nor" meaning or not.

-   A requested property MAY have a group attribute of type Integer that
    indicates expression grouping and operator precedence much like
    parenthetical groupings. A compliant implementation MAY support this
    attribute. A positive grouping indicates the number of nested
    expressions being opened with the property while a negative grouping
    indicates the number of nested expressions being closed with the
    property.

-   A requested property MAY have a preference attribute of type Integer
    that indicates a preference for the property along with a weight
    (the weight are taken as a ratio to the sum of all weights in the
    same group). A compliant implementation MAY support this attribute.
    If a group of positive valued preference alternatives are specified,
    at least one of the preferences must be satisfied for the job to
    run. If a group of negative valued preferences are specified, the
    preferences will try to be met according to their weights but the
    job will still run even if it can't satisfy any of the preferred
    properties. (Weight ranking can be removed by making all weights the
    same value (1 or -1 for example).

-   A requested property MAY have a performanceFactor attribute of type
    Float that provides a hint to the scheduler of what performance
    tradeoffs to make in terms of resources and start time. A compliant
    implementation MAY support this attribute.

The following are four examples of using Requested Properties:

\<Requested\>

\<Processors op="ge"\>8\</Processors\>

\<Processors op="le"\>16\</Processors\>

\<WallDuration\>3600\</WallDuration\>

\</Requested\>

\<Requested\>

\<NodeCount\>1\</NodeCount\>

\<NodeProperties\>

\<Name op="match"\>fr15.\*\</Name\>

\</NodeProperties\>

\<Requested\>

\<Requested\>

\<UserId group="1"\>scottmo\</UserId\>

\<AccountName group="-1"\>mscfops\</AccountName\>

\<UserId conj="or" group="1"\>monkeyboy\</UserId\>

\<AccountName group="-1"\>junglehunt\</AccountName\>

\</Requested\>

\<Requested\>

\<Memory preference="2"\>1024\</Memory\>

\<Memory preference="1"\>512\</Memory\>

\</Requested\>

## Utilized Element

A utilized property reflects properties as they were utilized, realized
or consumed. It reflects the actual amounts or values that are used or
currently allocated, as opposed to a limit, choice or pattern as may be
the case with a requested property, or a dedicated amount that prevents
sharing by other work requests.

-   A compliant implementation SHOULD support this element.

The following illustrates the syntax of this element:

\<Utilized\>

\<!\-- Utilized Properties \--\>+

\</Utilized\>

The following describes the attributes and elements for the example
above:

/Utilized

This element is used to encapsulate utilized properties.

/Utilized/\<Utilized Property\>

Utilized properties appear at this level.

Utilized Properties inherit some additional attributes.

-   A requested property MAY have a group attribute of type Integer that
    indicates expression grouping and operator precedence much like
    parenthetical groupings. A compliant implementation MAY support this
    attribute. A positive grouping indicates the number of nested
    expressions being opened with the property while a negative grouping
    indicates the number of nested expressions being closed with the
    property. The purpose of this attribute would be to logically group
    utilized properties if they were used in certain aggregations (like
    a job that spanned machines).

The following are the same four examples distinguishing the utilized
amounts and values:

\<Utilized\>

\<Processors\>12\</Processors\>

\<WallDuration\>1234\</WallDuration\>

\</Utilized\>

\<Utilized\>

\<NodeList\>

\<Node\>fr15n03\</Node\>

\</NodeList\>

\<Utilized\>

\<Utilized\>

\<UserId\>scottmo\</UserId\>

\<AccountName\>mscfops\</AccountName\>

\</Utilized\>

\<Utilized\>

\<Memory\>1024\</Memory\>

\</Utilized\>

## Dedicated Element

A dedicated property refers to a resource dedicated during the job. For
example, a job may request to restrict the usage of certain resources or
the entire node thereby preventing it from being shared by other work
requests even though the job only needs a portion of the resources to
accomplish its work. This may be to avoid conflict introduced when the
resource is shared, for confidentiality or any other reason. In such a
case, the amount of a resource requested and the amount actually
utilized may differ from the amount that was blocked. The dedicated
amount might be used for charge purposes, the utilized amount for usage
accounting, and requested amount for task assignment.

-   A compliant implementation MAY support this element.

The following illustrates the syntax of this element:

\<Dedicated\>

\<!\-- Dedicated Properties \--\>+

\</Dedicated\>

The following describes the attributes and elements for the example
above:

/Dedicated

This element is used to encapsulate dedicated properties.

/Dedicated/\<Dedicated Property\>

Dedicated properties appear at this level.

Dedicated Properties inherit some additional attributes.

-   A requested property MAY have a group attribute of type Integer that
    indicates expression grouping and operator precedence much like
    parenthetical groupings. A compliant implementation MAY support this
    attribute. A positive grouping indicates the number of nested
    expressions being opened with the property while a negative grouping
    indicates the number of nested expressions being closed with the
    property. The purpose of this attribute would be to logically group
    utilized properties if they were used in certain aggregations (like
    a job that spanned machines).

The following is an example of specifying dedicated properties:

\<Dedicated\>

\<Processors\>16\</Processors\>

\<Resource name="NetworkAdapter" type="ELAN4"\>2\</Resource\>

\</Dedicated\>

# AwarenessPolicy Attribute

A word or two should be said about compatibility mechanisms. With all
the leeway in the specification allowing implementers to choose to
implement or not implement various portions of the specification,
problems might arise if an implementation simply ignores a portion of a
job specification that is critical to the job function in certain
contexts, so we might want to enforce a policy in which a job is
rejected if any element or attribute contained in it was not supported
by the implementation. On the other hand, in many cases jobs will want
to interpret them the best they can and have the components implement a
best-effort approach to operating on the portions they understand.
Consequently, we define an awarenessPolicy attribute which can be added
as an optional attribute to the Job element or any other containment or
property element to indicate how the property (or the default action for
the elements that the containment element encloses) must react when the
implementation does not understand an element or attribute.

An awareness policy of "Reject" will cause the server to return a
failure if it receives a client request in which it does not support an
associated element name or attribute name or value. It is reasonable for
an implementation to ignore (not even look for) an element or attribute
that would not be critical to its function as long as ignoring this
attribute or element would not cause an incorrect result. However, any
element or attribute that was present that would be expected to be
handled in a manner that the implementation does not support must result
in a failure.

An awareness policy of "Warn" will accept the misunderstood element or
attribute and continue to process the job object on a best effort basis.
However a warning MUST be sent (if possible) to the requestor
enumerating the elements and attributes that are not understood.

An awareness policy of "Ignore" will accept the unsupported element or
attribute and continue to process the job object on a best effort basis.
The action could be to simply ignore the attribute.

-   This name of this attribute is awarenessPolicy.

-   This attribute is of type String.

-   This attribute can have values of "Reject", "Warn" or "Ignore".

-   A compliant implementation MAY support this attribute.

-   If an implementation does not support the attribute, it MUST adopt
    the policy of rejecting any job object which contains elements or
    attributes that it does not support. Furthermore, it SHOULD return a
    message to the requestor with an indication of the element or
    attribute name it did not understand.

-   This attribute MAY be present in a property or containment element.

-   If an implementation does support the attribute, but it is absent,
    the default value of "Reject" is implied.

-   Individual elements within the job object may override the
    containing object's awareness policy default by including this
    attribute. For example, a job might specify an awarenessPolicy of
    "Reject" at its root (the Job element) but may want to allow a
    particular subset of elements or attributes to be ignored if not
    understood. Conversely, a job with a default awarenessPolicy of
    "Ignore" might want to classify a subset of its optional elements as
    "Reject" if they are indispensible to its correct interpretation. An
    implementation can opt to check or not check for this attribute at
    any level it wants but must assume a "Reject" policy for any
    elements it does not check.

# References

**ISO 8601**

> ISO (International Organization for Standardization). *Representations
> of dates and times, 1988-06-15.* <http://www.iso.ch/markete/8601.pdf>

#  {#section .unnumbered}

**DATATYPES**

> XML Schema Part 2: Datatypes. *Recommendation, 02 MAY 2001.*
> <http://www.w3.org/TR/xmlschema-2/>

# Appendix A {#appendix-a .unnumbered}

## Units of Measure Abbreviations {#units-of-measure-abbreviations .unnumbered}

  ----------------------- ----------------------- -----------------------
  **Abbreviation**        **Definition**          **Quantity**

  B                       byte                    1 byte

  KB                      Kilobyte                2\^10 bytes

  MB                      Megabyte                2\^20 bytes

  GB                      Gigabyte                2\^30 bytes

  TB                      Terabyte                2\^40 bytes

  PB                      Petabyte                2\^50 bytes

  EB                      Exabyte                 2\^60 bytes

  ZB                      Zettabyte               2\^70 bytes

  YB                      Yottabyte               2\^80 bytes

  NB                      Nonabyte                2\^90 bytes

  DB                      Doggabyte               2\^100 bytes
  ----------------------- ----------------------- -----------------------