SSS Job Object Specification

Release Version 2.0

April 29, 2003

Scott Jackson, PNNL

David Jackson, Ames Lab

Brett Bode, Ames Lab

Scalable Systems Software Job Object Specification

Status of this Memo

This is an initial draft of the job object to be used by Scalable
Systems Software compliant components. It is envisioned for this
specification to be used in conjunction with the SSSRMAP protocol with
the job object passed in the Data field of Requests and Responses.
Queries can be issued to a job-cognizant component in the form of
modified XPATH expressions to the Get field to extract specific
information from the job object as described in the SSSRMAP protocol.

**Abstract**

This document describes the syntax and structure of the SSS job object.
A job model is described that is flexible enough to support the
specification of very simple jobs as well jobs with elaborate and
complex specification requirements in a way that avoids complex
structures and syntax when it is not needed. The basic assumption is
that a solitary job specification should be usable for all phases of the
job lifecycle and can be used at submission, queuing, staging,
reservations, quotations, execution, charging, accounting, etc. This job
specification provides support for multi-step jobs, as well as steps
with multiple task descriptions. It takes into account the aspects
necessary to operate within a grid or meta-scheduled environment where
the job might be executed at one (or more) destination among a variety
of different administrative domains and resource management systems.

# Introduction

This specification proposes a standard XML representation for a job
object for use by the various components in the SSS Resource Management
System. This object will be used in multiple contexts and by multiple
components. It is anticipated that this object will be passed via the
Data Element of SSSRMAP Requests and Responses.

## Goals

There are several goals motivating the design of this representation.

It needs to be inherently flexible. We recognize we will not be able to
exhaustively include the ever-changing job properties and capabilities
that constantly arise.

The same job object should be used at all stages of its lifecycle. This
object will be used at job submission, queuing, scheduling, and usage
charging and accounting, hence it needs to distinguish between
requested, utilized and allocated properties.

Its design takes into account the properties and structure required to
function in a meta or grid environment. It needs to include the
capability to support local mapping of properties, global namespaces,
etc.

This job object must support multi-step jobs. Each step can have
multiple logical task descriptions.

Many potential users of the specification will not be prepared to
implement the complex portions or fine-granularity that others need.
There needs to be a way to allow the more complicated structure to be
added as needed while leaving more straightforward cases simple.

Not all implementers will be interested in supporting properties at as
fine-grained level. It would be useful if the specification allowed the
properties to appear at the level an implementer considers them to vary
at.

There needs to be guidance for how to understand a given job object when
higher order features are not supported by an implementation, and which
parts are required, recommended and optional for implementors to
implement.

Needs to support composite resources,

Ability to specify preferences or fuzzy requirements.

## Non-Goals

The following topics are outside the scope of the job object

This specification does not attempt to specify namespaces or even naming
conventions for most property values.

## Examples

### Very Simple Example

This example shows that a very simple and direct representation can be
used when more complex features of this specification are not needed.

\<Job\>

\<JobId\>PBS.1234\</JobId\>

\<UserName\>scottmo\</UserName\>

\<Executable\>/bin/hostname\</Executable\>

\<Processors\>16\</Processors\>

\</Job\>

### Moderate Example

This example shows a single step job of medium complexity that makes use
of features such as an explicit step and required versus utilized
properties.

\<Job\>

\<JobId\>PBS.1234\</JobId\>

\<JobName\>Heavy Water\</JobName\>

\<AccountName\>nwchemdev\</AccountName\>

\<UserId\>peterk\</UserId\>

\<Application\>NWChem\</Application\>

\<Executable\>/usr/local/nwchem/bin/nwchem\</Executable\>

\<Arguments\>-input basis.in\</Arguments\>

\<InitialWorkingDirectory\>/home/peterk\</InitialWorkingDirectory\>

\<Step\>

\<StepId\>PBS.1234.0\</StepId\>

\<MachineName\>Colony\</MachineName\>

\<QualityOfService\>BottomFeeder\</QualityOfService\>

> \<Queue\>batch_normal\</Queue\>
>
> \<StepState\>Completed\</StepState\>

\<StartTime\>1051557713\</StartTime\>

> \<CompletionTime\>1051558868\</CompletionTime\>
>
> \<Charge\>25410\</Charge\>
>
> \<Requested\>
>
> \<Processors op="ge"\>12\</Processors\>
>
> \<Memory op="ge" units="GB"\>2\</Memory\>
>
> \<WallDuration\>3600\</WallDuration\>

\</Requested\>

\<Utilized\>

\<Processors\>16\</Processors\>

\<Memory metric="Average" units="GB"\>1.89\</Memory\>

> \<WallDuration\>1155\</WallDuration\>

\</Utilized\>

\<Environment\>

\<Variable name="PATH"\>/usr/bin:/home/peterk\</Variable\>

\</Environment\>

\</Step\>

\</Job\>

### Elaborate Example

This example shows that complex capabilities are expressible through
this specification when such capabilities are needed. Not all components
will care about this level of detail and components that use this
specification can include or discard the information that is relevant to
their function. Superfluous information can be ignored by the component
or filtered out (by XSLT for example).

\<Job\>

\<JobId\>fr15n05.1234\</JobId\>

\<JobState\>Active\</JobState\>

\<JobName\>ShuttleTakeoff\</JobName\>

\<GlobalJobId\>http://www.doesciencegrid.org/LBNL#12345678\</GlobalJobId\>

\<StagedTime\>1051557859\</StagedTime\>

\<SubmitHost\>asteroid.lbl.gov\</SubmitHost\>

\<SubmissionTime\>1051556734\</SubmissionTime\>

\<AccountName\>GrandChallenge18\</AccountName\>

\<GlobalUserId\>C=US,O=LBNL,CN=Keith Jackson\</GlobalUserId\>

\<UserId\>keith\</UserId\>

\<Environment\>

\<Variable name="LD_LIBRARY_PATH"\>/usr/lib\</Variable\>

\<Variable name="PATH"\>/usr/bin:\~/bin:\</Variable\>

\<Environment\>

\<Step\>

\<StepId\>fr15n05.1234.0\</StepId\>

\<StepName\>Launch Vector Initialization\</StepName\>

\<Executable\>/usr/local/gridphys/bin/lvcalc\</Executable\>

\<Queue\>batch\</Queue\>

\<StepState\>Completed\</StepState\>

\<MachineName\>SMP2.emsl.pnl.gov\</MachineName\>

\<StartTime\>1051557713\</StartTime\>

\<CompletionTime\>1051558868\</CompletionTime\>

\<QuoteId\>http://www.pnl.gov/SMP2#654321\</QuoteId\>

\<Charge units="USD"\>12.75\</Charge\>

\<Requested\>

\<WallDuration\>3600\</WallDuration\>

\<Processors\>2\</Processors\>

\<Memory\>1024\</Memory\>

\</Requested\>

\</Utilized\>

\<WallDuration\>1155\</WallDuration\>

\<Processors consumptionRate="0.78"\>2\</Processors\>

\<Memory metric="max"\>975\</Memory\>

\</Utilized\>

\<TaskGroup\>

\<TaskCount\>2\</TaskCount\>

\<TaskDistribution type="TasksPerNode"\>1\</TaskDistribution\>

\<Task\>

\<Node\>node1\</Node\>

\<ProcessId\>99353\</ProcessId\>

\</Task\>

\<Task\>

\<Node\>node12\</Node\>

\<ProcessId\>80209\</ProcessId\>

\</Task\>

\</TaskGroup\>

\</Step\>

\<Step\>

\<StepId\>fr15n05.1234.1\</StepId\>

\<StepName\>3-Phase Ascension\</StepName\>

\<Queue\>batch_normal\</Queue\>

\<StepState\>Idle\</StepState\>

\<MachineName\>Colony.emsl.pnl.gov\</MachineName\>

\<Priority\>1032847\</Priority\>

\<Hold\>System\</Hold\>

\<StatusMessage\>Insufficient funds to start job\</StatusMessage\>

\<Requested\>

\<WallDuration\>43200\</WallDuration\>

\</Requested\>

\<TaskGroup\>

\<TaskCount\>1\</TaskCount\>

\<TaskName\>Master\</TaskName\>

\<Executable\>/usr/local/bin/stage-coordinator\</Executable\>

\<Memory\>2048\<Memory\>

\<Resource name="License" type="ESSL2"\>1\</Resource\>

\<NodeProperties\>

\<Feature\>Jumbo-Frame\</Feature\>

\</NodeProperties\>

\</TaskGroup\>

\<TaskGroup\>

\<TaskName\>Slave\</TaskName\>

\<TaskDistribution type="Rule"\>RoundRobin\</TaskDistribution\>

\<Executable\>/usr/local/bin/stage-slave\</Executable\>

\<NodeCount\>4\</NodeCount\>

\<Requested\>

\<Processors group="-1"\>12\</Processors\>

\<Processors conj="or" group="1"\>16\</Processors\>

\<Memory\>512\</Memory\>

\<NodeProperties\>

\<Name op="match"\>fr15n.\*\</Name\>

\</NodeProperties\>

\</Requested\>

\</TaskGroup\>

\</Step\>

\</Job\>

# Conventions used in this document

## Keywords

The keywords "MUST", "MUST NOT", "REQUIIRED", "SHALL", "SHALL NOT",
"SHOULD", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to
be interpreted as described in RFC2119.

## Table Column Interpretations

In the property tables, the columns are interpreted to have the
following meanings:

Element Name: Name of the XML element (xsd:element)

Type: Data type defined by xsd (XML Schema Definition) as:

String xsd:string (a finite length sequence of printable characters)

Integer xsd:integer (a signed finite length sequence of decimal digits)

Float xsd:float (single-precision 32-bit floating point)

Boolean xsd:boolean (consists of the literals "true" or "false")

DateTime xsd:int (a 32-bit unsigned long in GMT seconds since the EPOCH)

Description: Brief description of the meaning of the property

Appearance: This column indicates whether the given property has to
appear within the parent element. It assumes the following meanings:

MUST This property is REQUIRED when the parent is specified.

SHOULD This propterty is RECOMMENDED when the parent is specified.

MAY This property is OPTIONAL when the parent is specified.

Compliance: The column indicates whether a compliant implementation has
to support the given property.

MUST A compliant implementation MUST support this property.

SHOULD A compliant implementation SHOULD support this property.

MAY A compliant implementation MAY support this property.

Categories: Some properties may be categorized into one of several
categories. Letters in this column indicate that the given property can
be classified in the the following property categories.

R This property can be encompassed in a Requested element.

U This property can be encompassed in a Utilized element.

D This property can be encompassed in a Dedicated element.

## Element Syntax Cardinality

The cardinality of elements in the element syntax sections may make use
of regular expression wildcards with the following meanings:

\* Zero or more occurrences

\+ One or more occurrences

? Zero or one occurrences

The absence of one of these symbols implies one and only one occurrence.

# The Job Model

The primary element within the job model is a job. A job can be
considered as an interdependent set of schedulable work entities. A job
is usually submitted at a single time and can consist of multiple
execution steps. A job step is a single schedulable entity and will be
the object normally seen in job queues. Job steps form a DAG (directed
acyclic graph) of dependencies on the status of previous jobs steps.

Job steps may consist of multiple tasks, which are the finest grained
work unit and represent an endpoint for executing a given process
instance. For example, a step that requests 3 nodes and 4 processors
will have 4 tasks, two on one node and one on each of two nodes. Tasks
may be grouped into TaskGroups, which are logical aggregations of tasks
and their common properties. Submit filters, prologs, epilogs,
notification scripts, etc. run once only for each step. Whereas
taskgroups function as logical descriptions of tasks and their
properties, they also describe the number of such tasks and the nodes
that they run on. As an example, a master taskgroup (consisting of a
single task) might ask for a node with a MATLAB license, 2GB of memory
and an internet connected network adapter while a slave taskgroup
(consisting of 12 tasks) could be targeted for nodes with more CPU
bandwidth \-- all within the same step and utilizing a common MPI ring.
Tasks (and hence taskgroups) can have different executables or
environments, specify different consumable resources or node properties.

A job will consist of at least one step, which will have at least one
taskgroup, which will have at least one task. Only the job has to be
specified explicitly. The others in the trivial cases (where there is
only one) can be implicit.

Default step properties, job-wide default taskgroup properties and
job-wide default task properties can be specified at the same level as
the job properties. In the case where a job has only one step and no
steps are explicitly specified, the default step properties are taken to
be the step properties. When one or more steps are specified, the
specified step properties within the steps override the default step
properties. If steps are not supported by an implementation, the step
properties are taken to be job properties.

Default taskgroup properties, and step-wide default task properties for
a particular step can be specified at the step level. In the case where
a step has only one taskgroup and no taskgroups are explicitly specified
the default taskgroup properties are taken to be the taskgroup
properties. When one or more taskgroups are specified, the specified
taskgroup properties within the taskgroup override the default taskgroup
properties for the step and job. If taskgroups are not supported by an
implementation, the taskgroup properties are taken to be step
properties.

Default task properties for a particular taskgroup can be specified at
the taskgroup level. In the case where a taskgroup has only one task and
no tasks are explicitly specified the default task properties are taken
to be the task properties. When one or more tasks are specified, the
specified task properties within the task override the default task
properties for the taskgroup, step and job. If tasks are not supported
by an implementation, the task properties are taken to be taskgroup
properties.

# Job Element

The root element of the Job object is the Job element and is used to
encapsulate a job.

-   A job object MUST have exactly one job element.

-   A compliant implementation MUST support this element.

-   A job MUST specify one or more Job Properties.

-   Any Step Property MAY appear at this level as a Default Step
    Property.

-   Any TaskGroup Property MAY appear at this level as a job-wide
    Default TaskGroup Property.

-   Any Task Property MAY appear at this level as a Job-wide Default
    Task Property

-   One or more Step elements can appear at this level as default steps
    (technically as a job property).

-   If no steps are explicitly specified, the job is implicitly defined
    to have one step, with its properties taken from the Default Step
    Properties.

-   A TaskGroup element MAY appear at this level as a job-wide default
    taskgroup (technically as a step property).

-   A Task element MAY appear at this level as a job-wide default task
    (technically a taskgroup property).

The following illustrates the syntax of this element:

\<Job\>

\<!\-- Job Properties \--\>+

\<!\-- Default Step Properties \--\>\*

\<!\-- Default TaskGroup Properties \--\>\*

\<!\-- Default Task Properties \--\>\*

\<Step/\>\* \# These are redundant and here for clarity

\<TaskGroup/\>\* \# Perhaps they add confusion and

\<Task/\>\* \# should be stripped out

\</Job\>

The following describes the attributes and elements for the example
above:

/Job

This required element is the root element encapsulating a job.

/Job/\<Job Property\>

Job properties appear at this level.

/Job/\<Step Property\>

Default step properties may be included at this level.

/Job/\<TaskGroup Property\>

Job-wide default taskgroup properties may be included at this level.

/Job/\<Task Property\>

Job-wide default task properties may be included at this level.

/Job/Step

This optional element encapsulates steps (technically a job property).

A job may specify zero or more steps.

/Job/TaskGroup

This optional element encapsulates taskgroups (technically a step
property).

> A taskgroup might appear at this level if it is a default taskgroup
> inherited by all steps, if one or more taskgroups are specified
> explicitly while the step is implied, or if steps are not defined for
> the implementation but taskgroups are.

A job may specify zero or more taskgroups.

/Job/Task

This optional element encapsulates tasks (technically a taskgroup
property).

> A task may appear at this level if it is a default task inherited by
> all taskgroups, if one or more tasks are specified explicitly at this
> level while the step and taskgroup are implied, or if steps or
> taskgroups are not defined for the implementation but tasks are.

/Job/{any}

This is a mechanism to allow extensible job properties to be specified.

/Job/@{any}

This is a mechanism to allow extensible attributes to be specified for
the job.

## Job Properties

Job Properties are properties that always apply to the job as a whole.
These include the job id, any explicit job steps, and other simple
optional job properties.

### Simple Job Properties

Simple (unstructured) job properties are enumerated in Table 1.

Table 1 Simple Job Properties

  ---------------- ---------- -------------- ------------ ------------ ------------
  Element Name     Type       Description    Appearance   Compliance   Categories

  CreationTime     DateTime   Date and time  MAY          MAY          
                              that the job                             
                              was submitted                            

  Description      String     Description of MAY          MAY          
                              the job                                  

  GlobalJobId      String     Globally       MAY          SHOULD       
                              unique job                               
                              identifier                               

  JobId            String     Local job      MUST         MUST         
                              identifier                               
                              assigned to                              
                              the job by the                           
                              local resource                           
                              manager.                                 

  JobName          String     Name of the    MAY          MAY          
                              job                                      

  JobState         String     State of the   MAY          MAY          
                              job as a                                 
                              whole. Valid                             
                              states may                               
                              include                                  
                              "NotQueued",                             
                              "Unstarted",                             
                              "Active",                                
                              "Completed".                             

  StagedTime       DateTime   Date and time  MAY          MAY          
                              that a job was                           
                              staged to the                            
                              local resource                           
                              management                               
                              system.                                  

  SubmitHost       String     FQDN of host   MAY          SHOULD       
                              where the job                            
                              was submitted                            
                              from.                                    

  SubmissionTime   DateTime   Date and time  MAY          SHOULD       
                              that a job was                           
                              submitted.                               
  ---------------- ---------- -------------- ------------ ------------ ------------

### Credentials

Credentials are a special group of job properties that relate to some
kind of authenticated token or id and are also distinct in the fact that
they can be categorized in both requested and utilized forms.

Credential job properties are enumerated in Table 2.

Table 2 Credential Job Properties

  --------------- --------- ------------- ------------ ------------ ------------
  Element Name    Type      Description   Appearance   Compliance   Categories

  AccountName     String    Name of the   MAY          SHOULD       RU
                            Project or                              
                            Charge                                  
                            Account                                 

  GlobalUserId    String    Globally      MAY          SHOULD       RU
                            unique user                             
                            identifier.                             
                            This may be                             
                            an X.509 DN                             
                            for example.                            

  GroupId         String    Name of the   MAY          MAY          RU
                            local group                             
                            id.                                     

  UserId          String    Name of the   MAY          MUST         RU
                            local userid                            
                            for the job.                            
  --------------- --------- ------------- ------------ ------------ ------------

### Step

A step is a job property.

-   A job MAY specify zero or more steps.

See the next section for element details.

# Step Element

The Step element is used to encapsulate a step.

-   A compliant implementation SHOULD support this element.

-   A step MUST specify one or more Step Properties.

-   Any TaskGroup Property MAY appear at this level as a Default
    TaskGroup Property.

-   One or more TaskGroup elements MAY appear at this level as default
    taskgroups (technically as a Step Property).

-   If no taskgroups are explicitly specified, the step is implicitly
    defined to have one taskgroup, with its properties taken from the
    Default TaskGroup Properties.

-   Default step properties are located as children of the encapsulating
    Job element.

-   A Task element MAY appear at this level as a step-wide default task
    (technically a taskgroup property).

-   If an implementation does not support the Step element, the Step
    Properties described in this section are used as Job Properties.

-   An implementation MAY choose to support individual properties
    defined in this section (Step Properties) as properties of a higher
    level only (Job Properties).

The following illustrates the syntax of this element:

\<Step\>

\<!\-- Step Properties \--\>+

\<!\-- Default TaskGroup Properties \--\>\*

\<!\-- Step-wide Default Task Properties \--\>\*

\<TaskGroup/\>\* \# Redundant

\<Task/\>\* \# Perhaps?

\</Step\>

The following describes the attributes and elements for the example
above:

/Step

This element is used to encapsulate the step.

/Step/\<Step Property\>

Step properties appear at this level.

/Step/\<TaskGroup Property\>

Default taskgroup properties for the step may be included at this level.

/Step/\<Task Property\>

Step-wide default task properties may be included at this level.

/Step/TaskGroup

This optional element encapsulates taskgroups (technically a step
property)

A step may specify zero or more taskgroups.

/Step/Task

This optional element encapsulates tasks (technically a taskgroup
property).

> A task may appear at this level if it is a default task inherited by
> all tasks within the step, if one or more tasks are specified
> explicitly at this level while the taskgroup is implied, or if
> taskgroups are not defined for the implementation but tasks are. A
> step may specify zero or more tasks.

/Step/{any}

This is a mechanism to allow extensible step properties to be specified.

/Step/@{any}

This is a mechanism to allow extensible attributes to be specified for
the step.

## Step Properties

Step Properties are properties that apply to a particular step or as
default properties to all steps. These properties include the step id,
any explicit taskgroups, and other simple optional step properties.

### Simple Step Properties

Simple (unstructured) step properties are enumerated in Table 3.

Table 3 Simple Step Properties

  ------------------ ---------- --------------- ------------- ------------ ------------
  Element Name       Type       Description     Appearance    Compliance   Categories

  Charge             Float      Amount charged  MAY           SHOULD       RU
                                for the step.                              

  CompletionTime     DateTime   Date and time   MAY           MUST         RU
                                that a step                                
                                completed                                  
                                (independent of                            
                                success or                                 
                                failure).                                  

  EligibleTime       DateTime   Date and time   MAY           MAY          
                                that a step                                
                                became eligible                            
                                to run.                                    

  Hold               String     Hold(s) on the  MAY           SHOULD       
                                job if any.                                

  MachineName        String     Name of the     MAY           MUST         RU
                                system or                                  
                                cluster that                               
                                the step runs                              
                                on.                                        

  NodeCount          Integer    Number of nodes MAY           MUST         RU
                                used by the                                
                                step.                                      

  Priority           Integer    Current queue   MAY           SHOULD       
                                priority (or                               
                                rank) for the                              
                                step.                                      

  QualityOfService   String     Name of the     MAY           SHOULD       RU
                                Quality of                                 
                                Service (QOS).                             

  Queue              String     Name of the     MAY           SHOULD       RU
                                Queue (or                                  
                                class) that the                            
                                job step runs                              
                                in.                                        

  QuoteId            String     Identifier for  MAY           MAY          
                                a guaranteed                               
                                charge rate                                
                                quote obtained                             
                                by the step.                               

  ReservationTime    DateTime   Date and time   MAY           MAY          
                                that a                                     
                                reservation was                            
                                placed for the                             
                                job step.                                  

  StartCount         Integer    Number of times MAY           MAY          
                                the scheduler                              
                                tried to start                             
                                the step.                                  

  StartTime          DateTime   Date and time   MAY           MUST         RU
                                that the step                              
                                started.                                   

  StatusMessage      String     Natural         MAY           SHOULD       
                                language                                   
                                message that                               
                                can be used to                             
                                provide detail                             
                                on why a job                               
                                failed, isn't                              
                                running, etc.                              

  StepId             String     A local step    SHOULD \* A   SHOULD       
                                identifier      step id can                
                                assigned to the be omitted in              
                                job by the      the trivial                
                                local resource  (single step)              
                                manager.        case if a                  
                                                rule exists                
                                                for                        
                                                canonically                
                                                generating                 
                                                the step id                
                                                from the job               
                                                id.                        

  StepName           String     Name of the     MAY           SHOULD       
                                step                                       

  StepState          String     State of the    MAY           MUST         
                                step. Valid                                
                                states may                                 
                                include "Idle",                            
                                "Hold",                                    
                                "Running",                                 
                                "Suspended",                               
                                "Completed".                               

  SuspendDuration    Integer    Number of       MAY           MAY          
                                seconds the job                            
                                was in the                                 
                                "Suspended"                                
                                state.                                     

  TimeCategory       String     This allows the MAY           MAY          
                                specification                              
                                of shifts like                             
                                "PrimeTime" for                            
                                charging                                   
                                purposes.                                  

  Type               String     Type of step    MAY           MAY          
                                (job). Meaning                             
                                of this                                    
                                extension                                  
                                property is                                
                                context                                    
                                specific.                                  

  WallDuration       Integer    Number of       MAY           MUST         RU
                                seconds the job                            
                                was in the                                 
                                "Running"                                  
                                state.                                     
  ------------------ ---------- --------------- ------------- ------------ ------------

### TaskGroup

A taskgroup is a step property.

-   A step MAY specify zero or more taskgroups.

See the next section for element details.

# TaskGroup Element

The TaskGroup element is used to encapsulate a taskgroup.

-   A compliant implementation MAY support this element.

-   A taskgroup MUST specify one or more TaskGroup Properties.

-   Default taskgroup properties are located as children of the
    encapsulating Step element or as children of the encapsulating Job
    element.

-   If an implementation does not support the TaskGroup element, the
    TaskGroup Properties described in this section are used as Step
    Properties if it is supported, or as Job Properties otherwise.

```{=html}
<!-- -->
```
-   An implementation MAY choose to support individual properties
    defined in this section (TaskGroup Properties) as properties of a
    higher level only (Step or Job Properties).

The following illustrates the syntax of this element:

\<TaskGroup\>

\<!\-- TaskGroup Properties \--\>+

\<Task\>+

\</TaskGroup\>

The following describes the attributes and elements for the example
above:

/TaskGroup

This element is used to encapsulate the taskgroup.

/TaskGroup/\<TaskGroup Property\>

TaskGroup properties appear at this level.

/TaskGroup/\<Task Property\>

Default task properties may appear at this level.

/TaskGroup/Task

This optional element encapsulates tasks (technically a taskgroup
property).

A taskgroup may specify one or more tasks.

/TaskGroup/{any}

This is a mechanism to allow extensible taskgroup properties to be
specified.

/TaskGroup/@{any}

This is a mechanism to allow extensible attributes to be specified for
the taskgroup.

## TaskGroup Properties

TaskGroup Properties are properties that apply to a particular taskgroup
or as default properties to encompassed taskgroups. These properties
include the taskgroup id, its tasks, and other taskgroup properties.

### Simple TaskGroup Properties

Simple (unstructured) taskgroup properties are enumerated in Table 4.

Table 4 Simple TaskGroup Properties

  -------------------- --------- ------------- ------------ ------------ ------------
  Element Name         Type      Description   Appearance   Compliance   Categories

  SessionId            Integer   Session id    MAY          MAY          
                                 for the                                 
                                 taskgroup or                            
                                 step.                                   

  TaskCount            Integer   Number of     MAY          MUST         
                                 tasks in this                           
                                 taskgroup                               

  TaskGroupId          String    A taskgroup   MAY          MAY          
                                 identifier                              
                                 unique within                           
                                 the step.                               

  TaskGroupName        String    A taskgroup   MAY          MAY          
                                 name (such as                           
                                 "Master").                              
  -------------------- --------- ------------- ------------ ------------ ------------

### NodeList Element

The NodeList element is used to encapsulate nodes.

-   A compliant implementation SHOULD support this element.

-   This element MAY appear zero or one times within a given set of
    TaskGroup Properties.

-   This element MUST contain one or more Node elements.

-   This element MAY be categorized as a requested or utilized property
    by being encompassed by the appropriate element.

The following illustrates the syntax of this element:

\<NodeList\>

\<Node/\>+

\</NodeList\>

The following describes the attributes and elements for the example
above:

/NodeList

This element is used to encapsulate nodes.

/NodeList/Node

A named node.

/NodeList/{any}

This is a mechanism to allow extensible NodeList properties to be
specified.

/NodeList/@{any}

This is a mechanism to allow extensible attributes to be specified for
the NodeList.

#### Node Element

The Node element represents a node.

-   This element is of type String.

The following is an example of a Node element:

\<Node\>node1\</Node\>

### TaskDistribution Element

The TaskDistribution element describes how tasks are to be mapped to
nodes.

This may be a rule name, a task per node ratio or an arbitrary geometry.

-   A compliant implementation SHOULD support this element.

-   This element MAY appear zero or one times within a given set of
    TaskGroup Properties.

-   This element is of type String.

-   This element MAY have a type attribute of type String that provides
    a hint as to the type of mapping guidance provided. It may have
    values including "Rule", "TasksPerNode", or "Geometry". A compliant
    implementation MAY support this attribute if this element is
    supported.

-   It is possible when specifying a task distribution along with
    Processors, NodeCount or TaskCount to encounter a contradiction. The
    components are responsible for resolving the contradictions.

The following are three examples of a TaskDistribution element:

\<TaskDistribution type="TaskPerNode"\>2\</TaskDistribution\>

\<TaskDistribution type="Rule"\>RoundRobin\</TaskDistribution\>

\<TaskDistribution type="Geometry"\>{1,4}{2}{3,5}\</TaskDistribution\>

### Task

A task is a taskgroup property.

-   A taskgroup MAY specify one or more tasks.

See the next section for element details.

# Task Element

The Task element is used to encapsulate a task.

-   A compliant implementation MUST support this element if it supports
    the TaskGroup element.

```{=html}
<!-- -->
```
-   A task MUST specify one or more Task Properties.

-   Default task properties are located as children of the encapsulating
    Task element or as or as children of the encapsulating Step or Job
    element.

-   If an implementation does not support the Task element, the Task
    Properties described in this section are used as TaskGroup
    Properties if it is supported, else as Step Properties if it is
    supported, or as Job Properties otherwise.

```{=html}
<!-- -->
```
-   An implementation MAY choose to support individual properties
    defined in this section (Task Properties) as properties of a higher
    level only (TaskGroup, Step or Job Properties).

The following illustrates the syntax of this element:

\<Task\>

\<!\-- Task Properties \--\>+

\</Task\>

The following describes the attributes and elements for the example
above:

/Task

This element is used to encapsulate the task.

/Task/\<Task Property\>

Task properties appear at this level.

/Task/{any}

This is a mechanism to allow extensible task properties to be specified.

/Task/@{any}

This is a mechanism to allow extensible attributes to be specified for
the task.

## Task Properties

Task Properties are properties that apply to a particular task or as
default properties to encompassed tasks. These properties include the
task id and other task properties.

### Simple Task Properties

Simple (unstructured) task properties are enumerated in Table 5.

Table 5 Simple Task Properties

  ------------------------- --------- ------------- ------------ ------------ ------------
  Element Name              Type      Description   Appearance   Compliance   Categories

  Application               String    Type of       MAY          MAY          
                                      application                             
                                      such as                                 
                                      "Gaussian" or                           
                                      "Nwchem".                               

  Arguments                 String    The arguments MAY          SHOULD       
                                      for the                                 
                                      executable.                             

  Executable                String    Executable.   MAY          MUST         
                                      This may be                             
                                      an absolute                             
                                      or relative                             
                                      path.                                   

  InitialWorkingDirectory   String    Initial       MAY          SHOULD       
                                      working                                 
                                      directory                               

  Node                      String    Name of the   MAY          MUST         
                                      node this                               
                                      task ran on.                            

  ProcessId                 Integer   Parent or     MAY          MAY          
                                      group process                           
                                      id for the                              
                                      task.                                   

  TaskId                    String    A task        MAY          MAY          
                                      identifier                              
                                      unique within                           
                                      the                                     
                                      taskgroup.                              
  ------------------------- --------- ------------- ------------ ------------ ------------

### Environment Element

The Environment element is used to encapsulate environment variables.

-   A compliant implementation SHOULD support this element.

-   An Environment element MAY appear zero or one times within a given
    set of Task Properties.

-   An Environment element MUST contain one or more Variable elements.

The following illustrates the syntax of this element:

\<Environment\>

\<Variable/\>+

\</Environment\>

The following describes the attributes and elements for the example
above:

/Environment

This element is used to encapsulate environment variables.

/Environment/Variable

A variable and its value.

/Environment/{any}

This is a mechanism to allow extensible environment properties to be
specified.

[/Environment/@{any](mailto:/Environment/@{any)}

This is a mechanism to allow extensible attributes to be specified for
the environment.

#### Variable Element

The Variable element represents an environment variable with its name
and value.

This element MUST have a name attribute that is of type String. A
compliant implementation MUST support this attribute if this element is
supported.

The character content of this element is the value of the environment
variable.

The following is an example of a Variable element:

\<Variable name="PATH"\>/usr/bin:/home/sssdemo\</Variable\>

### Consumable Resources

Consumable Resources are a special group of properties that can have
additional attributes and can be used in multiple contexts. In general a
consumable resource is a resource that can be consumed in a measurable
quantity.

-   A consumable resource MAY have a units attribute that is of type
    String that specifies the units by which it is being measured. If
    this attribute is omitted, a default unit is implied. A compliant
    implementation SHOULD support this attribute if the element is
    supported.

-   A consumable resource MAY have a metric attribute that is of type
    String that specifies the type of measurement being described. For
    example, the measurement may be a Total, an Average, a Min or a Max.
    A compliant implementation SHOULD support this attribute if the
    element is supported.

-   A consumable resource MAY have a wallDuration attribute of type
    Integer that indicates the amount of time for which that resource
    was used. This need only be specified if the resource was used for a
    different amount of time than the wallDuration for the step. A
    compliant implementation MAY support this attribute if the element
    is supported.

-   A consumable resource MAY have a consumptionRate attribute of type
    Float that indicates the average percentage that a resource was used
    over its wallDuration. For example, an overbooked SMP running 100
    jobs across 32 processors may wish to scale the usage and charge by
    the average fraction of processor usage actually delivered. A
    compliant implementation MAY support this attribute if the element
    is supported.

A list of simple consumable resources is listed in Table 8.

Table 8 Simple Consumable Resources

  ----------------- ---------- ------------- ------------ ------------ ------------
  Element Name      Type       Description   Appearance   Compliance   Categories

  Disk              Float      Amount of     MAY          SHOULD       RUD
                               disk.                                   

  Memory            Float      Amount of     MAY          SHOULD       RUD
                               memory.                                 

  Network           Float      Amount of     MAY          MAY          RUD
                               network.                                

  Processors        Integer    Number of     MAY          MUST         RUD
                               processors.                             

  Swap              Float      Amount of     MAY          MAY          RUD
                               virtual                                 
                               memory.                                 
  ----------------- ---------- ------------- ------------ ------------ ------------

The following are two examples for specifying a consumable resource:

\<Memory metric="Max" units="GB"\>483\</Memory\>

\<Processors wallDuration="1234"
consumptionRate="0.63"\>4\</Processors\>

### Resource Element

In addition to the consumable resources enumerated in the above table,
an extensible consumable resource is defined by the Resource element.

-   A compliant implementation SHOULD support this element.

-   This element MAY appear zero or more times within a given set of
    task properties.

-   Like the other consumable resources, this property MAY be
    categorized as a requested, utilized or dedicated properties by
    being encompassed in the appropriate elements.

-   This element is of type Float.

-   It shares the other same properties and attributes as the other
    consumable resources but it requires an additional name (and
    optional type) attribute to describe it.

-   This element MUST have a name attribute of type String that
    indicates the type of consumable resource being measured. A
    compliant implementation MUST support this attribute if the element
    is supported.

-   This element MAY have a type attribute of type String that
    distinguishes it within a general resource class. A compliant
    implementation SHOULD support this attribute if the element is
    supported.

The following are two examples for specifying a Resource element:

\<Resource name="License" type="MATLAB"\>1\</Resource\>

\<Resource name="Telescope" type="Zoom2000" wallDuration="750"
metric="KX"\>10\</Resource\>

### NodeProperties Element

The NodeProperties element is used to encapsulate node properties.

-   A compliant implementation MAY support this element.

-   This element MAY appear zero or one times within a given set of Task
    Properties.

-   This element MUST contain one or more Node Properties.

The following illustrates the syntax of this element:

\<NodeProperties\>

\<!\-- Node Properties \--\>+

\</NodeProperties\>

The following describes the attributes and elements for the example
above:

/NodeProperties

This element is used to encapsulate nodes properties.

/NodeProperties/\<Node Property\>

A node property.

/NodeProperty/{any}

This is a mechanism to allow extensible node properties to be specified.

[/NodeProperty/@{any](mailto:/NodeProperty/@{any)}

This is a mechanism to allow extensible attributes to be specified for
the NodeProperties.

#### Node Properties

Node Properties allow a distinction to be made between the resources or
properties you desire for a task, and the resources or properties you
want the node than the task runs on to have. For example, you may want
your task to only run on nodes that have a certain number of cpu's,
though you will not be using all of them and are still willing to share
them with other jobs.

##### Consumable Resources

Any of the Consumable Resources described above may be included as node
properties.

##### Other Optional Simple Node Properties

Other simple (unstructured) node properties are enumerated in Table 8.

Table 8 Optional Simple Node Properties

  ----------------- ---------- ------------- ------------ ------------ ------------
  Element Name      Type       Description   Appearance   Compliance   Categories

  Feature           String     Arbitrary     MAY          SHOULD       
                               named feature                           
                               of the node.                            

  Name              String     Node name or  MAY          MAY          
                               pattern.                                
  ----------------- ---------- ------------- ------------ ------------ ------------

### Extension Element

The Extension element provides a means to pass extensible properties
with the job object.

Some applications may find it easier to deal with a named extension
property than discover and handle elements for which they do not
understand or anticipate by name.

-   This element MUST have a name attribute that is of type String and
    represents the name of the extension property. A compliant
    implementation MUST support this attribute if this element is
    supported.

-   This element MAY have a type attribute that is of type String and
    provides a hint about the context within which the property should
    be understood. A compliant implementation SHOULD support this
    attribute if this element is supported.

-   The character content of this element is of type String and is the
    value of the extension property.

The following is an example of an Extension element:

\<Extension type="Scheduler" name="Restartable"\>true\</Extension\>

# Property Categories

Certain properties need to be classified as being in a particular
category. This is done when it is necessary to distinguish between a
property that is requested versus a property that was utilized or
dedicated. When no such distinction is necessary, it is recommended that
the property not be enveloped in one of these elements. In general, a
property should be enveloped in a category element only if it is
expected that the property will need to be attributed to more than one
property category, or if it needs to make use of some of the special
attributes inherited from the category.

## Requested Element

A requested property reflects properties as they were requested. A
disparity might occur between the requested value and the values
utilized or dedicated if a preference was expressed, if multiple options
were specified, or if ranges or pattern matching was specified.

-   A compliant implementation SHOULD support this element.

The following illustrates the syntax of this element:

\<Requested\>

\<!\-- Requested Properties \--\>+

\</Requested\>

The following describes the attributes and elements for the example
above:

/Requested

This element is used to encapsulate requested properties.

/Requested/\<Requested Property\>

Requested properties appear at this level.

Requested Properties inherit some additional attributes.

-   A requested property MAY have an op attribute of type String that
    indicates a conditional operation on the value. A compliant
    implementation SHOULD support this attribute. Valid values for the
    op attribute include "eq" meaning equals (which is the default),
    "ne" meaning not equal, "lt" meaning less than, "gt" meaning greater
    than, "le" meaning less than or equal to, "ge" meaning greater than
    or equal to, "match" which implies the value is a pattern to be
    matched.

-   A requested property MAY have a conj attribute of type String that
    indicates a conjunctive relationship with the previous element. A
    compliant implementation MAY support this attribute. Valid values
    for the conj attribute include "and" (which is the default), "or",
    "nand" meaning and not, and "nor" meaning or not.

-   A requested property MAY have a group attribute of type Integer that
    indicates expression grouping and operator precedence much like
    parenthetical groupings. A compliant implementation MAY support this
    attribute. A positive grouping indicates the number of nested
    expressions being opened with the property while a negative grouping
    indicates the number of nested expressions being closed with the
    property.

-   A requested property MAY have a preference attribute of type Integer
    that indicates a preference for the property along with a weight
    (the weight are taken as a ratio to the sum of all weights in the
    same group). A compliant implementation MAY support this attribute.
    If a group of positive valued preference alternatives are specified,
    at least one of the preferences must be satisfied for the job to
    run. If a group of negative valued preferences are specified, the
    preferences will try to be met according to their weights but the
    step will still run even if it can't satisfy any of the preferred
    properties. (Weight ranking can be removed by making all weights the
    same value (1 or -1 for example).

The following are four examples of using Requested Properties:

\<Requested\>

\<Processors op="ge"\>8\</Processors\>

\<Processors op="le"\>16\</Processors\>

\<WallDuration\>3600\</WallDuration\>

\</Reqested\>

\<Requested\>

\<NodeCount\>1\</NodeCount\>

\<NodeProperties\>

\<Name op="match"\>fr15.\*\</Name\>

\</NodeProperties\>

\<Requested\>

\<Requested\>

\<UserId group="1"\>scottmo\</UserId\>

\<AccountName group="-1"\>mscfops\</AccountName\>

\<UserId conj="or" group="1"\>monkeyboy\</UserId\>

\<AccountName group="-1"\>junglehunt\</AccountName\>

\</Requested\>

\<Requested\>

\<Memory preference="2"\>1024\</Memory\>

\<Memory preference="1"\>512\</Memory\>

\</Requested\>

## Utilized Element

A utilized property reflects properties as they were utilized, realized
or consumed. It reflects the actual amounts or values that are used or
currently allocated, as opposed to a limit, choice or pattern as may be
the case with a requested property, or a dedicated amount that prevents
sharing by other work requests.

-   A compliant implementation SHOULD support this element.

The following illustrates the syntax of this element:

\<Utilized\>

\<!\-- Utilized Properties \--\>+

\</Utilized\>

The following describes the attributes and elements for the example
above:

/Utilized

This element is used to encapsulate utilized properties.

/Utilized/\<Utilized Property\>

Utilized properties appear at this level.

Utilized Properties inherit some additional attributes.

-   A requested property MAY have a group attribute of type Integer that
    indicates expression grouping and operator precedence much like
    parenthetical groupings. A compliant implementation MAY support this
    attribute. A positive grouping indicates the number of nested
    expressions being opened with the property while a negative grouping
    indicates the number of nested expressions being closed with the
    property. The purpose of this attribute would be to logically group
    utilized properties if they were used in certain aggregations (like
    a job that spanned machines).

The following are the same four examples distinguishing the utilized
amounts and values:

\<Utilized\>

\<Processors\>12\</Processors\>

\<WallDuration\>1234\</WallDuration\>

\</Utilized\>

\<Utilized\>

\<NodeList\>

\<Node\>fr15n03\</Node\>

\</NodeList\>

\<Utilized\>

\<Utilized\>

\<UserId\>scottmo\</UserId\>

\<AccountName\>mscfops\</AccountName\>

\</Utilized\>

\<Utilized\>

\<Memory\>1024\</Memory\>

\</Utilized\>

## Dedicated Element

A dedicated property refers to a resource dedicated during the step. For
example, a step may request to restrict the usage of certain resources
or the entire node thereby preventing it from being shared by other work
requests even though the step only needs a portion of the resources to
accomplish its work. This may be to avoid conflict introduced when the
resource is shared, for confidentiality or any other reason. In such a
case, the amount of a resource requested and the amount actually
utilized may differ from the amount that was blocked. The dedicated
amount might be used for charge purposes, the utilized amount for usage
accounting, and requested amount for task assignment.

-   A compliant implementation MAY support this element.

The following illustrates the syntax of this element:

\<Dedicated\>

\<!\-- Dedicated Properties \--\>+

\</Dedicated\>

The following describes the attributes and elements for the example
above:

/Dedicated

This element is used to encapsulate dedicated properties.

/Dedicated/\<Dedicated Property\>

Dedicated properties appear at this level.

Dedicated Properties inherit some additional attributes.

-   A requested property MAY have a group attribute of type Integer that
    indicates expression grouping and operator precedence much like
    parenthetical groupings. A compliant implementation MAY support this
    attribute. A positive grouping indicates the number of nested
    expressions being opened with the property while a negative grouping
    indicates the number of nested expressions being closed with the
    property. The purpose of this attribute would be to logically group
    utilized properties if they were used in certain aggregations (like
    a job that spanned machines).

The following is an example of specifying dedicated properties:

\<Dedicated\>

\<Processors\>16\</Processors\>

\<Resource name="NetworkAdapter" type="ELAN4"\>2\</Resource\>

\</Dedicated\>

# AwarenessPolicy Attribute

A word or two should be said about compatibility mechanisms. With all
the leeway in the specification allowing implementers to choose to
implement or not implement various portions of the specification,
problems might arise if an implementation simply ignores a portion of a
job specification that is critical to the job function in certain
contexts, so we might want to enforce a policy in which a job is
rejected if any element or attribute contained in it was not understood
by the implementation. On the other hand, in many cases jobs will want
to interpret them the best they can and have the components implement a
best-effort approach to operating on the portions they understand.
Consequently, we define an awarenessPolicy attribute which can be added
as an optional attribute to the Job element or any other containment or
property element to indicate how the property (or the default action for
the elements that the containment element encloses) must react when the
implementation does not understand an element or attribute.

An awareness policy of "Reject" will cause the server to return a
failure if it receives a client request in which it does not support or
understand the associated element name or attribute name or value.

An awareness policy of "Warn" will accept the misunderstood element or
attribute and continue to process the job object on a best effort basis.
However a warning MUST be sent (if possible) to the requestor
enumerating the elements and attributes that are not understood.

An awareness policy of "Ignore" will accept the misunderstood element or
attribute and continue to process the job object on a best effort basis.
The action could be to simply ignore the attribute.

-   This name of this attribute is awarenessPolicy.

-   This attribute is of type String.

-   This attribute can have values of "Reject", "Warn" or "Ignore".

-   A compliant implementation MAY support this attribute.

-   If an implementation does not support the attribute, it MUST adopt
    the policy of rejecting any job object which contains elements or
    attributes that it does not support. Furthermore, it SHOULD return a
    message to the requestor with an indication of the element or
    attribute name it did not understand.

-   This attribute MAY be present in a property or containment element.

-   If an implementation does support the attribute, but it is absent,
    the default value of "Reject" is implied.

-   Individual elements within the job object may override the
    containing object's awareness policy default by including this
    attribute. For example, a job might specify an awarenessPolicy of
    "Reject" at its root (the Job element) but may want to allow a
    particular subset of elements or attributes to be ignored if not
    understood. Conversely, a job with a default awarenessPolicy of
    "Ignore" might want to classify a subset of its optional elements as
    "Reject" if they are indispensible to its correct interpretation. An
    implementation can opt to check or not check for this attribute at
    any level it wants but must assume a "Reject" policy for any
    elements it does not check.