# Presentation: 935868

## Scheduling & Resource ManagementGGF3 Area

- co-directors: Jennifer Schopf, Argonne National Lab & Bill Nitzberg, PBS/Veridian Systems
- www.cs.nwu.edu/~jms/sched-wg/
- sched-wg@gridforum.org
- Asia-Pacific Grid Workshop
- Tokyo, Japan
- 23 Oct 2001

## Scheduling on the Grid

**PBS**

**LSF**

**Globus **
**(over anything)**

**Run My Job**

## High-level Overview:“Solve Grid Resource Management”

- Who? – Developers, users
- What? -- Agreements / standards
- Capabilities, general protocols, APIs
- Why? -- Interoperability
- Reserving, allocating, & using resources
- Managing resources (owner’s pt-of-view)
- Support co-scheduling diverse resources
- Enable "better" "use of" resources

## Scheduling AreaObjectives & Progress

- “Look at what is done today, gather requirements...”
- “...refining protocols, interactions, etc.”
- “...work to standardize APIs and protocols”

## Active Groups

- Overview
- “10 Actions for Superscheduling” [in review]
- Standardizing “Run my job”
- “Advance Reservation API” [in review]
- BOF: Scheduling Command Line Interface WG
- BOF: DRM Application API WG
- WG: Grid Resource Management Protocol
- Language:
- WG: Scheduler Attributes
- WG: Scheduling Dictionary
- Other
- BOF: Scheduling Optimization RG

## Ten Actions for Super Scheduling

- Goal: List the basic steps and capabilities involved in resource reservation, acquisition, and usage
- Express core of current practices
- User’s point of view
- “How do people schedule over the grid today?”

## Super-scheduler Actions

**Phase 1: Resource Discovery**
- Step 1 Authorization Filtering
- Step 2 Application requirement definition
- Step 3 Minimal requirement filtering
**Phase 2 System Selection**
- Step 4 Gathering information (query)
- Step 5 Select the system(s) to run on
**Phase 3 Run job**
- Step 6 (optional) Make an advance reservation
- Step 7 Submit job to resources
- Step 8 Preparation Tasks
- Step 9 Monitor progress (maybe go back to 4)
- Step 10 Find out J is done
- Step 11 Completion tasks

## Current DocumentSchedWD 8.5

- Has been through 30 day internal review
- Will be going through document review process

## Active Groups

- Overview
- “10 Actions for Superscheduling” [in review]
- Standardizing “Run my job”
- “Advance Reservation API” [in review]
- BOF: Scheduling Command Line Interface WG
- BOF: DRM Application API WG
- WG: Grid Resource Management Protocol
- Language:
- Scheduler Attributes WG
- WG: Scheduling Dictionary
- Other
- BOF: Scheduling Optimization RG

## Run a Job (Steps from:  Ten Actions when SuperScheduling”, GGF SchedWD 8.5, J.M. Schopf, July 2001)

**Phase 1: Resource Discovery**
- Step 1 Authorization Filtering
- Step 2 Application requirement definition
- Step 3 Minimal requirement filtering
**Phase 2 System Selection**
- Step 4 Gathering information (query)
- Step 5 Select the system(s) to run on
**Phase 3 Run job**
- Step 6 (optional) Make an advance reservation
- Step 7 Submit job to resources
- Step 8 Preparation Tasks
- Step 9 Monitor progress (maybe go back to 4)
- Step 10 Find out J is done
- Step 11 Completion tasks

## Different Levels of Standards

- Command Line standards
- Script re-use
- API (Application Programmer Interface)
- Code re-use
- What the programmer sees
- Protocol
- Interoperability
- What the network sees
- Language/Syntax
- Re-use & interoperability

## A Protocol can have Multiple APIsE.g., TCP/IP

- TCP/IP APIs include BSD sockets, Winsock, System V streams, ...
- The protocol provides _interoperability_: programs using different APIs can exchange information
- I don’t need to know remote user’s API

**TCP/IP Protocol: Reliable byte streams**

**WinSock API**

**Berkeley Sockets API**

**Application**

**Application**

*[Slide courtesy of Globus Tutorial --  www.globus.org]*

## An API can have Multiple ProtocolsE.g., Message Passing Interface

- MPI provides portability: any correct program compiles & runs on a platform
- Does not provide interoperability: all processes must link against same SDK
- E.g., MPICH and LAM versions of MPI

*[Slide courtesy of Globus Tutorial – www.globus.org]*

## Advance Reservation API 

- “Advance Reservation & Co-Scheduling Workshop”, May 1999 (Pre-Grid Forum)
- Defined “reservation”:
- Resource + start + end + duration
- Enumerated desired capabilities
- de-coupled from job submission
- unique “printable” reservation ID
- query/response - returns list of available slots
- “hard” and “soft” reservations
- Enumerated harder stuff to put off til later, e.g., guarantee, cost model

## Advance Reservation API

- Defining an API to be used by local resource management systems
- API because we weren’t ready to define the protocol, but there was a fair amount of experience with APIs
- Volker Sander, Alain Roy led effort

## Reservation definition

*Start Time:*
- The earliest time  a reservation may begin.
*Duration:*
- How long the reservation last
*Resource Type:*
- The type of underlying resource (network, a computer, or a disk)
*Reservation Type*:
-  A particular kind of reservation  (esp. networking)
*Resource-Specific Parameters:*
- Parameters that are unique to each type of resource
- eg. bandwidth for a network reservation

## Defined APIs for

- Create a reservation
- Modify a reservation
- Query a reservation
- Commit a reservation
- Bind a reservation
- Revoke a reservation

## Creating a Reservation

- grid_reservation_create( reservation_manager_contact, &auth_info, request_rsl, time_rsl, &reservation_handle);
- request_rsl = “&(resource-type=compute) (number-of-nodes=14)”;
- time_rsl = “&(start-time=953158862) (duration=3600)”
- resource_manager_contact = "pitcairn.mcs.anl.gov:2119:/O=Grid/“ "O=Globus/CN=pitcairn.mcs.anl.gov"

## Modifying a Reservations

- Similar to creating a reservation, except that instead of providing a reservation manager contact, the application provides the handle to the reservation that was created earlier

## Querying a reservation

- To find out the status of a reservation, it can  be queried
- grid_reservation_status(&auth_info, reservation_handle, &status);

## Committing a reservation

- When a reservation is a two-phase commit reservation it needs to be committed before the reservation times out.
- Time-out period can be specified when the reservation is made, but it defaults to an implementation-specified time
- grid_reservation_commit(&auth_info, reservation_handle);

## Binding a reservation

- When the application ready to use a reservation, it may need to *bind* it in order to begin using the reservation.
- Binding a reservation is a required step if the creation was done without providing all required attributes to instantiate it.
- EG,  binds a process-id to a given reservation:
- grid_reservation_bind(&auth_info, reservation_handle,&bind_parameters);

## Advance Reservation API status

- Current Document SchedWD 9.4
- Has been through 30 day internal review
- Will be going through document review process

## “Run My Job” Efforts

- Command Line standards - BOF at GGF3
- Script re-use
- API (Application Programmer Interface) - BOF at GGF3
- Code re-use
- What the programmer sees
- Protocol - WG in progress
- Interoperability
- What the network sees

## BOF: Scheduling Command Line Interface

- Command line job submissions are different on every system
- Joe Werne’s qprep work as a starting point
- DoD
- Translator between existing queuing systems
- www.pstoolkit.org
- Posix defined this - but posix didn’t go far enough (eg. SMPs)

## The Problem

- Site A

- Site B

- Code A

- Code B

- Code C

- Site C

## The Solution

- Site A

- Site B

- Site C

- Archive is a
- Routine that
- Tr
- anslates from
- Your code to
- The native
-  syntax
- And semantics,
- Because archive
- Figures that
- You have better
- Things to do with
- Your tianslates from
- Your code to
- The native
-  syntax
- And semantics, to do with
- Your tianslates from
- Your code to
- The native
-  syntax
- And semantics,
- me ... like
- Sc
- me ... like
- Sc
- Ie
- nce!
- Because archive
- Figures that
- You have better
- Things to do with
- is a
- Routine that
- Tr
- anslates from
- Your code to
- The native
-  syntax
- And semantics,
- Because archive
- Figures that
- You have better
- Things to do with

## Current qprep

- Works in one of two ways, both of which are intended to be familiar to users:
- 1. Command-line arguments:
**qprep nodes=512 walltime=24:00 ... script**
- 2. Pseudo-comment directives in script preamble:
**	****#PSTQ nodes=512**
**	****#PSTQ walltime=24:00**
- qprep will edit the script, translating the preamble to the native queuing system
- To the extent possible, our plan is to implement qprep via a “translation table”

## Next Step

- Write charter
- Expand core group of people

## BOF: Distributed Resource Management Application API

- John Tollefsrud [john.tollefsrud@eng.sun.com], Sun
- Bill Nitzberg [bill@computer.org], Veridian
- Proposed charter
- ...standard API for “run my job”...
- Base API on existing work from existing DRM systems (Intel, Sun, Veridian, Globus, ...)
- Timeframe: ~9 months

## Goals

- Want to develop an API specification for the submission and control of jobs to one or more resource management systems
- High level functionality - start job, termination, suspension
- Standardize existing systems
- Don’t invent something new

## Next Steps...

- Solicit participation
- Especially resource management vendors & application developers
- Finalize charter, milestones, ...
- Discuss via sched-wg email list
- Submit to GGF chair to form DRMAA working group
- Timing: API defined in ~9 months
- E.g., Jul’02: DRMAA v1.0 GWD submitted for review

## Grid RM Protocol: Requirements

- Child of “Advance Reservation API” work
- Charter expanded from “Advance Reservation” to “Resource Management”
- GGF-2 consensus: you basically have to do most of a general RM protocol to do Advance Reservations anyway...
- Charter focused to start with “Requirements”

## Grid RMPerhaps a Document Series

- Grid RM Protocol: Requirements
- Grid RM Protocol: Operations
- Capabilities and semantics
- Leverage Other Grid Services Standards
- Transport services
- Security
- Language
- Grid RM Protocol: Bindings

## RequirementsSchedWD 12.1

- Control
- Extensibility
- Notification
- Reliability
- Protocol timers
- Negotiation
- Hierarchy
- Secure messaging
- Security language
- Resource language

## Grid Resource Management Protocol WG

- Presented revised document (SchedWD 12.1) focused on requirements
- Good feedback and discussion on the overall space of issues
- Next steps:
- Better grouping: capabilities vs. requirements of the protocol
- Include both requirements and non-requirements
- Identify “low hanging fruit” capabilities and draft protocols for them
- Perhaps a name change (which is more specific)
- As the protocol is not intended for administrative management of resource managers (e.g., defining policy)

- Authors have been asked to restructure doc
- Qualities of the protocol
- Capabilities that the protocol should support
- Stuff we should work on right away, stuff that can wait a bit....

## Next Steps

- Another draft, additional authors(?)
- Sched-wg@gridforum.org for comments
- New version ~2 weeks, telecon shortly thereafter (we hope)
- 1) Which of these are required vs suggested? Minimal set?
- 2) What about use cases or examples?

## Active Groups

- Overview
- “10 Actions for Superscheduling” [in review]
- Standardizing “Run my job”
- “Advance Reservation API” [in review]
- BOF: Scheduling Command Line Interface WG
- BOF: DRM Application API WG
- WG: Grid Resource Management Protocol
- Language:
- Scheduler Attributes WG
- WG: Scheduling Dictionary
- Other
- BOF: Scheduling Optimization RG

## Attributes for Communication between Scheduling Instances

- Led by Uwe Schwiegelshohn
- List some important features of lower level schedulers
- The functionality offered to a higher level scheduling instance may depend on
- the type of the scheduling system,
- the implementation of the system and
- the applied policy
- (There may be more than 2 scheduling instances)

## What functionality can be used by a higher level scheduling instance?

- This functionality is described by 4 classes of *attributes*
- properties of the allocation,
- revocable, guaranteed completion time, run-to-completion, exclusive, malleable
- available information,
- scheduling information, exclusive control, event notification
- manipulation of the allocation
- preemption, migration, checkpointing
- the request of resources
- offer support, cost/objective information, advanced reservation, allocation length, deallocation policy, remote co-scheduling, job dependencies

## Next Steps

- This document enters formal document process after minor revisions
- Then:
- Architecture that makes use of this
- API (maybe protocol) that uses this
- Identify what existing schedulers provide
- Will need a prototype to know if this is useful

## Scheduling Dictionary

- Led by Mary Roehrig, Wolfgang Ziegler
- What’s the point
- If a user wants “2 CPUs” there is an agreed on way (between schedulers) that this is defined
- Interoperability
- Define a grid language, say how it converts to your system’s language
- Still trying to get an overall set of terms, next step prioritizing and defining

## Active Groups

- Overview
- “10 Actions for Superscheduling” [in review]
- Standardizing “Run my job”
- “Advance Reservation API” [in review]
- BOF: Scheduling Command Line Interface WG
- BOF: DRM Application API WG
- WG: Grid Resource Management Protocol
- Language:
- Scheduler Attributes WG
- WG: Scheduling Dictionary
- Other
- BOF: Scheduling Optimization RG

## Optimization(Steps from:  Ten Actions when SuperScheduling”, GGF SchedWD 8.5, J.M. Schopf, July 2001)

**Phase 1: Resource Discovery**
- Step 1 Authorization Filtering
- Step 2 Application requirement definition
- Step 3 Minimal requirement filtering
**Phase 2 System Selection**
- Step 4 Gathering information (query)
- Step 5 Select the system(s) to run on
**Phase 3 Run job**
- Step 6 (optional) Make an advance reservation
- Step 7 Submit job to resources
- Step 8 Preparation Tasks
- Step 9 Monitor progress (maybe go back to 4)
- Step 10 Find out J is done
- Step 11 Completion tasks

## Scheduling Optimization RG

- Led by Vincenzo DiMartino and Marco Mililotti
- Enable "better" "use of" resources
- Research group - long term, investigative, not “standards” track

## What is Optimization?

- “The art of running as much job as possible with the minimum usage of resources”
- “How to avoid resources request conflict between Jobs”
- “How to maximize the GRID computing total troughput without to penalize the single job/user”.
- “To negotiate different cost/performance to “predictable” users”
- “To keep at minimum the Geographical area network  load”

## Possible focus includes:

- Taxonomy of scheduling approaches
- Taxonomy of optimization constraints
- How should constraints be described?
- List of capabilities that are required to do good optimization
- What do you assume is being furnished by the users?
- What information is needed by the optimization tech is being used
- What meta-data is supplied by the resources (nodes,hw,nw)  in a grid?
- How are jobs being described? How should they be described in the future

## Next Steps

- Define a charter: Identify which paper topics should be looked at first
- Issues
- Metrics/constraints for better
- Algorithms
- Information gathering
- Long term goal: better optimization for real schedulers

## Active Groups

- Overview
- “10 Actions for Superscheduling” [in review]
- Standardizing “Run my job”
- “Advance Reservation API” [in review]
- BOF: Scheduling Command Line Interface WG
- BOF: DRM Application API WG
- WG: Grid Resource Management Protocol
- Language:
- Scheduler Attributes WG
- WG: Scheduling Dictionary
- Other
- BOF: Scheduling Optimization RG

## How to Participate

- Our web page:
- www.cs.nwu.edu/~jms/sched-wg/
- Our mailing list:
- sched-wg@gridforum.org
- Co-leads for area:
- Jennifer Schopf, jms@mcs.anl.gov
- Bill Nitzberg, bill@computer.org