# Presentation: 001017

## Scheduling & Resource ManagementWorking Group

- Jennifer Schopf & Bill Nitzberg, co-chairs
- www.cs.nwu.edu/~jms/sched-wg/
- sched-wg@gridforum.org
- Grid Forum 5 Meeting
- Boston, MA
- Oct 15 - 18, 2000

## Scheduling Working GroupObjectives & Progress

## High-level Overview:“Solve Grid Resource Management”

- Who? -- Developers
- What? -- Agreements / standards
- Capabilities, general protocols, APIs
- Why? -- Interoperability
- Reserving, allocating, & using resources
- Managing resources (owner’s pt-of-view)
- Support co-scheduling diverse resources
- Enable "better" "use of" resources

## Charter

- “Look at what is done today, gather requirements...”
- “...refining protocols, interactions, etc.”
- “...work to standardize APIs”
- Current Focus Areas
- Advance reservations
- Super scheduling
- Token definition
- “information”: proper nouns, semantics, representation

## Progress

- _Grid Query and Reservation Interface_, SchedWD2.2, Q. Snell, D. Jackson, M. Clement
- _Generic Grid Resource Description_, SchedWD9, K. Czajkowski, G. von Laszewski, J. Gehring
- _Security Requirements of the Scheduling Working Group_, SchedWD6.9, K. Jackson & V. Sander
- _Ten Steps for Superscheduling_, SchedWD8, J. Schopf, Q. Snell, K. Czajkowski
_*(Advance Reservation API*_*, SchedWD 4, 5, & 7, A. Roy, V. Sander, Q. Snell, J. Gehring)*

## Scheduling Working GroupA Brief History

## “Advance Reservation & Co-Scheduling Workshop”, May 1999

- Defined “reservation”:
- Resource + start + end + duration
- Enumerated desired capabilities
- de-coupled from job submission
- unique “printable” reservation ID
- query/response - returns list of available slots
- “hard” and “soft” reservations
- Enumerated harder stuff to put off til later, e.g., guarantee, cost model

## Grid Forum 1 (NASA Ames)June 1999

- Initial Charter:
- “Solve Grid Resource Management”
- Three focus areas:
- Advance reservations
- Super scheduling
- Resource specification (semantics & tokens)

## Grid Forum 2 (Northwestern)October 1999

- Refined charter
- Requested:
- lists of “tokens” from different groups
- architecture pictures of existing systems
- Discussed “What is X?”
- e.g., “job”, “scheduler”

## Grid Forum 3 (UCSD)March 2000

- Adopted charter & “refocused”
- Decided _not_ to work on architecture
- Developed “Super-scheduler Model” (10 steps)
- Gave overviews of advance reservation systems (GARA, Maui, PBS, LSF)
- Commitments to draft several SchedRFCs

## Grid Forum 4 (Microsoft)July 2000

- Changed “Sched RFC” to “Sched Working Document”
- Revised working document drafts:
- Query Interface
- Resource Acquisition Steps
- “Security Requirements”
- “Advance Reservation API”
- Scheduling Information
- Suggested new working documents:
- “10 Steps” & “Run a job” API

## Grid Forum 5: AgendaOctober 2000

- Now: Working Group Update
- Mon 1300-1500: _Generic Grid Resource Description_ & _Ten Steps for Superscheduling_
- Mon 1500-1630: _Security Requirements of the Scheduling Working Group_
- Mon 1630-1800: _Grid Query and Reservation Interface_
- Tue 930-1100: Commercial Grid Systems Track
- Tue 1500-1630: Vendors, Standards, and Low Hanging Fruit

## Process

- Review working document (< 20 minutes)
- Focus on understanding the document (revisions) rather than “correcting” it
- Gather discussion items during/after review
- Prioritize & discuss each item
- Next steps (last 5 minutes)
- Revised documents posted to web site within 2 weeks.

## Generic Grid Resource Description

- Goal: Standardize scheduling information -- the “tokens” or ”labels”, semantics, and their representation

## Ten Steps for Superscheduling 

- Goal: List the basic steps and capabilities involved in resource reservation, acquisition, and use

## Security Requirements of the Scheduling Working Group

- Goal: Ensure interoperability between GF security standards and GF scheduling standards

## Grid Query Interface

- Goal: Refine interface to a scheduler to answer the question(s):
- When will my job start?
- What times can a reservation be guaranteed?

## Advance Reservation API

- Goal: API for advance reservations
- “Merge” existing interfaces...
- CCS
- GARA
- Legion
- Maui
- PBS

## Additional Sessions

- Commercial Grid Systems Track
- Vendors, Standards, and Low Hanging Fruit
- There are at least 3 “standards” efforts
- Are we working on the right “standards”?
- Coordinate / merge / move efforts

## Questions?

- Possible change:  1:00p - 3:00p
- Primary authors for “Resource Description” & “10 Steps” are unavailable...

## Generic Grid Resource Description

- Notes

## Generic Grid Resource Description

- Goals of this document:
- 1. Want same structured language for allocation & reservation requests
- 2. Point is not the syntax, but the capabilities that the syntax should support
- What could “we” talk about now?
- Is this the right general direction?
- List of resources?
- List of capabilities that the syntax must support?
- What is the target of the language?  Who is communicating?
- Sender (requester) is seeking resources from the receiver (provider); provider allocates/reserves resources
- Publisher/subscriber of information about resources
- Declarative vs. Procedural?
- No clear concensus (or agreement on definitions)

## GGRD Notes

- Requirements of language -- Ability to express
- constraints in users requests & in the resource base
- Fault domain
- Workflow
- May be overloading language by requiring the distinction between description and instantiation
- What about GIS GOS?
- An entity-relationship model might be better suited to our needs
- Place constraints on each entity and relation
- Need to deal with hard constraints, soft constraints, and priorities (or ranking)
- Should we be looking at UML?
- In literature for specifying automatic testing, there are formalisms for describing time constraints.

## GGRD Notes

- We are quickly increasing the domain of this “language”...
- What should the scope be?
- Perhaps the language should capture everything from the rough cut through the final “allocation” (and every step along the way)
- May want to focus on a smaller scope at first
- Proposal: have document reflect the smallest scope possible to stimulate/focus discussion for next time.
- What needs to be done with this documents:
- More clearly define the target(s) of this language:
- Focus this document on, and describe more explicitly, the requirements of the language (or the capabilities that the language must support)
- Then move onto language and syntax

## Generic Grid Resource DescriptionK. Czajkowski, (G. von Laszewski), (J. Gehring)

- Goal: Standardize scheduling information -- the “tokens” or ”labels”, semantics, and their representation
- Confusion over focus of the document
- Meant to list capabilities that the language should support (not the syntax)
- Next Steps:
- Refocus on requirements/capabilities of language, then move onto language & syntax

## Ten Steps for Superscheduling

- Notes

## Ten Steps for SuperschedulingJ. Schopf, (Q. Snell), (K. Czajkowski)

- Goal: List the basic steps and capabilities involved in resource reservation, acquisition, and use
- No major issues raised
- two minor editorial changes proposed
- Next Steps:
- Expand on last section
- Should be ready for presentation as proposed GF Document at the next GF

## Security Requirements of the Scheduling Working Group

- Notes

## Security Notes

- Suggestion: make a list of security requirement for scheduling group, and then “hand it over to the security group”
- Should this doc. be merged into the overall Security Requirements document?
- Request for a recommendation from the GF Security group as to which security stuff to use (e.g., GSI?)
- maybe a “best practices”
- Missing from the document?
- need to authenticate a user in a specific way (e.g., user has access because they have a contract with us) -- need mechanisms to handle legacy policies

## Security Notes

- Perhaps we need to begin policy discussions among different agencies (countries, etc.) to permit Grid computing (from the security perspective)
- Isn’t this happening in the CA discussions?
- Plus, there’s not really a need right now for DoE to interoperate with DoD
- Separate out issues (policy & mechanism) and focus on “the information needed” to do X
- (General concensus is that there are really only two choices for authentication: PKI and Kerberos)
- Goals of this document: bring out issues in scheduling to the security group (and vice versa).
- Proposal that we should re-visit scheduling/security from time to time

## Security Requirements of the Scheduling Working GroupK. Jackson & V. Sander 

- Goal: Ensure interoperability between GF security standards and GF scheduling standards
- No issues raised
- Next steps:
- To be merged into the Scenarios Security document
- Vauge non-commital commitment to look into a best practices paper

## Grid Query Interface

- Notes

## Grid Query Interface Q. Snell, (D. Jackson), M. Clement

- Goal: Refine interface to a scheduler to answer:
- When will my job start?
- What times can a reservation be guaranteed?
- No major issues raised; proposed APIs:
- QUERY UserID Duration ResourceList OtherInfo
- RESQUERY UserID TimeSpec ResourceList OtherInfo
- Next steps:
- Bind generic API to C
- Proposals expected from D. Jackson, P. Foley, & B. Nitzberg
- Proposed APIs after QUERY
- Reservation API (in progress)
- Binding a job to a reservation, cancellation, monitoring*

## Monitoring Discussion(focus on active (cancellable) jobs)

- Requires both query & event (publish/subscribe) model
- Failure semantics / recovery
- If the metascheduler/scheduler dies, should jobs still get events?
- Random set of stuff we’re interested in:
- API for a job to be able to “post” it’s job-specific state
- Get state of all jobs with “tag X”
- Get the original request for job J
- Get the list of allocated resources for job J
- Get environment for job J
- Get “info” on staging files
- Set thresholds and get notifications when thresholds are exceeded
- Register interest in events (publish/subscribe)
- Proposal expected from:  anand@virginia.edu
- should coordinate w/Performance Group documents

## Vendors, Standards, and Low Hanging Fruit

- Notes

## Vendors, Standards, and Low Hanging Fruit

- This group has been around for ~1 year
- Two questions:
- Are we focused on the “right stuff”
- What about the other standards efforts

## Current Effortse.g., people are expending time on these

- _Query_, API, Q. Snell, D. Jackson, M. Clement
- _Generic Grid Resource Description_, Syntax/Language, K. Czajkowski, G. von Laszewski, J. Gehring
- _Ten Steps_, Architecture/Framework, J. Schopf, Q. Snell, K. Czajkowski
- _Advance Reservation_, API, A. Roy, V. Sander, Q. Snell, J. Gehring

## Is There Consensus to Work on Other Stuff?

- Advance Reservation protocol?
- API was easier to agree on among groups writing advance reservation systems --- protocols were thought to be harder (to agree on)
- Tokens? (Uwe.Schwiegelshohn@udo.edu & haupt@erc.msstate.edu)
- Will the GGRD document include these?
- Scheduling Landscape document (P. Foley)
- Describe the different entries in the scheduling arena; “marketing” document representing all the different schedulers in “this room” -- framework on which to test scenarios

## What About theOther Standards Efforts

- Currently...
- Grid Forum
- Intel Peer-to-Peer Working Group (PtPWG)
- New Productivity Initiative (NPI)
- Broadband (?)

## Progress

- Teleconferences (every other week)
- Wed - Noon East Coast Time (9am Pacific, 6pm in Germany)
- Documents should be distributed <= 1 week before
- Nov 15:  _Ten Steps_, Architecture/Framework, J. Schopf, Q. Snell, K. Czajkowski
- Nov 29: _Advance Reservation_, API, A. Roy, V. Sander, Q. Snell, J. Gehring
- Dec 13:  _Generic Grid Resource Description_, Syntax/Language, K. Czajkowski, G. von Laszewski, J. Gehring
- Jan 10:  _Query_, API, Q. Snell, D. Jackson, M. Clement