# Presentation: 243217

## Resource Management Working Group

- SSS Quarterly Meeting
- November 28, 2001
- Dallas, Tx

## Resource Management and Accounting Working Group

- Working group scope and components
- Progress made
- Current and future issues
- Next steps

## Working Group Scope

- The Resource Management Working Group encompasses the areas of resource management, scheduling and accounting.
- This working group will focus on the following software components:
- Queue Manager
- Scheduler
- Allocation Manager
- Meta Scheduler
- Our charter will also encompass the following capabilities:
- Accounting
- Usage Reports

## Phase 1 Milestones

- 6 months:	Contribute to checkpoint/restart report with regard to scheduling related aspects
- 12 months: Establish and release initial resource management interface specifications
- 12 months: Establishment of the CVS repository and module structure, agreement on document conventions
- 12 months: Finalized API for system initiated checkpoint/restart of parallel MPI jobs on Linux systems
- 18 months: Release v1.0 of the Center’s resource management system based on existing open source code and the results of the scalability testing.

## High Level Progress

- Establishing high level design covering initial component functionality and required interfaces
- Determining inter-group requirements (GUI, security, IS, process management, etc)
- Preparing existing tools (Maui, Silver, QBank) for use within SSS
- Creating infrastructure within which to develop and test RM deliverables
- Creating infrastructure within which to develop and test intra- and inter-group interfaces

## Proposed Component Architecture

- Queue
- Manager

- Allocation
- Manager

- Collector

- Meta
- Scheduler

- Scheduler

- Node
- Manager

- Process
- Manager

- Security
- System

- Information
- Service

- Discovery
- Service

| Color Key | Working Group |
| --- | --- |
|  | Resource Management and Accounting |
|  | Execution Management and Monitoring |
|  | Node Config and Infrastructure |

## Component Interaction DiagramJob submitted to Queue Manager

- User
- Interface

- Collector

- Meta
- Scheduler

- Queue Manager

- Allocation Manager

- Scheduler

- Process
- Manager

- 2

- 1

- 3

- 4

- 6

- 5

- 7

- 9

- 8

- 10

- 11

## Component Interaction TraceJob submitted to Queue Manager

- A user submits a job to the Queue Manager
- The Queue Manager does a sanity balance check with the Bank
- The Queue Manager notifies the Scheduler that a new job has arrived
- The Scheduler queries node and job status until job can run
- A bank reservation is made with the Allocation Manager
- The Scheduler requests the Queue Manager to run the job
- The Queue Manager passes job control to the Process Manager
- The Process Manager notifies Queue Manager of job completion
- The Queue Manager notifies Scheduler of job completion
- A bank withdrawal is made with the Allocation Manager
- The user is notified of job completion

## Component Interaction Diagram Job submitted to Meta Scheduler

- User
- Interface

- Collector

- Meta
- Scheduler

- Queue Manager

- Allocation Manager

- Scheduler

- Process
- Manager

- 2

- 1

- 3

- 4

- 6

- 5

- 8

- 7

- 10

- 9

- 11

- 13

- 12

- 14

- 15

## Component Interaction TraceJob submitted to Meta Scheduler

- A user submits a job to the Meta Scheduler
- The Meta Scheduler contacts Schedulers to determine which systems could run the job the soonest
- The Schedulers request quotes from Allocation Banks to determine which systems would run the job for the lowest cost
- A Scheduler reservation is created for the job on the resource providing the best service -- this reservation can be moved or improved upon until the job is staged
- The job is staged and queued at the system where it is to run
- The Queue Manager notifies the Scheduler that a new job has arrived
- The Scheduler queries node and job status until job can run
- A bank reservation is made with the Allocation Manager
- The Scheduler requests the Queue Manager to run the job
- The Queue Manager passes job control to the Process Manager
- The Process Manager notifies Queue Manager of job completion
- The Queue Manager notifies Scheduler of job completion
- A bank withdrawal is made with the Allocation Manager
- The Scheduler notifies the Meta Scheduler of job completion
- The user is notified of job completion

## Design/Interface Progress

- Initial high level RMS architecture defined
- Resource management dictionary created defining objects within resource management ‘world’
- Object ‘tokens’ declared for major objects
- Component functional interfaces identified
- Initial XML request/response syntax proposed
- Prototypes being constructed to test communication protocols
- Initial detailed extra-group component requirements document created

## Local Scheduler Rationale

- Local interfaces with majority of inter and intra RM components
- Establish test platform from which interfaces can be tested
- Leverage existing capabilities to accelerate SSS development
- Establish infrastructure within which scheduling and metascheduling services and capabilities can be developed
- Establish ‘driver’ to evaluate other resource management components

## Local Scheduler Progress

- Baseline scheduler established (Maui 3.2) for SSS scheduling services integrating production and development capabilities
- Prototype interface enabling XML communication with queue manager, metascheduler, and node manager
- Extended QoS infrastructure integrated
- Extended Job prioritization infrastructure integrated
- Prototype created for object-oriented data access
- Advanced metascheduling interface integrated

## Meta Scheduler Progress

- Initial distribution packaging created to allow collaborative development
- Documentation enhanced and extended
- Prototype XML scheduler to metascheduler query interface developed
- Initial fault tolerance framework designed

## Queue Manager Design

- Established need for unified queue manager design common to Scheduler and Metascheduler
- Queue manager will interface directly with Process manager
- In process of refining the queue manager tasks
- Queue manager will provide an interface to obtain information about any job regardless of job state including completed jobs (i.e. it will maintain a job information archive)

## Allocation Manager Progress

- QBank placed under revision control
- Java prototype created which sends requests in XML
- Experimenting with protocol frameworks (simple octet-counting, octet-stuffing, SOAP, BEEP)

## Next Steps (In Progress)

- Software Lifecycle Infrastructure
- Online intra-RM schedule and dependencies document
- Detailed extra-RM working group requirements
- Coordinate creation of component level regression test suite
- Bug tracking systems activated (used to track internal defects and development plans)
- Interface
- Produce validating intra-RM XML schema
- Produce prototype RM components communicating in initial protocol
- Feature Enhancements
- Contribution to checkpoint/restart report
- Creation of queue manager prototype

## Next Steps (6 Months)

- Usability
- GUI-server interface, GUI format, security determined and prototypes created
- Documentation of initial meta job constraints/features and specification language
- Inter-group Collaboration
- Creation of early scheduler XML implementation for use as RM driver
- Development of initial dynamic job scheduler-queue manager interface
- Extension of RM specifications/requirement document
- Extension of internal component test infrastructure
- Determination of ‘best practices’ in documentation maintenance
- Evaluation and adoption of web project management and collaboration tools
- Creation of prototype queue manager with scheduler/task manager interfaces

## Next Steps (6 Months)

- Fault Tolerance
- Enhance metascheduler to ‘survive’ local daemon failure
- Enhancement of threaded scheduling interface.
- Development of threaded metascheduling interface.
- Resource Optimization
- Development of local optimization features of meta workload
- Feature Enhancements
- Creation of resource manager extension features.
- Development of direct metascheduler to queue manager staging roadmap.
- Interfaces
- Specification of ‘best guess’ security infrastructure and evaluation of impact on system internals and communication protocols

## Next Steps (1 year)

- Software Lifecycle Infrastructure
- Create multi-component regression tests
- Generate ‘alpha’ package of scheduling, metascheduling, and allocation management packages.
- Interfaces
- Development of functional XML interfaces for all components
- Early adoption of security infrastructure
- Creation of optional information service interfaces
- Admin and end-user GUI’s proposed to enable use of new functionality
- Inter-group Collaboration
- Enhanced suspend/resume and checkpoint/restart features with detailed roadmap specified for all remaining suspend/resume and checkpoint restart deliverables

## Current Issues

- Should there be an enveloping protocol framework which handles framing (where the XML document begins and ends), authentication, multiplexing, streaming data, etc? (should we look at something like BEEP, or start from scratch and invent something of our own?)
- The queue manager/collector to node/process manager functionality and data interface requires further refinement.
- Queue manager/collector and node/process manager development schedules must be determined and coordinated.

##  Issues

- Continued effort is required to complete an ‘intra-RM’ XML schema to handle initial RMS interaction needs.  Boundaries between internal ‘intra-RM’ and global XML schema is needed.
- Understanding of open source requirements (I.e. can software be included in SSS distribution that requires registration and usage agreements)

## Inter-Group Issues

- Need for coordination of resource management system across working groups – so that the pieces all function together properly and no part is overlooked. Need to coordinate schedules for delivery of RMWG-dependent non-RMWG components.
- Early vendor/industry collaborations (We’d better do this while it can still influence our design. Need to talk to decision makers and develop business plans)

## Inter-group Issues

- Information service – should we rather be looking for something existing? (i.e. MDS2)
- Need to solidify SSS-wide standards for packaging, revision control, documentation content, format, and packaging, problem tracking, ... and establish mechanisms and places to home them.
- Creation of regression and integration test suite (w/ Validation and Testing WG – we need this from an early stage)

## Conclusions

- Questions...