# Presentation: 175344

## Scalable Systems Softwarefor Terascale Computer Centers

**Al Geist**
**Narayan Desai**
**Rusty Lusk**
**Brett Bode**
**Paul Hargrove**

- SciDAC ISIC Review March 12, 2003

- Research sponsored by MICS Office of DOE

## Scalable Systems Softwarefor Terascale Computer Centers

**www.scidac.org/ScalableSystems**

**Scalable Systems Software****for Terascale Computer Centers**

**Coordinator: Al Geist**

**Participating Organizations**
- Includes DOE Labs, NSF Supercomputer Centers, Vendors

**ORNL**
**ANL**
**LBNL**
**PNNL**

**PSC**
**SDSC**
**IBM**
**Open to all like MPI forum**

**SNL**
**LANL**
**Ames**
**NCSA**

**Cray**
**Intel**
**Unlimited Scale**

**SGI**
**HP**

- Copy of these slides

## The Problem Today

**System administrators and managers of terascale computer centers are facing a crisis:**

**The Problem Today**

## Do nothing and ...

**Each computer center rewrites their home-grown software **
**Redundant effort and delayed availability**
**End result leaves the community no better off  ****and science suffers**
**Example: San Diego Supercomputer Center has no automatic way of having their queue manager interact with their accounting system.**** **

**Do nothing and ...**

- Date: Tue, 18 Feb 2003 10:22:59 -0800
- From: Donald Frederick <frederik@sdsc.edu_[>](mailto:frederik@sdsc.edu)_
- To: dubey@tagore.uchicago.edu
- Subject: UChicago ASCI Account on SDSC BH
- Dear Anshu - Please ask your colleagues not to submit any new jobs until we can replenish the Chicago ASCI account.
-  Thanks. - -Don F

## Our Three Goals

-  MPI-like process to promote interoperability, portability, and long-term usability.
**Produce a fully integrated suite of systems software** **and tools**
- Reference Implementation for the management and utilization of terascale computational resources.
**Research and development of more advanced versions of the components**
- To support the scalability, fault tolerance, and performance requirements of large science applications. Up to 10,000 nodes.

**Our Three Goals**

## Scope of the Effort

- restart

- Resource & Queue
- Management

- Accounting
- & user mgmt

- System
- Build &
- Configure

- Job management

- System
- Monitoring

- Security

- Allocation
- management

- Fault Tolerance

**Scope of the Effort**

## Impact

- Reduced facility management costs
-  reduce need to support ad hoc software
-  better systems tools available
-  able to get machines up and running faster and keep running
- More effective use of machines by scientific applications
-  scalable launch of jobs and checkpoint/restart
-  job monitoring and management tools
-  allocation management interface

**Impact**

## Vision:

- database

**A Common Integrated **
**Interface Framework**

- common pool of attributes

- XML format for attributes
- Standardized request protocol
- Choose an existing transfer protocol-TCP
- Every component uses the same framework

- Vendor
- optimized

- highly scalable version

- Easy to swap components

- User
- Host
- Node
- Job
- Allocation
- Etc...

**Vision:**

## System Software Architecture

**System Software Architecture**

**User DB**

**Accounting**

**Scheduler**

**System**
**Monitor**

- Components not in our mission

**High**

**Performance**

**Communication**

**& I/O**

**File**

**System**

- Interacts with

- all components

**Node Configuration &Build Manager **

**Allocation Management **

**Queue Manager**

**Job Manager &**
**Monitor**

**Data Migration**

**Usage Reports**

**User utilities**

**Checkpoint/ restart**

## Three Phase Plan

- Specify standardized interfaces between system components through a series of open meetings organized and run like the meetings that defined the MPI standard

**18 ****months**

**Three Phase Plan**

**18 months**

**24 months**

- Research and creation of components that have no existing versions
- Continuing improvements in efficiency and scalability of the most critical system components.
- Explore  long-term support for the distribution through the industry participants

- Starting in 2005 focus on the next generation of computers and the creation of system software components needed for this future technology
- Getting vendors to support the Scalable System Software distribution on their platforms

- Project is on schedule

## Project Management

- www.scidac.org/ScalableSystems

- Node build, configuration, and information service
- Resource management, scheduling, and allocation
- Process management, system monitoring, and checkpointing
- Validation and Integration

- A main notebook for general information & mtg notes And individual notebooks for each working group

**Web-based Project Notebooks ****(over 200 pages and growing)**

**Project Management**

**Quarterly Face-to Face Meetings**

**Four different Working Groups**

- To discuss and vote on interface proposals

## Voting and Concensus

- Basic Wire Protocol
- Cluster Bios support
- XML schema formalization
- Single method monitoring XML interface
- Authentication support
- Add Event Manager to architecture
- Support HTTP protocol
- And so on...

**Voting and Concensus**

**We Have Active Vendor Participation**

**Sample of proposals voted on:**

- In discuss and voting on interface proposals

**Same voting rules used by MPI forum**

## Four Working Groups 

**Overall Leader: ****Al Geist ORNL   ****gst@ornl.gov**
**Node build, configuration working group**
**    ****Leader: Narayan Desai ANL**
**Proccess management working group **
**    ****Leader: Paul Hargrove LBNL**
**Resource management working group **
**    ****Leader: Scott Jackson PNNL**
**Validation and testing working group**
**    ****Leader: Eric DeBenedictis SNL**

- Each working group is composed of members from several organizations
- Leaders of the working groups come from across the organizations
- Working groups are cross-fertilized with members from other groups

## Build and Configure WG

**Build and Configure WG**

**Hardware manager**

- manages all physical node services, including basic node identification, hardware inventory functions, power controller access, and management hardware topology information.

- handles all aspects of software installation and configuration management on all nodes in the system.

**Build and Configuration manager**

- an administrative control panel for the cluster. It keeps track of and allows direct control of all individual node states and availability.

**Node state manager**

- Compatible with OSCAR, Rocks, City, Cplant, and Scyld-like systems

## Integrating Components

**Integrating Components**

**SSSlib communication library**

**Event Manager**

- Supplies five wire protocols and is extensible
- Supplies basic authentication
- Compatible with C, C++, Java, Perl, and Python components

**Service Directory**

- Stores location and protocol information for all components
- Provides this information when components need to communicate

- Scalable solution to asynchronous events between components
- Also provides framework for system monitoring

## Process Management WG

**Process Management WG**

**Process manager**

- scalable startup
- runtime communication establishment
- knows where all the processes of a parallel job have been started.

- unified framework for collecting data across systems
- provides real-time state data of components and jobs
- focus on scalability and extensibility into new environments

**System and Job Monitors**

*Checkpoint*—an MPI-LAM or serial job on demand
*Suspend*—a job to temporarily cease use of resources
*Migrate*—a running job to different (overlapping) nodes

**Checkpoint manager**

## Resource Management WG

**Resource Management WG**

**Scheduler**

- based on Maui with Scalable Systems interfaces
- extensive advance reservation support and policy control
- support for Loadleveler, PBS, SGE, LSF, and BProc systems

- maintains information about jobs, both present and past
- handles job submission and deletion in the system
- node setup, data staging, and teardown

**Queue Manager**

- supports management features for accounts, users, machines, allocations, jobs, resources, usage and charging w/ simple GUI

**Accounting & Allocation manager**

- based on Silver with Scalable Systems interfaces and Grid interfaces
- distribute workload across HPC systems at computer center

**Meta-scheduler**

## Validation and Testing WG

**Validation and Testing WG**

**APITEST**

- unit-test driver for black-box testing components through their communications interfaces

- Scriptable test driver for whole-system tests
- Tests defined with XML
- http interface
- Matches stdout, stderr, and exit codes

**QMTest**

- whole-system test framework
- applications are run on the machine in various manners for the purpose of finding bugs, and giving confidence that the system will not crash during production use

**Coupled-system tester**

- QMTest

- Make

## System Software Components Presently under construction

**Authentication &**
**Communication**
***N. Desai, A. Lusk***

**Meta**
**Scheduler**
***D. Jackson***

**Meta**
**Manager**
***S. Scott***

**Accounting**
***S. Jackson***

**Scheduler**
***D. Jackson***

**System/Job**
**Monitors**
***M. Showerman***

**Package**
**Services**
***J. Mugler***

**Information**
**Services**
***J.P. Navarro***

**Allocation**
**Management**
***S. Jackson***

**Queue**
**Manager**
***B. Bode***

**Job**
**Manager**
***B. Bode***** **

**Checkpoint /**
**Restart**
***P. Hargrove***

**Process**
**Manager**
***R.Lusk***** **

**Service**
**Directory**
***N. Desai***

**Node**
**Manager**
***T. Naughton***

**C-Plant**
**XML interface**
***E. Debenedictis***** **

**Resource Mgmt**
**Working Group**

**Build & Configure**
**Working Group**

**Process Mgmt**
**Working Group**

**Strong Emphasis on multi-lab cooperation and team effort**

## Progress on Integrated Suite

**Grid Interfaces**

**Accounting**

**Event **
**Manager**

**Service**
**Directory**

**Meta**
**Scheduler**

**Meta**
**Monitor**

**Meta**
**Manager**

**Scheduler**

**Node State**
**Manager**

**Allocation**
**Management**

**Process **
**Manager **

**Usage**
**Reports**

**Meta Services**

**System &**
**Job Monitor**

**Job Queue**
**Manager**

**Node**
**Configuration **
**& Build**
**Manager**

**Standard XML**
**interfaces**

**Working Components and Interfaces (bold)**

**authentication **
**communication**

**Components written in any mixture of C, C++, Java, Perl, and Python.**

**Checkpoint /**
**Restart**

**Validation & Testing**

**Hardware**
**Infrastructure**
**Manager**

**Progress on Integrated Suite**

## Interactions and Users

- Process Manager, Event Manager, and Service Directory in production use on Chiba City at ANL
- XML wrapped Cplant tools used at Sandia
- Allocation Management component under evaluation by CCS at ORNL
- Adrian Wong from NERSC is an advisor to Scalable Systems on the NERSC requirements.

**Interactions and Users**

**Vendors Interest**

**Early use in DOE Computer Centers**

- They consider these tools commodity - would like to use ours They are asking what license we’ll use – told BSD
- Particular interest in validation & testing framework

## Leveraging Other Work

**Leveraging Other Work**

**OSCAR**

- Packaging technology
- Incorporating Scalable Systems components
- Large user base to spread the Scalable Systems interfaces

**City**

- Developing, and incorporating Scalable Systems components
- Production scale testing

**CPlant**

- Micro-kernel clusters
- Validation and testing framework
- Incorporating Scalable Systems interfaces

**Science Appliance**

- Bproc based clusters compatible with Scalable Systems Architecture
- Supermon monitoring component

## To Learn More – Five Project Notebooks

**A main notebook for general information**
**And individual notebooks for each working group**
- Allows groups to keep track of other groups progress and comment on the items of overlap
- Allows Center members and interested parties to see what is being defined and implemented
- Over 200 total pages

- Get to all notebooks through main web site
**www.scidac.org/ScalableSystems**
- Click on side bar or at “project notebooks” at bottom of page** **

## Component Demonstration

**Component Demonstration**

- Queue
- Manager

- Allocation
- Manager

- Process
- Manager

- Scheduler

- Event
- Manager

- Service
- Directory

- System
- Monitor