# Presentation: 118605

## Scalable Cluster Management:Frameworks, Tools, and Systems

- David A. Evensky
- Ann C. Gentile
- Pete Wyckoff
- Robert C. Armstrong
- Robert L. Clay
- Ron Brightwell

- Sandia National Laboratories

## Lilith: a tool framework for very large clusters

- Most current tools for clusters are designed as monolithic programs, to do one task well.
- If you need a new task, you need a new tool.
- The Lilith framework allows users to easily construct new tools using a component framework.

## Control of large distributed systems

- System administration
- Auditing & job control by users
- Interrogation of processes
- Simple Applications

- 1 sec program on 1000 nodes

## Lilith: Scalable component framework

- Lilith spans a tree of machines executing user-defined code.
- User code (Lilim/Lilly) provides component functionality on a single node
- Provides scalable distribution, result collection

**Notes:**
Lilith was designed for rapid tool development. 
In order to write a tool, the writer creates code that provides the tool functionality on a single node. Lilith distributes the code down the tree and provides scalable execution.
The user code thus undergoes a three-phase process: distribution, execution, and result collection.
Lilith handles all the details of the code distribution, inter-machine communications, and result collection. By freeing the tool writer of these details, Lilith-based tools can be rapidly developed.

## Component Methods

**MO[] distributeOnTree(MO, int[])**
- data distribution down the tree
**MO onTree(MO)**
- component action on the node
**MO collateOnTree(MO[])**
- result collection and condensation

## Security

- Uses purely Java 2 mechanisms at
- this time....

- User sends
- credential with call

- LilithHost creates
- ProtectionDomain from
- user credential

- LilithHost calls checkPermission

- Method invocation

- Sandbox setup similarly using
- the User credential and PolicyFile

## Prototypical tools

## Lilith Lights tool

- Snake toy app
- demo that draws a snake over front panel
- no global repository for state --- all info distributed
- Snake’s movement was limited to left half of machine
- program error in declaration of drand48() biased results

## Who serves who?

- Programmers adapt to:
- The OS that runs on the machine,
- The system configuration chosen by the admins
- Changing system environments
- economically driven to heterogeneous distributed computing
- Why can’t the user dictate the software environment as a resource request?

## DASE

- Dynamically Adaptive Software Environment
- Provide multi-OS/multi-environment capability
- Manage multiple SW environments
- “save” user environment for reuse later
- Integration with SW component architectures

## DASE Service Object Model

- Mapping

- Resource Space

- Physical system

- Logical** **partitioning

- “system”model

- Partitioner

- App Object- resource spec
- - data/map objects

- Solver

- Visualizer

- App Space

- Mesher

- Scheduler

- Resource
- Request

## Flexible Resource Management

## Scalable Unit

## System Support Hierarchy

## Hardware  Management

- Discovery and Control
- Perl scripts that
- control individual devices (power controller, terminal server, machine, switch)
- build a database of configuration info (MAC and IP addresses, serial numbers, etc.)
- Roles
- database is augmented with each components role in the system (compute, sss0, terminal server, etc.)

## “Virtual Machines”

- Allows arbitrary grouping of scalable units that use the same system software
- Operations to update system software and boot nodes, scalable units, or machines
- Updates system software on an SU in 1 min.
- Update system software on 24 SUs in 1.5 min.
- Boot an SU in 5 min. (staged for power drain)
- Boot 24 SUs in 10 min.

## “Virtual Machines”

## http://dancer.ca.sandia.govhttp://www.cplant.ca.sandia.govhttp://www.cs.sandia.gov/cplant