# Presentation: 316956

## The ORNL Cluster

**The ORNL Cluster**
**	****Computing Experience...**

**Stephen L. Scott**
**Oak Ridge National Laboratory**
**Computer Science and Mathematics Division**
**Network and Cluster Computing Group**
- December 6, 2004
- RAMS Workshop
- Oak Ridge, TN

**scottsl@ornl.gov     www.csm.ornl.gov/~sscott**

## OSCAR

## What is OSCAR?

**What is OSCAR?**

- OSCAR Framework (cluster installation configuration and management)
- Remote installation facility
- Small set of “core” components
- Modular package & test facility
- Package repositories
- Use “best known methods”
- Leverage existing technology where possible
- Wizard based cluster software installation
- Operating system
- Cluster environment
- Administration
- Operation
- Automatically configures cluster components
- Increases consistency among cluster builds
- Reduces time to build / install a cluster
- Reduces need for expertise

**O****pen**** ****S****ource**** ****C****luster**** ****A****pplication**** ****R****esources**

***Step 5***

***Step 8 Done!***

***Step 6***

## OSCAR Components

- Administration/Configuration
- SIS, C3, OPIUM, Kernel-Picker, NTPconfig cluster services (dhcp, nfs, ...)
- Security: Pfilter, OpenSSH
- HPC Services/Tools
- Parallel Libs: MPICH, LAM/MPI, PVM
- Torque, Maui, OpenPBS
- HDF5
- Ganglia, Clumon, ... [monitoring systems]
*Other 3**rd** party OSCAR Packages*
- Core Infrastructure/Management
- System Installation Suite (SIS), Cluster Command & Control (C3), Env-Switcher,
- OSCAR DAtabase (ODA), OSCAR Package Downloader (OPD)

## Open Source Community Development Effort

- Open Cluster Group (OCG)
- Informal group formed to make cluster computing more practical for HPC research and development
- Membership is open, direct by steering committee
- OCG working groups
- OSCAR  (core group)
- Thin-OSCAR (Diskless Beowulf)
- HA-OSCAR (High Availability)
- SSS-OSCAR (Scalable Systems Software)
- SSI-OSCAR (Single System Image)
- BIO-OSCAR (Bioinformatics cluster system)

## OSCAR Core Partners

- Dell
- IBM
- Intel
- Bald Guy Software
- RevolutionLinux

- Indiana University
- NCSA
- Oak Ridge National Laboratory
- Université de Sherbrooke
- Louisiana Tech Univ.

*November 2004*

## eXtreme TORC powered by OSCAR

**eXtreme TORC powered by OSCAR**

** ****65 Pentium IV Machines**
** ****Peak Performance: 129.7 GFLOPS**
** ****RAM memory: 50.152 GB**

** ****Disk Capacity: 2.68 TB**
** ****Dual interconnects**
**   ****- Gigabit & Fast Ethernet**

## Slide 8

## HA-OSCAR: 

**HA-OSCAR: **

- The first known field-grade open source HA Beowulf cluster release
- Self-configuration Multi-head Beowulf system
- HA and HPC clustering techniques to enable critical HPC infrastructure
- Active/Hot Standby
- Self-healing with 3-5 sec automatic failover time

**RAS Management for HPC cluster**

## Slide 10

## Scalable Systems Software

## Scalable Systems Software

- IBM
- Cray
- Intel
- SGI

**Participating Organizations**

- ORNL
- ANL
- LBNL
- PNNL

- NCSA
- PSC
- SDSC

- SNL
- LANL
- Ames

- Collectively (with industry) define standard interfaces between systems components for interoperability
- Create scalable, standardized management tools for efficiently running our large computing centers

**Problem**

**Goals**

**Impact**

- Computer centers use incompatible, ad hoc set of systems tools
- Present tools are not designed to scale to multi-Teraflop systems

- Reduced facility  mgmt costs.
- More effective use of machines by scientific applications.

**www.scidac.org/ScalableSystems**

**To learn more visit**

**Notes:**
* Project Coordinator: Al Geist (ORNL)
* Collaboration among DOE labs, NSF supercomputing centers and Industry.
* This shows lots of stuff but key points are:
     - participating orgs
     - working groups
     - summary of problem/goal/impact (mentioned already)

## SSS-OSCAR

*Scalable System Software*

- Leverage OSCAR framework to package and distribute the Scalable System Software (SSS) suite,  *sss-oscar*.
*sss-oscar* – A release of OSCAR containing all SSS software in single downloadable bundle.

**SSS project developing standard interface for scalable tools**
- 	Improve interoperability
- 	Improve long-term usability & manageability
- 	Reduce costs for supercomputing centers
**Map out functional areas**
- 	Schedulers, Job Mangers
- 	System Monitors
- 	Accounting & User management
- 	Checkpoint/Restart
- 	Build & Configuration systems
**Standardize the system interfaces**
- 	Open forum of universities, labs, industry reps
- 	Define component interfaces in XML
- 	Develop communication infrastructure

**Notes:**
TJN: (11/5/2004) Condensed lots of stuff into just a few slides, draft#1

## OSCAR-ized  SSS Components

- Bamboo – *Queue/Job Manager** *
- BLCR – *Berkeley Checkpoint/Restart*
- Gold – *Accounting & Allocation Management System*
- LAM/MPI (w/ BLCR) – *Checkpoint/Restart enabled MPI*
- MAUI-SSS – *Job Scheduler*
- SSSLib – * SSS Communication library*
*Includes: SD, EM, PM, BCM, NSM, NWI*
- Warehouse – *Distributed System Monitor*
- MPD2 – *MPI Process Manager*

**Notes:**
TJN: (5/15/04)
Notes:
  - SSSLib – also contains the Build & Configuration components (node state mgr, etc.)
      However, the Service Directory, Event Mgr, and underlying communication infrastructure 
      are the primary aspects of this components.

## Cluster Power Tools

## C3 Power Tools

- Command-line interface for cluster system administration and parallel user tools.
- Parallel execution **cexec**
- Execute across a single cluster or multiple clusters at same time
- Scatter/gather operations **cpush****/****cget**
- Distribute or fetch files for all node(s)/cluster(s)
- Used throughout OSCAR and as underlying mechanism for tools like OPIUM’s *useradd* enhancements.

## C3 Building Blocks

** **System administration
-  cpushimage - “push” image across cluster
-  cshutdown - Remote shutdown to reboot or halt cluster
-  User & system tools
-  cpush - push single file -to- directory
-  crm - delete single file -to- directory
-  cget - retrieve files from each node
-  ckill - kill a process on each node
-  cexec - execute arbitrary command on each node
-  cexecs – serial mode, useful for debugging
-  clist – list each cluster available and it’s type
-  cname – returns a node name from a given node position
-  cnum – returns a node position from a given node name

**Notes:**
none

## C3 Power Tools

- Example to run hostname on all nodes of default cluster:
- $ cexec hostname
- Example to push an RPM to /tmp on the first 3 nodes
- $ cpush :1-3 helloworld-1.0.i386.rpm /tmp
- Example to get a file from node1 and nodes 3-6
- $ cget :1,3-6 /tmp/results.dat  /tmp
-  * Can leave off the destination with cget and will use the same location as source.

## Motivation for Success!

## RAMS – Summer 2004

* *

## Preparation for Success!

**Personality & Attitude**
- Adventurous
- Self starter
- Self learner
- Dedication
- Willing to work long hours
- Able to manage time
- Willing to fail
- Work experience
- Responsible
- Mature personal and professional behavior

**Academic**
- Minimum of Sophomore standing
- CS major
- Above average GPA
- Extremely high faculty recommendations
- Good communication skills
- Two or more programming languages
- Data structures
- Software engineering