# Presentation: 406524

## The ORNL Cluster Computing Experience...

**The ORNL Cluster Computing Experience...**

**John L. Mugler**
**Stephen L. Scott**
**Oak Ridge National Laboratory**
**Computer Science and Mathematics Division**
**Network and Cluster Computing Group**
**December 2, 2003**
**RAM Workshop**
**Oak Ridge, TN**

**scottsl@ornl.gov     www.csm.ornl.gov/~sscott**

## Introduction

- Cluster computing has become popular
- Clusters abound!
- Price/performance
- hardware cost decrease in exchange for administration costs
- Enter the cluster distributions/toolkits
- OSCAR, Scyld, Rocks, ...

## eXtreme TORC powered by OSCAR

**eXtreme TORC powered by OSCAR**

** ****65 Pentium IV Machines**
** ****Peak Performance: 129.7 GFLOPS**
** ****RAM memory: 50.152 GB**

** ****Disk Capacity: 2.68 TB**
** ****Dual interconnects**
**   ****- Gigabit & Fast Ethernet**

## Cluster Projects

## OSCAROpen Source Cluster Application Resources

- Snapshot of best known methods for building, programming and using clusters.
- Consortium of academic/research & industry members.

## Project Organization

- Open Cluster Group (OCG)
- Informal group formed to make cluster computing more practical for HPC research and development
- Membership is open, direct by steering committee
- OCG working groups
- OSCAR
- Thin-OSCAR (diskless)
- HA-OSCAR (high availability)

## OSCAR 2003 Core Members

- Dell
- IBM
- Intel
- MSC.Software
- Bald Guy Software

- Indiana University
- NCSA
- Oak Ridge National Lab
- Université de Sherbrooke

**Notes:**
Just use english pronunciation for “Université de Sherbrooke” --- Sherbrooke University.

## What does OSCAR do?

- Wizard based cluster software installation
-  Operating system
-  Cluster environment
- Automatically configures cluster components
- Increases consistency among cluster builds
- Reduces time to build / install a cluster
- Reduces need for expertise

## OSCAR: Basic Design

- Use “best known methods”
- Leverage existing technology where possible
- OSCAR framework
- Remote installation facility
- Small set of “core” components
- Modular package & test facility
- Package repositories

**Notes:**
OSCAR Packages
Simple way to wrap software & configuration
Keep simple for package authors
Timely release/updates
Modular packaging (each self contained)
Leverage RPM + meta file + scripts, docs, ...
Repositories for downloading via OPD

## Slide 10

## OSCAR Summary

- Toolkit / framework to build and maintain cluster computers.
- Reduce duplication of effort
- Leverages existing tools & methods
- Simplifies process

## C3 Power Tools

- Command-line interface for cluster system administration and parallel user tools.
- Parallel execution **cexec**
- Execute across a single cluster or multiple clusters at same time
- Scatter/gather operations **cpush**/**cget**
- Distribute or fetch files for all node(s)/cluster(s)
- Used throughout OSCAR and as underlying mechanism for tools like OPIUM’s *useradd* enhancements.

## C3 Power Tools

- Example to run hostname on all nodes of default cluster:
**$ cexec hostname**
- Example to push an RPM to /tmp on the first 3 nodes
**$ cpush :1-3 helloworld-1.0.i386.rpm /tmp**
- Example to get a file from node1 and nodes 3-6
**$ cget :1,3-6 /tmp/results.dat  /tmp**
-  * Can leave off the destination with cget and will use the same location as source.

## Goal of the SSS project

** **
*“...**fundamentally change the way future *
*high-end systems software is developed to*
* **make it *_*more cost effective*_* and *_*robust*_*.”*
- * “Scalable Systems Software for Terascale Computer Centers” document

## SSS Problem Summary

- Supercomputing centers have incompatible system software
- Tools not designed for multi-teraflop scale
- Duplication of work to try and scale tools
- System growth vs.  administrator growth

## Scalable Systems Software

- IBM
- Cray
- Intel
- Unlimited Scale

**Participating Organizations**

- ORNL
- ANL
- LBNL
- PNNL

- NCSA
- PSC
- SDSC

- SNL
- LANL
- Ames

- Collectively (with industry) define standard interfaces between systems components for interoperability
- Create scalable, standardized management tools for efficiently running our large computing centers

**Problem**

**Goals**

**Impact**

- Computer centers use incompatible, ad hoc set of systems tools
- Present tools are not designed to scale to multi-Teraflop systems

- Reduced facility  mgmt costs.
- More effective use of machines by scientific applications.

**www.scidac.org/ScalableSystems**

**To learn more visit**

**Notes:**
* Project Coordinator: Al Geist (ORNL)
* Collaboration among DOE labs, NSF supercomputing centers and Industry.
* This shows lots of stuff but key points are:
     - participating orgs
     - working groups
     - summary of problem/goal/impact (mentioned already)

## SSS Overview

- Standard interface for multi-terascale tools
- Improve interoperability
- Improve long-term usability & manageability
- Reduce costs for supercomputing centers
- Ultimately more cost effective & robust

## Resource Allocation & Tracking System (RATS)

- What is RATS?
- Software system for managing resource usage
- Project Team

- _ETSU_
-    Smitha Chennu
-    Mitchell Griffith
-    David Hulse
-    Robert Whitten

- _ORNL_
-    Tom Barron
-    Rebecca Fahey
-    Phil Pfeiffer
-    Stephen Scott

## Motivation for Success!

## Student & Faculty Research Experiences inHigh-Performance Cluster Computing

**Summer 2003**
**4 undergraduate (RATS)**
**3 undergraduate (RAM)**
**1 faculty sabbatical**
**1 undergraduate**
**2 post-MS (ORISE)**
**Spring 2003**
**4 undergraduate (RATS)**
**1 faculty sabbatical**
**3 post-MS (ORISE)**
**1 offsite MS student**
**1 offsite undergraduate**
**Fall 2002**
**4 undergraduate (RATS)**
**3 post-MS (ORISE)**
**1 offsite MS student**
**1 offsite undergraduate**
**Summer 2002**
**3 undergraduate (RAM)**
**3 post-MS**
**1 undergraduate**

**Summer 2001**
**1 faculty (HERE)**
**3 MS students (HERE)**
**5 undergraduate (HERE / ERULF)**
**Spring 2001**
**1 MS student**
**2 undergraduate**
**Fall 2000**
**2 undergraduate**
**Summer 2000**
**1 faculty (HERE)**
**1 MS student (HERE)**
**1 undergraduate (HERE)**
**1 undergraduate (RAM)**
**5 undergraduate (ERULF)**
**Spring 2000**
**1 undergraduate (ERULF)**
**Summer 1999**
**1 undergraduate (ERULF)**

## RAM – Summer 2002 & 2003

* *

## DOE Nanoscale Science Research Centers WorkshopWashington, DC February 26-28, 2003

## Preparation for Success!

**Personality & Attitude**
- Adventurous
- Self starter
- Self learner
- Dedication
- Willing to work long hours
- Able to manage time
- Willing to fail
- Work experience
- Responsible
- Mature personal and professional behavior

**Academic**
- Minimum of Sophomore standing
- CS major
- Above average GPA
- Extremely high faculty recommendations
- Good communication skills
- Two or more programming languages
- Data structures
- Software engineering

## Resources

**Open Cluster Group Projects**
- www.OpenClusterGroup.org/OSCAR
- www.OpenClusterGroup.org/Thin-OSCAR
- www.OpenClusterGroup.org/HA-OSCAR
**OSCAR Development site**
- sourceforge.net/projects/oscar/
**C3 Project Page**
- www.csm.ornl.gov/torc/C3
**SSS Project**
- www.scidac.org/ScalableSystems
**SSS Electronic Notebooks**
- www.scidac.org/ScalableSystems/chapters.html