# Presentation: 356576

## Working Group updates, Suite Tests and Scalability, Race conditions, SSS-OSCAR Releases and Hackerfest

**Working Group updates, Suite Tests and Scalability, Race conditions, ****SSS-OSCAR Releases and Hackerfest**

- Al Geist
- August 17-19, 2005
- Oak Ridge, TN

**Welcome to Oak Ridge National Lab!****First quarterly meeting here**

## Demonstration: Faster than Light Computer

- Able to calculate the answer
- before the problem is specified

**Faster than Light Computer**

**ORNL**

**ORNL**

**ORNL**

**ORNL**

**ORNL**

## Scalable Systems Software

- IBM
- Cray
- Intel
- SGI

**Scalable Systems Software**

**Participating Organizations**

- ORNL
- ANL
- LBNL
- PNNL

- NCSA
- PSC

- SNL
- LANL
- Ames

- Collectively (with industry) define standard interfaces between systems components for interoperability
- Create scalable, standardized management tools for efficiently running our large computing centers

**Problem**

**Goals**

- Computer centers use incompatible, ad hoc set of systems tools
- Present tools are not designed to scale to multi-Teraflop systems

**www.scidac.org/ScalableSystems**

**To learn more visit**

## Scalable Systems Software Suite

**Grid Interfaces**

**Accounting**

**Event **
**Manager**

**Service**
**Directory**

**Meta**
**Scheduler**

**Meta**
**Monitor**

**Meta**
**Manager**

**Scheduler**

**Node State**
**Manager**

**Allocation**
**Management**

**Process **
**Manager **

**Usage**
**Reports**

**Meta Services**

**System &**
**Job Monitor**

**Job Queue**
**Manager**

**Node**
**Configuration **
**& Build**
**Manager**

**Standard XML**
**interfaces**

**authentication **
**communication**

**Components written in any mixture of C, C++, Java, Perl, and Python can be integrated into the Scalable Systems Software Suite**** **

**Checkpoint /**
**Restart**

**Validation & Testing**

**Hardware**
**Infrastructure**
**Manager**

**SSS-OSCAR**

**Scalable Systems Software Suite**

**Any Updates to this diagram?**

## Components in Suites

**Gold**

**EM**

**SD**

**Grid scheduler**

**Warehouse**

**Meta**
**Manager**

**Maui**
**sched**

**NSM**

**Gold**

**PM **

**Usage**
**Reports**

**Meta Services**

**Warehouse**
**(****superMon**
**NWPerf****)**

**Bamboo**
**QM**

**BCM**

**Multiple Component Implementations exits**

**ssslib**

**BLCR**

**APITest**

**HIM**

**Compliant with PBS, Loadlever job scripts**

## Scalable Systems Users

**Scalable Systems Users**

**Production use today:**
**Running an SSS suite at ANL, and Ames**
**ORNL industrial cluster (soon)**
**Running components at PNNL**
**Maui w/ SSS API  (3000/mo), Moab (Amazon, Ford, TeraGrid, ...)**
**Who can we involve before the end of the project?**
**- ****National Leadership-class facility?**
- NLCF is a partnership between
- ORNL (Cray), ANL (BG), PNNL (cluster)
**NERSC and NSF centers**
- NCSA cluster(s)
- NERSC cluster?
- NCAR BG

## Goals for This Meeting

**Updates on the Integrated Software Suite components**
** **
**Change in Resource Management Group**
**Scott Jackson left PNNL**
**Planning for SciDAC phase 2 – **
**       ****discuss new directions **
**Preparing for next SSS-OSCAR software suite release**
**       ****What needs to be done at hackerfest?**
**Getting more outside Users.**
**      ****Production and feedback to suite**

## Highlights of Last Meeting (May in ANL)

**Highlights of Last Meeting (****May in ANL)**

- Details in Main project notebook

**Fred Attended**  - he gave state of MICS, SciDAC-2, and his vision for changed focus
**Discussion of whitepaper and presentation for Strayer** ideas and Fred feedback
**API Discussions**
- voted for Process Manager API (12 yes 0 no 0 abstain)
- New Warehouse protocol presented
**Agreed to Quarterly Suite Releases this year**– and dates.

## Since Last Meeting

**Since Last Meeting**

**FastOS Meeting in DC**
- Any chatting about leveraging our System Software?
**SciDAC 2 Meeting in San Francisco**
- Scalable Systems Poster
- Talk on ISICS
- Several SSS members there. Anything to report?
**Telecoms and New entries in Electronic Notebooks **
- Pretty sparse since last meeting

## Agenda – August 17

- 8:00 Continental Breakfast CSB room B226
- 8:30 Al Geist - Project Status
- 9:00 Craig Steffen – Race Conditions in Suite
- 9:30 Paul Hargrove Process Management and Monitoring
- 10:30 Break
- 11:00 Todd Kordenbrock – Robustness and Scalability Testing
- 12:00 Lunch (on own at cafeteria )
- 1:30 Brett Bode - Resource Management components
- 2:30 Narayan Desai - Node Build, Configure, Cobalt status
- 3:30 Break
- 4:00 Craig Steffen – SSSRMAP in ssslib
- 4:30 Discuss proposal ideas for SciDAC 2
- 4:30 Discussion of getting SSS users and feedback
- 5:30 Adjourn for dinner

## Agenda – August 18

-   8:00 Continental Breakfast
-   8:30 Thomas Naughton - SSS OSCAR software releases
-   9:30 Discussion and voting
- Your name here
- 10:30 Group discussion of ideas for SciDAC-2.
- 11:30 Discussion of Hackerfest goals
- 	Set next meeting date/location:
- 12:00 Lunch (walk over to cafeteria)
-   1:30 Hackerfest begins room B226
-   3:00 Break
-   5:30 or whenever break for dinner

## Agenda – August 19

-   8:00 Continental Breakfast
-   8:30 Hackfest continues
- 12:00 Hackerfest ends

## What is going on inSciDAC 2Executive PanelFive Workshops in past 5 wksPreparing a SciDAC 2 program plan at LBNL today!ISIC section has words about system software and tools

## View to the Future

**HW, CS, and Science Teams all contribute to the science breakthroughs**

**Leadership-class Platforms**

**Breakthrough****Science**

**Software & Libs**

- SciDAC CS teams

**Tuned codes**

**Research team**

**High-End science problem**

**Computing Environment**
- Common look&feel across diverse HW

**SciDAC**
**Science Teams**

## SciDAC Phase 2 and CS ISICs

**Future CS ISICs need to be mindful of needs of **
- National Leadership Computing facility
- w/ Cray, IBM BG, SGI, clusters, multiple OS
- No one architecture is best for all applications
- SciDAC Science Teams
- Needs depend on application areas chosen
- End stations? Do they have special SW needs?
- FastOS Research Projects
- Complement, don’t duplicate these efforts
- Cray software roadmap
- Making the Leadership computers usable, efficient, fast

## Gaps and potential next steps

- Heterogeneous leadership-class machines
- science teams need to have a robust environment that presents similar programming interfaces and tools across the different machines.
- Fault tolerance requirements in apps and systems software
- particularly as systems scale up to petascale around 2010
- Support for application users submitting interactive jobs
- computational steering as means of scientific discovery
- High performance File System and I/O research
- increasing demands of security, scalability, and fault tolerance
- Security
- One-time-passwords and impact on scientific progress

## Heterogeneous Machines

- Heterogeneous Architectures
- Vector architectures, Scalar, SMP, Hybrids, Clusters
- How is a science team to know what is best for them?
- Multiple OS
- Even within one machine, eg. Blue Gene, Red Storm
- How to effectively and efficiently administer such systems?
- Diverse programming environment
- science teams need to have a robust environment that presents similar programming interfaces and tools across the different machines
- Diverse system management environment
- Managing and scheduling multiple node types
- System updates, accounting, ... everything will be harder in round 2

## Fault Tolerance

- Holistic Fault Tolerance
- Research into schemes that take into account the full impact of faults: application, middleware, OS, and hardware
- Fault tolerance in systems software
- Research into prediction and prevention
- Survivability and resiliency when faults can not be avoided
- Application recovery
- transparent failure recovery
- Research into Intelligent checkpointing based on active monitoring, sophisticated rule-based recoverys, diskless checkpointing...
- For petascale systems research into recovery w/o checkpointing

## Interactive Computing

- Batch jobs are not the always the best for Science
- Good for large numbers of users, wide mix of jobs, but
- National Leadership Computing Facility has different focus
- Computational Steering as a paradigm for discovery
- Break the cycle: simulate, dump results, analyze, rerun simulation
- More efficient use of the computer resources
- Needed for Application development
- Scaling studies on terascale systems
- Debugging applications which only fail at scale

## File System and I/O Research

- Lustre is today’s answer
- There are already concerns about its capabilities as systems scale up to 100+ TF
- What is the answer for 2010?
- Research is needed to explore the file system and I/O requirements for petascale systems that will be here in 5 years
- I/O continues to be a bottleneck in large systems
- Hitting the memory access wall on a node
- To expensive to scale I/O bandwidth with Teraflops across nodes
- Research needed to understand how to structure applications or modify I/O to allow applications to run efficiently

## Security

- New stricter access policies to computer centers
- Attacks on supercomputer centers have gotten worse.
- One-Time-Passwords, PIV?
- Sites are shifting policies, tightening firewalls, going to SecureID tokens
- Impact on scientific progress
- Collaborations within international teams
- Foreign nationals clearance delays
- Access to data and computational resources
- Advances required in system software
- To allow compliance with different site policies and be able to handle tightest requirements
- Study how to reduce impact on scientists

## Meeting notes

**Al Geist** – see slides
- Craig Steffen – Exciting new race condition!
- Nodes go offline – Warehouse doesn’t know quick enough
- Event manager, scheduler, lots of components affected
- Problem grows linear with system size. C
- Order of operations need to be considered – something we haven’t considered before. Issue can be reduced, can’t be solved
- Good discussion on ways to reduce race conditions.
- SSS use at NCSA
- Paul Egli rewrote Warehouse – many new features added, Sandia user
- Now monitoring sessions
- All configuration is dynamic
- Multiple debugging channels
- Sandia user – tested to 1024 virtual nodes
- Web site – _[http://arrakis.ncsa.uiuc.edu/warehouse/](http://arrakis.ncsa.uiuc.edu/warehouse/)_
- New hire full time on SSS
- Lining up T2 scheduling (500 proc)

## Meeting notes

**Paul Hargrove** – Checkpoint Manager BLCR status
- AMD64/EM64T port now in beta (crashes some users machines)
- Recently discovered kernel panic during signal interaction
- (must fix at hackerfest)
- Next step process groups/sessions – begin next week
- LRS-XML and Events “real soon now”
- Open MPI chpt/restart support by SC2005
- Torque integration done at U. Mich. for phd thesis (needs hardening)
- Process manager – MPD rewrite “refactoring”
- Getting a PM stable and working on BG.
**Todd K – **Scalability and Robustness tests
- ESP2 Efficiency ratio 0.9173 on 64 nodes
- Scalability – Bamboo 1000 job submission
- 	Gold (java version) – reservation slow – PERL version not tested
- 	Warehouse – up to 1024 nodes
- 	Maui on 64 nodes (need more testing)
- Durability – Node Warm stop – 30 seconds to Maui notification
- 	Node Warm start – 10 seconds
- 	Node Cold stop – 30 seconds

## Meeting notes

**Todd K** – testing continued
- Single node failure – good
- Resource hog (stress)
- Resource exhaustion – service node (Gold fails in logging package)
- Anomalies
- 	Maui
- 	Warehouse
- 	Gold
- 	happynsm
- ToDo
- Test BLCR module
- Retest on larger cluster
- Get latest release of all software and retest
- Write report on results.

## Meeting notes

**Brett Bode** – RM status
- New release of components
- Bamboo v1.1
- Maui 3.2.6p13
- Gold 2b2.10.2
- Gold being used on Utah cluster
- SSS suite on several systems at Ames
- New fountain component – to front end Supermon, ganglia, etc.
- Demos new tool called Goanna for looking at fountain output
- Has same interface as Warehouse – could plug right in
- General release of GOLD 2.0.0.0 available. New perl cgi gui
- 	no Java dependency at all in Gold
- X509 support in Mcom (for Maui and Silver)
- Cluster scheduler bunch of new features
- Grid scheduler – enabled basic accounting for grid jobs.
- Future work – Gary needs to get up to speed on Gold code
- 	make it all work with LRS

## Meeting notes

**Narian ** – LRS Conversion status
- All components in center cloud converted to LRS
- 	Service directory, Event manager, BCM stack, Processor Manager
- Targeted for SC05 release
- SSSlib changeover – completed
- SDK support – completed
- Cobalt Overview
- SSS suite on Chiba and BG
- Motivations – scalability, flexibility, simplicity, support for research ideas
- Tools included: parallel programming tools
- Porting has been easy – now running on Linux, MacOS, and BG/L
- Only about 5K lines of code.
- Targeted for Cray xt3, x1, zeptoOS
- Unique features- small partition support on BG/L, OS Spec support
- Agile – swap out components. User and admin requests easier to satisfy
- Running on ANL and NCAR (evaluation at other BG sites)
- May be running on JAZZ soon.
- Future- better scheduler, new platforms, more front ends, better docs

## Meeting notes

**Narian ** – Parallel tool development
- parallel Unix tools suite
- File staging
- Parallel rsync