# Presentation: 928239

## MaryDell Nochumson

- Distributed High Performance Computing Environments group
- Los Alamos National Laboratory

##   Group  provides tools for debugging, performance 	analysis, and memory utilization 

- 	My main focus has been MPI and Bluemountain stability.
- 	Working very closely with the vendor to provide a stable, 	scaleable, robust, message passing library that gives 	correct answers
- 	Identify stability problems and track stability progress
- 	on the Bluemountain cluster.

## LANL Bluemountain Cluster

- 48 rack mounted Origin 2000 SMP
- 128 nodes per SMP
- Hypercube
- A node has 2 R10000
- 	4MB of secondary cache per processor
- 	Main Memory 32MB
- Cray link Interconnect
- HIPPi Bypass

## Different cluster user roles 

**Local users of production codes**:
- 	Run codes in a production environment
**Developers of production codes**:
- 	Develop scalable high performance applications
**System administration**:
- 	To provide a stable rich computing environment
**System development**:
- 	To provide tools to look at memory placement, etc.
**Network services**:
- 	Provide a reliable delivery layer for HIPPI 800
**Remote users **
- 	Doing code development and running production codes

## Each roles metric of success 

**Users of production codes.  **
- 	Stable computing environment
- 	Lots of consulting support and education of the different
- 		layers of software they must interact with.
- 	Cycles to run
- 	Analysis tools
**Developers of codes and system/network development**
- 	Stable computing environment
- 	Good compilers, debuggers, performance and analysis tools
- 	Consulting support and education of the different layers of
- 		software they must interact with.
- 	System support
- 	Dedicated time on the machine to scale and debug large processor jobs.
- 	A forum for obtaining current status and changes and to express
- 		their concerns and problems.
- 	Formal Bug tracking

-  **System Administrators**
- 	Take the machine for system upgrades and preventive 			maintenance.
- 	Need to solve problems in a timely fashion, minimize down time.
- 	Advise users of any changes to the system.
- 	Formal Bug tracking

## What are the priorities of the metrics 

**Critical:**
** ****	**Robust Message Passing Library, a runtime system, and
- 	a stable cluster.** **
**Must: **
**	**Rich Development environment (Compilers,debuggers,
- 	resource manager**, ** consulting)
**Nice To Have:**
- 	Dynamic buffer allocation in MPI library
- 	Better compilers
-     	More performance and analysis tools

## What are the workarounds/tradeoffs?

**What are the workarounds/tradeoffs****?**

- Workarounds are never an expectable solution to users
- 	They get them anyway! 
- Tradeoff is that users can continue to get work done.	No debugger can work over very large numbers of PE’s so the
- 	workaround is printf() and dbx.

## What examples of cluster SW environments excel at delivering one or more of these user roles/perspectives? 

- Our major breakthrough was with the SGI/MPI library and stabilizing
- the Bluemountain cluster.
**April, 1999**
- MPI assumed a reliable delivery layer below it, but HIPPi 800 was
- not reliable.
- Code team could not assume correctness of answers.
- Cluster was very unstable, many machine crashes and HIPPI problems
- and other unidentified problems.
- Little work was being done.

## Kinds of  failures on 1024 and 2048 pe runs

- Total Percent
- 45% job failure 55% job success

- File system
- Mem Dims

## Where are we now

- 3024 PE – 6 hr run
- ~85% success
- No unknown failures

- 2016 – 3 hr run
- ~90% success
- No unknown failures

## How did we get there:

- Identified the stability problems with two weeks of diagnostics
- runs. Continued to measure stability with controlled runs.
-  HIPPi swat team
-  Equipment was sent back to vendor for further analysis of failure.
-  Failure analysis with full identification of each failure
-  Weekly telecom to identify problem area
-  Complete database of machine activity (when machines are down, what the problem
-     was and what was done to fix the problem)

**Notes:**
Fixed many internal library bugs
Multiboard with deadlock detection, ping at startup
1GB aggregated buffers
Matching header/data cookie, look for dropped headers
Down board detection 
Detect downed machines
Optimized collectives 
Better NUMA placement of buffers.
Multiple internal lists
Killing mpirun how kills your job
Error data written to the syslog

## Identified MPI problems and worked with vendor to incorporate features and fix bugs

-  Multiboard with deadlock detection, ping at startup
-  1GB aggregated buffers
-  Matching header/data cookie, look for dropped headers
-  Down board detection
-  Detect downed machines
-  Optimized collectives
-  Better NUMA placement of buffers.
-  Multiple internal lists
-  Killing mpirun how kills your job
-  Error data written to the syslog
-  Fixed many internal library bugs