# Presentation: 294001

## The ASCI Q System at Los Alamos 

- John Morrison
- CCN Division Leader

**Nicholas C. Metropolis Center **
**for Modeling and Simulation**

- 7th Workshop on Distributed Supercomputing
- March 4, 2003

**ASCI Q**

**LA-UR-03-0541**

**The ASCI Q System at Los Alamos **

- LA-UR-

## Q is operational for stewardship applications (1st 10T)

** ****Many ASCI applications are experiencing significant  performance increases over Blue Mountain.**
** ****Linpack performance run of 7.727 TeraOps (more than 75% efficiency)**
** ****Initial user response is very positive (with some issues!)**
**(Users want more cycles...)**
** ****Users from the tri-lab community are also using the system**

** ****Available to users for Classified ASCI codes since August 2002**
** ****Smaller initial system available since April 2002**
** ****Los Alamos has run its December 2002 ASCI Milestone calculation on Q**

- LA-UR-

## Question 1: 
Is your machine living up to the performance expectations? If yes, how? If not, what is the root cause?

**Is your machine living up to the performance expectations? If yes, how? If not, what is the root cause?**

- LA-UR-

## Performance ComparisonQ vs. White vs. Blue Mountain

**Cycle-time : lower is better**
**Weak-scaling of SAGE (problem per processor is constant )**
**-> ideal cycle-time is a constant for all PEs (but have parallel overheads)**

- LA-UR-

## Performance ComparisonQ vs. White vs. Blue Mountain

**Ratio: Higher the better**
**i.e. ASCI Q is between 5 and 6.5 times better than Blue Mountain**
**      ****ASCI Q is between 2 and 2.8 times better than ASCI White**

- LA-UR-

## Modeled and Measured Performance

**Unique capability for performance prediction developed in the Performance and Architecture Lab (PAL) at Los Alamos **
**Latest two sets of measurements are consistent **
**	****(~70% longer than model)**

- Lower is
- better!

- LA-UR-

## Using fewer PEs per Node	

**Test performance using 1,2,3 and 4 PEs per node**
**Reduces the number of compute processors available **
**Performance degradation appears when using all 4 procs in a node!**

- LA-UR-

## Performance Variability

**Lots of noise on the nodes: daemons and kernel activity**
**This noise was analyzed, quantified, modeled, and included back in the application model**
**This system activity has structure: it was identified and modeled**
**Cycle-time varies from cycle to cycle**

- LA-UR-

**Notes:**
This is time to perform a very well calibrated operation: it’s all over the place, whereas it should be the red line

## Performance Variability (2)

**Histogram of cycle-time over 1000 cycles**
**Minimum cycle-time is very close to model! (0.75 vs 0.70)**

**Performance is variable (some cycles are not affected!)**

- LA-UR-

## Modeled and Experimental Data

**The model is a close approximation of the experimental data**
**The primary bottleneck is the noise generated by the compute nodes (Tru64)**

- Lower
- Is better

- LA-UR-

**Notes:**
Performance if we were to remove (or optimize or reduce) various kinds of system activity

## Performance after System Optimization

- After system mods (both kernel and daemons and Quadrics RMS:
- right on target! After these optimizations, Q will deliver the performance
- that it’s supposed to. Modeling works!

- LA-UR-

**Notes:**
This is where the performance is going to be after HP will include out mods in the official system software release: exactly where it should be!

## Resources

**Performance and Architecture Lab (PAL in CCS-3**
**Work by Petrini, Kerbyson, Hoisie**
**Publications on this work and other architecture and performance topics at**
**           **
**                **** ****www.c3.lanl.gov/par_arch**

- LA-UR-

## Plan of Attack

**Find low hanging fruit (common problems with high payback) to attack first**
**Kill unnecessary daemons **
**Look at member 1 and 2 for CFS related activities**
**Member 31 noise!! **

- LA-UR-

## Kill Daemons

***HP SC engineering is checking that there are no operational problems with permanently switching them off.***

| Daemons | status |
| --- | --- |
| envmond | /sbin/init.d/envmon stop |
| insightd | sbin/init.d/insightd stop |
| snmpd | /sbin/init.d/snmpd stop |
| advfsd | Not running at LANL |
| smsd | Not running at LANL |
| lat | Already off |
| lpd | /sbin/init.d/lpd stop |
| xlogin | Not running at LANL |
| niff | /sbin/init.d/niffd stop |

- LA-UR-

## Summary on Performance

**Performance of Q machine is meeting and exceeding performance expectations**
**Performance Modeling Integral part of Q machine system deployment**
**Performance testing done at each major contractual milestone**
**FS-QB used in the unclassified environment for performance variability testing.**
**Approach is to systematically evaluate and implement recommendations of performance variability testing**

- LA-UR-

## Question 2: What is the MTBI? What are the topmost reasons for interrupts? What is the average utilization rate?

- LA-UR-

## Machine Q Interrupts and Overall MTBI

- LA-UR-

## Topmost Reasons for HW interrupts

- LA-UR-

## Interrupts for CPUs

- LA-UR-

## Scientific Investigation of Cosmic Rays Impact on CPU Failures

**L2 Btag memory parity checked but not corrected**
**	****At altitude at Los Alamos the number of neutrons is about 6-10 times higher than at sea level**
**	****With large number of ES45 systems and altitude we could be finding neutron induced CPU failures due to single bit soft errors**
**Neutron Monitors installed with Q to measure neutron flux**
**LANSCE Beam line testing of different memories**
**	****Two classes of programs used**
**	****Some discrepancies between results, trying to figure out**
**	****Only testing for neutron impact, other particles being evaulated**
**Statistical analysis for predicted error rates**
**Attempting to map beam line test output to predicted # of CPU failures on Q based on neutron flux at SCC**

- LA-UR-

## Scientific Investigation of Cosmic Rays Impact on CPU Failures - continued

**Initial results seem to indicate that the system is being impacted by neutrons hitting the L2 btag memory**
**Mapping of beam line results to predict # of CPU failures is not yet fully understood**
**We are managing around this problem from an applications perspective as demonstrated by the recent success of the milestone runs.**

- LA-UR-

## Memory Interrupts

- LA-UR-

## Overall utilization rate for initial

- Few months is between 50-60%

- LA-UR-

## Over 4.3 Million processor hours for Science Runs 

- Over 4.3 Million processor hours for Science Runs
- System Utilization over 85% some weeks

- LA-UR-

## Question 3:

What is the primary complaint, if any, from the users?

**What is the primary complaint, if any, from the users?**

- LA-UR-

## Historical Top Issues

**Reliability and Availability**
**Message Passing Interface (MPI)**
**LSF integration**
**File systems**
**Code development tools**

- LA-UR-

## Current Top User IssuesOctober 2002 Q Technical Quarterly Review

**Highest Priority**
**File system problems**
**System Availability & Reliability**
**HPSS “store” performance**

- Note the absence of MPI issues

- LA-UR-

## Current Top User IssuesOctober, 2002

**Medium Priority**
**Serial jobs (& login sessions) on QA**
**Q file system performance poor for many small serial files**
**Formal change control and preventative maintenance on all Q systems**
**QA viz users need non-purged file system**

- LA-UR-

## Current Top User IssuesOctober, 2002

**Lower Priority **
**LSF configurations on all Q systems**
**Early-Beta nature of QA versus User count**
**White-to-LANL(Q & HPSS) connectivity**
**DFS on Q (for Sandia users)**
**MPI CRC on Q**
**Q “devq” 2-login limit**

- LA-UR-

## Highest Priority

**File system problems**
**Loss of all /scratch files (multiple times)**
**Local component failures impact entire file system**
**Files not always visible (PFS & NFS)**
**Slow performance (e.g. simple “ls” command)**
**System Availability & Reliability**
**Whole machine impact**
**Long (4-8 hr) reboot time!**
**Many hung “services” require reboots**

- LA-UR-

## Highest Priority

**HPSS “store” performance**
**HPSS rates too low for QA capability**
**< 50MB/s**
**100’s GB (not unusual) require **_hours_** to store**
**SW & HW upgrades (relief is coming)**
**150MB/s Nov. target; 600MB/s Jan. target**
**Parallel clients; new HW & 4- & 16-way stripes**
**Totalview & F90 data in modules on Q**
**Can’t see F90 data located in modules**
**Workaround cumbersome & sometimes even crashes**
**Issue is over 1yr old!**

- LA-UR-

## Medium Priority

**Serial jobs (& login sessions) on QA**
**4 PE minimum due to RMS/LSF config**
**Q file system performance poor for many small serial files**
**Many codes write serial files from 1 PE**
**Some codes write 1 serial file per PE per dump time**
**Some codes write multiple sets of files at each dump time**
**Formal change control and preventative maintenance on all Q systems**
**Machine needs to move to more production-like status**
**QA viz users need non-purged file system**
**Interactive viz requires all files be resident simultaneously**
**No special “viz” file systems as on BlueMtn**

- LA-UR-