# Presentation: 525185

## Planned Machines: ASCI Purple, ALC and M&IC MCR

- Presented to SOS7
- Mark Seager
- _[seager@llnl.gov](mailto:seager@llnl.gov)_
- 925-423-3141
- ICCD ADH for Advanced Technology
- Lawrence Livermore National Laboratory

- This work was performed under the auspices of the U.S. Department of Energy by the University of California, Lawrence Livermore National Laboratory under Contract No. W-7405-Eng-48.

## Q1: What is unique in structure and function of your machine?

- Purple’s unique structure is fat SMPs with 16 rails of Federation interconnect
- MCR+ALC’s unique structure is the shared global file system
- However, most important point is that applications are highly mobile between Purple, MCR+ALC, White, Q and other clusters of SMP systems...

## Purple’s unique structure is fat SMPs with 16 rails of interconnect

**Purple System**
- 100 TF/s + 30-45 TF/s delivered on sPPM+UMT2000
- 50 TB memory, 2.0 PB of disk @ 108 GB/s delivered
- 197 x 64-way Armada SMP w 16 Federation Links
- 4 Login/network nodes
- Login/network nodes for login/NFS
- 8x10 Gb/s for parallel FTP on each Login
- All external networking is 1-10 Gb/s Ethernet
- Clustered I/O services for cluster wide file system
- Fibre Channel2 I/O attach does not extend

**Programming/Usage Model**
- Application launch over all compute nodes up to 8,192 tasks
- 1 MPI task/CPU and Shared Memory, full 64b support
- Scalable MPI (MPI_allreduce, buffer space)
- Likely usage
- multiple MPI tasks/node with 4-16 OpenMP/MPI task
- Single STDIO interface
- Parallel I/O to single file, multiple serial I/O (1 file/MPI task)

**191 Parallel Batch/Interactive/Visualization Nodes**

- System Data and Control Networks

- System Data and Control Networks

- System Data and Control Networks

- ...

- ...

- 16 Federation links per SMP in four switch planes

- Fibre Channel 2 I/O Network

## Unique feature of ALC+MCR is Lustre Lite shared file system†

- †Cluster wide file system leverages DOE/NNSA ASCI PathForward Open Source Lustre development

**Notes:**
960 nodes = 32 gateway + 2 MDS (fail-over pair) + 2 Login.  2 Service nodes are not on the QsNet swtich.

## Q2: What characterizes your applications? Examples are: Intensities of message passing, memory utilization, computing, IO, and data.

- Applications characterized as multi-physics package simulations
- All applications compute/comms intensive
- Each package pushes performance envelope along a different dimension
- Some packages are MPI latency dominated
- Some packages are MPI BW dominated
- Memory BW is critical factor, but expensive memory subsystems don’t perform much better than commodity ones...

## Q3: What prior experience guided you to this choice?

- Mission and Applications
- Budgets
- Politics
- Delivered performance
- Balanced risk and cost performance

## Strategic Approach: straddle multiple curves to balance risk and opportunity of new disruptive technologies 

- Three complementary curves...
- Delivers to today’s stockpile’s demanding needs
- Production environment
- For “must have” deliverables now
- Delivers transition for next generation
- “Near production” environment
- Provides cycles for science
- Provides cycles for stockpile
- Leading to next generation production systems
- These are the capacity systems in a strategic capacity/capability mix
- Delivers affordable path to petaFLOP/s
- Research environment, leading transition to petaflop systems?
- Are there other paths to a breakthrough regime by 2006-7?

**Performance**

**Time**

**Mainframes**
**(RIP)**

**Vendor integrated **
** ****SMP Cluster **
**(IBM SP, HP SC)**

**IA32/ IA64/AMD **
**+ Linux**

**Cell-Based**
**(IBM BG/L)**

**Today**

**FY05**

**Straddle strategy**
**for stability **_**and**_** **
**preeminence**

**$2M/TF**** (Purple C)**

**$1.2M/TF**
**(MCR)**

**$170K/TF**

**$10 M/TF (White)**

**$7M/TF (Q)**

**$ 500K /TF**

***Any given technology curve is ultimately ******limited****** by Moore’s Law***

## Q4. Other than your own machine, for your needs what are the best and worst machines? And, why?

- Clusters of SMPs with full node OS makes system administration and programming much easier, but scalability is an issue
- Vectors suck
- 10x potential speed-up from vectorization on Cray YMP class machines yielded only 1.5-2x in delivered performance boost to stockpile codes