# Presentation: 693159

## Science Driven System Architectures(and a few words about benchmarking)

- John Shalf
- NERSC Users Group Meeting
- Princeton Plasma Physics Laboratory
- June 2006

## Overview

- Overview of the SSP for procurement benchmarks
- Recent Benchmarking Activities
- Workload Characterization
- Wild Ideas

## NERSC Mission

- Support large scale computational science that cannot be done elsewhere
- Support wide variety of science and computational methods
- Provide a stable production environment to deliver these services

## NERSC’s Diverse Workload

## Benchmarking for NERSC System Procurements

- Require a uniform/scientific metric for system “value” over the lifetime of the system that;
- Assesses effective/delivered system performance
- Representative of NERSC workload
- Takes into account system availability and delivery time
- Focus on the total value of the system to the DOE science community!
- Full Application based benchmark methodology
- SSP: Sustained System Performance
- ESP: Effective System Performance
- Same methodology (SSP/ESP) employed for “validation” of the delivered system
- Factory testing
- Acceptance testing
- Continuing testing through the lifetime of the system to assess impact of all system upgrades

## SSP Metric

- 7 production applications provide representative subset of NERSC workload
- Immunity to performance “tweaking”
- Jobs scaled to match typical/target problem sizes
- Emphasis on capability jobs
- Uses weighted harmonic mean of job performance
*add wallclock times together and divide by total flop count*
- Total “value” is the area under the SSP performance curve!

## SSP v2 Applications

| Application
 | Scientific Discipline | Algorithm or Method | MPI Tasks | System Size | Wallclock
Timing (sec) |
| --- | --- | --- | --- | --- | --- |
| GTC* | Plasma Physics – (SciDAC) | Particle-in-cell | 256 | 107 
ions | 1682 |
| MADCAP* | Cosmology
(SciDAC) | Matrix inversion | 484 | 40000x
40000 | 903 |
| Milc* | Particle Physics (SciDAC) | Lattice QCD | 512 | 323x64 | 1031 |
| NAMD | Biophysics | Molecular dynamics | 1024 | 92224 atoms | 379 |
| NWChem | Chemistry | Density functional | 256 | 125 atoms | 2367 |
| Paratec* | Material Science | Density functional | 128 | 432 atoms | 1386 |
| SEAM* | Climate
(SciDAC) | Finite element | 1024 | 30 days | 494 |

## ESP

- Throughput of system under normal operating conditions (nontrivial)
- Batch Scheduler efficiency and validation
- Job Launcher efficiency
- Effect of job fragmentation on system performance
- Issues with < full bisection interconnects
- Even fat-trees suffer from fragmentation issues
- Job migration overhead (remediation)

## Application Selection Issues

- It is difficult to get good coverage
- Some scientists will not part with “crown jewels”
- Hopelessly **un**-portable codes
- Huge time investment for porting and packaging
- Tuning requirements for novel/unique architectures
- Difficult for vendors to find test systems of appropriate scale
- Must test applications at reasonable/native scale
- Rotation of benchmarks to prevent “performance islands”
- Rotation of benchmarks to follow workload trends
- SSP applications will have turn-over as science evolves
- Vendor “non-compliance” during bidding process
- Motivates us to simplify benchmarking procedures and methods
- Perhaps we need a persistent effort to manage the SSP?

## Emerging Concerns (apps not in SSP/ESP)

- Some good science doesn’t scale to thousands of processors
- AMR
- Load balancing
- Locality constraints for prolongation and restriction
- Pointer chasing (and lots of it!) (Little’s Law limitations)
- Sparse Matrix / SuperLU
- Domain decomp limits strong scaling efficiency
- Emerging issues with existing applications
- Implicit Methods
- Vector inner product required by Krylov subspace algorithms is hampered by latency-bound fast global reductions at massive parallelism
- Climate Models
- When science that depends on parameter studies and ensemble runs, capacity and capability are intimately linked! *(capacity vs. capability is a bogus metric)*
- Growth in experimental and sensor data processing requires more attention to I/O performance and global filesystems

## Science Driven System Architecture (SDSA)

**Stewardship of NERSC SSP Benchmark suite** *(Harvey Wasserman, Lenny Oliker)*
- Workload characterization (benchmarks only sensible in context of the workload they are intended to model:
- Selection and packaging of benchmarks + collaboration with other govt. agencies
- Benchmarking and data collection on available systems (no surprises)
**Development of New Benchmark Areas** *(Mike Welcome, Hongzhang Shan)*
- I/O Benchmarking
- AMR Benchmarks
**Performance Modeling** *(Erich Strohmaier, Andrew Canning)*
- Develop microbenchmarks that act as proxies to full application code
- Develop performance predictive performance models that enable us to predict performance of systems that do not yet exist
- Use predictive performance models to answer “what-if” architectural questions.
**Algorithm Tracking and Computer Architecture Evaluation** *(Lin-Wang Wang, Esmond Ng)*
- What are current resource requirements for current algorithms and how will they affect future computer system architectures?
- How will future system architecture choices affect the development of future numerical algorithms?
**Vendor Engagement** *(everyone)*
- Vendor development cycle 18-24 months!
- Provide detailed performance analysis & discussion w/vendors to effect changes early in the development cycle *(when it really matters)!*
- Bring feedback from vendors back to application groups (vendor code tuning assistance)

## Multicore? (Or not?)

- Bassi SSP on Power5+ showed only 8.27% performance degradation when run using dual-core mode.

## Multicore? (Or not?)

- However, AMD Opteron X2 shows >50% degradation for many NAS benchmarks compared to single-core???

-  100% would be perfect speedup
-  50% means each core runs 50% as    fast as single core case
-  <50% means performance degrades **more** than memory bandwidth alone can explain!!!

## Understanding Interconnects

- CPU clock scaling bonanza has ended
- Heat density
- New physics below 90nm *(departure from bulk material properties)*
- Yet, by end of decade mission critical applications expected to have 100X computational demands of current levels *(PITAC Report, Feb 1999)*
- The path forward for high end computing is increasingly reliant on massive parallelism
- Petascale platforms will likely have hundreds of thousands of processors
- System costs and performance may soon be dominated by interconnect
- What kind of an interconnect is required for a >100k processor system?
- What topological requirements? *(fully connected, mesh)*
- Bandwidth/Latency characteristics?
- Specialized support for collective communications?

**Notes:**
Slowing down the CPU to match the interconnect performance is acceptable if you are latency bound (there is nothing you can do about that), but it seems a travesty if the real problem is that you are bandwidth bound.  Our analysis (shown later in this document) demonstrates that Point2Point messages are primarily bandwidth bound.  As such, the path to more power-efficient computing probably lies in investing more money and power budget in the interconnect rather than slowing down the CPUs to match an under-powered interconnection network.

## Questions(How do we determine appropriate interconnect requirements?)

- Topology: *will the apps inform us what kind of topology to use?*
- Crossbars: Not scalable
- Fat-Trees: Cost scales superlinearly with number of processors
- Lower Degree Interconnects: *(n-Dim Mesh, Torus, Hypercube, Cayley)*
- Costs scale linearly with number of processors
- Problems with application mapping/scheduling fault tolerance
- Bandwidth/Latency/Overhead
- Which is most important? *(trick question: they are intimately connected)*
- Requirements for a “balanced” machine? *(eg. performance is not dominated by communication costs)*
- Collectives
- How important/what type?
- Do they deserve a dedicated interconnect?
- Should we put floating point hardware into the NIC?

**Notes:**
The batch scheduler must play 3D tetris all day long with job migration etc...

## Approach

- Identify candidate set of “Ultrascale Applications” that span scientific disciplines
- Applications demanding enough to require Ultrascale computing resources
- Applications that are capable of scaling up to hundreds of thousands of processors
*Not every application is “Ultrascale!” (not all good science is Ultrascale)*
- Find communication profiling methodology that is
- Scalable: N*eed to be able to run for a long time with many processors.  Traces are too large*
- Non-invasive: *Some of these codes are large and can be difficult to instrument even using automated tools*
- Low-impact on performance: *Full scale apps... not proxies!*

## IPM (the “hammer”)

**Integrated**
**Performance**
**Monitoring**
- portable, lightweight, scalable profiling
- fast hash method
- profiles MPI topology
- profiles code regions
- open source

- MPI_Pcontrol(1,”W”);
-  ...code...
- MPI_Pcontrol(-1,”W”);

**###########################################**
**# IPMv0.7 :: csnode041 256 tasks  ES/ESOS**
**# madbench.x (completed) 10/27/04/14:45:56**
**#**
**#       <mpi>      <user>      <wall> (sec)**
**#      171.67      352.16      393.80      **
**# ...**
**###############################################**
**# W**
**#       <mpi>      <user>      <wall> (sec)**
**#       36.40      198.00      198.36**
**#**
**# call            [time]      %mpi   %wall**
**# MPI_Reduce      2.395e+01   65.8     6.1**
**# MPI_Recv        9.625e+00   26.4     2.4**
**# MPI_Send        2.708e+00    7.4     0.7**
**# MPI_Testall     7.310e-02    0.2     0.0**
**# MPI_Isend       2.597e-02    0.1     0.0**
**###############################################**
**...**

*Developed by David Skinner, NERSC*

## Application Overview (the “nails”)

| NAME | Discipline | Problem/Method | Structure |
| --- | --- | --- | --- |
| MADCAP | Cosmology | CMB Analysis | Dense Matrix |
| FVCAM | Climate Modeling | AGCM | 3D Grid |
| CACTUS | Astrophysics | General Relativity | 3D Grid |
| LBMHD | Plasma Physics | MHD  | 2D/3D Lattice |
| GTC | Magnetic Fusion | Vlasov-Poisson | Particle in Cell |
| PARATEC | Material Science | DFT | Fourier/Grid |
| SuperLU | Multi-Discipline | LU Factorization | Sparse Matrix |
| PMEMD | Life Sciences | Molecular Dynamics | Particle |

## Call Counts

- Wait

- Wait

- Wait

- WaitAll

- WaitAll

- Isend

- Isend

- Isend

- Isend

- Irecv

- Irecv

- Irecv

- Irecv

- Isend

- Isend

- Wait

- Irecv

- Recv

- Recv

- Send

- Send

- WaitAny

- Irecv

- SendRecv

- Allreduce

- Gather

- Reduce

## P2P Topology Overview

## Cactus CommunicationPDE Solvers on Block Structured Grids 

## GTC Communication

**GTC Communication**

- Call Counts

## SuperLU Communication

**SuperLU Communication**

## PARATEC Communication

- 3D FFT

## Collective Buffer Sizes

- 95% Latency Bound!!!

## Latency/Balance Diagrams

- Latency Bound

- Bandwidth Bound

- Communication

- Computation Bound

- Communication Bound

- Computation

- Faster Processors

- More Interconnect
- Bandwidth

- Lower Interconnect
- Latency

## Revisiting Original Questions

- Topology
- Most codes require far less than full connectivity
- PARATEC is the only code requiring full connectivity
- Many require low degree (<12 neighbors)
- Codes with low topological degree of communication not necessarily isomorphic to a mesh!
- Non-isotropic communication pattern
- Non-uniform requirements
- Bandwidth/Delay/Overhead requirements
- Scalable codes primarily bandwidth-bound messages
- Average message sizes several Kbytes
- Collectives
- Most payloads less than 1k *(8-100 bytes!)*
- Well below the bandwidth delay product
- Primarily latency-bound *(requires different kind of interconnect)*
- Math operations limited primarily to reductions involving sum, max, and min operations.
- Deserves a dedicated network *(significantly different reqs.)*

**Notes:**
(if prob scale maintained.. Eg don’t make procs too weak or use less memory/proc)

## Algorithm Tracking

- Can’t track an algorithm until you define what constitutes an algorithm that is worth tracking
- Which algorithms or libraries are important?
- Workload analysis precedes drill-down into algorithm

## Materials Science Workload

## Materials Science Workload

- Lin-Wang Wang ERCAP analysis
- PARATEC is a good proxy for MatSci apps.
- A massively parallel future (Petascale) may push us to methods that exhibit more Spatial Locality in their communication patterns
- Are real-space methods a good replacement, or is it just going to waste more CPU cycles to get the same quality answer?